[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 01:59:28.905637 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:35975,127.0.99.61:45483,127.0.99.60:37793
I20240503 01:59:28.906680 396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:28.907263 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:28.914777 401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:28.915498 402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:28.916708 396 server_base.cc:1034] running on GCE node
W20240503 01:59:28.917618 404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:28.918190 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:28.918298 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:28.918350 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701568918348 us; error 0 us; skew 500 ppm
I20240503 01:59:28.925587 396 webserver.cc:460] Webserver started at http://127.0.99.62:41433/ using document root <none> and password file <none>
I20240503 01:59:28.926158 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:28.926224 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:28.926451 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:28.928269 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-0-root/instance:
uuid: "d47b384da7a84de4b92d3a65b2d30841"
format_stamp: "Formatted at 2024-05-03 01:59:28 on dist-test-slave-82l4"
I20240503 01:59:28.931917 396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 01:59:28.934010 409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:28.935122 396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 01:59:28.935228 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-0-root
uuid: "d47b384da7a84de4b92d3a65b2d30841"
format_stamp: "Formatted at 2024-05-03 01:59:28 on dist-test-slave-82l4"
I20240503 01:59:28.935330 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:28.953370 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:28.954109 396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:28.954291 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:28.961959 461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:35975 every 8 connection(s)
I20240503 01:59:28.961974 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:35975
I20240503 01:59:28.963306 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 01:59:28.965428 462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 01:59:28.966135 467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:28.967281 464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:28.968183 465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:28.968813 396 server_base.cc:1034] running on GCE node
I20240503 01:59:28.969009 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:28.969046 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:28.969062 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701568969062 us; error 0 us; skew 500 ppm
I20240503 01:59:28.970034 396 webserver.cc:460] Webserver started at http://127.0.99.61:34947/ using document root <none> and password file <none>
I20240503 01:59:28.970180 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:28.970223 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:28.970301 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:28.970126 462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:28.970633 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-1-root/instance:
uuid: "b6ab12222bb64b1982db8c7169f8e400"
format_stamp: "Formatted at 2024-05-03 01:59:28 on dist-test-slave-82l4"
I20240503 01:59:28.972122 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 01:59:28.973126 474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:28.973374 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:28.973446 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-1-root
uuid: "b6ab12222bb64b1982db8c7169f8e400"
format_stamp: "Formatted at 2024-05-03 01:59:28 on dist-test-slave-82l4"
I20240503 01:59:28.973506 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:28.980307 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:28.980633 462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:28.980719 396 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240503 01:59:28.981285 410 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.61:45483: connect: Connection refused (error 111)
W20240503 01:59:28.982115 462 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.61:45483: Network error: Client connection negotiation failed: client connection to 127.0.99.61:45483: connect: Connection refused (error 111)
I20240503 01:59:28.985875 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.61:45483
I20240503 01:59:28.985900 527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.61:45483 every 8 connection(s)
I20240503 01:59:28.986593 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 01:59:28.986644 528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.987812 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } has no permanent_uuid. Determining permanent_uuid...
W20240503 01:59:28.989044 531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:28.989967 530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:28.990656 533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:28.991755 396 server_base.cc:1034] running on GCE node
I20240503 01:59:28.991909 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:28.991947 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:28.991969 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701568991969 us; error 0 us; skew 500 ppm
I20240503 01:59:28.992234 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:28.992900 396 webserver.cc:460] Webserver started at http://127.0.99.60:42685/ using document root <none> and password file <none>
I20240503 01:59:28.993047 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:28.993091 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:28.993147 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:28.993553 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-2-root/instance:
uuid: "60a40ee749c34842a9ecdaeef65196fa"
format_stamp: "Formatted at 2024-05-03 01:59:28 on dist-test-slave-82l4"
I20240503 01:59:28.994896 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:28.996102 540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:28.996361 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:28.996416 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-2-root
uuid: "60a40ee749c34842a9ecdaeef65196fa"
format_stamp: "Formatted at 2024-05-03 01:59:28 on dist-test-slave-82l4"
I20240503 01:59:28.996469 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:28.996476 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240503 01:59:28.997566 528 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.60:37793: Network error: Client connection negotiation failed: client connection to 127.0.99.60:37793: connect: Connection refused (error 111)
I20240503 01:59:29.011948 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:29.012353 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:29.017202 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.60:37793
I20240503 01:59:29.017275 593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.60:37793 every 8 connection(s)
I20240503 01:59:29.017645 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 01:59:29.018101 594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.019268 594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:29.023381 594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:29.026894 594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:29.031342 462 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } attempt: 1
I20240503 01:59:29.033182 594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa: Bootstrap starting.
I20240503 01:59:29.035111 462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:29.035976 594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.036813 594 log.cc:824] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:29.038539 594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa: No bootstrap required, opened a new log
I20240503 01:59:29.039691 462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841: Bootstrap starting.
I20240503 01:59:29.040609 462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.041685 462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841: No bootstrap required, opened a new log
I20240503 01:59:29.042061 594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } }
I20240503 01:59:29.042075 462 raft_consensus.cc:348] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } }
I20240503 01:59:29.042248 462 raft_consensus.cc:374] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.042248 594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.042325 462 raft_consensus.cc:729] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47b384da7a84de4b92d3a65b2d30841, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.042325 594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60a40ee749c34842a9ecdaeef65196fa, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.042814 462 consensus_queue.cc:260] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } }
I20240503 01:59:29.042814 594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } }
I20240503 01:59:29.043455 600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } } }
I20240503 01:59:29.043576 600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:29.043711 601 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } } }
I20240503 01:59:29.043857 601 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:29.044051 462 sys_catalog.cc:564] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:29.044051 594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:29.046363 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240503 01:59:29.048573 622 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 01:59:29.048570 623 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 01:59:29.048696 622 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 01:59:29.048710 623 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 01:59:29.049714 528 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } attempt: 1
I20240503 01:59:29.054467 528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400: Bootstrap starting.
I20240503 01:59:29.055653 528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.056893 528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400: No bootstrap required, opened a new log
I20240503 01:59:29.057350 528 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } }
I20240503 01:59:29.057459 528 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.057500 528 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ab12222bb64b1982db8c7169f8e400, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.057649 528 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } }
I20240503 01:59:29.057950 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } } }
I20240503 01:59:29.058058 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:29.058143 528 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:29.060102 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240503 01:59:29.060601 637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 01:59:29.060657 637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 01:59:29.060698 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:29.063874 639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:29.064703 638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:29.065717 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:29.066702 396 server_base.cc:1034] running on GCE node
I20240503 01:59:29.066866 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:29.066903 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:29.066918 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701569066918 us; error 0 us; skew 500 ppm
I20240503 01:59:29.067960 396 webserver.cc:460] Webserver started at http://127.0.99.1:38271/ using document root <none> and password file <none>
I20240503 01:59:29.068128 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:29.068171 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:29.068231 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:29.068624 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-0-root/instance:
uuid: "f5a665731b5f460d8d62f3ac897d5cac"
format_stamp: "Formatted at 2024-05-03 01:59:29 on dist-test-slave-82l4"
I20240503 01:59:29.070008 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:29.070880 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:29.071147 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.071211 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-0-root
uuid: "f5a665731b5f460d8d62f3ac897d5cac"
format_stamp: "Formatted at 2024-05-03 01:59:29 on dist-test-slave-82l4"
I20240503 01:59:29.071276 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:29.090430 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:29.091229 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:29.091673 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:29.092463 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:29.092518 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:29.092574 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:29.092602 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:29.099202 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:35751
I20240503 01:59:29.099249 709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:35751 every 8 connection(s)
I20240503 01:59:29.100554 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:29.103832 719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:29.104499 717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:29.105389 710 heartbeater.cc:347] Connected to a master server at 127.0.99.60:37793
I20240503 01:59:29.105525 710 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:29.105595 711 heartbeater.cc:347] Connected to a master server at 127.0.99.62:35975
I20240503 01:59:29.105672 711 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:29.106024 396 server_base.cc:1034] running on GCE node
I20240503 01:59:29.106173 710 heartbeater.cc:511] Master 127.0.99.60:37793 requested a full tablet report, sending...
I20240503 01:59:29.106173 711 heartbeater.cc:511] Master 127.0.99.62:35975 requested a full tablet report, sending...
I20240503 01:59:29.107623 426 ts_manager.cc:194] Registered new tserver with Master: f5a665731b5f460d8d62f3ac897d5cac (127.0.99.1:35751)
I20240503 01:59:29.107846 558 ts_manager.cc:194] Registered new tserver with Master: f5a665731b5f460d8d62f3ac897d5cac (127.0.99.1:35751)
W20240503 01:59:29.108443 716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:29.108763 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:29.108820 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:29.108843 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701569108843 us; error 0 us; skew 500 ppm
I20240503 01:59:29.108983 712 heartbeater.cc:347] Connected to a master server at 127.0.99.61:45483
I20240503 01:59:29.109061 712 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:29.109249 712 heartbeater.cc:511] Master 127.0.99.61:45483 requested a full tablet report, sending...
I20240503 01:59:29.109853 396 webserver.cc:460] Webserver started at http://127.0.99.2:33173/ using document root <none> and password file <none>
I20240503 01:59:29.109867 492 ts_manager.cc:194] Registered new tserver with Master: f5a665731b5f460d8d62f3ac897d5cac (127.0.99.1:35751)
I20240503 01:59:29.110028 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:29.110087 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:29.110162 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:29.110535 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-1-root/instance:
uuid: "bcaec0fda27b4cbe991d5bc62d14b670"
format_stamp: "Formatted at 2024-05-03 01:59:29 on dist-test-slave-82l4"
I20240503 01:59:29.111994 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 01:59:29.112916 725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:29.113191 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:29.113247 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-1-root
uuid: "bcaec0fda27b4cbe991d5bc62d14b670"
format_stamp: "Formatted at 2024-05-03 01:59:29 on dist-test-slave-82l4"
I20240503 01:59:29.113314 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:29.123996 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:29.124406 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:29.124688 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:29.125172 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:29.125211 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:29.125247 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:29.125276 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:29.131837 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:38973
I20240503 01:59:29.131879 788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:38973 every 8 connection(s)
I20240503 01:59:29.133013 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:29.135948 799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:29.136749 797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:29.137465 626 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:29.137646 626 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } }
I20240503 01:59:29.138783 626 leader_election.cc:288] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d47b384da7a84de4b92d3a65b2d30841 (127.0.99.62:35975), 60a40ee749c34842a9ecdaeef65196fa (127.0.99.60:37793)
I20240503 01:59:29.139274 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6ab12222bb64b1982db8c7169f8e400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d47b384da7a84de4b92d3a65b2d30841" is_pre_election: true
I20240503 01:59:29.139423 396 server_base.cc:1034] running on GCE node
I20240503 01:59:29.139521 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6ab12222bb64b1982db8c7169f8e400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60a40ee749c34842a9ecdaeef65196fa" is_pre_election: true
I20240503 01:59:29.139652 566 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ab12222bb64b1982db8c7169f8e400 in term 0.
W20240503 01:59:29.139712 796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:29.139456 436 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6ab12222bb64b1982db8c7169f8e400 in term 0.
I20240503 01:59:29.139967 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:29.140000 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:29.140022 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701569140021 us; error 0 us; skew 500 ppm
I20240503 01:59:29.140058 475 leader_election.cc:302] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60a40ee749c34842a9ecdaeef65196fa, b6ab12222bb64b1982db8c7169f8e400; no voters:
I20240503 01:59:29.140348 626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:29.140408 626 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:29.140447 626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.141225 790 heartbeater.cc:347] Connected to a master server at 127.0.99.62:35975
I20240503 01:59:29.141294 396 webserver.cc:460] Webserver started at http://127.0.99.3:43649/ using document root <none> and password file <none>
I20240503 01:59:29.141458 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:29.141510 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:29.141477 626 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } }
I20240503 01:59:29.141580 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:29.141785 626 leader_election.cc:288] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [CANDIDATE]: Term 1 election: Requested vote from peers d47b384da7a84de4b92d3a65b2d30841 (127.0.99.62:35975), 60a40ee749c34842a9ecdaeef65196fa (127.0.99.60:37793)
I20240503 01:59:29.141989 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-2-root/instance:
uuid: "c329ed9fbea84bf78e9e31797e6e2565"
format_stamp: "Formatted at 2024-05-03 01:59:29 on dist-test-slave-82l4"
I20240503 01:59:29.142302 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6ab12222bb64b1982db8c7169f8e400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d47b384da7a84de4b92d3a65b2d30841"
I20240503 01:59:29.142383 789 heartbeater.cc:347] Connected to a master server at 127.0.99.60:37793
I20240503 01:59:29.142418 436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.142463 789 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:29.142706 789 heartbeater.cc:511] Master 127.0.99.60:37793 requested a full tablet report, sending...
I20240503 01:59:29.143318 436 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ab12222bb64b1982db8c7169f8e400 in term 1.
I20240503 01:59:29.143350 558 ts_manager.cc:194] Registered new tserver with Master: bcaec0fda27b4cbe991d5bc62d14b670 (127.0.99.2:38973)
I20240503 01:59:29.141304 790 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:29.143625 477 leader_election.cc:302] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6ab12222bb64b1982db8c7169f8e400, d47b384da7a84de4b92d3a65b2d30841; no voters:
I20240503 01:59:29.143834 790 heartbeater.cc:511] Master 127.0.99.62:35975 requested a full tablet report, sending...
I20240503 01:59:29.143949 626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.144107 626 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [term 1 LEADER]: Becoming Leader. State: Replica: b6ab12222bb64b1982db8c7169f8e400, State: Running, Role: LEADER
I20240503 01:59:29.144273 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6ab12222bb64b1982db8c7169f8e400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60a40ee749c34842a9ecdaeef65196fa"
I20240503 01:59:29.144376 566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.144392 626 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } }
I20240503 01:59:29.144430 426 ts_manager.cc:194] Registered new tserver with Master: bcaec0fda27b4cbe991d5bc62d14b670 (127.0.99.2:38973)
I20240503 01:59:29.145323 566 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6ab12222bb64b1982db8c7169f8e400 in term 1.
I20240503 01:59:29.145797 396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:29.145880 791 heartbeater.cc:347] Connected to a master server at 127.0.99.61:45483
I20240503 01:59:29.145972 791 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:29.146148 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ab12222bb64b1982db8c7169f8e400. Latest consensus state: current_term: 1 leader_uuid: "b6ab12222bb64b1982db8c7169f8e400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } } }
I20240503 01:59:29.146263 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:29.146204 791 heartbeater.cc:511] Master 127.0.99.61:45483 requested a full tablet report, sending...
I20240503 01:59:29.146683 806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:29.146875 492 ts_manager.cc:194] Registered new tserver with Master: bcaec0fda27b4cbe991d5bc62d14b670 (127.0.99.2:38973)
I20240503 01:59:29.147786 806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:29.149902 807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:29.150164 396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20240503 01:59:29.150229 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-2-root
uuid: "c329ed9fbea84bf78e9e31797e6e2565"
format_stamp: "Formatted at 2024-05-03 01:59:29 on dist-test-slave-82l4"
I20240503 01:59:29.150295 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:29.152669 436 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [term 1 FOLLOWER]: Refusing update from remote peer b6ab12222bb64b1982db8c7169f8e400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.152669 566 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [term 1 FOLLOWER]: Refusing update from remote peer b6ab12222bb64b1982db8c7169f8e400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.153244 626 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.153388 804 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.155041 601 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ab12222bb64b1982db8c7169f8e400. Latest consensus state: current_term: 1 leader_uuid: "b6ab12222bb64b1982db8c7169f8e400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } } }
I20240503 01:59:29.155179 601 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:29.155041 600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ab12222bb64b1982db8c7169f8e400. Latest consensus state: current_term: 1 leader_uuid: "b6ab12222bb64b1982db8c7169f8e400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } } }
I20240503 01:59:29.155287 600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:29.156672 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6ab12222bb64b1982db8c7169f8e400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } } }
I20240503 01:59:29.156767 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:29.156834 601 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6ab12222bb64b1982db8c7169f8e400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } } }
I20240503 01:59:29.156909 601 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:29.157250 808 mvcc.cc:205] Tried to move back new op lower bound from 7023417627243364352 to 7023417627218198528. Current Snapshot: MvccSnapshot[applied={T|T < 7023417627243364352}]
I20240503 01:59:29.159013 804 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6ab12222bb64b1982db8c7169f8e400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } } }
I20240503 01:59:29.159116 804 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:29.159842 600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6ab12222bb64b1982db8c7169f8e400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47b384da7a84de4b92d3a65b2d30841" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 35975 } } peers { permanent_uuid: "b6ab12222bb64b1982db8c7169f8e400" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 45483 } } peers { permanent_uuid: "60a40ee749c34842a9ecdaeef65196fa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 37793 } } }
I20240503 01:59:29.159950 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:29.159938 600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:29.160593 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:29.160869 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:29.160893 806 catalog_manager.cc:1347] Generated new cluster ID: 627fe9b158b44677853fe11dabdcb35c
I20240503 01:59:29.160943 806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:29.161401 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:29.161448 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:29.161490 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:29.161517 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:29.167778 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:40867
I20240503 01:59:29.173295 879 heartbeater.cc:347] Connected to a master server at 127.0.99.60:37793
I20240503 01:59:29.173434 879 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:29.173653 879 heartbeater.cc:511] Master 127.0.99.60:37793 requested a full tablet report, sending...
I20240503 01:59:29.174342 558 ts_manager.cc:194] Registered new tserver with Master: c329ed9fbea84bf78e9e31797e6e2565 (127.0.99.3:40867)
I20240503 01:59:29.175104 878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:40867 every 8 connection(s)
I20240503 01:59:29.180804 881 heartbeater.cc:347] Connected to a master server at 127.0.99.61:45483
I20240503 01:59:29.180958 881 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:29.181270 881 heartbeater.cc:511] Master 127.0.99.61:45483 requested a full tablet report, sending...
I20240503 01:59:29.181859 880 heartbeater.cc:347] Connected to a master server at 127.0.99.62:35975
I20240503 01:59:29.181888 492 ts_manager.cc:194] Registered new tserver with Master: c329ed9fbea84bf78e9e31797e6e2565 (127.0.99.3:40867)
I20240503 01:59:29.181928 880 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:29.182224 880 heartbeater.cc:511] Master 127.0.99.62:35975 requested a full tablet report, sending...
I20240503 01:59:29.182838 426 ts_manager.cc:194] Registered new tserver with Master: c329ed9fbea84bf78e9e31797e6e2565 (127.0.99.3:40867)
I20240503 01:59:29.183357 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014992412s
I20240503 01:59:29.196398 806 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:29.197300 806 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:29.209327 806 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400: Generated new TSK 0
I20240503 01:59:29.210034 806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:30.049710 622 catalog_manager.cc:1259] Loaded cluster ID: 627fe9b158b44677853fe11dabdcb35c
I20240503 01:59:30.049710 623 catalog_manager.cc:1259] Loaded cluster ID: 627fe9b158b44677853fe11dabdcb35c
I20240503 01:59:30.049813 622 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841: loading cluster ID for follower catalog manager: success
I20240503 01:59:30.049867 623 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa: loading cluster ID for follower catalog manager: success
I20240503 01:59:30.051223 622 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841: acquiring CA information for follower catalog manager: success
I20240503 01:59:30.051244 623 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa: acquiring CA information for follower catalog manager: success
I20240503 01:59:30.051859 623 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 01:59:30.051859 622 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 01:59:30.111795 492 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60150
I20240503 01:59:30.148455 492 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60164
I20240503 01:59:30.183720 492 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60166
I20240503 01:59:31.060271 636 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 01:59:31.112979 712 heartbeater.cc:503] Master 127.0.99.61:45483 was elected leader, sending a full tablet report...
I20240503 01:59:31.149348 791 heartbeater.cc:503] Master 127.0.99.61:45483 was elected leader, sending a full tablet report...
I20240503 01:59:31.184669 881 heartbeater.cc:503] Master 127.0.99.61:45483 was elected leader, sending a full tablet report...
I20240503 01:59:31.337774 396 test_util.cc:268] Using random seed: 302919209
I20240503 01:59:31.351145 492 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60172:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 01:59:31.368389 674 tablet_service.cc:1467] Processing CreateTablet for tablet 31fa6e01d3fa4c25b7562eef70ca9b79 (DEFAULT_TABLE table=test-workload [id=7d9c6b7d234f437dae0b6c755e21b1b0]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 01:59:31.368686 753 tablet_service.cc:1467] Processing CreateTablet for tablet f2b6d348ccce4ad0bd65977cd3ba8c37 (DEFAULT_TABLE table=test-workload [id=7d9c6b7d234f437dae0b6c755e21b1b0]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 01:59:31.368870 674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31fa6e01d3fa4c25b7562eef70ca9b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.368914 753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b6d348ccce4ad0bd65977cd3ba8c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.371210 673 tablet_service.cc:1467] Processing CreateTablet for tablet 38d0f1a35eb6415fa317d58c98d14ea0 (DEFAULT_TABLE table=test-workload [id=7d9c6b7d234f437dae0b6c755e21b1b0]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 01:59:31.371394 914 tablet_bootstrap.cc:492] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670: Bootstrap starting.
I20240503 01:59:31.371523 843 tablet_service.cc:1467] Processing CreateTablet for tablet 3eebe628409d45b78c2b113949c1db11 (DEFAULT_TABLE table=test-workload [id=7d9c6b7d234f437dae0b6c755e21b1b0]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 01:59:31.371780 843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eebe628409d45b78c2b113949c1db11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.372231 914 tablet_bootstrap.cc:654] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.373117 914 tablet_bootstrap.cc:492] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670: No bootstrap required, opened a new log
I20240503 01:59:31.373188 914 ts_tablet_manager.cc:1367] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:31.373411 916 tablet_bootstrap.cc:492] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565: Bootstrap starting.
I20240503 01:59:31.373524 914 raft_consensus.cc:348] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaec0fda27b4cbe991d5bc62d14b670" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38973 } }
I20240503 01:59:31.373672 914 raft_consensus.cc:374] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.373929 917 tablet_bootstrap.cc:492] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac: Bootstrap starting.
I20240503 01:59:31.374061 673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d0f1a35eb6415fa317d58c98d14ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.374198 914 raft_consensus.cc:729] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcaec0fda27b4cbe991d5bc62d14b670, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.374498 914 consensus_queue.cc:260] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaec0fda27b4cbe991d5bc62d14b670" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38973 } }
I20240503 01:59:31.374594 914 raft_consensus.cc:388] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:31.374663 914 raft_consensus.cc:482] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:31.374715 914 raft_consensus.cc:3037] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.375243 917 tablet_bootstrap.cc:654] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.375659 914 raft_consensus.cc:504] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaec0fda27b4cbe991d5bc62d14b670" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38973 } }
I20240503 01:59:31.374293 916 tablet_bootstrap.cc:654] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.375805 914 leader_election.cc:302] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcaec0fda27b4cbe991d5bc62d14b670; no voters:
I20240503 01:59:31.376026 914 leader_election.cc:288] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:31.376363 914 ts_tablet_manager.cc:1398] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:31.376432 917 tablet_bootstrap.cc:492] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac: No bootstrap required, opened a new log
I20240503 01:59:31.376552 917 ts_tablet_manager.cc:1367] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:31.376339 920 raft_consensus.cc:2781] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.376821 920 raft_consensus.cc:686] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [term 1 LEADER]: Becoming Leader. State: Replica: bcaec0fda27b4cbe991d5bc62d14b670, State: Running, Role: LEADER
I20240503 01:59:31.376909 916 tablet_bootstrap.cc:492] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565: No bootstrap required, opened a new log
I20240503 01:59:31.377177 916 ts_tablet_manager.cc:1367] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:31.377163 920 consensus_queue.cc:237] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaec0fda27b4cbe991d5bc62d14b670" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38973 } }
I20240503 01:59:31.377214 917 raft_consensus.cc:348] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a665731b5f460d8d62f3ac897d5cac" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35751 } }
I20240503 01:59:31.377300 917 raft_consensus.cc:374] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.377327 917 raft_consensus.cc:729] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a665731b5f460d8d62f3ac897d5cac, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.377430 917 consensus_queue.cc:260] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a665731b5f460d8d62f3ac897d5cac" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35751 } }
I20240503 01:59:31.377524 917 raft_consensus.cc:388] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:31.377561 917 raft_consensus.cc:482] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:31.377599 917 raft_consensus.cc:3037] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.379513 492 catalog_manager.cc:5484] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcaec0fda27b4cbe991d5bc62d14b670 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "bcaec0fda27b4cbe991d5bc62d14b670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaec0fda27b4cbe991d5bc62d14b670" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38973 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:31.379817 917 raft_consensus.cc:504] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a665731b5f460d8d62f3ac897d5cac" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35751 } }
I20240503 01:59:31.379915 917 leader_election.cc:302] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5a665731b5f460d8d62f3ac897d5cac; no voters:
I20240503 01:59:31.380077 917 leader_election.cc:288] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:31.380360 917 ts_tablet_manager.cc:1398] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:31.380529 917 tablet_bootstrap.cc:492] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac: Bootstrap starting.
I20240503 01:59:31.380681 916 raft_consensus.cc:348] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c329ed9fbea84bf78e9e31797e6e2565" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 40867 } }
I20240503 01:59:31.380759 916 raft_consensus.cc:374] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.380789 916 raft_consensus.cc:729] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c329ed9fbea84bf78e9e31797e6e2565, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.381251 917 tablet_bootstrap.cc:654] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.381467 916 consensus_queue.cc:260] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c329ed9fbea84bf78e9e31797e6e2565" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 40867 } }
I20240503 01:59:31.381553 916 raft_consensus.cc:388] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:31.381587 916 raft_consensus.cc:482] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:31.381621 916 raft_consensus.cc:3037] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.382748 916 raft_consensus.cc:504] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c329ed9fbea84bf78e9e31797e6e2565" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 40867 } }
I20240503 01:59:31.383647 916 leader_election.cc:302] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c329ed9fbea84bf78e9e31797e6e2565; no voters:
I20240503 01:59:31.383850 916 leader_election.cc:288] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:31.384168 916 ts_tablet_manager.cc:1398] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:31.384318 928 raft_consensus.cc:2781] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 01:59:31.384461 792 tablet.cc:2369] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:31.384577 928 raft_consensus.cc:686] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [term 1 LEADER]: Becoming Leader. State: Replica: c329ed9fbea84bf78e9e31797e6e2565, State: Running, Role: LEADER
I20240503 01:59:31.384748 928 consensus_queue.cc:237] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c329ed9fbea84bf78e9e31797e6e2565" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 40867 } }
I20240503 01:59:31.385650 924 raft_consensus.cc:2781] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.385861 924 raft_consensus.cc:686] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [term 1 LEADER]: Becoming Leader. State: Replica: f5a665731b5f460d8d62f3ac897d5cac, State: Running, Role: LEADER
I20240503 01:59:31.385986 924 consensus_queue.cc:237] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a665731b5f460d8d62f3ac897d5cac" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35751 } }
I20240503 01:59:31.386931 917 tablet_bootstrap.cc:492] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac: No bootstrap required, opened a new log
I20240503 01:59:31.387043 917 ts_tablet_manager.cc:1367] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:31.387441 917 raft_consensus.cc:348] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a665731b5f460d8d62f3ac897d5cac" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35751 } }
I20240503 01:59:31.387523 917 raft_consensus.cc:374] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.387555 917 raft_consensus.cc:729] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a665731b5f460d8d62f3ac897d5cac, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.387650 917 consensus_queue.cc:260] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a665731b5f460d8d62f3ac897d5cac" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35751 } }
I20240503 01:59:31.387727 917 raft_consensus.cc:388] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:31.387764 917 raft_consensus.cc:482] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:31.387800 917 raft_consensus.cc:3037] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.388684 917 raft_consensus.cc:504] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a665731b5f460d8d62f3ac897d5cac" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35751 } }
I20240503 01:59:31.388806 917 leader_election.cc:302] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5a665731b5f460d8d62f3ac897d5cac; no voters:
I20240503 01:59:31.388900 917 leader_election.cc:288] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:31.388959 924 raft_consensus.cc:2781] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.389057 924 raft_consensus.cc:686] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [term 1 LEADER]: Becoming Leader. State: Replica: f5a665731b5f460d8d62f3ac897d5cac, State: Running, Role: LEADER
I20240503 01:59:31.389132 917 ts_tablet_manager.cc:1398] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 01:59:31.389223 924 consensus_queue.cc:237] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a665731b5f460d8d62f3ac897d5cac" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35751 } }
I20240503 01:59:31.389983 492 catalog_manager.cc:5484] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac reported cstate change: term changed from 0 to 1, leader changed from <none> to f5a665731b5f460d8d62f3ac897d5cac (127.0.99.1). New cstate: current_term: 1 leader_uuid: "f5a665731b5f460d8d62f3ac897d5cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a665731b5f460d8d62f3ac897d5cac" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35751 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:31.389983 491 catalog_manager.cc:5484] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 reported cstate change: term changed from 0 to 1, leader changed from <none> to c329ed9fbea84bf78e9e31797e6e2565 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "c329ed9fbea84bf78e9e31797e6e2565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c329ed9fbea84bf78e9e31797e6e2565" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 40867 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:31.394680 492 catalog_manager.cc:5484] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac reported cstate change: term changed from 0 to 1, leader changed from <none> to f5a665731b5f460d8d62f3ac897d5cac (127.0.99.1). New cstate: current_term: 1 leader_uuid: "f5a665731b5f460d8d62f3ac897d5cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a665731b5f460d8d62f3ac897d5cac" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35751 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:31.401139 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 01:59:31.405057 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 01:59:31.405292 396 tablet_replica.cc:330] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac: stopping tablet replica
I20240503 01:59:31.405519 396 raft_consensus.cc:2229] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:31.405640 396 raft_consensus.cc:2258] T 38d0f1a35eb6415fa317d58c98d14ea0 P f5a665731b5f460d8d62f3ac897d5cac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:31.406596 396 tablet_replica.cc:330] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac: stopping tablet replica
I20240503 01:59:31.406674 396 raft_consensus.cc:2229] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:31.406759 396 pending_rounds.cc:62] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac: Trying to abort 1 pending ops.
I20240503 01:59:31.406809 396 pending_rounds.cc:68] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023417636397445120 op_type: NO_OP noop_request { }
I20240503 01:59:31.406884 396 raft_consensus.cc:2866] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 01:59:31.406950 396 raft_consensus.cc:2258] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 01:59:31.423398 934 consensus_queue.cc:1159] T 31fa6e01d3fa4c25b7562eef70ca9b79 P f5a665731b5f460d8d62f3ac897d5cac [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f5a665731b5f460d8d62f3ac897d5cac" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20240503 01:59:31.437427 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 01:59:31.441033 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 01:59:31.444519 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 01:59:31.444730 396 tablet_replica.cc:330] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670: stopping tablet replica
I20240503 01:59:31.444844 396 raft_consensus.cc:2229] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:31.444954 396 raft_consensus.cc:2258] T f2b6d348ccce4ad0bd65977cd3ba8c37 P bcaec0fda27b4cbe991d5bc62d14b670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:31.457701 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 01:59:31.459537 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 01:59:31.462714 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 01:59:31.462941 396 tablet_replica.cc:330] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565: stopping tablet replica
I20240503 01:59:31.463078 396 raft_consensus.cc:2229] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:31.463197 396 raft_consensus.cc:2258] T 3eebe628409d45b78c2b113949c1db11 P c329ed9fbea84bf78e9e31797e6e2565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:31.475922 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 01:59:31.477609 396 master.cc:537] Master@127.0.99.62:35975 shutting down...
I20240503 01:59:31.480710 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:31.480847 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:31.480918 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P d47b384da7a84de4b92d3a65b2d30841: stopping tablet replica
I20240503 01:59:31.501281 396 master.cc:552] Master@127.0.99.62:35975 shutdown complete.
I20240503 01:59:31.503895 396 master.cc:537] Master@127.0.99.61:45483 shutting down...
I20240503 01:59:31.509095 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:31.509362 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:31.509420 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P b6ab12222bb64b1982db8c7169f8e400: stopping tablet replica
I20240503 01:59:31.522270 396 master.cc:552] Master@127.0.99.61:45483 shutdown complete.
I20240503 01:59:31.524917 396 master.cc:537] Master@127.0.99.60:37793 shutting down...
I20240503 01:59:31.528213 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:31.528352 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:31.528414 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 60a40ee749c34842a9ecdaeef65196fa: stopping tablet replica
I20240503 01:59:31.540796 396 master.cc:552] Master@127.0.99.60:37793 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2643 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240503 01:59:31.548336 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:43229
I20240503 01:59:31.548638 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:31.551270 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:31.552031 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.552897 396 server_base.cc:1034] running on GCE node
W20240503 01:59:31.553633 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.553844 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:31.553894 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:31.553920 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701571553918 us; error 0 us; skew 500 ppm
I20240503 01:59:31.554745 396 webserver.cc:460] Webserver started at http://127.0.99.62:36429/ using document root <none> and password file <none>
I20240503 01:59:31.554881 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:31.554930 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:31.555027 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:31.555397 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/master-0-root/instance:
uuid: "4010630478884cbba7144c4f99e15ae3"
format_stamp: "Formatted at 2024-05-03 01:59:31 on dist-test-slave-82l4"
I20240503 01:59:31.556713 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:31.557490 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.557727 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.557790 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/master-0-root
uuid: "4010630478884cbba7144c4f99e15ae3"
format_stamp: "Formatted at 2024-05-03 01:59:31 on dist-test-slave-82l4"
I20240503 01:59:31.557854 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:31.572113 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:31.572424 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:31.576189 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:43229
I20240503 01:59:31.577685 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.579381 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:43229 every 8 connection(s)
I20240503 01:59:31.580453 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3: Bootstrap starting.
I20240503 01:59:31.581151 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.582029 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3: No bootstrap required, opened a new log
I20240503 01:59:31.582362 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4010630478884cbba7144c4f99e15ae3" member_type: VOTER }
I20240503 01:59:31.582444 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.582477 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4010630478884cbba7144c4f99e15ae3, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.582605 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4010630478884cbba7144c4f99e15ae3" member_type: VOTER }
I20240503 01:59:31.582679 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:31.582713 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:31.582762 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.583441 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4010630478884cbba7144c4f99e15ae3" member_type: VOTER }
I20240503 01:59:31.583551 998 leader_election.cc:302] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4010630478884cbba7144c4f99e15ae3; no voters:
I20240503 01:59:31.583727 998 leader_election.cc:288] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:31.583813 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.584008 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [term 1 LEADER]: Becoming Leader. State: Replica: 4010630478884cbba7144c4f99e15ae3, State: Running, Role: LEADER
I20240503 01:59:31.584168 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:31.584141 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4010630478884cbba7144c4f99e15ae3" member_type: VOTER }
I20240503 01:59:31.584563 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4010630478884cbba7144c4f99e15ae3. Latest consensus state: current_term: 1 leader_uuid: "4010630478884cbba7144c4f99e15ae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4010630478884cbba7144c4f99e15ae3" member_type: VOTER } }
I20240503 01:59:31.584539 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4010630478884cbba7144c4f99e15ae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4010630478884cbba7144c4f99e15ae3" member_type: VOTER } }
I20240503 01:59:31.584641 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:31.584651 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:31.585870 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
W20240503 01:59:31.586275 1017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 01:59:31.586331 1017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 01:59:31.586405 1006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:31.587044 1006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:31.588591 1006 catalog_manager.cc:1347] Generated new cluster ID: 297e6fa516a3414d85d25826b260651d
I20240503 01:59:31.588647 1006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:31.597620 1006 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:31.598104 1006 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:31.606340 1006 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3: Generated new TSK 0
I20240503 01:59:31.606506 1006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:31.618011 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:31.620460 1019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:31.621377 1020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:31.622054 1022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.623131 396 server_base.cc:1034] running on GCE node
I20240503 01:59:31.623282 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:31.623328 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:31.623344 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701571623344 us; error 0 us; skew 500 ppm
I20240503 01:59:31.624177 396 webserver.cc:460] Webserver started at http://127.0.99.1:42857/ using document root <none> and password file <none>
I20240503 01:59:31.624302 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:31.624344 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:31.624411 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:31.624750 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-0-root/instance:
uuid: "de7ea1e0c0f442bc9ca238a03b9b4922"
format_stamp: "Formatted at 2024-05-03 01:59:31 on dist-test-slave-82l4"
I20240503 01:59:31.626067 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.626834 1027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.627075 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.627133 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-0-root
uuid: "de7ea1e0c0f442bc9ca238a03b9b4922"
format_stamp: "Formatted at 2024-05-03 01:59:31 on dist-test-slave-82l4"
I20240503 01:59:31.627198 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:31.653700 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:31.654064 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:31.654335 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:31.654781 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:31.654819 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.654866 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:31.654901 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.658600 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:36289
I20240503 01:59:31.658632 1090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:36289 every 8 connection(s)
I20240503 01:59:31.659412 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:31.662078 1095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:31.663120 1098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.663635 1091 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43229
I20240503 01:59:31.663720 1091 heartbeater.cc:464] Registering TS with master...
W20240503 01:59:31.663899 1096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.664029 1091 heartbeater.cc:511] Master 127.0.99.62:43229 requested a full tablet report, sending...
I20240503 01:59:31.664619 396 server_base.cc:1034] running on GCE node
I20240503 01:59:31.664623 962 ts_manager.cc:194] Registered new tserver with Master: de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289)
I20240503 01:59:31.664768 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:31.664808 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:31.664834 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701571664832 us; error 0 us; skew 500 ppm
I20240503 01:59:31.665395 962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33822
I20240503 01:59:31.665740 396 webserver.cc:460] Webserver started at http://127.0.99.2:33181/ using document root <none> and password file <none>
I20240503 01:59:31.665886 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:31.665936 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:31.666011 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:31.666373 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-1-root/instance:
uuid: "21c3f3afa337498aba4d3fea01d2c6d8"
format_stamp: "Formatted at 2024-05-03 01:59:31 on dist-test-slave-82l4"
I20240503 01:59:31.667788 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:31.668617 1103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.668875 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.668934 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-1-root
uuid: "21c3f3afa337498aba4d3fea01d2c6d8"
format_stamp: "Formatted at 2024-05-03 01:59:31 on dist-test-slave-82l4"
I20240503 01:59:31.668987 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:31.680130 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:31.680450 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:31.680692 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:31.681123 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:31.681160 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.681195 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:31.681224 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.685079 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:45487
I20240503 01:59:31.685161 1166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:45487 every 8 connection(s)
I20240503 01:59:31.685897 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:31.688652 1171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:31.689455 1170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.689693 1167 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43229
I20240503 01:59:31.689779 1167 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:31.689976 1167 heartbeater.cc:511] Master 127.0.99.62:43229 requested a full tablet report, sending...
W20240503 01:59:31.690371 1173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:31.690565 962 ts_manager.cc:194] Registered new tserver with Master: 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487)
I20240503 01:59:31.691093 396 server_base.cc:1034] running on GCE node
I20240503 01:59:31.691238 962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33830
I20240503 01:59:31.691287 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:31.691321 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:31.691349 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701571691349 us; error 0 us; skew 500 ppm
I20240503 01:59:31.692250 396 webserver.cc:460] Webserver started at http://127.0.99.3:44829/ using document root <none> and password file <none>
I20240503 01:59:31.692375 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:31.692420 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:31.692476 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:31.692811 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-2-root/instance:
uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8"
format_stamp: "Formatted at 2024-05-03 01:59:31 on dist-test-slave-82l4"
I20240503 01:59:31.694159 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.694927 1178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.695191 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.695250 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-2-root
uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8"
format_stamp: "Formatted at 2024-05-03 01:59:31 on dist-test-slave-82l4"
I20240503 01:59:31.695317 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:31.704277 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:31.704613 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:31.704877 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:31.705298 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:31.705335 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.705379 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:31.705413 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:31.709645 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:45325
I20240503 01:59:31.709697 1241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:45325 every 8 connection(s)
I20240503 01:59:31.713737 1242 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43229
I20240503 01:59:31.713848 1242 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:31.714095 1242 heartbeater.cc:511] Master 127.0.99.62:43229 requested a full tablet report, sending...
I20240503 01:59:31.714704 962 ts_manager.cc:194] Registered new tserver with Master: 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3:45325)
I20240503 01:59:31.715288 962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33834
I20240503 01:59:31.715608 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005583052s
I20240503 01:59:32.666345 1091 heartbeater.cc:503] Master 127.0.99.62:43229 was elected leader, sending a full tablet report...
I20240503 01:59:32.692073 1167 heartbeater.cc:503] Master 127.0.99.62:43229 was elected leader, sending a full tablet report...
I20240503 01:59:32.716197 1242 heartbeater.cc:503] Master 127.0.99.62:43229 was elected leader, sending a full tablet report...
I20240503 01:59:32.741935 396 test_util.cc:268] Using random seed: 304323372
I20240503 01:59:32.750537 962 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33842:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 01:59:32.751214 962 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 01:59:32.766314 1206 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6a4d2a93ea474f84a24d27fc8dd23e (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 01:59:32.766633 1201 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba9735a19ce44d9a2e72e151d7a8d27 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 01:59:32.766731 1051 tablet_service.cc:1467] Processing CreateTablet for tablet ee2fd9be51514278913c0a2db93f26d7 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 01:59:32.766816 1055 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6a4d2a93ea474f84a24d27fc8dd23e (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 01:59:32.766937 1205 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7f84458d244a67bb3404ccff14c861 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 01:59:32.767072 1199 tablet_service.cc:1467] Processing CreateTablet for tablet 840460feafde4b52b8bb4248f8bc0866 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 01:59:32.767175 1054 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7f84458d244a67bb3404ccff14c861 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 01:59:32.767128 1051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee2fd9be51514278913c0a2db93f26d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.767405 1054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7f84458d244a67bb3404ccff14c861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.767371 1204 tablet_service.cc:1467] Processing CreateTablet for tablet c20f17a418074b6885e26f1cf3aae794 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 01:59:32.767628 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6a4d2a93ea474f84a24d27fc8dd23e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.767771 1203 tablet_service.cc:1467] Processing CreateTablet for tablet 1dcf8b9c35cb42709aefb1596ae31006 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 01:59:32.767854 1053 tablet_service.cc:1467] Processing CreateTablet for tablet c20f17a418074b6885e26f1cf3aae794 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 01:59:32.768078 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20f17a418074b6885e26f1cf3aae794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.768151 1202 tablet_service.cc:1467] Processing CreateTablet for tablet ee2fd9be51514278913c0a2db93f26d7 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 01:59:32.768556 1200 tablet_service.cc:1467] Processing CreateTablet for tablet 69a0a37b42f649ce90219bb888c60e41 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 01:59:32.769047 1049 tablet_service.cc:1467] Processing CreateTablet for tablet 69a0a37b42f649ce90219bb888c60e41 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 01:59:32.769270 1049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a0a37b42f649ce90219bb888c60e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.766718 1206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6a4d2a93ea474f84a24d27fc8dd23e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.770148 1052 tablet_service.cc:1467] Processing CreateTablet for tablet 1dcf8b9c35cb42709aefb1596ae31006 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 01:59:32.770423 1052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dcf8b9c35cb42709aefb1596ae31006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.770613 1262 tablet_bootstrap.cc:492] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922: Bootstrap starting.
I20240503 01:59:32.771436 1048 tablet_service.cc:1467] Processing CreateTablet for tablet 840460feafde4b52b8bb4248f8bc0866 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 01:59:32.771692 1048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840460feafde4b52b8bb4248f8bc0866. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.772315 1262 tablet_bootstrap.cc:654] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.772478 1050 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba9735a19ce44d9a2e72e151d7a8d27 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 01:59:32.772732 1050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba9735a19ce44d9a2e72e151d7a8d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.773608 1262 tablet_bootstrap.cc:492] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922: No bootstrap required, opened a new log
I20240503 01:59:32.773692 1262 ts_tablet_manager.cc:1367] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:32.774134 1262 raft_consensus.cc:348] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.774271 1262 raft_consensus.cc:374] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.774307 1262 raft_consensus.cc:729] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7ea1e0c0f442bc9ca238a03b9b4922, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.774457 1262 consensus_queue.cc:260] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.770228 1199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840460feafde4b52b8bb4248f8bc0866. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.774812 1204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20f17a418074b6885e26f1cf3aae794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.774859 1262 ts_tablet_manager.cc:1398] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.775015 1205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7f84458d244a67bb3404ccff14c861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.775411 1201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba9735a19ce44d9a2e72e151d7a8d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.775658 1262 tablet_bootstrap.cc:492] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922: Bootstrap starting.
I20240503 01:59:32.776672 1262 tablet_bootstrap.cc:654] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.777525 1203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dcf8b9c35cb42709aefb1596ae31006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.780593 1265 tablet_bootstrap.cc:492] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Bootstrap starting.
I20240503 01:59:32.780746 1200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a0a37b42f649ce90219bb888c60e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.781342 1262 tablet_bootstrap.cc:492] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922: No bootstrap required, opened a new log
I20240503 01:59:32.781442 1262 ts_tablet_manager.cc:1367] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 01:59:32.781839 1131 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6a4d2a93ea474f84a24d27fc8dd23e (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 01:59:32.781931 1262 raft_consensus.cc:348] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } }
I20240503 01:59:32.782032 1265 tablet_bootstrap.cc:654] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.782059 1262 raft_consensus.cc:374] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.782191 1262 raft_consensus.cc:729] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7ea1e0c0f442bc9ca238a03b9b4922, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.782332 1262 consensus_queue.cc:260] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } }
I20240503 01:59:32.782357 1130 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7f84458d244a67bb3404ccff14c861 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 01:59:32.782665 1262 ts_tablet_manager.cc:1398] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.782748 1202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee2fd9be51514278913c0a2db93f26d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.782810 1129 tablet_service.cc:1467] Processing CreateTablet for tablet c20f17a418074b6885e26f1cf3aae794 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 01:59:32.782866 1262 tablet_bootstrap.cc:492] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922: Bootstrap starting.
I20240503 01:59:32.783268 1128 tablet_service.cc:1467] Processing CreateTablet for tablet 1dcf8b9c35cb42709aefb1596ae31006 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 01:59:32.783623 1265 tablet_bootstrap.cc:492] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: No bootstrap required, opened a new log
I20240503 01:59:32.783694 1265 ts_tablet_manager.cc:1367] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:32.783737 1127 tablet_service.cc:1467] Processing CreateTablet for tablet ee2fd9be51514278913c0a2db93f26d7 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 01:59:32.783973 1262 tablet_bootstrap.cc:654] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.784128 1126 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba9735a19ce44d9a2e72e151d7a8d27 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 01:59:32.784170 1265 raft_consensus.cc:348] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.784265 1265 raft_consensus.cc:374] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.784307 1265 raft_consensus.cc:729] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.784425 1265 consensus_queue.cc:260] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.784554 1125 tablet_service.cc:1467] Processing CreateTablet for tablet 69a0a37b42f649ce90219bb888c60e41 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 01:59:32.784770 1265 ts_tablet_manager.cc:1398] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.785019 1124 tablet_service.cc:1467] Processing CreateTablet for tablet 840460feafde4b52b8bb4248f8bc0866 (DEFAULT_TABLE table=test-workload [id=dbe4632473c248cc99342947b55cab90]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 01:59:32.785166 1265 tablet_bootstrap.cc:492] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Bootstrap starting.
I20240503 01:59:32.785880 1265 tablet_bootstrap.cc:654] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.782116 1131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6a4d2a93ea474f84a24d27fc8dd23e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.786867 1265 tablet_bootstrap.cc:492] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: No bootstrap required, opened a new log
I20240503 01:59:32.786952 1265 ts_tablet_manager.cc:1367] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:32.787343 1265 raft_consensus.cc:348] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.787429 1265 raft_consensus.cc:374] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.787465 1265 raft_consensus.cc:729] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.787587 1265 consensus_queue.cc:260] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.787804 1265 ts_tablet_manager.cc:1398] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.787956 1265 tablet_bootstrap.cc:492] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Bootstrap starting.
I20240503 01:59:32.788523 1271 tablet_bootstrap.cc:492] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8: Bootstrap starting.
I20240503 01:59:32.788781 1265 tablet_bootstrap.cc:654] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.789203 1130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7f84458d244a67bb3404ccff14c861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.789417 1127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee2fd9be51514278913c0a2db93f26d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.789680 1126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba9735a19ce44d9a2e72e151d7a8d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.790271 1271 tablet_bootstrap.cc:654] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.791441 1262 tablet_bootstrap.cc:492] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922: No bootstrap required, opened a new log
I20240503 01:59:32.791492 1125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69a0a37b42f649ce90219bb888c60e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.791532 1262 ts_tablet_manager.cc:1367] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 01:59:32.791563 1265 tablet_bootstrap.cc:492] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: No bootstrap required, opened a new log
I20240503 01:59:32.791618 1265 ts_tablet_manager.cc:1367] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 01:59:32.791985 1265 raft_consensus.cc:348] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.791985 1262 raft_consensus.cc:348] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.792129 1265 raft_consensus.cc:374] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.792136 1262 raft_consensus.cc:374] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.792176 1265 raft_consensus.cc:729] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.792187 1262 raft_consensus.cc:729] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7ea1e0c0f442bc9ca238a03b9b4922, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.792305 1265 consensus_queue.cc:260] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.792306 1262 consensus_queue.cc:260] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.792577 1262 ts_tablet_manager.cc:1398] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.792577 1265 ts_tablet_manager.cc:1398] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.792744 1262 tablet_bootstrap.cc:492] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922: Bootstrap starting.
I20240503 01:59:32.792744 1265 tablet_bootstrap.cc:492] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Bootstrap starting.
I20240503 01:59:32.793432 1128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dcf8b9c35cb42709aefb1596ae31006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.793653 1262 tablet_bootstrap.cc:654] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.793653 1265 tablet_bootstrap.cc:654] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.794934 1271 tablet_bootstrap.cc:492] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8: No bootstrap required, opened a new log
I20240503 01:59:32.795044 1271 ts_tablet_manager.cc:1367] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:32.795058 1124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840460feafde4b52b8bb4248f8bc0866. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.795280 1262 tablet_bootstrap.cc:492] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922: No bootstrap required, opened a new log
I20240503 01:59:32.795349 1262 ts_tablet_manager.cc:1367] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:32.795508 1271 raft_consensus.cc:348] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } }
I20240503 01:59:32.795603 1271 raft_consensus.cc:374] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.795636 1271 raft_consensus.cc:729] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c3f3afa337498aba4d3fea01d2c6d8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.795717 1262 raft_consensus.cc:348] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.795809 1262 raft_consensus.cc:374] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.795838 1262 raft_consensus.cc:729] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7ea1e0c0f442bc9ca238a03b9b4922, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.795965 1262 consensus_queue.cc:260] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.796217 1262 ts_tablet_manager.cc:1398] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.796347 1262 tablet_bootstrap.cc:492] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922: Bootstrap starting.
I20240503 01:59:32.796885 1129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20f17a418074b6885e26f1cf3aae794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.797075 1262 tablet_bootstrap.cc:654] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.795833 1271 consensus_queue.cc:260] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } }
I20240503 01:59:32.797989 1271 ts_tablet_manager.cc:1398] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:32.798156 1262 tablet_bootstrap.cc:492] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922: No bootstrap required, opened a new log
I20240503 01:59:32.798242 1262 ts_tablet_manager.cc:1367] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:32.798667 1265 tablet_bootstrap.cc:492] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: No bootstrap required, opened a new log
I20240503 01:59:32.798754 1265 ts_tablet_manager.cc:1367] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 01:59:32.798722 1262 raft_consensus.cc:348] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.798806 1262 raft_consensus.cc:374] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.798851 1262 raft_consensus.cc:729] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7ea1e0c0f442bc9ca238a03b9b4922, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.799147 1265 raft_consensus.cc:348] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.799252 1265 raft_consensus.cc:374] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.799288 1265 raft_consensus.cc:729] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.799401 1265 consensus_queue.cc:260] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.799407 1262 consensus_queue.cc:260] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.799696 1265 ts_tablet_manager.cc:1398] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.799759 1262 ts_tablet_manager.cc:1398] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.799849 1265 tablet_bootstrap.cc:492] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Bootstrap starting.
I20240503 01:59:32.799925 1262 tablet_bootstrap.cc:492] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922: Bootstrap starting.
I20240503 01:59:32.800813 1265 tablet_bootstrap.cc:654] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.801010 1262 tablet_bootstrap.cc:654] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.798161 1271 tablet_bootstrap.cc:492] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8: Bootstrap starting.
I20240503 01:59:32.802443 1271 tablet_bootstrap.cc:654] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.802592 1269 raft_consensus.cc:482] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.802678 1269 raft_consensus.cc:504] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.803248 1269 leader_election.cc:288] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487)
I20240503 01:59:32.804549 1265 tablet_bootstrap.cc:492] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: No bootstrap required, opened a new log
I20240503 01:59:32.804597 1262 tablet_bootstrap.cc:492] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922: No bootstrap required, opened a new log
I20240503 01:59:32.804646 1265 ts_tablet_manager.cc:1367] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:32.804682 1262 ts_tablet_manager.cc:1367] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:32.805027 1265 raft_consensus.cc:348] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } }
I20240503 01:59:32.805141 1265 raft_consensus.cc:374] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.805133 1262 raft_consensus.cc:348] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.805224 1265 raft_consensus.cc:729] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.805261 1262 raft_consensus.cc:374] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.805299 1262 raft_consensus.cc:729] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7ea1e0c0f442bc9ca238a03b9b4922, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.805356 1265 consensus_queue.cc:260] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } }
I20240503 01:59:32.805418 1262 consensus_queue.cc:260] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.805660 1265 ts_tablet_manager.cc:1398] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.805696 1262 ts_tablet_manager.cc:1398] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.805832 1265 tablet_bootstrap.cc:492] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Bootstrap starting.
I20240503 01:59:32.805866 1262 tablet_bootstrap.cc:492] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922: Bootstrap starting.
I20240503 01:59:32.806701 1265 tablet_bootstrap.cc:654] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.806878 1262 tablet_bootstrap.cc:654] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.808166 1262 tablet_bootstrap.cc:492] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922: No bootstrap required, opened a new log
I20240503 01:59:32.808275 1262 ts_tablet_manager.cc:1367] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:32.808470 1271 tablet_bootstrap.cc:492] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8: No bootstrap required, opened a new log
I20240503 01:59:32.808543 1271 ts_tablet_manager.cc:1367] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 01:59:32.808743 1262 raft_consensus.cc:348] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.808869 1262 raft_consensus.cc:374] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.808913 1262 raft_consensus.cc:729] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7ea1e0c0f442bc9ca238a03b9b4922, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.808979 1271 raft_consensus.cc:348] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.809036 1262 consensus_queue.cc:260] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.809082 1271 raft_consensus.cc:374] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.809163 1271 raft_consensus.cc:729] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c3f3afa337498aba4d3fea01d2c6d8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.809289 1271 consensus_queue.cc:260] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.809357 1262 ts_tablet_manager.cc:1398] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.809547 1271 ts_tablet_manager.cc:1398] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.809728 1271 tablet_bootstrap.cc:492] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8: Bootstrap starting.
I20240503 01:59:32.809995 1262 tablet_bootstrap.cc:492] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922: Bootstrap starting.
I20240503 01:59:32.810739 1271 tablet_bootstrap.cc:654] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.811014 1262 tablet_bootstrap.cc:654] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.812397 1265 tablet_bootstrap.cc:492] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: No bootstrap required, opened a new log
I20240503 01:59:32.812481 1265 ts_tablet_manager.cc:1367] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:32.812893 1265 raft_consensus.cc:348] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.813009 1265 raft_consensus.cc:374] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.813051 1265 raft_consensus.cc:729] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.813187 1265 consensus_queue.cc:260] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.813556 1265 ts_tablet_manager.cc:1398] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.813731 1265 tablet_bootstrap.cc:492] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Bootstrap starting.
I20240503 01:59:32.814003 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dcf8b9c35cb42709aefb1596ae31006" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" is_pre_election: true
I20240503 01:59:32.814075 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dcf8b9c35cb42709aefb1596ae31006" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" is_pre_election: true
I20240503 01:59:32.814157 1065 raft_consensus.cc:2454] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 0.
I20240503 01:59:32.814643 1265 tablet_bootstrap.cc:654] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Neither blocks nor log segments found. Creating new log.
W20240503 01:59:32.814590 1181 leader_election.cc:341] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:32.814776 1180 leader_election.cc:302] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, de7ea1e0c0f442bc9ca238a03b9b4922; no voters: 21c3f3afa337498aba4d3fea01d2c6d8
I20240503 01:59:32.815102 1269 raft_consensus.cc:2781] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.815168 1269 raft_consensus.cc:482] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.815227 1269 raft_consensus.cc:3037] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.815676 1271 tablet_bootstrap.cc:492] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8: No bootstrap required, opened a new log
I20240503 01:59:32.815781 1262 tablet_bootstrap.cc:492] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922: No bootstrap required, opened a new log
I20240503 01:59:32.815819 1271 ts_tablet_manager.cc:1367] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 01:59:32.815876 1262 ts_tablet_manager.cc:1367] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 01:59:32.816360 1262 raft_consensus.cc:348] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.816360 1271 raft_consensus.cc:348] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.816498 1262 raft_consensus.cc:374] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.816546 1262 raft_consensus.cc:729] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7ea1e0c0f442bc9ca238a03b9b4922, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.816555 1271 raft_consensus.cc:374] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.816670 1271 raft_consensus.cc:729] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c3f3afa337498aba4d3fea01d2c6d8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.816681 1262 consensus_queue.cc:260] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.816820 1271 consensus_queue.cc:260] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.816972 1262 ts_tablet_manager.cc:1398] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.817159 1271 ts_tablet_manager.cc:1398] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.817329 1271 tablet_bootstrap.cc:492] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8: Bootstrap starting.
I20240503 01:59:32.817677 1265 tablet_bootstrap.cc:492] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: No bootstrap required, opened a new log
I20240503 01:59:32.817763 1265 ts_tablet_manager.cc:1367] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:32.818140 1271 tablet_bootstrap.cc:654] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.818205 1265 raft_consensus.cc:348] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.818307 1265 raft_consensus.cc:374] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.818341 1265 raft_consensus.cc:729] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.818450 1265 consensus_queue.cc:260] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.818852 1265 ts_tablet_manager.cc:1398] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.819067 1265 tablet_bootstrap.cc:492] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Bootstrap starting.
I20240503 01:59:32.819164 1271 tablet_bootstrap.cc:492] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8: No bootstrap required, opened a new log
I20240503 01:59:32.819253 1271 ts_tablet_manager.cc:1367] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:32.819589 1271 raft_consensus.cc:348] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.819669 1271 raft_consensus.cc:374] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.819705 1271 raft_consensus.cc:729] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c3f3afa337498aba4d3fea01d2c6d8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.819823 1271 consensus_queue.cc:260] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.819960 1265 tablet_bootstrap.cc:654] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.820065 1271 ts_tablet_manager.cc:1398] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.820226 1271 tablet_bootstrap.cc:492] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8: Bootstrap starting.
I20240503 01:59:32.821038 1271 tablet_bootstrap.cc:654] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.821694 1265 tablet_bootstrap.cc:492] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: No bootstrap required, opened a new log
I20240503 01:59:32.821771 1265 ts_tablet_manager.cc:1367] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:32.822177 1265 raft_consensus.cc:348] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.822283 1265 raft_consensus.cc:374] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.822328 1265 raft_consensus.cc:729] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.822455 1265 consensus_queue.cc:260] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.822746 1265 ts_tablet_manager.cc:1398] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.823390 1271 tablet_bootstrap.cc:492] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8: No bootstrap required, opened a new log
I20240503 01:59:32.823477 1271 ts_tablet_manager.cc:1367] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:32.823817 1271 raft_consensus.cc:348] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.823915 1271 raft_consensus.cc:374] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.823952 1271 raft_consensus.cc:729] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c3f3afa337498aba4d3fea01d2c6d8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.824072 1271 consensus_queue.cc:260] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.824326 1271 ts_tablet_manager.cc:1398] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.824635 1271 tablet_bootstrap.cc:492] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8: Bootstrap starting.
I20240503 01:59:32.825407 1271 tablet_bootstrap.cc:654] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.826448 1271 tablet_bootstrap.cc:492] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8: No bootstrap required, opened a new log
I20240503 01:59:32.826526 1271 ts_tablet_manager.cc:1367] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:32.826884 1271 raft_consensus.cc:348] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.827028 1271 raft_consensus.cc:374] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.827077 1271 raft_consensus.cc:729] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c3f3afa337498aba4d3fea01d2c6d8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.827193 1271 consensus_queue.cc:260] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.827469 1271 ts_tablet_manager.cc:1398] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.827684 1271 tablet_bootstrap.cc:492] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8: Bootstrap starting.
I20240503 01:59:32.828603 1271 tablet_bootstrap.cc:654] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.829602 1271 tablet_bootstrap.cc:492] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8: No bootstrap required, opened a new log
I20240503 01:59:32.829690 1271 ts_tablet_manager.cc:1367] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:32.830030 1271 raft_consensus.cc:348] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.830111 1271 raft_consensus.cc:374] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.830132 1271 raft_consensus.cc:729] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c3f3afa337498aba4d3fea01d2c6d8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.830226 1271 consensus_queue.cc:260] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.830487 1271 ts_tablet_manager.cc:1398] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.830710 1271 tablet_bootstrap.cc:492] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8: Bootstrap starting.
I20240503 01:59:32.832190 1271 tablet_bootstrap.cc:654] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.833220 1271 tablet_bootstrap.cc:492] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8: No bootstrap required, opened a new log
I20240503 01:59:32.833299 1271 ts_tablet_manager.cc:1367] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:32.833670 1271 raft_consensus.cc:348] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.833758 1271 raft_consensus.cc:374] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.833809 1271 raft_consensus.cc:729] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c3f3afa337498aba4d3fea01d2c6d8, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.833935 1271 consensus_queue.cc:260] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.834169 1271 ts_tablet_manager.cc:1398] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.835224 1269 raft_consensus.cc:504] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.835641 1269 leader_election.cc:288] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 election: Requested vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487)
I20240503 01:59:32.835916 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dcf8b9c35cb42709aefb1596ae31006" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922"
I20240503 01:59:32.836028 1065 raft_consensus.cc:3037] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.835974 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dcf8b9c35cb42709aefb1596ae31006" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8"
I20240503 01:59:32.836170 1141 raft_consensus.cc:3037] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.836951 1141 raft_consensus.cc:2454] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 1.
I20240503 01:59:32.837257 1181 leader_election.cc:302] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, 21c3f3afa337498aba4d3fea01d2c6d8; no voters:
I20240503 01:59:32.837255 1065 raft_consensus.cc:2454] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 1.
I20240503 01:59:32.837568 1269 raft_consensus.cc:2781] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.837646 1269 raft_consensus.cc:686] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Becoming Leader. State: Replica: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, State: Running, Role: LEADER
I20240503 01:59:32.837785 1269 consensus_queue.cc:237] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.840600 961 catalog_manager.cc:5484] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:32.857049 1289 raft_consensus.cc:482] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.857230 1289 raft_consensus.cc:504] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.857766 1289 leader_election.cc:288] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487)
I20240503 01:59:32.858260 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20f17a418074b6885e26f1cf3aae794" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" is_pre_election: true
I20240503 01:59:32.858404 1141 raft_consensus.cc:2454] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 0.
I20240503 01:59:32.858681 1181 leader_election.cc:302] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, 21c3f3afa337498aba4d3fea01d2c6d8; no voters:
I20240503 01:59:32.858848 1289 raft_consensus.cc:2781] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.858906 1289 raft_consensus.cc:482] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.858984 1289 raft_consensus.cc:3037] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.858907 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20f17a418074b6885e26f1cf3aae794" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" is_pre_election: true
I20240503 01:59:32.859129 1065 raft_consensus.cc:2454] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 0.
I20240503 01:59:32.886523 1269 raft_consensus.cc:482] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.886672 1269 raft_consensus.cc:504] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.887195 1269 leader_election.cc:288] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487)
I20240503 01:59:32.887286 1296 raft_consensus.cc:482] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.887372 1296 raft_consensus.cc:504] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.887627 1264 raft_consensus.cc:482] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.887692 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a0a37b42f649ce90219bb888c60e41" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" is_pre_election: true
I20240503 01:59:32.887764 1296 leader_election.cc:288] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487)
I20240503 01:59:32.887745 1264 raft_consensus.cc:504] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.887809 1141 raft_consensus.cc:2454] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 0.
I20240503 01:59:32.888164 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba9735a19ce44d9a2e72e151d7a8d27" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" is_pre_election: true
I20240503 01:59:32.888249 1264 leader_election.cc:288] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487), 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3:45325)
I20240503 01:59:32.888332 1141 raft_consensus.cc:2454] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 0.
I20240503 01:59:32.888166 1181 leader_election.cc:302] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, 21c3f3afa337498aba4d3fea01d2c6d8; no voters:
I20240503 01:59:32.888797 1064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba9735a19ce44d9a2e72e151d7a8d27" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" is_pre_election: true
I20240503 01:59:32.888864 1181 leader_election.cc:302] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, 21c3f3afa337498aba4d3fea01d2c6d8; no voters:
I20240503 01:59:32.888916 1064 raft_consensus.cc:2454] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 0.
I20240503 01:59:32.889068 1269 raft_consensus.cc:2781] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.889118 1269 raft_consensus.cc:482] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.889163 1269 raft_consensus.cc:3037] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.889312 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a0a37b42f649ce90219bb888c60e41" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" is_pre_election: true
I20240503 01:59:32.889407 1065 raft_consensus.cc:2454] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 0.
I20240503 01:59:32.889631 1296 raft_consensus.cc:2781] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.889688 1296 raft_consensus.cc:482] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.889743 1296 raft_consensus.cc:3037] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.892859 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba9735a19ce44d9a2e72e151d7a8d27" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" is_pre_election: true
I20240503 01:59:32.893005 1141 raft_consensus.cc:2454] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de7ea1e0c0f442bc9ca238a03b9b4922 in term 0.
I20240503 01:59:32.893035 1216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba9735a19ce44d9a2e72e151d7a8d27" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" is_pre_election: true
I20240503 01:59:32.893325 1030 leader_election.cc:302] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c3f3afa337498aba4d3fea01d2c6d8, de7ea1e0c0f442bc9ca238a03b9b4922; no voters:
I20240503 01:59:32.893514 1264 raft_consensus.cc:2781] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.893579 1264 raft_consensus.cc:482] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.893625 1264 raft_consensus.cc:3037] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.935806 1289 raft_consensus.cc:504] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.935806 1264 raft_consensus.cc:504] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.936431 1289 leader_election.cc:288] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 election: Requested vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487)
I20240503 01:59:32.936529 1264 leader_election.cc:288] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 1 election: Requested vote from peers 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487), 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3:45325)
I20240503 01:59:32.936690 1269 raft_consensus.cc:504] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.936978 1269 leader_election.cc:288] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 election: Requested vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487)
I20240503 01:59:32.937444 1141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba9735a19ce44d9a2e72e151d7a8d27" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8"
I20240503 01:59:32.937575 1141 raft_consensus.cc:3037] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.937649 1140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20f17a418074b6885e26f1cf3aae794" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8"
I20240503 01:59:32.937758 1140 raft_consensus.cc:3037] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.937932 1296 raft_consensus.cc:504] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.938145 1216 raft_consensus.cc:2379] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de7ea1e0c0f442bc9ca238a03b9b4922 in current term 1: Already voted for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in this term.
I20240503 01:59:32.938652 1141 raft_consensus.cc:2454] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de7ea1e0c0f442bc9ca238a03b9b4922 in term 1.
I20240503 01:59:32.938774 1140 raft_consensus.cc:2454] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 1.
I20240503 01:59:32.938823 1296 leader_election.cc:288] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 election: Requested vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487)
I20240503 01:59:32.939033 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a0a37b42f649ce90219bb888c60e41" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8"
I20240503 01:59:32.939141 1139 raft_consensus.cc:3037] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.939231 1030 leader_election.cc:302] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c3f3afa337498aba4d3fea01d2c6d8, de7ea1e0c0f442bc9ca238a03b9b4922; no voters:
I20240503 01:59:32.939447 1181 leader_election.cc:302] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, 21c3f3afa337498aba4d3fea01d2c6d8; no voters:
I20240503 01:59:32.939549 1140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba9735a19ce44d9a2e72e151d7a8d27" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8"
I20240503 01:59:32.939667 1140 raft_consensus.cc:2379] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in current term 1: Already voted for candidate de7ea1e0c0f442bc9ca238a03b9b4922 in this term.
I20240503 01:59:32.940169 1139 raft_consensus.cc:2454] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 1.
I20240503 01:59:32.940356 1264 raft_consensus.cc:2781] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.940413 1064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69a0a37b42f649ce90219bb888c60e41" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922"
I20240503 01:59:32.940435 1181 leader_election.cc:302] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, 21c3f3afa337498aba4d3fea01d2c6d8; no voters:
I20240503 01:59:32.940518 1064 raft_consensus.cc:3037] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.940593 1269 raft_consensus.cc:2781] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.940737 1296 raft_consensus.cc:2781] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.940179 1216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba9735a19ce44d9a2e72e151d7a8d27" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8"
I20240503 01:59:32.940958 1216 raft_consensus.cc:2379] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate de7ea1e0c0f442bc9ca238a03b9b4922 in current term 1: Already voted for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in this term.
I20240503 01:59:32.941061 1296 raft_consensus.cc:686] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Becoming Leader. State: Replica: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, State: Running, Role: LEADER
I20240503 01:59:32.940601 1264 raft_consensus.cc:686] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 LEADER]: Becoming Leader. State: Replica: de7ea1e0c0f442bc9ca238a03b9b4922, State: Running, Role: LEADER
I20240503 01:59:32.941180 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20f17a418074b6885e26f1cf3aae794" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922"
I20240503 01:59:32.941283 1065 raft_consensus.cc:3037] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.940357 1063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba9735a19ce44d9a2e72e151d7a8d27" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922"
I20240503 01:59:32.940668 1269 raft_consensus.cc:686] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Becoming Leader. State: Replica: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, State: Running, Role: LEADER
I20240503 01:59:32.941509 1269 consensus_queue.cc:237] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.941385 1296 consensus_queue.cc:237] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.941753 1180 leader_election.cc:302] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8; no voters: 21c3f3afa337498aba4d3fea01d2c6d8, de7ea1e0c0f442bc9ca238a03b9b4922
I20240503 01:59:32.941308 1264 consensus_queue.cc:237] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.941922 1300 raft_consensus.cc:2726] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 01:59:32.943919 960 catalog_manager.cc:5484] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 reported cstate change: term changed from 0 to 1, leader changed from <none> to de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:32.944064 961 catalog_manager.cc:5484] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:32.944342 961 catalog_manager.cc:5484] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:32.963986 1277 raft_consensus.cc:482] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.964133 1277 raft_consensus.cc:504] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:32.964686 1277 leader_election.cc:288] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3:45325)
I20240503 01:59:32.968071 1063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840460feafde4b52b8bb4248f8bc0866" candidate_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" is_pre_election: true
I20240503 01:59:32.968071 1216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840460feafde4b52b8bb4248f8bc0866" candidate_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" is_pre_election: true
I20240503 01:59:32.968241 1063 raft_consensus.cc:2454] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c3f3afa337498aba4d3fea01d2c6d8 in term 0.
I20240503 01:59:32.968295 1216 raft_consensus.cc:2454] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c3f3afa337498aba4d3fea01d2c6d8 in term 0.
I20240503 01:59:32.968541 1107 leader_election.cc:302] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, 21c3f3afa337498aba4d3fea01d2c6d8; no voters:
I20240503 01:59:32.968694 1277 raft_consensus.cc:2781] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.968760 1277 raft_consensus.cc:482] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.968791 1277 raft_consensus.cc:3037] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.020449 1269 raft_consensus.cc:482] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.020643 1269 raft_consensus.cc:504] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:33.021152 1269 leader_election.cc:288] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487)
I20240503 01:59:33.021631 1063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f84458d244a67bb3404ccff14c861" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" is_pre_election: true
I20240503 01:59:33.021631 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f84458d244a67bb3404ccff14c861" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" is_pre_election: true
I20240503 01:59:33.021802 1063 raft_consensus.cc:2454] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 0.
I20240503 01:59:33.021831 1139 raft_consensus.cc:2454] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 0.
I20240503 01:59:33.022120 1181 leader_election.cc:302] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, 21c3f3afa337498aba4d3fea01d2c6d8; no voters:
I20240503 01:59:33.022315 1269 raft_consensus.cc:2781] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.022375 1269 raft_consensus.cc:482] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.022411 1269 raft_consensus.cc:3037] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.035449 1277 raft_consensus.cc:504] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:33.035761 1269 raft_consensus.cc:504] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:33.035442 1064 raft_consensus.cc:2454] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 1.
I20240503 01:59:33.036536 1065 raft_consensus.cc:2454] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 1.
I20240503 01:59:33.036641 1064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f84458d244a67bb3404ccff14c861" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922"
I20240503 01:59:33.036762 1064 raft_consensus.cc:3037] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.036048 1269 leader_election.cc:288] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 election: Requested vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487)
I20240503 01:59:33.036048 1277 leader_election.cc:288] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [CANDIDATE]: Term 1 election: Requested vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3:45325)
I20240503 01:59:33.037037 1216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840460feafde4b52b8bb4248f8bc0866" candidate_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8"
I20240503 01:59:33.037037 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f84458d244a67bb3404ccff14c861" candidate_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8"
I20240503 01:59:33.037165 1216 raft_consensus.cc:3037] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.037205 1139 raft_consensus.cc:3037] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.037353 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840460feafde4b52b8bb4248f8bc0866" candidate_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922"
I20240503 01:59:33.037452 1065 raft_consensus.cc:3037] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.037808 1064 raft_consensus.cc:2454] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 1.
I20240503 01:59:33.038197 1180 leader_election.cc:302] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, de7ea1e0c0f442bc9ca238a03b9b4922; no voters:
I20240503 01:59:33.038336 1065 raft_consensus.cc:2454] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c3f3afa337498aba4d3fea01d2c6d8 in term 1.
I20240503 01:59:33.038468 1216 raft_consensus.cc:2454] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c3f3afa337498aba4d3fea01d2c6d8 in term 1.
I20240503 01:59:33.038516 1269 raft_consensus.cc:2781] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.038622 1269 raft_consensus.cc:686] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Becoming Leader. State: Replica: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, State: Running, Role: LEADER
I20240503 01:59:33.038694 1105 leader_election.cc:302] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c3f3afa337498aba4d3fea01d2c6d8, de7ea1e0c0f442bc9ca238a03b9b4922; no voters:
I20240503 01:59:33.038853 1277 raft_consensus.cc:2781] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.038806 1269 consensus_queue.cc:237] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:33.039098 1277 raft_consensus.cc:686] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 LEADER]: Becoming Leader. State: Replica: 21c3f3afa337498aba4d3fea01d2c6d8, State: Running, Role: LEADER
I20240503 01:59:33.039248 1277 consensus_queue.cc:237] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:33.040678 961 catalog_manager.cc:5484] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:33.040681 960 catalog_manager.cc:5484] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:33.064071 1139 raft_consensus.cc:2454] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 106cab8aa6eb4a7ba92ecc6eb6daa9f8 in term 1.
I20240503 01:59:33.166939 1264 raft_consensus.cc:482] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.167129 1264 raft_consensus.cc:504] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } }
I20240503 01:59:33.167766 1264 leader_election.cc:288] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3:45325), 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487)
I20240503 01:59:33.168195 1216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6a4d2a93ea474f84a24d27fc8dd23e" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" is_pre_election: true
I20240503 01:59:33.168207 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6a4d2a93ea474f84a24d27fc8dd23e" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" is_pre_election: true
I20240503 01:59:33.168352 1216 raft_consensus.cc:2454] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de7ea1e0c0f442bc9ca238a03b9b4922 in term 0.
I20240503 01:59:33.168386 1139 raft_consensus.cc:2454] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de7ea1e0c0f442bc9ca238a03b9b4922 in term 0.
I20240503 01:59:33.168713 1031 leader_election.cc:302] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, de7ea1e0c0f442bc9ca238a03b9b4922; no voters:
I20240503 01:59:33.168941 1264 raft_consensus.cc:2781] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.168998 1264 raft_consensus.cc:482] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.169050 1264 raft_consensus.cc:3037] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.179176 1277 raft_consensus.cc:482] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.179386 1277 raft_consensus.cc:504] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:33.179476 1299 raft_consensus.cc:482] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.179616 1299 raft_consensus.cc:504] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:33.179888 1277 leader_election.cc:288] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3:45325)
I20240503 01:59:33.180037 1299 leader_election.cc:288] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487), 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3:45325)
I20240503 01:59:33.180320 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2fd9be51514278913c0a2db93f26d7" candidate_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" is_pre_election: true
I20240503 01:59:33.180397 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2fd9be51514278913c0a2db93f26d7" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" is_pre_election: true
I20240503 01:59:33.180320 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2fd9be51514278913c0a2db93f26d7" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" is_pre_election: true
I20240503 01:59:33.180500 1065 raft_consensus.cc:2454] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c3f3afa337498aba4d3fea01d2c6d8 in term 0.
I20240503 01:59:33.180562 1139 raft_consensus.cc:2454] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de7ea1e0c0f442bc9ca238a03b9b4922 in term 0.
I20240503 01:59:33.180537 1215 raft_consensus.cc:2454] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de7ea1e0c0f442bc9ca238a03b9b4922 in term 0.
I20240503 01:59:33.180842 1030 leader_election.cc:302] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c3f3afa337498aba4d3fea01d2c6d8, de7ea1e0c0f442bc9ca238a03b9b4922; no voters:
I20240503 01:59:33.180842 1105 leader_election.cc:302] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c3f3afa337498aba4d3fea01d2c6d8, de7ea1e0c0f442bc9ca238a03b9b4922; no voters:
I20240503 01:59:33.180320 1216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2fd9be51514278913c0a2db93f26d7" candidate_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" is_pre_election: true
I20240503 01:59:33.181061 1216 raft_consensus.cc:2454] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c3f3afa337498aba4d3fea01d2c6d8 in term 0.
I20240503 01:59:33.181118 1277 raft_consensus.cc:2781] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.181037 1299 raft_consensus.cc:2781] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.181192 1277 raft_consensus.cc:482] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.181217 1299 raft_consensus.cc:482] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.181263 1277 raft_consensus.cc:3037] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.181268 1299 raft_consensus.cc:3037] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.238816 1269 consensus_queue.cc:1035] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.242265 1269 consensus_queue.cc:1035] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.261238 1264 raft_consensus.cc:504] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } }
I20240503 01:59:33.261238 1277 raft_consensus.cc:504] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:33.261870 1264 leader_election.cc:288] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 1 election: Requested vote from peers 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3:45325), 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487)
I20240503 01:59:33.261896 1277 leader_election.cc:288] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [CANDIDATE]: Term 1 election: Requested vote from peers de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1:36289), 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3:45325)
I20240503 01:59:33.262151 1299 raft_consensus.cc:504] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:33.262444 1299 leader_election.cc:288] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 1 election: Requested vote from peers 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487), 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3:45325)
I20240503 01:59:33.262419 1216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6a4d2a93ea474f84a24d27fc8dd23e" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8"
I20240503 01:59:33.262610 1216 raft_consensus.cc:3037] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.262419 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2fd9be51514278913c0a2db93f26d7" candidate_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922"
I20240503 01:59:33.262738 1065 raft_consensus.cc:2379] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 21c3f3afa337498aba4d3fea01d2c6d8 in current term 1: Already voted for candidate de7ea1e0c0f442bc9ca238a03b9b4922 in this term.
I20240503 01:59:33.263056 1140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2fd9be51514278913c0a2db93f26d7" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8"
I20240503 01:59:33.263168 1140 raft_consensus.cc:2379] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate de7ea1e0c0f442bc9ca238a03b9b4922 in current term 1: Already voted for candidate 21c3f3afa337498aba4d3fea01d2c6d8 in this term.
I20240503 01:59:33.263566 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2fd9be51514278913c0a2db93f26d7" candidate_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8"
I20240503 01:59:33.263676 1215 raft_consensus.cc:3037] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.263646 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6a4d2a93ea474f84a24d27fc8dd23e" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8"
I20240503 01:59:33.263659 1216 raft_consensus.cc:2454] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de7ea1e0c0f442bc9ca238a03b9b4922 in term 1.
I20240503 01:59:33.263831 1139 raft_consensus.cc:3037] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.264459 1031 leader_election.cc:302] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, de7ea1e0c0f442bc9ca238a03b9b4922; no voters:
I20240503 01:59:33.264775 1216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2fd9be51514278913c0a2db93f26d7" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8"
I20240503 01:59:33.264925 1215 raft_consensus.cc:2454] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c3f3afa337498aba4d3fea01d2c6d8 in term 1.
I20240503 01:59:33.264992 1299 raft_consensus.cc:2781] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.265074 1299 raft_consensus.cc:686] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 LEADER]: Becoming Leader. State: Replica: de7ea1e0c0f442bc9ca238a03b9b4922, State: Running, Role: LEADER
I20240503 01:59:33.264935 1139 raft_consensus.cc:2454] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de7ea1e0c0f442bc9ca238a03b9b4922 in term 1.
I20240503 01:59:33.265245 1299 consensus_queue.cc:237] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } }
I20240503 01:59:33.265286 1107 leader_election.cc:302] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, 21c3f3afa337498aba4d3fea01d2c6d8; no voters: de7ea1e0c0f442bc9ca238a03b9b4922
I20240503 01:59:33.265614 1277 raft_consensus.cc:2781] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.265712 1277 raft_consensus.cc:686] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 LEADER]: Becoming Leader. State: Replica: 21c3f3afa337498aba4d3fea01d2c6d8, State: Running, Role: LEADER
I20240503 01:59:33.265806 1277 consensus_queue.cc:237] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:33.265619 1031 leader_election.cc:302] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de7ea1e0c0f442bc9ca238a03b9b4922; no voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, 21c3f3afa337498aba4d3fea01d2c6d8
I20240503 01:59:33.266395 1299 raft_consensus.cc:2726] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 01:59:33.266949 960 catalog_manager.cc:5484] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 reported cstate change: term changed from 0 to 1, leader changed from <none> to de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:33.267374 961 catalog_manager.cc:5484] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:33.318753 1299 consensus_queue.cc:1035] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.321705 1299 consensus_queue.cc:1035] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.333045 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:33.335716 1323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:33.336659 1321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:33.337456 1320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:33.338569 396 server_base.cc:1034] running on GCE node
I20240503 01:59:33.338800 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:33.338842 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:33.338860 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701573338859 us; error 0 us; skew 500 ppm
I20240503 01:59:33.339951 396 webserver.cc:460] Webserver started at http://127.0.99.4:45245/ using document root <none> and password file <none>
I20240503 01:59:33.340093 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:33.340143 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:33.340209 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:33.340639 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-3-root/instance:
uuid: "9ed7ed50af5d435baeb6ffdf8e227e81"
format_stamp: "Formatted at 2024-05-03 01:59:33 on dist-test-slave-82l4"
I20240503 01:59:33.342092 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:33.343106 1328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:33.343381 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.343456 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-3-root
uuid: "9ed7ed50af5d435baeb6ffdf8e227e81"
format_stamp: "Formatted at 2024-05-03 01:59:33 on dist-test-slave-82l4"
I20240503 01:59:33.343533 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714701568895932-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:33.360886 1269 consensus_queue.cc:1035] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.365146 1269 consensus_queue.cc:1035] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.378315 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:33.378747 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:33.379105 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:33.379609 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:33.379654 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:33.379705 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:33.379739 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:33.384200 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:36583
I20240503 01:59:33.384260 1393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:36583 every 8 connection(s)
I20240503 01:59:33.385548 1269 consensus_queue.cc:1035] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.389529 1296 consensus_queue.cc:1035] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.391036 1394 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43229
I20240503 01:59:33.391172 1394 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:33.391441 1394 heartbeater.cc:511] Master 127.0.99.62:43229 requested a full tablet report, sending...
I20240503 01:59:33.391974 961 ts_manager.cc:194] Registered new tserver with Master: 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4:36583)
I20240503 01:59:33.392565 961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33844
I20240503 01:59:33.528839 1269 consensus_queue.cc:1035] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.532358 1300 consensus_queue.cc:1035] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.586120 1016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 01:59:33.591674 1215 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1d7f84458d244a67bb3404ccff14c861"
dest_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8"
mode: GRACEFUL
new_leader_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922"
from {username='slave'} at 127.0.0.1:59920
I20240503 01:59:33.591874 1215 raft_consensus.cc:595] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Received request to transfer leadership to TS de7ea1e0c0f442bc9ca238a03b9b4922
I20240503 01:59:33.592298 1016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 01:59:33.592535 1016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 01:59:33.626314 1307 consensus_queue.cc:1035] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.627187 1307 consensus_queue.cc:1035] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.651979 1299 consensus_queue.cc:1035] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.654819 1310 consensus_queue.cc:1035] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.661301 1406 consensus_queue.cc:1035] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.664405 1277 consensus_queue.cc:1035] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.079325 1269 raft_consensus.cc:979] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: : Instructing follower de7ea1e0c0f442bc9ca238a03b9b4922 to start an election
I20240503 01:59:34.079453 1269 raft_consensus.cc:1067] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Signalling peer de7ea1e0c0f442bc9ca238a03b9b4922 to start an election
I20240503 01:59:34.080027 1065 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d7f84458d244a67bb3404ccff14c861"
dest_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922"
from {username='slave'} at 127.0.0.1:54738
I20240503 01:59:34.080180 1065 raft_consensus.cc:482] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:34.080245 1065 raft_consensus.cc:3037] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.150897 1065 raft_consensus.cc:504] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:34.151491 1065 leader_election.cc:288] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 2 election: Requested vote from peers 21c3f3afa337498aba4d3fea01d2c6d8 (127.0.99.2:45487), 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3:45325)
I20240503 01:59:34.151955 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f84458d244a67bb3404ccff14c861" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8"
I20240503 01:59:34.151983 1139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f84458d244a67bb3404ccff14c861" candidate_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "21c3f3afa337498aba4d3fea01d2c6d8"
I20240503 01:59:34.152089 1215 raft_consensus.cc:3032] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:34.152133 1215 raft_consensus.cc:729] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, State: Running, Role: LEADER
I20240503 01:59:34.152125 1139 raft_consensus.cc:3037] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.152346 1215 consensus_queue.cc:260] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:34.152525 1215 raft_consensus.cc:3037] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:34.153275 1215 raft_consensus.cc:2454] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de7ea1e0c0f442bc9ca238a03b9b4922 in term 2.
I20240503 01:59:34.153366 1139 raft_consensus.cc:2454] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de7ea1e0c0f442bc9ca238a03b9b4922 in term 2.
I20240503 01:59:34.153610 1031 leader_election.cc:302] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106cab8aa6eb4a7ba92ecc6eb6daa9f8, de7ea1e0c0f442bc9ca238a03b9b4922; no voters:
I20240503 01:59:34.153782 1310 raft_consensus.cc:2781] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:34.153878 1310 raft_consensus.cc:686] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 2 LEADER]: Becoming Leader. State: Replica: de7ea1e0c0f442bc9ca238a03b9b4922, State: Running, Role: LEADER
I20240503 01:59:34.154035 1310 consensus_queue.cc:237] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } }
I20240503 01:59:34.155524 960 catalog_manager.cc:5484] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 reported cstate change: term changed from 1 to 2, leader changed from 106cab8aa6eb4a7ba92ecc6eb6daa9f8 (127.0.99.3) to de7ea1e0c0f442bc9ca238a03b9b4922 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:34.393457 1394 heartbeater.cc:503] Master 127.0.99.62:43229 was elected leader, sending a full tablet report...
I20240503 01:59:34.664381 1215 raft_consensus.cc:1261] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 2 FOLLOWER]: Refusing update from remote peer de7ea1e0c0f442bc9ca238a03b9b4922: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 01:59:34.664862 1310 consensus_queue.cc:1035] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:34.667493 1139 raft_consensus.cc:1261] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 2 FOLLOWER]: Refusing update from remote peer de7ea1e0c0f442bc9ca238a03b9b4922: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 01:59:34.667827 1310 consensus_queue.cc:1035] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:35.592978 1016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 01:59:35.594626 1016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 01:59:35.594766 1016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 01:59:37.595196 1016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 01:59:37.596813 1016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 01:59:37.596971 1016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 01:59:38.624228 1216 consensus_queue.cc:237] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } attrs { replace: true } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } }
I20240503 01:59:38.626030 1139 raft_consensus.cc:1261] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Refusing update from remote peer 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:38.626034 1065 raft_consensus.cc:1261] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Refusing update from remote peer 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:38.627542 1432 consensus_queue.cc:1035] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:38.629284 1432 consensus_queue.cc:1035] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:38.630090 1430 raft_consensus.cc:2932] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } attrs { replace: true } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
I20240503 01:59:38.631919 1139 raft_consensus.cc:2932] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } attrs { replace: true } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
I20240503 01:59:38.632102 1065 raft_consensus.cc:2932] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } attrs { replace: true } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
I20240503 01:59:38.634137 1216 consensus_queue.cc:237] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } }
I20240503 01:59:38.634730 961 catalog_manager.cc:5484] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 reported cstate change: config changed from index -1 to 2, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:38.634938 1179 consensus_peers.cc:485] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 -> Peer 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4:36583): Couldn't send request to peer 9ed7ed50af5d435baeb6ffdf8e227e81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c20f17a418074b6885e26f1cf3aae794. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.635663 1065 raft_consensus.cc:1261] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Refusing update from remote peer 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:38.635927 1139 raft_consensus.cc:1261] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Refusing update from remote peer 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 01:59:38.636140 1179 consensus_peers.cc:485] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 -> Peer 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4:36583): Couldn't send request to peer 9ed7ed50af5d435baeb6ffdf8e227e81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69a0a37b42f649ce90219bb888c60e41. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.636242 1432 consensus_queue.cc:1035] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:38.636400 1426 consensus_queue.cc:1035] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:38.638264 1426 raft_consensus.cc:2932] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
I20240503 01:59:38.638924 1139 raft_consensus.cc:2932] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
I20240503 01:59:38.641752 962 catalog_manager.cc:5484] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
I20240503 01:59:38.642578 1065 raft_consensus.cc:2932] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
I20240503 01:59:38.647989 1064 consensus_queue.cc:237] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } }
I20240503 01:59:38.649669 1216 raft_consensus.cc:1261] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 2 FOLLOWER]: Refusing update from remote peer de7ea1e0c0f442bc9ca238a03b9b4922: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 01:59:38.650153 1139 raft_consensus.cc:1261] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 2 FOLLOWER]: Refusing update from remote peer de7ea1e0c0f442bc9ca238a03b9b4922: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 01:59:38.650182 1449 consensus_queue.cc:1035] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:38.650533 1449 consensus_queue.cc:1035] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:38.652673 1216 raft_consensus.cc:2932] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
I20240503 01:59:38.653091 1139 raft_consensus.cc:2932] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
I20240503 01:59:38.655272 960 catalog_manager.cc:5484] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 reported cstate change: config changed from index -1 to 3, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New cstate: current_term: 2 leader_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
I20240503 01:59:38.652294 1448 raft_consensus.cc:2932] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
W20240503 01:59:38.657263 1028 consensus_peers.cc:485] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 -> Peer 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4:36583): Couldn't send request to peer 9ed7ed50af5d435baeb6ffdf8e227e81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d7f84458d244a67bb3404ccff14c861. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.659744 1216 consensus_queue.cc:237] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } }
I20240503 01:59:38.661338 1065 raft_consensus.cc:1261] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Refusing update from remote peer 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:38.661410 1139 raft_consensus.cc:1261] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Refusing update from remote peer 106cab8aa6eb4a7ba92ecc6eb6daa9f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 01:59:38.661635 1179 consensus_peers.cc:485] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 -> Peer 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4:36583): Couldn't send request to peer 9ed7ed50af5d435baeb6ffdf8e227e81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1dcf8b9c35cb42709aefb1596ae31006. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.661851 1432 consensus_queue.cc:1035] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:38.662250 1432 consensus_queue.cc:1035] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:38.663862 1444 raft_consensus.cc:2932] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
I20240503 01:59:38.664288 1139 raft_consensus.cc:2932] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
I20240503 01:59:38.664803 1065 raft_consensus.cc:2932] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } attrs { replace: true } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } } }
I20240503 01:59:38.666297 962 catalog_manager.cc:5484] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 reported cstate change: config changed from index -1 to 2, NON_VOTER 9ed7ed50af5d435baeb6ffdf8e227e81 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "de7ea1e0c0f442bc9ca238a03b9b4922" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21c3f3afa337498aba4d3fea01d2c6d8" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45487 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "106cab8aa6eb4a7ba92ecc6eb6daa9f8" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 45325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed7ed50af5d435baeb6ffdf8e227e81" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 36583 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:38.906061 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
W20240503 01:59:38.910892 947 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20240503 01:59:38.912050 1010 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 01:59:38.912079 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240503 01:59:38.912123 1010 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 01:59:38.912374 396 tablet_replica.cc:330] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922: stopping tablet replica
I20240503 01:59:38.912557 396 raft_consensus.cc:2229] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.912801 396 raft_consensus.cc:2258] T 8ba9735a19ce44d9a2e72e151d7a8d27 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.913443 396 tablet_replica.cc:330] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922: stopping tablet replica
I20240503 01:59:38.913585 396 raft_consensus.cc:2229] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.913738 396 raft_consensus.cc:2258] T 1dcf8b9c35cb42709aefb1596ae31006 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.914435 396 tablet_replica.cc:330] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922: stopping tablet replica
I20240503 01:59:38.914584 396 raft_consensus.cc:2229] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 2 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.914835 396 raft_consensus.cc:2258] T 1d7f84458d244a67bb3404ccff14c861 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.915673 396 tablet_replica.cc:330] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922: stopping tablet replica
I20240503 01:59:38.915810 396 raft_consensus.cc:2229] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.916003 396 raft_consensus.cc:2258] T 6f6a4d2a93ea474f84a24d27fc8dd23e P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.916481 396 tablet_replica.cc:330] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922: stopping tablet replica
I20240503 01:59:38.916649 396 raft_consensus.cc:2229] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.916761 396 raft_consensus.cc:2258] T ee2fd9be51514278913c0a2db93f26d7 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.917172 396 tablet_replica.cc:330] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922: stopping tablet replica
I20240503 01:59:38.917301 396 raft_consensus.cc:2229] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.917420 396 raft_consensus.cc:2258] T c20f17a418074b6885e26f1cf3aae794 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.918018 396 tablet_replica.cc:330] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922: stopping tablet replica
I20240503 01:59:38.918187 396 raft_consensus.cc:2229] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.918347 396 raft_consensus.cc:2258] T 840460feafde4b52b8bb4248f8bc0866 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.918783 396 tablet_replica.cc:330] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922: stopping tablet replica
I20240503 01:59:38.918947 396 raft_consensus.cc:2229] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.919154 396 raft_consensus.cc:2258] T 69a0a37b42f649ce90219bb888c60e41 P de7ea1e0c0f442bc9ca238a03b9b4922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.935359 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 01:59:38.939440 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 01:59:38.949383 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 01:59:38.949875 396 tablet_replica.cc:330] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8: stopping tablet replica
I20240503 01:59:38.950042 396 raft_consensus.cc:2229] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.950198 396 raft_consensus.cc:2258] T 1dcf8b9c35cb42709aefb1596ae31006 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.950858 396 tablet_replica.cc:330] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8: stopping tablet replica
I20240503 01:59:38.951067 396 raft_consensus.cc:2229] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.951195 396 raft_consensus.cc:2258] T 1d7f84458d244a67bb3404ccff14c861 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.951694 396 tablet_replica.cc:330] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8: stopping tablet replica
I20240503 01:59:38.951797 396 raft_consensus.cc:2229] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.951964 396 raft_consensus.cc:2258] T ee2fd9be51514278913c0a2db93f26d7 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.952394 396 tablet_replica.cc:330] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8: stopping tablet replica
I20240503 01:59:38.952550 396 raft_consensus.cc:2229] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.952647 396 raft_consensus.cc:2258] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.952978 396 tablet_replica.cc:330] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8: stopping tablet replica
I20240503 01:59:38.953076 396 raft_consensus.cc:2229] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.953167 396 raft_consensus.cc:2258] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.953572 396 tablet_replica.cc:330] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8: stopping tablet replica
I20240503 01:59:38.953718 396 raft_consensus.cc:2229] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.953819 396 raft_consensus.cc:2258] T 69a0a37b42f649ce90219bb888c60e41 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.954267 396 tablet_replica.cc:330] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8: stopping tablet replica
I20240503 01:59:38.954358 396 raft_consensus.cc:2229] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.954515 396 raft_consensus.cc:2258] T 840460feafde4b52b8bb4248f8bc0866 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.954869 396 tablet_replica.cc:330] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8: stopping tablet replica
I20240503 01:59:38.955008 396 raft_consensus.cc:2229] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.955173 396 raft_consensus.cc:2258] T c20f17a418074b6885e26f1cf3aae794 P 21c3f3afa337498aba4d3fea01d2c6d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.959249 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 01:59:38.963268 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
W20240503 01:59:38.964213 1010 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 01:59:38.964274 1010 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 01:59:38.965441 1010 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 01:59:38.965498 1010 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 01:59:38.966717 1010 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 01:59:38.966876 1010 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 01:59:38.972630 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 01:59:38.972893 396 tablet_replica.cc:330] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: stopping tablet replica
I20240503 01:59:38.973066 396 raft_consensus.cc:2229] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.973420 396 raft_consensus.cc:2258] T c20f17a418074b6885e26f1cf3aae794 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.974241 396 tablet_replica.cc:330] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: stopping tablet replica
I20240503 01:59:38.974399 396 raft_consensus.cc:2229] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.974531 396 raft_consensus.cc:2258] T 8ba9735a19ce44d9a2e72e151d7a8d27 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.975116 396 tablet_replica.cc:330] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: stopping tablet replica
I20240503 01:59:38.975271 396 raft_consensus.cc:2229] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.975569 396 raft_consensus.cc:2258] T 1dcf8b9c35cb42709aefb1596ae31006 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.976176 396 tablet_replica.cc:330] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: stopping tablet replica
I20240503 01:59:38.976317 396 raft_consensus.cc:2229] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.976430 396 raft_consensus.cc:2258] T ee2fd9be51514278913c0a2db93f26d7 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.976871 396 tablet_replica.cc:330] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: stopping tablet replica
I20240503 01:59:38.976994 396 raft_consensus.cc:2229] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.977098 396 raft_consensus.cc:2258] T 6f6a4d2a93ea474f84a24d27fc8dd23e P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.977547 396 tablet_replica.cc:330] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: stopping tablet replica
I20240503 01:59:38.977675 396 raft_consensus.cc:2229] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.977798 396 raft_consensus.cc:2258] T 1d7f84458d244a67bb3404ccff14c861 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.978399 396 tablet_replica.cc:330] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: stopping tablet replica
I20240503 01:59:38.978530 396 raft_consensus.cc:2229] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.978636 396 raft_consensus.cc:2258] T 840460feafde4b52b8bb4248f8bc0866 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.979115 396 tablet_replica.cc:330] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8: stopping tablet replica
I20240503 01:59:38.979228 396 raft_consensus.cc:2229] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.979516 396 raft_consensus.cc:2258] T 69a0a37b42f649ce90219bb888c60e41 P 106cab8aa6eb4a7ba92ecc6eb6daa9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.096660 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 01:59:39.099284 396 tablet_server.cc:178] TabletServer@127.0.99.4:0 shutting down...
I20240503 01:59:39.103276 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 01:59:39.115339 396 tablet_server.cc:194] TabletServer@127.0.99.4:0 shutdown complete.
I20240503 01:59:39.116907 396 master.cc:537] Master@127.0.99.62:43229 shutting down...
I20240503 01:59:39.120100 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:39.120242 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.120303 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4010630478884cbba7144c4f99e15ae3: stopping tablet replica
I20240503 01:59:39.132901 396 master.cc:552] Master@127.0.99.62:43229 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7594 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240503 01:59:39.143675 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:42821,127.0.99.61:38845,127.0.99.60:44739
I20240503 01:59:39.144117 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:39.147092 1458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:39.147822 1457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:39.149066 1460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:39.150046 396 server_base.cc:1034] running on GCE node
I20240503 01:59:39.150233 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:39.150285 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:39.150307 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701579150307 us; error 0 us; skew 500 ppm
I20240503 01:59:39.151298 396 webserver.cc:460] Webserver started at http://127.0.99.62:41413/ using document root <none> and password file <none>
I20240503 01:59:39.151460 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:39.151515 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:39.151611 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:39.152001 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-0-root/instance:
uuid: "6b7f8e9f68b649be9f7f91938e8f5180"
format_stamp: "Formatted at 2024-05-03 01:59:39 on dist-test-slave-82l4"
I20240503 01:59:39.153534 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:39.154407 1465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:39.154625 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.154693 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-0-root
uuid: "6b7f8e9f68b649be9f7f91938e8f5180"
format_stamp: "Formatted at 2024-05-03 01:59:39 on dist-test-slave-82l4"
I20240503 01:59:39.154767 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:39.167131 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:39.167517 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:39.171525 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:42821
I20240503 01:59:39.172984 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 01:59:39.174693 1517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:42821 every 8 connection(s)
W20240503 01:59:39.177328 1520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:39.177816 1518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.179065 1518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:39.179183 396 server_base.cc:1034] running on GCE node
W20240503 01:59:39.180356 1523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:39.181212 1521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:39.181403 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:39.181452 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:39.181476 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701579181475 us; error 0 us; skew 500 ppm
I20240503 01:59:39.182523 396 webserver.cc:460] Webserver started at http://127.0.99.61:45489/ using document root <none> and password file <none>
I20240503 01:59:39.182688 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:39.182742 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:39.182817 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:39.183269 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-1-root/instance:
uuid: "b4eca598e5084cf9ad3a8c804d506bbc"
format_stamp: "Formatted at 2024-05-03 01:59:39 on dist-test-slave-82l4"
I20240503 01:59:39.183965 1518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:39.185135 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
W20240503 01:59:39.185237 1518 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.61:38845: Network error: Client connection negotiation failed: client connection to 127.0.99.61:38845: connect: Connection refused (error 111)
I20240503 01:59:39.186101 1531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:39.186348 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:39.186415 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-1-root
uuid: "b4eca598e5084cf9ad3a8c804d506bbc"
format_stamp: "Formatted at 2024-05-03 01:59:39 on dist-test-slave-82l4"
I20240503 01:59:39.186483 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:39.198253 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:39.198668 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:39.202597 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.61:38845
I20240503 01:59:39.202621 1583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.61:38845 every 8 connection(s)
I20240503 01:59:39.203369 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 01:59:39.203593 1584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.205052 1584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } has no permanent_uuid. Determining permanent_uuid...
W20240503 01:59:39.206091 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:39.206884 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:39.208000 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:39.209007 396 server_base.cc:1034] running on GCE node
I20240503 01:59:39.209142 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 01:59:39.209102 1584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } has no permanent_uuid. Determining permanent_uuid...
W20240503 01:59:39.209192 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:39.209220 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701579209220 us; error 0 us; skew 500 ppm
I20240503 01:59:39.210134 396 webserver.cc:460] Webserver started at http://127.0.99.60:43007/ using document root <none> and password file <none>
I20240503 01:59:39.210284 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:39.210345 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:39.210422 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:39.210798 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-2-root/instance:
uuid: "afcca03838f34648b1a69b1a11d504d3"
format_stamp: "Formatted at 2024-05-03 01:59:39 on dist-test-slave-82l4"
I20240503 01:59:39.212390 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:39.213246 1584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:39.213479 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:39.213726 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:39.213799 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-2-root
uuid: "afcca03838f34648b1a69b1a11d504d3"
format_stamp: "Formatted at 2024-05-03 01:59:39 on dist-test-slave-82l4"
I20240503 01:59:39.213862 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240503 01:59:39.214453 1584 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.60:44739: Network error: Client connection negotiation failed: client connection to 127.0.99.60:44739: connect: Connection refused (error 111)
I20240503 01:59:39.223389 1518 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } attempt: 1
I20240503 01:59:39.227445 1518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } has no permanent_uuid. Determining permanent_uuid...
W20240503 01:59:39.228597 1518 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.60:44739: Network error: Client connection negotiation failed: client connection to 127.0.99.60:44739: connect: Connection refused (error 111)
I20240503 01:59:39.240424 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:39.240607 1584 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } attempt: 1
I20240503 01:59:39.240826 396 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240503 01:59:39.241932 1584 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.60:44739: Network error: Client connection negotiation failed: client connection to 127.0.99.60:44739: connect: Connection refused (error 111)
I20240503 01:59:39.244962 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.60:44739
I20240503 01:59:39.244998 1649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.60:44739 every 8 connection(s)
I20240503 01:59:39.245364 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 01:59:39.245752 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.246774 1650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:39.250267 1650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:39.251744 1518 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } attempt: 1
I20240503 01:59:39.253726 1650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } has no permanent_uuid. Determining permanent_uuid...
I20240503 01:59:39.256897 1518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180: Bootstrap starting.
I20240503 01:59:39.257876 1518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.258136 1650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3: Bootstrap starting.
I20240503 01:59:39.258913 1518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180: No bootstrap required, opened a new log
I20240503 01:59:39.258945 1650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.259318 1518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } }
I20240503 01:59:39.259410 1518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.259443 1518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7f8e9f68b649be9f7f91938e8f5180, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.259579 1518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } }
I20240503 01:59:39.259872 1657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:39.259941 1650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3: No bootstrap required, opened a new log
I20240503 01:59:39.259985 1657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:39.260082 1518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:39.260332 1650 raft_consensus.cc:348] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } }
I20240503 01:59:39.260429 1650 raft_consensus.cc:374] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.260452 1650 raft_consensus.cc:729] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afcca03838f34648b1a69b1a11d504d3, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.260566 1650 consensus_queue.cc:260] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } }
I20240503 01:59:39.260857 1659 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:39.261008 1659 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:39.261011 1650 sys_catalog.cc:564] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:39.261855 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240503 01:59:39.262542 1674 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 01:59:39.262590 1674 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 01:59:39.263109 1680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 01:59:39.263161 1680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 01:59:39.317095 1584 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } attempt: 2
I20240503 01:59:39.322082 1584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc: Bootstrap starting.
I20240503 01:59:39.323020 1584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.324108 1584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc: No bootstrap required, opened a new log
I20240503 01:59:39.324517 1584 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } }
I20240503 01:59:39.324620 1584 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.324654 1584 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4eca598e5084cf9ad3a8c804d506bbc, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.324802 1584 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } }
I20240503 01:59:39.325119 1682 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:39.325212 1682 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:39.325258 1584 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:39.326714 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 01:59:39.327220 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:39.327257 1693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 01:59:39.327329 1693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 01:59:39.330049 1694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:39.330880 1697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:39.332036 1695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:39.333123 396 server_base.cc:1034] running on GCE node
I20240503 01:59:39.333333 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:39.333374 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:39.333400 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701579333399 us; error 0 us; skew 500 ppm
I20240503 01:59:39.334399 396 webserver.cc:460] Webserver started at http://127.0.99.1:46313/ using document root <none> and password file <none>
I20240503 01:59:39.334548 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:39.334600 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:39.334681 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:39.335213 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-0-root/instance:
uuid: "17c7ff67dd6b450cacde411ee7ca6dd5"
format_stamp: "Formatted at 2024-05-03 01:59:39 on dist-test-slave-82l4"
I20240503 01:59:39.336653 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.337532 1702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:39.337749 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.337814 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-0-root
uuid: "17c7ff67dd6b450cacde411ee7ca6dd5"
format_stamp: "Formatted at 2024-05-03 01:59:39 on dist-test-slave-82l4"
I20240503 01:59:39.337885 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:39.346351 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:39.346730 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:39.347047 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:39.347504 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:39.347543 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:39.347592 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:39.347623 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:39.352135 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:43039
I20240503 01:59:39.352174 1765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:43039 every 8 connection(s)
I20240503 01:59:39.353308 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:39.356557 1776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:39.357415 1767 heartbeater.cc:347] Connected to a master server at 127.0.99.62:42821
I20240503 01:59:39.357517 1767 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:39.357757 1766 heartbeater.cc:347] Connected to a master server at 127.0.99.60:44739
I20240503 01:59:39.357766 1767 heartbeater.cc:511] Master 127.0.99.62:42821 requested a full tablet report, sending...
I20240503 01:59:39.357849 1766 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:39.357864 1768 heartbeater.cc:347] Connected to a master server at 127.0.99.61:38845
W20240503 01:59:39.357885 1774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:39.357985 1768 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:39.358088 1766 heartbeater.cc:511] Master 127.0.99.60:44739 requested a full tablet report, sending...
I20240503 01:59:39.358176 1768 heartbeater.cc:511] Master 127.0.99.61:38845 requested a full tablet report, sending...
I20240503 01:59:39.358502 1482 ts_manager.cc:194] Registered new tserver with Master: 17c7ff67dd6b450cacde411ee7ca6dd5 (127.0.99.1:43039)
W20240503 01:59:39.358582 1773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:39.358626 1614 ts_manager.cc:194] Registered new tserver with Master: 17c7ff67dd6b450cacde411ee7ca6dd5 (127.0.99.1:43039)
I20240503 01:59:39.358775 1548 ts_manager.cc:194] Registered new tserver with Master: 17c7ff67dd6b450cacde411ee7ca6dd5 (127.0.99.1:43039)
I20240503 01:59:39.359416 396 server_base.cc:1034] running on GCE node
I20240503 01:59:39.359571 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:39.359607 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:39.359635 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701579359635 us; error 0 us; skew 500 ppm
I20240503 01:59:39.360641 396 webserver.cc:460] Webserver started at http://127.0.99.2:43113/ using document root <none> and password file <none>
I20240503 01:59:39.360844 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:39.360900 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:39.360960 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:39.361379 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-1-root/instance:
uuid: "c725a04955854e3a848a12408840f442"
format_stamp: "Formatted at 2024-05-03 01:59:39 on dist-test-slave-82l4"
I20240503 01:59:39.363078 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:39.363948 1781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:39.364169 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.364235 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-1-root
uuid: "c725a04955854e3a848a12408840f442"
format_stamp: "Formatted at 2024-05-03 01:59:39 on dist-test-slave-82l4"
I20240503 01:59:39.364306 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:39.395283 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:39.395779 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:39.396107 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:39.396600 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:39.396639 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:39.396688 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:39.396725 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:39.401167 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:41509
I20240503 01:59:39.401226 1844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:41509 every 8 connection(s)
I20240503 01:59:39.402175 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:39.405539 1852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:39.406605 1846 heartbeater.cc:347] Connected to a master server at 127.0.99.62:42821
W20240503 01:59:39.406622 1853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:39.406719 1846 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:39.406808 1847 heartbeater.cc:347] Connected to a master server at 127.0.99.61:38845
I20240503 01:59:39.406899 1847 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:39.407007 1846 heartbeater.cc:511] Master 127.0.99.62:42821 requested a full tablet report, sending...
I20240503 01:59:39.407145 1847 heartbeater.cc:511] Master 127.0.99.61:38845 requested a full tablet report, sending...
W20240503 01:59:39.407719 1855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:39.407775 1548 ts_manager.cc:194] Registered new tserver with Master: c725a04955854e3a848a12408840f442 (127.0.99.2:41509)
I20240503 01:59:39.407799 1482 ts_manager.cc:194] Registered new tserver with Master: c725a04955854e3a848a12408840f442 (127.0.99.2:41509)
I20240503 01:59:39.407940 1845 heartbeater.cc:347] Connected to a master server at 127.0.99.60:44739
I20240503 01:59:39.408015 1845 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:39.408195 1845 heartbeater.cc:511] Master 127.0.99.60:44739 requested a full tablet report, sending...
I20240503 01:59:39.408564 396 server_base.cc:1034] running on GCE node
I20240503 01:59:39.408746 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:39.408784 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:39.408807 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701579408807 us; error 0 us; skew 500 ppm
I20240503 01:59:39.408972 1614 ts_manager.cc:194] Registered new tserver with Master: c725a04955854e3a848a12408840f442 (127.0.99.2:41509)
I20240503 01:59:39.409852 396 webserver.cc:460] Webserver started at http://127.0.99.3:41987/ using document root <none> and password file <none>
I20240503 01:59:39.410003 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:39.410077 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:39.410157 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:39.410564 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-2-root/instance:
uuid: "828b4de0f6224c058c85a91a874cb725"
format_stamp: "Formatted at 2024-05-03 01:59:39 on dist-test-slave-82l4"
I20240503 01:59:39.412115 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 01:59:39.413061 1860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:39.413327 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:39.413410 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-2-root
uuid: "828b4de0f6224c058c85a91a874cb725"
format_stamp: "Formatted at 2024-05-03 01:59:39 on dist-test-slave-82l4"
I20240503 01:59:39.413480 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714701568895932-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:39.424813 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:39.425294 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:39.425617 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:39.426097 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:39.426177 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:39.426234 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:39.426249 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:39.430647 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:35743
I20240503 01:59:39.430707 1923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:35743 every 8 connection(s)
I20240503 01:59:39.435834 1924 heartbeater.cc:347] Connected to a master server at 127.0.99.60:44739
I20240503 01:59:39.435956 1924 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:39.436180 1926 heartbeater.cc:347] Connected to a master server at 127.0.99.61:38845
I20240503 01:59:39.436223 1924 heartbeater.cc:511] Master 127.0.99.60:44739 requested a full tablet report, sending...
I20240503 01:59:39.436280 1926 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:39.436497 1926 heartbeater.cc:511] Master 127.0.99.61:38845 requested a full tablet report, sending...
I20240503 01:59:39.436910 1925 heartbeater.cc:347] Connected to a master server at 127.0.99.62:42821
I20240503 01:59:39.436978 1925 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:39.437090 1614 ts_manager.cc:194] Registered new tserver with Master: 828b4de0f6224c058c85a91a874cb725 (127.0.99.3:35743)
I20240503 01:59:39.437156 1925 heartbeater.cc:511] Master 127.0.99.62:42821 requested a full tablet report, sending...
I20240503 01:59:39.437206 1548 ts_manager.cc:194] Registered new tserver with Master: 828b4de0f6224c058c85a91a874cb725 (127.0.99.3:35743)
I20240503 01:59:39.437791 1482 ts_manager.cc:194] Registered new tserver with Master: 828b4de0f6224c058c85a91a874cb725 (127.0.99.3:35743)
I20240503 01:59:39.438330 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007102905s
I20240503 01:59:39.695756 1659 raft_consensus.cc:482] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:39.695941 1659 raft_consensus.cc:504] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } }
I20240503 01:59:39.696440 1659 leader_election.cc:288] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7f8e9f68b649be9f7f91938e8f5180 (127.0.99.62:42821), b4eca598e5084cf9ad3a8c804d506bbc (127.0.99.61:38845)
I20240503 01:59:39.696873 1558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afcca03838f34648b1a69b1a11d504d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" is_pre_election: true
I20240503 01:59:39.696878 1492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afcca03838f34648b1a69b1a11d504d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" is_pre_election: true
I20240503 01:59:39.697010 1492 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afcca03838f34648b1a69b1a11d504d3 in term 0.
I20240503 01:59:39.697011 1558 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afcca03838f34648b1a69b1a11d504d3 in term 0.
I20240503 01:59:39.697336 1600 leader_election.cc:302] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7f8e9f68b649be9f7f91938e8f5180, afcca03838f34648b1a69b1a11d504d3; no voters:
I20240503 01:59:39.697506 1659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:39.697577 1659 raft_consensus.cc:482] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:39.697615 1659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.698395 1659 raft_consensus.cc:504] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } }
I20240503 01:59:39.698719 1659 leader_election.cc:288] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7f8e9f68b649be9f7f91938e8f5180 (127.0.99.62:42821), b4eca598e5084cf9ad3a8c804d506bbc (127.0.99.61:38845)
I20240503 01:59:39.699043 1492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afcca03838f34648b1a69b1a11d504d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7f8e9f68b649be9f7f91938e8f5180"
I20240503 01:59:39.699100 1558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afcca03838f34648b1a69b1a11d504d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4eca598e5084cf9ad3a8c804d506bbc"
I20240503 01:59:39.699144 1492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.699183 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.700003 1558 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afcca03838f34648b1a69b1a11d504d3 in term 1.
I20240503 01:59:39.700001 1492 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afcca03838f34648b1a69b1a11d504d3 in term 1.
I20240503 01:59:39.700277 1600 leader_election.cc:302] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7f8e9f68b649be9f7f91938e8f5180, afcca03838f34648b1a69b1a11d504d3; no voters:
I20240503 01:59:39.700461 1659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:39.700642 1659 raft_consensus.cc:686] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [term 1 LEADER]: Becoming Leader. State: Replica: afcca03838f34648b1a69b1a11d504d3, State: Running, Role: LEADER
I20240503 01:59:39.700781 1659 consensus_queue.cc:237] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } }
I20240503 01:59:39.701372 1932 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afcca03838f34648b1a69b1a11d504d3. Latest consensus state: current_term: 1 leader_uuid: "afcca03838f34648b1a69b1a11d504d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:39.701488 1932 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:39.701750 1934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:39.702450 1934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:39.704159 1492 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 1 FOLLOWER]: Refusing update from remote peer afcca03838f34648b1a69b1a11d504d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:39.704170 1558 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 1 FOLLOWER]: Refusing update from remote peer afcca03838f34648b1a69b1a11d504d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:39.704531 1659 consensus_queue.cc:1035] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:39.704632 1932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:39.705797 1657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afcca03838f34648b1a69b1a11d504d3. Latest consensus state: current_term: 1 leader_uuid: "afcca03838f34648b1a69b1a11d504d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:39.705811 1682 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader afcca03838f34648b1a69b1a11d504d3. Latest consensus state: current_term: 1 leader_uuid: "afcca03838f34648b1a69b1a11d504d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:39.705929 1657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:39.705938 1682 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:39.707041 1940 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "afcca03838f34648b1a69b1a11d504d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:39.707208 1940 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:39.707563 1657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "afcca03838f34648b1a69b1a11d504d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:39.707664 1657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:39.707798 1682 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "afcca03838f34648b1a69b1a11d504d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:39.707881 1682 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:39.707994 1940 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "afcca03838f34648b1a69b1a11d504d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:39.708117 1940 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:39.708159 1934 catalog_manager.cc:1347] Generated new cluster ID: 08fbe19afd374a468680c34d66f6b164
I20240503 01:59:39.708202 1934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:39.723817 1934 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:39.724637 1934 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:39.732863 1934 catalog_manager.cc:5857] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3: Generated new TSK 0
I20240503 01:59:39.733080 1934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:40.263393 1674 catalog_manager.cc:1259] Loaded cluster ID: 08fbe19afd374a468680c34d66f6b164
I20240503 01:59:40.263479 1674 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180: loading cluster ID for follower catalog manager: success
I20240503 01:59:40.264731 1674 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180: acquiring CA information for follower catalog manager: success
I20240503 01:59:40.265259 1674 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 01:59:40.328089 1693 catalog_manager.cc:1259] Loaded cluster ID: 08fbe19afd374a468680c34d66f6b164
I20240503 01:59:40.328183 1693 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc: loading cluster ID for follower catalog manager: success
I20240503 01:59:40.329445 1693 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc: acquiring CA information for follower catalog manager: success
I20240503 01:59:40.329983 1693 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 01:59:40.361135 1614 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59698
I20240503 01:59:40.410377 1614 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59714
I20240503 01:59:40.439069 1614 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59718
I20240503 01:59:41.262734 1679 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 01:59:41.281498 396 test_util.cc:268] Using random seed: 312862935
I20240503 01:59:41.290930 1614 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59728:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 01:59:41.304287 1809 tablet_service.cc:1467] Processing CreateTablet for tablet cd91660b06d342e28029524daf0b4c67 (DEFAULT_TABLE table=test-workload [id=bccde525464d4e5db68d140097a25566]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 01:59:41.304656 1809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd91660b06d342e28029524daf0b4c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.305016 1888 tablet_service.cc:1467] Processing CreateTablet for tablet f8864cae3b654f4cb3d64a7b4c8874b7 (DEFAULT_TABLE table=test-workload [id=bccde525464d4e5db68d140097a25566]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 01:59:41.305179 1730 tablet_service.cc:1467] Processing CreateTablet for tablet e0caa4dee698466a9b65e38a424bf132 (DEFAULT_TABLE table=test-workload [id=bccde525464d4e5db68d140097a25566]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 01:59:41.305292 1888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8864cae3b654f4cb3d64a7b4c8874b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.305416 1730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0caa4dee698466a9b65e38a424bf132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.306720 1971 tablet_bootstrap.cc:492] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442: Bootstrap starting.
I20240503 01:59:41.307673 1972 tablet_bootstrap.cc:492] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725: Bootstrap starting.
I20240503 01:59:41.307687 1971 tablet_bootstrap.cc:654] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.308459 1972 tablet_bootstrap.cc:654] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.308694 1971 tablet_bootstrap.cc:492] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442: No bootstrap required, opened a new log
I20240503 01:59:41.308775 1971 ts_tablet_manager.cc:1367] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:41.309043 1973 tablet_bootstrap.cc:492] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5: Bootstrap starting.
I20240503 01:59:41.309132 1971 raft_consensus.cc:348] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c725a04955854e3a848a12408840f442" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 41509 } }
I20240503 01:59:41.309242 1971 raft_consensus.cc:374] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.309335 1971 raft_consensus.cc:729] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c725a04955854e3a848a12408840f442, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.309494 1971 consensus_queue.cc:260] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c725a04955854e3a848a12408840f442" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 41509 } }
I20240503 01:59:41.309561 1972 tablet_bootstrap.cc:492] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725: No bootstrap required, opened a new log
I20240503 01:59:41.309592 1971 raft_consensus.cc:388] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:41.309625 1972 ts_tablet_manager.cc:1367] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:41.309656 1971 raft_consensus.cc:482] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:41.309737 1971 raft_consensus.cc:3037] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.309933 1973 tablet_bootstrap.cc:654] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.310179 1972 raft_consensus.cc:348] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "828b4de0f6224c058c85a91a874cb725" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35743 } }
I20240503 01:59:41.310295 1972 raft_consensus.cc:374] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.310482 1972 raft_consensus.cc:729] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 828b4de0f6224c058c85a91a874cb725, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.310658 1971 raft_consensus.cc:504] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c725a04955854e3a848a12408840f442" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 41509 } }
I20240503 01:59:41.310690 1972 consensus_queue.cc:260] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "828b4de0f6224c058c85a91a874cb725" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35743 } }
I20240503 01:59:41.310814 1971 leader_election.cc:302] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c725a04955854e3a848a12408840f442; no voters:
I20240503 01:59:41.310839 1972 raft_consensus.cc:388] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:41.310899 1972 raft_consensus.cc:482] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:41.310993 1972 raft_consensus.cc:3037] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.311019 1973 tablet_bootstrap.cc:492] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5: No bootstrap required, opened a new log
I20240503 01:59:41.311040 1971 leader_election.cc:288] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:41.311157 1973 ts_tablet_manager.cc:1367] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:41.311118 1977 raft_consensus.cc:2781] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:41.311343 1977 raft_consensus.cc:686] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [term 1 LEADER]: Becoming Leader. State: Replica: c725a04955854e3a848a12408840f442, State: Running, Role: LEADER
I20240503 01:59:41.311475 1971 ts_tablet_manager.cc:1398] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:41.311455 1977 consensus_queue.cc:237] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c725a04955854e3a848a12408840f442" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 41509 } }
I20240503 01:59:41.311576 1973 raft_consensus.cc:348] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c7ff67dd6b450cacde411ee7ca6dd5" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43039 } }
I20240503 01:59:41.311647 1973 raft_consensus.cc:374] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.311686 1973 raft_consensus.cc:729] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c7ff67dd6b450cacde411ee7ca6dd5, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.311650 1845 heartbeater.cc:503] Master 127.0.99.60:44739 was elected leader, sending a full tablet report...
I20240503 01:59:41.311784 1973 consensus_queue.cc:260] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c7ff67dd6b450cacde411ee7ca6dd5" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43039 } }
I20240503 01:59:41.311870 1973 raft_consensus.cc:388] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:41.311904 1973 raft_consensus.cc:482] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:41.311941 1973 raft_consensus.cc:3037] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.312742 1614 catalog_manager.cc:5484] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 reported cstate change: term changed from 0 to 1, leader changed from <none> to c725a04955854e3a848a12408840f442 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "c725a04955854e3a848a12408840f442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c725a04955854e3a848a12408840f442" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 41509 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:41.312748 1973 raft_consensus.cc:504] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c7ff67dd6b450cacde411ee7ca6dd5" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43039 } }
I20240503 01:59:41.313416 1973 leader_election.cc:302] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17c7ff67dd6b450cacde411ee7ca6dd5; no voters:
I20240503 01:59:41.313575 1973 leader_election.cc:288] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:41.313851 1973 ts_tablet_manager.cc:1398] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:41.313868 1982 raft_consensus.cc:2781] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:41.314016 1766 heartbeater.cc:503] Master 127.0.99.60:44739 was elected leader, sending a full tablet report...
I20240503 01:59:41.314106 1982 raft_consensus.cc:686] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [term 1 LEADER]: Becoming Leader. State: Replica: 17c7ff67dd6b450cacde411ee7ca6dd5, State: Running, Role: LEADER
I20240503 01:59:41.314208 1982 consensus_queue.cc:237] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c7ff67dd6b450cacde411ee7ca6dd5" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43039 } }
I20240503 01:59:41.316947 1972 raft_consensus.cc:504] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "828b4de0f6224c058c85a91a874cb725" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35743 } }
I20240503 01:59:41.317713 1613 catalog_manager.cc:5484] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17c7ff67dd6b450cacde411ee7ca6dd5 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "17c7ff67dd6b450cacde411ee7ca6dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c7ff67dd6b450cacde411ee7ca6dd5" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43039 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:41.319293 1972 leader_election.cc:302] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 828b4de0f6224c058c85a91a874cb725; no voters:
I20240503 01:59:41.319943 1972 leader_election.cc:288] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:41.320947 1988 raft_consensus.cc:2781] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:41.321136 1988 raft_consensus.cc:686] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [term 1 LEADER]: Becoming Leader. State: Replica: 828b4de0f6224c058c85a91a874cb725, State: Running, Role: LEADER
I20240503 01:59:41.321260 1988 consensus_queue.cc:237] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "828b4de0f6224c058c85a91a874cb725" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35743 } }
I20240503 01:59:41.321524 1924 heartbeater.cc:503] Master 127.0.99.60:44739 was elected leader, sending a full tablet report...
I20240503 01:59:41.322985 1972 ts_tablet_manager.cc:1398] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 01:59:41.323288 1614 catalog_manager.cc:5484] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 reported cstate change: term changed from 0 to 1, leader changed from <none> to 828b4de0f6224c058c85a91a874cb725 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "828b4de0f6224c058c85a91a874cb725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "828b4de0f6224c058c85a91a874cb725" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35743 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:41.326928 396 master.cc:537] Master@127.0.99.60:44739 shutting down...
W20240503 01:59:41.327674 1924 heartbeater.cc:644] Failed to heartbeat to 127.0.99.60:44739 (0 consecutive failures): Remote error: Failed to send heartbeat to master: Service unavailable: service kudu.master.MasterService not registered on Master
I20240503 01:59:41.329741 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:41.329908 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.329970 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P afcca03838f34648b1a69b1a11d504d3: stopping tablet replica
I20240503 01:59:41.343425 396 master.cc:552] Master@127.0.99.60:44739 shutdown complete.
I20240503 01:59:42.826637 1996 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader afcca03838f34648b1a69b1a11d504d3)
I20240503 01:59:42.826824 1996 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } }
I20240503 01:59:42.827282 1996 leader_election.cc:288] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b7f8e9f68b649be9f7f91938e8f5180 (127.0.99.62:42821), afcca03838f34648b1a69b1a11d504d3 (127.0.99.60:44739)
I20240503 01:59:42.827579 1997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader afcca03838f34648b1a69b1a11d504d3)
I20240503 01:59:42.827704 1997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } }
I20240503 01:59:42.827773 1492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" is_pre_election: true
I20240503 01:59:42.827877 1492 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4eca598e5084cf9ad3a8c804d506bbc in term 1.
I20240503 01:59:42.828073 1997 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4eca598e5084cf9ad3a8c804d506bbc (127.0.99.61:38845), afcca03838f34648b1a69b1a11d504d3 (127.0.99.60:44739)
I20240503 01:59:42.828235 1534 leader_election.cc:302] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7f8e9f68b649be9f7f91938e8f5180, b4eca598e5084cf9ad3a8c804d506bbc; no voters:
I20240503 01:59:42.828470 1996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 01:59:42.828538 1996 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 1 FOLLOWER]: Starting leader election (detected failure of leader afcca03838f34648b1a69b1a11d504d3)
I20240503 01:59:42.828583 1996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:42.828572 1558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" is_pre_election: true
W20240503 01:59:42.829033 1535 leader_election.cc:334] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer afcca03838f34648b1a69b1a11d504d3 (127.0.99.60:44739): Network error: Client connection negotiation failed: client connection to 127.0.99.60:44739: connect: Connection refused (error 111)
W20240503 01:59:42.829382 1469 leader_election.cc:334] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer afcca03838f34648b1a69b1a11d504d3 (127.0.99.60:44739): Network error: Client connection negotiation failed: client connection to 127.0.99.60:44739: connect: Connection refused (error 111)
I20240503 01:59:42.829756 1996 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } }
I20240503 01:59:42.829947 1558 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b7f8e9f68b649be9f7f91938e8f5180 in current term 2: Already voted for candidate b4eca598e5084cf9ad3a8c804d506bbc in this term.
I20240503 01:59:42.830080 1996 leader_election.cc:288] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [CANDIDATE]: Term 2 election: Requested vote from peers 6b7f8e9f68b649be9f7f91938e8f5180 (127.0.99.62:42821), afcca03838f34648b1a69b1a11d504d3 (127.0.99.60:44739)
I20240503 01:59:42.830329 1492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6b7f8e9f68b649be9f7f91938e8f5180"
I20240503 01:59:42.830387 1469 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b7f8e9f68b649be9f7f91938e8f5180; no voters: afcca03838f34648b1a69b1a11d504d3, b4eca598e5084cf9ad3a8c804d506bbc
I20240503 01:59:42.830438 1492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 1 FOLLOWER]: Advancing to term 2
W20240503 01:59:42.831034 1535 leader_election.cc:334] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer afcca03838f34648b1a69b1a11d504d3 (127.0.99.60:44739): Network error: Client connection negotiation failed: client connection to 127.0.99.60:44739: connect: Connection refused (error 111)
I20240503 01:59:42.831358 1492 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4eca598e5084cf9ad3a8c804d506bbc in term 2.
I20240503 01:59:42.831459 1997 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 01:59:42.831642 1534 leader_election.cc:302] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b7f8e9f68b649be9f7f91938e8f5180, b4eca598e5084cf9ad3a8c804d506bbc; no voters: afcca03838f34648b1a69b1a11d504d3
I20240503 01:59:42.831780 1996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:42.831918 1996 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 2 LEADER]: Becoming Leader. State: Replica: b4eca598e5084cf9ad3a8c804d506bbc, State: Running, Role: LEADER
I20240503 01:59:42.832051 1996 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } }
I20240503 01:59:42.832614 2002 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4eca598e5084cf9ad3a8c804d506bbc. Latest consensus state: current_term: 2 leader_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:42.832696 2002 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:42.832931 2004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:42.833973 2004 catalog_manager.cc:669] Loaded metadata for table test-workload [id=bccde525464d4e5db68d140097a25566]
I20240503 01:59:42.834682 2004 tablet_loader.cc:96] loaded metadata for tablet cd91660b06d342e28029524daf0b4c67 (table test-workload [id=bccde525464d4e5db68d140097a25566])
I20240503 01:59:42.834897 2004 tablet_loader.cc:96] loaded metadata for tablet e0caa4dee698466a9b65e38a424bf132 (table test-workload [id=bccde525464d4e5db68d140097a25566])
I20240503 01:59:42.835116 2004 tablet_loader.cc:96] loaded metadata for tablet f8864cae3b654f4cb3d64a7b4c8874b7 (table test-workload [id=bccde525464d4e5db68d140097a25566])
I20240503 01:59:42.835208 2004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:42.835590 2004 catalog_manager.cc:1259] Loaded cluster ID: 08fbe19afd374a468680c34d66f6b164
I20240503 01:59:42.835635 2004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:42.836171 2004 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:42.836565 2004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc: Loaded TSK: 0
I20240503 01:59:42.836997 2004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:43.327069 1692 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 01:59:43.327178 1692 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 01:59:43.366734 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 01:59:43.370321 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 01:59:43.370565 396 tablet_replica.cc:330] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5: stopping tablet replica
I20240503 01:59:43.370677 396 raft_consensus.cc:2229] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:43.370817 396 raft_consensus.cc:2258] T e0caa4dee698466a9b65e38a424bf132 P 17c7ff67dd6b450cacde411ee7ca6dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 01:59:43.382870 1535 consensus_peers.cc:485] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc -> Peer afcca03838f34648b1a69b1a11d504d3 (127.0.99.60:44739): Couldn't send request to peer afcca03838f34648b1a69b1a11d504d3. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.60:44739: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:43.383347 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 01:59:43.385246 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 01:59:43.389269 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 01:59:43.389494 396 tablet_replica.cc:330] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442: stopping tablet replica
I20240503 01:59:43.389636 396 raft_consensus.cc:2229] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:43.389779 396 raft_consensus.cc:2258] T cd91660b06d342e28029524daf0b4c67 P c725a04955854e3a848a12408840f442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.402360 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 01:59:43.404896 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 01:59:43.410451 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 01:59:43.410625 1492 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 2 FOLLOWER]: Refusing update from remote peer b4eca598e5084cf9ad3a8c804d506bbc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:43.410943 396 tablet_replica.cc:330] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725: stopping tablet replica
I20240503 01:59:43.411162 396 raft_consensus.cc:2229] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:43.411334 396 raft_consensus.cc:2258] T f8864cae3b654f4cb3d64a7b4c8874b7 P 828b4de0f6224c058c85a91a874cb725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.411319 2008 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:43.412451 2011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4eca598e5084cf9ad3a8c804d506bbc. Latest consensus state: current_term: 2 leader_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:43.412591 2011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:43.413611 2013 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:43.413760 2013 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:43.414256 2011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7f8e9f68b649be9f7f91938e8f5180" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 42821 } } peers { permanent_uuid: "b4eca598e5084cf9ad3a8c804d506bbc" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 38845 } } peers { permanent_uuid: "afcca03838f34648b1a69b1a11d504d3" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 44739 } } }
I20240503 01:59:43.414371 2011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 01:59:43.424698 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 01:59:43.426721 396 master.cc:537] Master@127.0.99.62:42821 shutting down...
I20240503 01:59:43.430480 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:43.430675 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.430739 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6b7f8e9f68b649be9f7f91938e8f5180: stopping tablet replica
I20240503 01:59:43.443804 396 master.cc:552] Master@127.0.99.62:42821 shutdown complete.
I20240503 01:59:43.446919 396 master.cc:537] Master@127.0.99.61:38845 shutting down...
I20240503 01:59:43.450100 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 2 LEADER]: Raft consensus shutting down.
I20240503 01:59:43.450357 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.450481 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P b4eca598e5084cf9ad3a8c804d506bbc: stopping tablet replica
I20240503 01:59:43.463258 396 master.cc:552] Master@127.0.99.61:38845 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (4327 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240503 01:59:43.470722 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:33223
I20240503 01:59:43.471155 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:43.473861 2017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:43.474797 2020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:43.475754 2018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:43.476704 396 server_base.cc:1034] running on GCE node
I20240503 01:59:43.476871 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:43.476910 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:43.476925 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701583476925 us; error 0 us; skew 500 ppm
I20240503 01:59:43.477942 396 webserver.cc:460] Webserver started at http://127.0.99.62:40713/ using document root <none> and password file <none>
I20240503 01:59:43.478112 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:43.478163 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:43.478255 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:43.478646 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/master-0-root/instance:
uuid: "3d6048f112714ba68d6fd1a6fc595e68"
format_stamp: "Formatted at 2024-05-03 01:59:43 on dist-test-slave-82l4"
I20240503 01:59:43.480188 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:43.481289 2025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:43.481546 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:43.481631 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/master-0-root
uuid: "3d6048f112714ba68d6fd1a6fc595e68"
format_stamp: "Formatted at 2024-05-03 01:59:43 on dist-test-slave-82l4"
I20240503 01:59:43.481694 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:43.489220 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:43.489519 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:43.493208 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:33223
I20240503 01:59:43.496098 2077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:33223 every 8 connection(s)
I20240503 01:59:43.496542 2078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.498298 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68: Bootstrap starting.
I20240503 01:59:43.499130 2078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.500093 2078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68: No bootstrap required, opened a new log
I20240503 01:59:43.500474 2078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6048f112714ba68d6fd1a6fc595e68" member_type: VOTER }
I20240503 01:59:43.500572 2078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.500608 2078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6048f112714ba68d6fd1a6fc595e68, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.500756 2078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6048f112714ba68d6fd1a6fc595e68" member_type: VOTER }
I20240503 01:59:43.500841 2078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:43.500883 2078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:43.500932 2078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.501583 2078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6048f112714ba68d6fd1a6fc595e68" member_type: VOTER }
I20240503 01:59:43.501708 2078 leader_election.cc:302] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d6048f112714ba68d6fd1a6fc595e68; no voters:
I20240503 01:59:43.501874 2078 leader_election.cc:288] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:43.502025 2081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:43.502213 2081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [term 1 LEADER]: Becoming Leader. State: Replica: 3d6048f112714ba68d6fd1a6fc595e68, State: Running, Role: LEADER
I20240503 01:59:43.502290 2078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:43.502336 2081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6048f112714ba68d6fd1a6fc595e68" member_type: VOTER }
I20240503 01:59:43.502761 2082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d6048f112714ba68d6fd1a6fc595e68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6048f112714ba68d6fd1a6fc595e68" member_type: VOTER } }
I20240503 01:59:43.502856 2082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:43.502806 2083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d6048f112714ba68d6fd1a6fc595e68. Latest consensus state: current_term: 1 leader_uuid: "3d6048f112714ba68d6fd1a6fc595e68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6048f112714ba68d6fd1a6fc595e68" member_type: VOTER } }
I20240503 01:59:43.502923 2083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:43.503170 2086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:43.504069 2086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:43.504266 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 01:59:43.505856 2086 catalog_manager.cc:1347] Generated new cluster ID: 9ca9ab71645d4733b880ab45b5b36259
I20240503 01:59:43.505901 2086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:43.515837 2086 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:43.516357 2086 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:43.524283 2086 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68: Generated new TSK 0
I20240503 01:59:43.524446 2086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:43.536584 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:43.539733 2099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:43.540864 2102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:43.541639 2100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:43.542573 396 server_base.cc:1034] running on GCE node
I20240503 01:59:43.542769 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:43.542812 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:43.542835 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701583542834 us; error 0 us; skew 500 ppm
I20240503 01:59:43.543834 396 webserver.cc:460] Webserver started at http://127.0.99.1:45617/ using document root <none> and password file <none>
I20240503 01:59:43.544014 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:43.544072 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:43.544174 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:43.544595 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-0-root/instance:
uuid: "b32a463b2df340f7b2b4d32b8e45e291"
format_stamp: "Formatted at 2024-05-03 01:59:43 on dist-test-slave-82l4"
I20240503 01:59:43.546065 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:43.546942 2107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:43.547225 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:43.547289 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-0-root
uuid: "b32a463b2df340f7b2b4d32b8e45e291"
format_stamp: "Formatted at 2024-05-03 01:59:43 on dist-test-slave-82l4"
I20240503 01:59:43.547380 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:43.561226 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:43.561668 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:43.562011 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:43.562525 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:43.562650 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:43.562744 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:43.562788 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:43.566788 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:46789
I20240503 01:59:43.566802 2170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:46789 every 8 connection(s)
I20240503 01:59:43.567657 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:43.570601 2175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:43.571427 2176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:43.572116 2171 heartbeater.cc:347] Connected to a master server at 127.0.99.62:33223
W20240503 01:59:43.572212 2178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:43.572224 2171 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:43.572485 2171 heartbeater.cc:511] Master 127.0.99.62:33223 requested a full tablet report, sending...
I20240503 01:59:43.573088 2042 ts_manager.cc:194] Registered new tserver with Master: b32a463b2df340f7b2b4d32b8e45e291 (127.0.99.1:46789)
I20240503 01:59:43.573344 396 server_base.cc:1034] running on GCE node
I20240503 01:59:43.573590 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:43.573639 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:43.573659 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701583573659 us; error 0 us; skew 500 ppm
I20240503 01:59:43.574077 2042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35092
I20240503 01:59:43.574775 396 webserver.cc:460] Webserver started at http://127.0.99.2:43139/ using document root <none> and password file <none>
I20240503 01:59:43.574927 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:43.575033 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:43.575134 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:43.575518 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-1-root/instance:
uuid: "edc497971b3c4b07ad6666b3025689c1"
format_stamp: "Formatted at 2024-05-03 01:59:43 on dist-test-slave-82l4"
I20240503 01:59:43.576944 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 01:59:43.577836 2183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:43.578060 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:43.578126 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-1-root
uuid: "edc497971b3c4b07ad6666b3025689c1"
format_stamp: "Formatted at 2024-05-03 01:59:43 on dist-test-slave-82l4"
I20240503 01:59:43.578200 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:43.585098 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:43.585407 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:43.585667 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:43.586088 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:43.586128 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:43.586179 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:43.586210 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:43.590186 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:44563
I20240503 01:59:43.590226 2246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:44563 every 8 connection(s)
I20240503 01:59:43.591029 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:43.594046 2253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:43.594759 2250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:43.594775 2247 heartbeater.cc:347] Connected to a master server at 127.0.99.62:33223
I20240503 01:59:43.594902 2247 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:43.595130 2247 heartbeater.cc:511] Master 127.0.99.62:33223 requested a full tablet report, sending...
W20240503 01:59:43.595490 2251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:43.595767 2042 ts_manager.cc:194] Registered new tserver with Master: edc497971b3c4b07ad6666b3025689c1 (127.0.99.2:44563)
I20240503 01:59:43.596232 2042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35098
I20240503 01:59:43.596632 396 server_base.cc:1034] running on GCE node
I20240503 01:59:43.596828 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:43.596886 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:43.596908 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701583596908 us; error 0 us; skew 500 ppm
I20240503 01:59:43.597888 396 webserver.cc:460] Webserver started at http://127.0.99.3:37123/ using document root <none> and password file <none>
I20240503 01:59:43.598011 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:43.598053 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:43.598109 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:43.598443 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-2-root/instance:
uuid: "876084c657b14a5e9d364226fda4f9ad"
format_stamp: "Formatted at 2024-05-03 01:59:43 on dist-test-slave-82l4"
I20240503 01:59:43.599879 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 01:59:43.600669 2258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:43.600914 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:43.600975 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-2-root
uuid: "876084c657b14a5e9d364226fda4f9ad"
format_stamp: "Formatted at 2024-05-03 01:59:43 on dist-test-slave-82l4"
I20240503 01:59:43.601034 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:43.626129 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:43.626555 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:43.626863 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:43.627353 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:43.627395 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:43.627444 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:43.627473 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:43.633993 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:41541
I20240503 01:59:43.634016 2321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:41541 every 8 connection(s)
I20240503 01:59:43.639086 2322 heartbeater.cc:347] Connected to a master server at 127.0.99.62:33223
I20240503 01:59:43.639186 2322 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:43.639400 2322 heartbeater.cc:511] Master 127.0.99.62:33223 requested a full tablet report, sending...
I20240503 01:59:43.639997 2042 ts_manager.cc:194] Registered new tserver with Master: 876084c657b14a5e9d364226fda4f9ad (127.0.99.3:41541)
I20240503 01:59:43.640223 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005613351s
I20240503 01:59:43.640513 2042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35104
I20240503 01:59:44.575045 2171 heartbeater.cc:503] Master 127.0.99.62:33223 was elected leader, sending a full tablet report...
I20240503 01:59:44.597324 2247 heartbeater.cc:503] Master 127.0.99.62:33223 was elected leader, sending a full tablet report...
I20240503 01:59:44.641378 2322 heartbeater.cc:503] Master 127.0.99.62:33223 was elected leader, sending a full tablet report...
I20240503 01:59:44.669297 396 test_util.cc:268] Using random seed: 316250735
I20240503 01:59:44.678828 2042 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55764:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 01:59:44.679386 2042 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 01:59:44.690562 2286 tablet_service.cc:1467] Processing CreateTablet for tablet 1841194d17d64058acfe79adc5d647a0 (DEFAULT_TABLE table=test-workload [id=98779ae19b0d457f91eb9d29c2e2769f]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 01:59:44.691002 2286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1841194d17d64058acfe79adc5d647a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.692308 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 96179fcafd744bb88e96985236ee8ec5 (DEFAULT_TABLE table=test-workload [id=98779ae19b0d457f91eb9d29c2e2769f]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 01:59:44.692308 2211 tablet_service.cc:1467] Processing CreateTablet for tablet 1841194d17d64058acfe79adc5d647a0 (DEFAULT_TABLE table=test-workload [id=98779ae19b0d457f91eb9d29c2e2769f]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 01:59:44.692652 2211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1841194d17d64058acfe79adc5d647a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.692834 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96179fcafd744bb88e96985236ee8ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.690562 2285 tablet_service.cc:1467] Processing CreateTablet for tablet 96179fcafd744bb88e96985236ee8ec5 (DEFAULT_TABLE table=test-workload [id=98779ae19b0d457f91eb9d29c2e2769f]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 01:59:44.694059 2285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96179fcafd744bb88e96985236ee8ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.695986 2342 tablet_bootstrap.cc:492] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad: Bootstrap starting.
I20240503 01:59:44.696913 2342 tablet_bootstrap.cc:654] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.698015 2342 tablet_bootstrap.cc:492] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad: No bootstrap required, opened a new log
I20240503 01:59:44.698117 2342 ts_tablet_manager.cc:1367] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:44.698499 2342 raft_consensus.cc:348] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } }
I20240503 01:59:44.698601 2342 raft_consensus.cc:374] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.698635 2342 raft_consensus.cc:729] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 876084c657b14a5e9d364226fda4f9ad, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.698773 2342 consensus_queue.cc:260] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } }
I20240503 01:59:44.699805 2343 tablet_bootstrap.cc:492] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1: Bootstrap starting.
I20240503 01:59:44.701179 2342 ts_tablet_manager.cc:1398] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:44.701567 2343 tablet_bootstrap.cc:654] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.703209 2342 tablet_bootstrap.cc:492] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad: Bootstrap starting.
I20240503 01:59:44.703514 2135 tablet_service.cc:1467] Processing CreateTablet for tablet 1841194d17d64058acfe79adc5d647a0 (DEFAULT_TABLE table=test-workload [id=98779ae19b0d457f91eb9d29c2e2769f]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 01:59:44.703626 2134 tablet_service.cc:1467] Processing CreateTablet for tablet 96179fcafd744bb88e96985236ee8ec5 (DEFAULT_TABLE table=test-workload [id=98779ae19b0d457f91eb9d29c2e2769f]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 01:59:44.703792 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1841194d17d64058acfe79adc5d647a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.703948 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96179fcafd744bb88e96985236ee8ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.704172 2342 tablet_bootstrap.cc:654] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.705884 2348 tablet_bootstrap.cc:492] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291: Bootstrap starting.
I20240503 01:59:44.706946 2348 tablet_bootstrap.cc:654] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.707480 2342 tablet_bootstrap.cc:492] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad: No bootstrap required, opened a new log
I20240503 01:59:44.707608 2342 ts_tablet_manager.cc:1367] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:44.708086 2348 tablet_bootstrap.cc:492] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291: No bootstrap required, opened a new log
I20240503 01:59:44.708055 2342 raft_consensus.cc:348] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } }
I20240503 01:59:44.708165 2348 ts_tablet_manager.cc:1367] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:44.708154 2342 raft_consensus.cc:374] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.708245 2342 raft_consensus.cc:729] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 876084c657b14a5e9d364226fda4f9ad, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.708392 2342 consensus_queue.cc:260] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } }
I20240503 01:59:44.708590 2348 raft_consensus.cc:348] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } }
I20240503 01:59:44.708647 2342 ts_tablet_manager.cc:1398] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:44.708686 2348 raft_consensus.cc:374] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.708724 2348 raft_consensus.cc:729] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32a463b2df340f7b2b4d32b8e45e291, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.708863 2348 consensus_queue.cc:260] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } }
I20240503 01:59:44.709177 2348 ts_tablet_manager.cc:1398] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:44.709440 2348 tablet_bootstrap.cc:492] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291: Bootstrap starting.
I20240503 01:59:44.710743 2348 tablet_bootstrap.cc:654] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.711798 2348 tablet_bootstrap.cc:492] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291: No bootstrap required, opened a new log
I20240503 01:59:44.711886 2348 ts_tablet_manager.cc:1367] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:44.711961 2343 tablet_bootstrap.cc:492] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1: No bootstrap required, opened a new log
I20240503 01:59:44.712059 2343 ts_tablet_manager.cc:1367] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 01:59:44.712270 2348 raft_consensus.cc:348] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } }
I20240503 01:59:44.712353 2348 raft_consensus.cc:374] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.712390 2348 raft_consensus.cc:729] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32a463b2df340f7b2b4d32b8e45e291, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.712520 2348 consensus_queue.cc:260] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } }
I20240503 01:59:44.712589 2343 raft_consensus.cc:348] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } }
I20240503 01:59:44.712726 2343 raft_consensus.cc:374] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.712778 2343 raft_consensus.cc:729] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc497971b3c4b07ad6666b3025689c1, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.712826 2348 ts_tablet_manager.cc:1398] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:44.712929 2343 consensus_queue.cc:260] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } }
I20240503 01:59:44.713371 2343 ts_tablet_manager.cc:1398] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:44.713572 2343 tablet_bootstrap.cc:492] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1: Bootstrap starting.
I20240503 01:59:44.714661 2343 tablet_bootstrap.cc:654] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.716722 2343 tablet_bootstrap.cc:492] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1: No bootstrap required, opened a new log
I20240503 01:59:44.716821 2343 ts_tablet_manager.cc:1367] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:44.717326 2343 raft_consensus.cc:348] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } }
I20240503 01:59:44.717460 2343 raft_consensus.cc:374] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.717510 2343 raft_consensus.cc:729] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc497971b3c4b07ad6666b3025689c1, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.717653 2343 consensus_queue.cc:260] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } }
I20240503 01:59:44.717988 2343 ts_tablet_manager.cc:1398] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:44.718760 2350 raft_consensus.cc:482] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:44.718859 2350 raft_consensus.cc:504] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } }
I20240503 01:59:44.719453 2350 leader_election.cc:288] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edc497971b3c4b07ad6666b3025689c1 (127.0.99.2:44563), 876084c657b14a5e9d364226fda4f9ad (127.0.99.3:41541)
I20240503 01:59:44.723487 2221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96179fcafd744bb88e96985236ee8ec5" candidate_uuid: "b32a463b2df340f7b2b4d32b8e45e291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc497971b3c4b07ad6666b3025689c1" is_pre_election: true
I20240503 01:59:44.723616 2221 raft_consensus.cc:2454] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b32a463b2df340f7b2b4d32b8e45e291 in term 0.
I20240503 01:59:44.723907 2109 leader_election.cc:302] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b32a463b2df340f7b2b4d32b8e45e291, edc497971b3c4b07ad6666b3025689c1; no voters:
I20240503 01:59:44.724063 2350 raft_consensus.cc:2781] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:44.724115 2350 raft_consensus.cc:482] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:44.724160 2350 raft_consensus.cc:3037] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.724942 2350 raft_consensus.cc:504] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } }
I20240503 01:59:44.725261 2350 leader_election.cc:288] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [CANDIDATE]: Term 1 election: Requested vote from peers edc497971b3c4b07ad6666b3025689c1 (127.0.99.2:44563), 876084c657b14a5e9d364226fda4f9ad (127.0.99.3:41541)
I20240503 01:59:44.725653 2221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96179fcafd744bb88e96985236ee8ec5" candidate_uuid: "b32a463b2df340f7b2b4d32b8e45e291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc497971b3c4b07ad6666b3025689c1"
I20240503 01:59:44.725775 2221 raft_consensus.cc:3037] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.726941 2221 raft_consensus.cc:2454] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b32a463b2df340f7b2b4d32b8e45e291 in term 1.
I20240503 01:59:44.727228 2109 leader_election.cc:302] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b32a463b2df340f7b2b4d32b8e45e291, edc497971b3c4b07ad6666b3025689c1; no voters:
I20240503 01:59:44.727550 2350 raft_consensus.cc:2781] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:44.727542 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96179fcafd744bb88e96985236ee8ec5" candidate_uuid: "b32a463b2df340f7b2b4d32b8e45e291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "876084c657b14a5e9d364226fda4f9ad" is_pre_election: true
I20240503 01:59:44.727630 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96179fcafd744bb88e96985236ee8ec5" candidate_uuid: "b32a463b2df340f7b2b4d32b8e45e291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "876084c657b14a5e9d364226fda4f9ad"
I20240503 01:59:44.727655 2296 raft_consensus.cc:2454] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b32a463b2df340f7b2b4d32b8e45e291 in term 0.
I20240503 01:59:44.727757 2350 raft_consensus.cc:686] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [term 1 LEADER]: Becoming Leader. State: Replica: b32a463b2df340f7b2b4d32b8e45e291, State: Running, Role: LEADER
I20240503 01:59:44.727886 2350 consensus_queue.cc:237] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } }
I20240503 01:59:44.729509 2041 catalog_manager.cc:5484] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 reported cstate change: term changed from 0 to 1, leader changed from <none> to b32a463b2df340f7b2b4d32b8e45e291 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "b32a463b2df340f7b2b4d32b8e45e291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:44.772378 2356 raft_consensus.cc:482] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:44.772541 2356 raft_consensus.cc:504] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } }
I20240503 01:59:44.773147 2356 leader_election.cc:288] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 876084c657b14a5e9d364226fda4f9ad (127.0.99.3:41541), edc497971b3c4b07ad6666b3025689c1 (127.0.99.2:44563)
I20240503 01:59:44.773476 2221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1841194d17d64058acfe79adc5d647a0" candidate_uuid: "b32a463b2df340f7b2b4d32b8e45e291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc497971b3c4b07ad6666b3025689c1" is_pre_election: true
I20240503 01:59:44.773459 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1841194d17d64058acfe79adc5d647a0" candidate_uuid: "b32a463b2df340f7b2b4d32b8e45e291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "876084c657b14a5e9d364226fda4f9ad" is_pre_election: true
I20240503 01:59:44.773633 2221 raft_consensus.cc:2454] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b32a463b2df340f7b2b4d32b8e45e291 in term 0.
I20240503 01:59:44.773633 2295 raft_consensus.cc:2454] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b32a463b2df340f7b2b4d32b8e45e291 in term 0.
I20240503 01:59:44.773926 2109 leader_election.cc:302] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b32a463b2df340f7b2b4d32b8e45e291, edc497971b3c4b07ad6666b3025689c1; no voters:
I20240503 01:59:44.774075 2356 raft_consensus.cc:2781] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:44.774137 2356 raft_consensus.cc:482] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:44.774183 2356 raft_consensus.cc:3037] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.775128 2356 raft_consensus.cc:504] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } }
I20240503 01:59:44.775471 2356 leader_election.cc:288] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [CANDIDATE]: Term 1 election: Requested vote from peers 876084c657b14a5e9d364226fda4f9ad (127.0.99.3:41541), edc497971b3c4b07ad6666b3025689c1 (127.0.99.2:44563)
I20240503 01:59:44.775779 2221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1841194d17d64058acfe79adc5d647a0" candidate_uuid: "b32a463b2df340f7b2b4d32b8e45e291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc497971b3c4b07ad6666b3025689c1"
I20240503 01:59:44.775779 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1841194d17d64058acfe79adc5d647a0" candidate_uuid: "b32a463b2df340f7b2b4d32b8e45e291" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "876084c657b14a5e9d364226fda4f9ad"
I20240503 01:59:44.775913 2295 raft_consensus.cc:3037] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.775913 2221 raft_consensus.cc:3037] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.776784 2295 raft_consensus.cc:2454] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b32a463b2df340f7b2b4d32b8e45e291 in term 1.
I20240503 01:59:44.776784 2221 raft_consensus.cc:2454] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b32a463b2df340f7b2b4d32b8e45e291 in term 1.
I20240503 01:59:44.777107 2111 leader_election.cc:302] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 876084c657b14a5e9d364226fda4f9ad, b32a463b2df340f7b2b4d32b8e45e291; no voters:
I20240503 01:59:44.777262 2356 raft_consensus.cc:2781] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:44.777350 2356 raft_consensus.cc:686] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 1 LEADER]: Becoming Leader. State: Replica: b32a463b2df340f7b2b4d32b8e45e291, State: Running, Role: LEADER
I20240503 01:59:44.777483 2356 consensus_queue.cc:237] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } }
I20240503 01:59:44.779296 2041 catalog_manager.cc:5484] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 reported cstate change: term changed from 0 to 1, leader changed from <none> to b32a463b2df340f7b2b4d32b8e45e291 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "b32a463b2df340f7b2b4d32b8e45e291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:45.218941 2356 consensus_queue.cc:1035] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:45.221649 2295 raft_consensus.cc:3037] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.223225 2356 consensus_queue.cc:1035] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:45.320838 2356 consensus_queue.cc:1035] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:45.323807 2356 consensus_queue.cc:1035] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:45.504173 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 01:59:45.508622 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1841194d17d64058acfe79adc5d647a0"
dest_uuid: "b32a463b2df340f7b2b4d32b8e45e291"
mode: GRACEFUL
new_leader_uuid: "876084c657b14a5e9d364226fda4f9ad"
from {username='slave'} at 127.0.0.1:59470
I20240503 01:59:45.508781 2145 raft_consensus.cc:595] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 1 LEADER]: Received request to transfer leadership to TS 876084c657b14a5e9d364226fda4f9ad
I20240503 01:59:45.509135 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 01:59:45.509214 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 01:59:45.928849 2356 raft_consensus.cc:979] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291: : Instructing follower 876084c657b14a5e9d364226fda4f9ad to start an election
I20240503 01:59:45.929013 2356 raft_consensus.cc:1067] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 1 LEADER]: Signalling peer 876084c657b14a5e9d364226fda4f9ad to start an election
I20240503 01:59:45.929503 2295 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1841194d17d64058acfe79adc5d647a0"
dest_uuid: "876084c657b14a5e9d364226fda4f9ad"
from {username='slave'} at 127.0.0.1:58076
I20240503 01:59:45.929627 2295 raft_consensus.cc:482] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:45.929689 2295 raft_consensus.cc:3037] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:45.930505 2295 raft_consensus.cc:504] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } }
I20240503 01:59:45.931005 2295 leader_election.cc:288] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [CANDIDATE]: Term 2 election: Requested vote from peers edc497971b3c4b07ad6666b3025689c1 (127.0.99.2:44563), b32a463b2df340f7b2b4d32b8e45e291 (127.0.99.1:46789)
I20240503 01:59:45.934543 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1841194d17d64058acfe79adc5d647a0" candidate_uuid: "876084c657b14a5e9d364226fda4f9ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b32a463b2df340f7b2b4d32b8e45e291"
I20240503 01:59:45.934543 2221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1841194d17d64058acfe79adc5d647a0" candidate_uuid: "876084c657b14a5e9d364226fda4f9ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "edc497971b3c4b07ad6666b3025689c1"
I20240503 01:59:45.934727 2145 raft_consensus.cc:3032] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:45.934727 2221 raft_consensus.cc:3037] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:45.934814 2145 raft_consensus.cc:729] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b32a463b2df340f7b2b4d32b8e45e291, State: Running, Role: LEADER
I20240503 01:59:45.935026 2145 consensus_queue.cc:260] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } }
I20240503 01:59:45.935231 2145 raft_consensus.cc:3037] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:45.935956 2221 raft_consensus.cc:2454] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 876084c657b14a5e9d364226fda4f9ad in term 2.
I20240503 01:59:45.936162 2145 raft_consensus.cc:2454] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 876084c657b14a5e9d364226fda4f9ad in term 2.
I20240503 01:59:45.936332 2260 leader_election.cc:302] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 876084c657b14a5e9d364226fda4f9ad, edc497971b3c4b07ad6666b3025689c1; no voters:
I20240503 01:59:45.936630 2374 raft_consensus.cc:2781] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:45.936813 2374 raft_consensus.cc:686] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 2 LEADER]: Becoming Leader. State: Replica: 876084c657b14a5e9d364226fda4f9ad, State: Running, Role: LEADER
I20240503 01:59:45.936964 2374 consensus_queue.cc:237] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } }
I20240503 01:59:45.938329 2042 catalog_manager.cc:5484] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad reported cstate change: term changed from 1 to 2, leader changed from b32a463b2df340f7b2b4d32b8e45e291 (127.0.99.1) to 876084c657b14a5e9d364226fda4f9ad (127.0.99.3). New cstate: current_term: 2 leader_uuid: "876084c657b14a5e9d364226fda4f9ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:46.508288 2145 raft_consensus.cc:1261] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 2 FOLLOWER]: Refusing update from remote peer 876084c657b14a5e9d364226fda4f9ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 01:59:46.508795 2380 consensus_queue.cc:1035] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:46.511606 2221 raft_consensus.cc:1261] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 2 FOLLOWER]: Refusing update from remote peer 876084c657b14a5e9d364226fda4f9ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 01:59:46.512003 2382 consensus_queue.cc:1035] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:47.509459 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 01:59:47.509909 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 01:59:47.509979 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 01:59:49.510309 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 01:59:49.510713 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 01:59:49.510782 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 01:59:51.511092 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 01:59:51.511483 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 01:59:51.511547 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 01:59:53.511832 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 01:59:53.512223 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 01:59:53.512297 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 01:59:55.512634 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 01:59:55.513017 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 01:59:55.513075 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 01:59:55.842396 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:55.845752 2391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:55.846585 2392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:55.847347 2394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:55.848394 396 server_base.cc:1034] running on GCE node
I20240503 01:59:55.848560 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 01:59:55.848600 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 01:59:55.848624 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701595848623 us; error 0 us; skew 500 ppm
I20240503 01:59:55.849579 396 webserver.cc:460] Webserver started at http://127.0.99.4:45367/ using document root <none> and password file <none>
I20240503 01:59:55.849725 396 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:55.849773 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:55.849851 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:55.850337 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-3-root/instance:
uuid: "35ac17d4656c4a38bfef8dcef07f1ab3"
format_stamp: "Formatted at 2024-05-03 01:59:55 on dist-test-slave-82l4"
I20240503 01:59:55.851867 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 01:59:55.852731 2399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:55.852941 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:55.853005 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-3-root
uuid: "35ac17d4656c4a38bfef8dcef07f1ab3"
format_stamp: "Formatted at 2024-05-03 01:59:55 on dist-test-slave-82l4"
I20240503 01:59:55.853066 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:55.878417 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:55.878787 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:55.879146 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:55.879602 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:55.879644 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:55.879684 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:55.879735 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:55.884341 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:38235
I20240503 01:59:55.884390 2462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:38235 every 8 connection(s)
I20240503 01:59:55.888900 2463 heartbeater.cc:347] Connected to a master server at 127.0.99.62:33223
I20240503 01:59:55.888996 2463 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:55.889207 2463 heartbeater.cc:511] Master 127.0.99.62:33223 requested a full tablet report, sending...
I20240503 01:59:55.889780 2041 ts_manager.cc:194] Registered new tserver with Master: 35ac17d4656c4a38bfef8dcef07f1ab3 (127.0.99.4:38235)
I20240503 01:59:55.890288 2041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44822
I20240503 01:59:56.519356 2295 consensus_queue.cc:237] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: true } }
I20240503 01:59:56.520720 2145 raft_consensus.cc:1261] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 2 FOLLOWER]: Refusing update from remote peer 876084c657b14a5e9d364226fda4f9ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 01:59:56.520758 2221 raft_consensus.cc:1261] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 2 FOLLOWER]: Refusing update from remote peer 876084c657b14a5e9d364226fda4f9ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 01:59:56.521116 2390 consensus_queue.cc:1035] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:56.521302 2470 consensus_queue.cc:1035] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:56.522913 2470 raft_consensus.cc:2932] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 35ac17d4656c4a38bfef8dcef07f1ab3 (127.0.99.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: true } } }
I20240503 01:59:56.523310 2221 raft_consensus.cc:2932] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 35ac17d4656c4a38bfef8dcef07f1ab3 (127.0.99.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: true } } }
I20240503 01:59:56.523586 2145 raft_consensus.cc:2932] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 35ac17d4656c4a38bfef8dcef07f1ab3 (127.0.99.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: true } } }
W20240503 01:59:56.524705 2261 consensus_peers.cc:485] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad -> Peer 35ac17d4656c4a38bfef8dcef07f1ab3 (127.0.99.4:38235): Couldn't send request to peer 35ac17d4656c4a38bfef8dcef07f1ab3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1841194d17d64058acfe79adc5d647a0. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:56.525943 2041 catalog_manager.cc:5484] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 reported cstate change: config changed from index -1 to 3, NON_VOTER 35ac17d4656c4a38bfef8dcef07f1ab3 (127.0.99.4) added. New cstate: current_term: 2 leader_uuid: "876084c657b14a5e9d364226fda4f9ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: true } } }
I20240503 01:59:56.891057 2463 heartbeater.cc:503] Master 127.0.99.62:33223 was elected leader, sending a full tablet report...
I20240503 01:59:56.932346 2483 ts_tablet_manager.cc:902] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3: Initiating tablet copy from peer 876084c657b14a5e9d364226fda4f9ad (127.0.99.3:41541)
I20240503 01:59:56.933184 2483 tablet_copy_client.cc:308] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:41541
I20240503 01:59:56.937345 2306 tablet_copy_service.cc:140] P 876084c657b14a5e9d364226fda4f9ad: Received BeginTabletCopySession request for tablet 1841194d17d64058acfe79adc5d647a0 from peer 35ac17d4656c4a38bfef8dcef07f1ab3 ({username='slave'} at 127.0.0.1:44622)
I20240503 01:59:56.937453 2306 tablet_copy_service.cc:161] P 876084c657b14a5e9d364226fda4f9ad: Beginning new tablet copy session on tablet 1841194d17d64058acfe79adc5d647a0 from peer 35ac17d4656c4a38bfef8dcef07f1ab3 at {username='slave'} at 127.0.0.1:44622: session id = 35ac17d4656c4a38bfef8dcef07f1ab3-1841194d17d64058acfe79adc5d647a0
I20240503 01:59:56.938741 2306 tablet_copy_source_session.cc:205] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:56.939926 2483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1841194d17d64058acfe79adc5d647a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.942036 2483 tablet_copy_client.cc:783] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:56.942303 2483 tablet_copy_client.cc:647] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:56.943697 2483 tablet_copy_client.cc:523] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:56.945147 2483 tablet_bootstrap.cc:492] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3: Bootstrap starting.
I20240503 01:59:56.949970 2483 tablet_bootstrap.cc:492] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:56.950261 2483 tablet_bootstrap.cc:492] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3: Bootstrap complete.
I20240503 01:59:56.950434 2483 ts_tablet_manager.cc:1367] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:59:56.950910 2483 raft_consensus.cc:348] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: true } }
I20240503 01:59:56.951100 2483 raft_consensus.cc:729] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 35ac17d4656c4a38bfef8dcef07f1ab3, State: Initialized, Role: LEARNER
I20240503 01:59:56.951220 2483 consensus_queue.cc:260] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: true } }
I20240503 01:59:56.951575 2483 ts_tablet_manager.cc:1398] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:56.952085 2306 tablet_copy_service.cc:342] P 876084c657b14a5e9d364226fda4f9ad: Request end of tablet copy session 35ac17d4656c4a38bfef8dcef07f1ab3-1841194d17d64058acfe79adc5d647a0 received from {username='slave'} at 127.0.0.1:44622
I20240503 01:59:56.952198 2306 tablet_copy_service.cc:433] P 876084c657b14a5e9d364226fda4f9ad: ending tablet copy session 35ac17d4656c4a38bfef8dcef07f1ab3-1841194d17d64058acfe79adc5d647a0 on tablet 1841194d17d64058acfe79adc5d647a0 with peer 35ac17d4656c4a38bfef8dcef07f1ab3
I20240503 01:59:57.507462 2437 raft_consensus.cc:1203] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 01:59:57.513329 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 01:59:57.513729 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 01:59:57.513806 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 01:59:57.945966 2477 raft_consensus.cc:1050] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad: attempting to promote NON_VOTER 35ac17d4656c4a38bfef8dcef07f1ab3 to VOTER
I20240503 01:59:57.946460 2477 consensus_queue.cc:237] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: false } }
I20240503 01:59:57.947824 2437 raft_consensus.cc:1261] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3 [term 2 LEARNER]: Refusing update from remote peer 876084c657b14a5e9d364226fda4f9ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 01:59:57.947911 2221 raft_consensus.cc:1261] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 2 FOLLOWER]: Refusing update from remote peer 876084c657b14a5e9d364226fda4f9ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 01:59:57.947858 2145 raft_consensus.cc:1261] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 2 FOLLOWER]: Refusing update from remote peer 876084c657b14a5e9d364226fda4f9ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 01:59:57.948189 2495 consensus_queue.cc:1035] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:57.948372 2492 consensus_queue.cc:1035] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:57.948690 2492 consensus_queue.cc:1035] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:57.950469 2492 raft_consensus.cc:2932] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 35ac17d4656c4a38bfef8dcef07f1ab3 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: false } } }
I20240503 01:59:57.951792 2437 raft_consensus.cc:2932] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 35ac17d4656c4a38bfef8dcef07f1ab3 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: false } } }
I20240503 01:59:57.952723 2042 catalog_manager.cc:5484] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad reported cstate change: config changed from index 3 to 4, 35ac17d4656c4a38bfef8dcef07f1ab3 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "876084c657b14a5e9d364226fda4f9ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:57.953415 2145 raft_consensus.cc:2932] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 35ac17d4656c4a38bfef8dcef07f1ab3 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: false } } }
I20240503 01:59:57.953692 2221 raft_consensus.cc:2932] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 35ac17d4656c4a38bfef8dcef07f1ab3 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "edc497971b3c4b07ad6666b3025689c1" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 44563 } attrs { replace: true } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: false } } }
I20240503 01:59:57.958777 2295 consensus_queue.cc:237] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: false } }
I20240503 01:59:57.959708 2437 raft_consensus.cc:1261] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3 [term 2 FOLLOWER]: Refusing update from remote peer 876084c657b14a5e9d364226fda4f9ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:57.959806 2145 raft_consensus.cc:1261] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 2 FOLLOWER]: Refusing update from remote peer 876084c657b14a5e9d364226fda4f9ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:57.960027 2495 consensus_queue.cc:1035] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:57.960206 2477 consensus_queue.cc:1035] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:57.961580 2492 raft_consensus.cc:2932] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER edc497971b3c4b07ad6666b3025689c1 (127.0.99.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: false } } }
I20240503 01:59:57.961895 2437 raft_consensus.cc:2932] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER edc497971b3c4b07ad6666b3025689c1 (127.0.99.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: false } } }
I20240503 01:59:57.962153 2145 raft_consensus.cc:2932] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER edc497971b3c4b07ad6666b3025689c1 (127.0.99.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: false } } }
I20240503 01:59:57.963444 2029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1841194d17d64058acfe79adc5d647a0 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:57.963722 2041 catalog_manager.cc:5484] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad reported cstate change: config changed from index 4 to 5, VOTER edc497971b3c4b07ad6666b3025689c1 (127.0.99.2) evicted. New cstate: current_term: 2 leader_uuid: "876084c657b14a5e9d364226fda4f9ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "876084c657b14a5e9d364226fda4f9ad" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 41541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b32a463b2df340f7b2b4d32b8e45e291" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 46789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35ac17d4656c4a38bfef8dcef07f1ab3" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 38235 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:57.966063 2211 tablet_service.cc:1514] Processing DeleteTablet for tablet 1841194d17d64058acfe79adc5d647a0 with delete_type TABLET_DATA_TOMBSTONED (TS edc497971b3c4b07ad6666b3025689c1 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34976
I20240503 01:59:57.966545 2504 tablet_replica.cc:330] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1: stopping tablet replica
I20240503 01:59:57.966665 2504 raft_consensus.cc:2229] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:57.966799 2504 raft_consensus.cc:2258] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:57.967409 2504 ts_tablet_manager.cc:1875] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:57.969185 2504 ts_tablet_manager.cc:1888] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 01:59:57.969269 2504 log.cc:1196] T 1841194d17d64058acfe79adc5d647a0 P edc497971b3c4b07ad6666b3025689c1: Deleting WAL directory at /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714701568895932-396-0/minicluster-data/ts-1-root/wals/1841194d17d64058acfe79adc5d647a0
I20240503 01:59:57.969977 2027 catalog_manager.cc:4830] TS edc497971b3c4b07ad6666b3025689c1 (127.0.99.2:44563): tablet 1841194d17d64058acfe79adc5d647a0 (table test-workload [id=98779ae19b0d457f91eb9d29c2e2769f]) successfully deleted
I20240503 01:59:59.514168 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 01:59:59.514603 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 01:59:59.514673 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:01.515076 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:01.515447 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:01.515511 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:03.515863 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:03.516271 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:03.516333 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:05.516693 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:05.517074 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:05.517132 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:07.517545 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:07.517944 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:07.518002 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:09.518390 2096 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:09.518815 2096 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:09.518875 2096 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:10.036442 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 02:00:10.040305 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:10.040571 396 tablet_replica.cc:330] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291: stopping tablet replica
I20240503 02:00:10.040706 396 raft_consensus.cc:2229] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:10.040953 396 raft_consensus.cc:2258] T 96179fcafd744bb88e96985236ee8ec5 P b32a463b2df340f7b2b4d32b8e45e291 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.041465 396 tablet_replica.cc:330] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291: stopping tablet replica
I20240503 02:00:10.041574 396 raft_consensus.cc:2229] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:10.041704 396 raft_consensus.cc:2258] T 1841194d17d64058acfe79adc5d647a0 P b32a463b2df340f7b2b4d32b8e45e291 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.055820 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 02:00:10.058125 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 02:00:10.062917 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:10.063194 396 tablet_replica.cc:330] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1: stopping tablet replica
I20240503 02:00:10.063308 396 raft_consensus.cc:2229] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:10.063416 396 raft_consensus.cc:2258] T 96179fcafd744bb88e96985236ee8ec5 P edc497971b3c4b07ad6666b3025689c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.076391 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 02:00:10.078693 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 02:00:10.083696 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:10.083968 396 tablet_replica.cc:330] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad: stopping tablet replica
I20240503 02:00:10.084098 396 raft_consensus.cc:2229] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:10.084242 396 raft_consensus.cc:2258] T 96179fcafd744bb88e96985236ee8ec5 P 876084c657b14a5e9d364226fda4f9ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.084751 396 tablet_replica.cc:330] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad: stopping tablet replica
I20240503 02:00:10.084859 396 raft_consensus.cc:2229] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 2 LEADER]: Raft consensus shutting down.
I20240503 02:00:10.085067 396 raft_consensus.cc:2258] T 1841194d17d64058acfe79adc5d647a0 P 876084c657b14a5e9d364226fda4f9ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.098027 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 02:00:10.100517 396 tablet_server.cc:178] TabletServer@127.0.99.4:0 shutting down...
I20240503 02:00:10.104640 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:10.104857 396 tablet_replica.cc:330] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3: stopping tablet replica
I20240503 02:00:10.104969 396 raft_consensus.cc:2229] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:10.105100 396 raft_consensus.cc:2258] T 1841194d17d64058acfe79adc5d647a0 P 35ac17d4656c4a38bfef8dcef07f1ab3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.118664 396 tablet_server.cc:194] TabletServer@127.0.99.4:0 shutdown complete.
I20240503 02:00:10.120908 396 master.cc:537] Master@127.0.99.62:33223 shutting down...
I20240503 02:00:10.124694 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:10.124863 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.124930 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3d6048f112714ba68d6fd1a6fc595e68: stopping tablet replica
I20240503 02:00:10.138252 396 master.cc:552] Master@127.0.99.62:33223 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26675 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240503 02:00:10.145973 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:38547
I20240503 02:00:10.146327 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:10.149307 2508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:10.150532 2509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:10.151427 2511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:10.152235 396 server_base.cc:1034] running on GCE node
I20240503 02:00:10.152415 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:10.152454 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:10.152470 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701610152470 us; error 0 us; skew 500 ppm
I20240503 02:00:10.159346 396 webserver.cc:460] Webserver started at http://127.0.99.62:41907/ using document root <none> and password file <none>
I20240503 02:00:10.159533 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:10.159595 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:10.159668 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:10.160105 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714701568895932-396-0/minicluster-data/master-0-root/instance:
uuid: "d9249abbcc0744f9b5163b930135542d"
format_stamp: "Formatted at 2024-05-03 02:00:10 on dist-test-slave-82l4"
I20240503 02:00:10.161630 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:00:10.162590 2516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:10.162907 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.163038 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714701568895932-396-0/minicluster-data/master-0-root
uuid: "d9249abbcc0744f9b5163b930135542d"
format_stamp: "Formatted at 2024-05-03 02:00:10 on dist-test-slave-82l4"
I20240503 02:00:10.163117 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714701568895932-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714701568895932-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714701568895932-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:10.185812 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:10.186183 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:10.190237 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:38547
I20240503 02:00:10.193151 2568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:38547 every 8 connection(s)
I20240503 02:00:10.193585 2569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.195358 2569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d: Bootstrap starting.
I20240503 02:00:10.196094 2569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.197115 2569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d: No bootstrap required, opened a new log
I20240503 02:00:10.197480 2569 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9249abbcc0744f9b5163b930135542d" member_type: VOTER }
I20240503 02:00:10.197569 2569 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.197605 2569 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9249abbcc0744f9b5163b930135542d, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.197741 2569 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9249abbcc0744f9b5163b930135542d" member_type: VOTER }
I20240503 02:00:10.197811 2569 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:10.197849 2569 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:10.197897 2569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.198550 2569 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9249abbcc0744f9b5163b930135542d" member_type: VOTER }
I20240503 02:00:10.198670 2569 leader_election.cc:302] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9249abbcc0744f9b5163b930135542d; no voters:
I20240503 02:00:10.198840 2569 leader_election.cc:288] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:10.198993 2572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.199182 2572 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [term 1 LEADER]: Becoming Leader. State: Replica: d9249abbcc0744f9b5163b930135542d, State: Running, Role: LEADER
I20240503 02:00:10.199334 2572 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9249abbcc0744f9b5163b930135542d" member_type: VOTER }
I20240503 02:00:10.199344 2569 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:00:10.199767 2573 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9249abbcc0744f9b5163b930135542d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9249abbcc0744f9b5163b930135542d" member_type: VOTER } }
I20240503 02:00:10.199821 2574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9249abbcc0744f9b5163b930135542d. Latest consensus state: current_term: 1 leader_uuid: "d9249abbcc0744f9b5163b930135542d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9249abbcc0744f9b5163b930135542d" member_type: VOTER } }
I20240503 02:00:10.199940 2573 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:10.200040 2574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:10.200516 2579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:00:10.201269 2579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:00:10.201427 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:00:10.203125 2579 catalog_manager.cc:1347] Generated new cluster ID: f7297f9fb0174cf893fcb9f27619d9e6
I20240503 02:00:10.203171 2579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:00:10.250631 2579 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:00:10.251225 2579 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:00:10.263296 2579 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d: Generated new TSK 0
I20240503 02:00:10.263475 2579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:00:10.265492 396 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 5.6011e-05s
I20240503 02:00:22.348955 396 master.cc:537] Master@127.0.99.62:38547 shutting down...
I20240503 02:00:22.352437 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:22.352623 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:22.352675 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P d9249abbcc0744f9b5163b930135542d: stopping tablet replica
I20240503 02:00:22.364830 396 master.cc:552] Master@127.0.99.62:38547 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12223 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240503 02:00:22.369208 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:37049
I20240503 02:00:22.369526 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:22.372743 2592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:22.373880 2591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:22.374715 2594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:22.375560 396 server_base.cc:1034] running on GCE node
I20240503 02:00:22.375695 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:22.375727 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:22.375743 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701622375743 us; error 0 us; skew 500 ppm
I20240503 02:00:22.376710 396 webserver.cc:460] Webserver started at http://127.0.99.62:40239/ using document root <none> and password file <none>
I20240503 02:00:22.376865 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:22.376921 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:22.377008 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:22.377413 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/master-0-root/instance:
uuid: "90b098306e6b4750b0dc4199c0286152"
format_stamp: "Formatted at 2024-05-03 02:00:22 on dist-test-slave-82l4"
I20240503 02:00:22.378850 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:00:22.379797 2599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:22.380048 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:22.380108 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/master-0-root
uuid: "90b098306e6b4750b0dc4199c0286152"
format_stamp: "Formatted at 2024-05-03 02:00:22 on dist-test-slave-82l4"
I20240503 02:00:22.380163 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:22.397008 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:22.397418 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:22.401979 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:37049
I20240503 02:00:22.402627 2651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:37049 every 8 connection(s)
I20240503 02:00:22.403227 2652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:22.405086 2652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152: Bootstrap starting.
I20240503 02:00:22.405835 2652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:22.406924 2652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152: No bootstrap required, opened a new log
I20240503 02:00:22.407359 2652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b098306e6b4750b0dc4199c0286152" member_type: VOTER }
I20240503 02:00:22.407451 2652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:22.407472 2652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b098306e6b4750b0dc4199c0286152, State: Initialized, Role: FOLLOWER
I20240503 02:00:22.407598 2652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b098306e6b4750b0dc4199c0286152" member_type: VOTER }
I20240503 02:00:22.407676 2652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:22.407704 2652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:22.407737 2652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:22.408353 2652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b098306e6b4750b0dc4199c0286152" member_type: VOTER }
I20240503 02:00:22.408463 2652 leader_election.cc:302] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90b098306e6b4750b0dc4199c0286152; no voters:
I20240503 02:00:22.408612 2652 leader_election.cc:288] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:22.408727 2655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:22.409049 2655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [term 1 LEADER]: Becoming Leader. State: Replica: 90b098306e6b4750b0dc4199c0286152, State: Running, Role: LEADER
I20240503 02:00:22.409103 2652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:00:22.409197 2655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b098306e6b4750b0dc4199c0286152" member_type: VOTER }
I20240503 02:00:22.409628 2656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90b098306e6b4750b0dc4199c0286152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b098306e6b4750b0dc4199c0286152" member_type: VOTER } }
I20240503 02:00:22.409647 2657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b098306e6b4750b0dc4199c0286152. Latest consensus state: current_term: 1 leader_uuid: "90b098306e6b4750b0dc4199c0286152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b098306e6b4750b0dc4199c0286152" member_type: VOTER } }
I20240503 02:00:22.409852 2657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:22.409821 2656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:22.410346 2664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:00:22.411325 2664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:00:22.411530 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:00:22.413089 2664 catalog_manager.cc:1347] Generated new cluster ID: b00deae96bff4595aa7de54856c459f2
I20240503 02:00:22.413144 2664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:00:22.417366 2664 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:00:22.417891 2664 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:00:22.424371 2664 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152: Generated new TSK 0
I20240503 02:00:22.424546 2664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:00:22.427665 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:22.430788 2674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:22.431491 2673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:22.432313 396 server_base.cc:1034] running on GCE node
W20240503 02:00:22.433425 2676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:22.433642 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:22.433691 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:22.433707 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701622433707 us; error 0 us; skew 500 ppm
I20240503 02:00:22.434684 396 webserver.cc:460] Webserver started at http://127.0.99.1:40647/ using document root <none> and password file <none>
I20240503 02:00:22.434835 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:22.434885 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:22.434993 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:22.435410 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-0-root/instance:
uuid: "c74b634989f9485299019708592f9bd2"
format_stamp: "Formatted at 2024-05-03 02:00:22 on dist-test-slave-82l4"
I20240503 02:00:22.436913 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:22.437865 2681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:22.438094 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.438160 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-0-root
uuid: "c74b634989f9485299019708592f9bd2"
format_stamp: "Formatted at 2024-05-03 02:00:22 on dist-test-slave-82l4"
I20240503 02:00:22.438236 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:22.459446 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:22.459846 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:22.460147 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:22.460654 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:22.460693 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:22.460739 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:22.460773 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:22.464943 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:43447
I20240503 02:00:22.464969 2744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:43447 every 8 connection(s)
I20240503 02:00:22.465783 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:22.468631 2749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:22.469460 2750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:22.470211 2745 heartbeater.cc:347] Connected to a master server at 127.0.99.62:37049
I20240503 02:00:22.470296 2745 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:22.470520 2745 heartbeater.cc:511] Master 127.0.99.62:37049 requested a full tablet report, sending...
W20240503 02:00:22.470628 2752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:22.471184 2616 ts_manager.cc:194] Registered new tserver with Master: c74b634989f9485299019708592f9bd2 (127.0.99.1:43447)
I20240503 02:00:22.471726 396 server_base.cc:1034] running on GCE node
I20240503 02:00:22.471928 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:22.471971 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:22.471990 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701622471990 us; error 0 us; skew 500 ppm
I20240503 02:00:22.472174 2616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37628
I20240503 02:00:22.472967 396 webserver.cc:460] Webserver started at http://127.0.99.2:35827/ using document root <none> and password file <none>
I20240503 02:00:22.473116 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:22.473162 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:22.473215 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:22.473560 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-1-root/instance:
uuid: "fd2f340a2b8a4e99816c74162de7e251"
format_stamp: "Formatted at 2024-05-03 02:00:22 on dist-test-slave-82l4"
I20240503 02:00:22.474871 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:22.475746 2757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:22.475960 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:22.476016 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-1-root
uuid: "fd2f340a2b8a4e99816c74162de7e251"
format_stamp: "Formatted at 2024-05-03 02:00:22 on dist-test-slave-82l4"
I20240503 02:00:22.476063 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:22.486253 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:22.486625 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:22.486879 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:22.487396 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:22.487435 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:22.487473 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:22.487505 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:22.491803 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:40913
I20240503 02:00:22.491845 2820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:40913 every 8 connection(s)
I20240503 02:00:22.492558 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:22.495596 2824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:22.496368 2821 heartbeater.cc:347] Connected to a master server at 127.0.99.62:37049
I20240503 02:00:22.496464 2821 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:22.496707 2821 heartbeater.cc:511] Master 127.0.99.62:37049 requested a full tablet report, sending...
W20240503 02:00:22.496776 2825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:22.497375 2616 ts_manager.cc:194] Registered new tserver with Master: fd2f340a2b8a4e99816c74162de7e251 (127.0.99.2:40913)
W20240503 02:00:22.497634 2827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:22.498001 2616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37644
I20240503 02:00:22.498448 396 server_base.cc:1034] running on GCE node
I20240503 02:00:22.498639 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:22.498678 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:22.498694 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701622498694 us; error 0 us; skew 500 ppm
I20240503 02:00:22.499696 396 webserver.cc:460] Webserver started at http://127.0.99.3:43029/ using document root <none> and password file <none>
I20240503 02:00:22.499853 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:22.499924 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:22.500008 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:22.500380 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-2-root/instance:
uuid: "a8457388262248999c581fb7aae06d95"
format_stamp: "Formatted at 2024-05-03 02:00:22 on dist-test-slave-82l4"
I20240503 02:00:22.501730 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 02:00:22.502610 2832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:22.502843 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:22.502907 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-2-root
uuid: "a8457388262248999c581fb7aae06d95"
format_stamp: "Formatted at 2024-05-03 02:00:22 on dist-test-slave-82l4"
I20240503 02:00:22.502995 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714701568895932-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:22.511067 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:22.511422 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:22.511715 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:22.512118 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:22.512157 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:22.512212 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:22.512238 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:22.516628 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:35851
I20240503 02:00:22.516664 2895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:35851 every 8 connection(s)
I20240503 02:00:22.521147 2896 heartbeater.cc:347] Connected to a master server at 127.0.99.62:37049
I20240503 02:00:22.521245 2896 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:22.521448 2896 heartbeater.cc:511] Master 127.0.99.62:37049 requested a full tablet report, sending...
I20240503 02:00:22.522015 2616 ts_manager.cc:194] Registered new tserver with Master: a8457388262248999c581fb7aae06d95 (127.0.99.3:35851)
I20240503 02:00:22.522531 2616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37646
I20240503 02:00:22.522671 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005657684s
I20240503 02:00:23.473215 2745 heartbeater.cc:503] Master 127.0.99.62:37049 was elected leader, sending a full tablet report...
I20240503 02:00:23.499078 2821 heartbeater.cc:503] Master 127.0.99.62:37049 was elected leader, sending a full tablet report...
I20240503 02:00:23.523483 2896 heartbeater.cc:503] Master 127.0.99.62:37049 was elected leader, sending a full tablet report...
I20240503 02:00:24.411144 2670 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:26.411532 2670 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:28.411921 2670 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:30.412300 2670 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:32.412652 2670 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:34.413040 2670 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:34.600808 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 02:00:34.604797 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:34.618292 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 02:00:34.620434 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 02:00:34.623781 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:34.635828 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 02:00:34.637643 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 02:00:34.641239 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:34.653306 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 02:00:34.655230 396 master.cc:537] Master@127.0.99.62:37049 shutting down...
I20240503 02:00:34.658180 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:34.658339 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:34.658389 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 90b098306e6b4750b0dc4199c0286152: stopping tablet replica
I20240503 02:00:34.670476 396 master.cc:552] Master@127.0.99.62:37049 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12306 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240503 02:00:34.675976 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:45463
I20240503 02:00:34.676303 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:00:34.679459 396 server_base.cc:1034] running on GCE node
W20240503 02:00:34.680352 2903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:34.681277 2900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:34.682144 2901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:34.682436 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:34.682488 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:34.682505 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701634682505 us; error 0 us; skew 500 ppm
I20240503 02:00:34.683581 396 webserver.cc:460] Webserver started at http://127.0.99.62:38823/ using document root <none> and password file <none>
I20240503 02:00:34.683734 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:34.683781 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:34.683841 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:34.684247 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/master-0-root/instance:
uuid: "a4ecf3bcbac5400f935f4ac33b05837d"
format_stamp: "Formatted at 2024-05-03 02:00:34 on dist-test-slave-82l4"
I20240503 02:00:34.685760 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:00:34.686784 2908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:34.687139 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:34.687211 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/master-0-root
uuid: "a4ecf3bcbac5400f935f4ac33b05837d"
format_stamp: "Formatted at 2024-05-03 02:00:34 on dist-test-slave-82l4"
I20240503 02:00:34.687291 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:34.701435 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:34.701834 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:34.706046 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:45463
I20240503 02:00:34.707638 2960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:45463 every 8 connection(s)
I20240503 02:00:34.708086 2961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:34.709921 2961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d: Bootstrap starting.
I20240503 02:00:34.710700 2961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:34.711714 2961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d: No bootstrap required, opened a new log
I20240503 02:00:34.712126 2961 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ecf3bcbac5400f935f4ac33b05837d" member_type: VOTER }
I20240503 02:00:34.712219 2961 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:34.712241 2961 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ecf3bcbac5400f935f4ac33b05837d, State: Initialized, Role: FOLLOWER
I20240503 02:00:34.712358 2961 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ecf3bcbac5400f935f4ac33b05837d" member_type: VOTER }
I20240503 02:00:34.712436 2961 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:34.712466 2961 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:34.712500 2961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:34.713263 2961 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ecf3bcbac5400f935f4ac33b05837d" member_type: VOTER }
I20240503 02:00:34.713377 2961 leader_election.cc:302] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4ecf3bcbac5400f935f4ac33b05837d; no voters:
I20240503 02:00:34.713534 2961 leader_election.cc:288] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:34.713662 2964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:34.713868 2964 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [term 1 LEADER]: Becoming Leader. State: Replica: a4ecf3bcbac5400f935f4ac33b05837d, State: Running, Role: LEADER
I20240503 02:00:34.713974 2961 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:00:34.714015 2964 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ecf3bcbac5400f935f4ac33b05837d" member_type: VOTER }
I20240503 02:00:34.714442 2965 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4ecf3bcbac5400f935f4ac33b05837d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ecf3bcbac5400f935f4ac33b05837d" member_type: VOTER } }
I20240503 02:00:34.714468 2966 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4ecf3bcbac5400f935f4ac33b05837d. Latest consensus state: current_term: 1 leader_uuid: "a4ecf3bcbac5400f935f4ac33b05837d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ecf3bcbac5400f935f4ac33b05837d" member_type: VOTER } }
I20240503 02:00:34.714558 2965 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:34.714565 2966 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:34.715014 2969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:00:34.715868 2969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:00:34.716116 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:00:34.717725 2969 catalog_manager.cc:1347] Generated new cluster ID: d371d64422ef4240811ca74d814c35f7
I20240503 02:00:34.717770 2969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:00:34.724036 2969 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:00:34.724643 2969 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:00:34.734927 2969 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d: Generated new TSK 0
I20240503 02:00:34.735157 2969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:00:34.748564 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:34.751621 2983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:34.752607 2982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:34.753410 2985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:34.754205 396 server_base.cc:1034] running on GCE node
I20240503 02:00:34.754381 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:34.754415 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:34.754432 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701634754432 us; error 0 us; skew 500 ppm
I20240503 02:00:34.755484 396 webserver.cc:460] Webserver started at http://127.0.99.1:37497/ using document root <none> and password file <none>
I20240503 02:00:34.755647 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:34.755700 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:34.755833 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:34.756249 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-0-root/instance:
uuid: "2907e13d9e0e4a5f917cf74dad4dd114"
format_stamp: "Formatted at 2024-05-03 02:00:34 on dist-test-slave-82l4"
I20240503 02:00:34.757745 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240503 02:00:34.758757 2990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:34.759040 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:34.759110 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-0-root
uuid: "2907e13d9e0e4a5f917cf74dad4dd114"
format_stamp: "Formatted at 2024-05-03 02:00:34 on dist-test-slave-82l4"
I20240503 02:00:34.759182 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:34.766196 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:34.766615 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:34.766932 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:34.767484 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:34.767525 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:34.767577 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:34.767604 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:34.771689 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:41789
I20240503 02:00:34.771713 3053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:41789 every 8 connection(s)
I20240503 02:00:34.772516 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:34.775661 3059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:34.776460 3058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:34.776899 3054 heartbeater.cc:347] Connected to a master server at 127.0.99.62:45463
I20240503 02:00:34.777000 3054 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:34.777240 3054 heartbeater.cc:511] Master 127.0.99.62:45463 requested a full tablet report, sending...
W20240503 02:00:34.777362 3061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:34.777864 2925 ts_manager.cc:194] Registered new tserver with Master: 2907e13d9e0e4a5f917cf74dad4dd114 (127.0.99.1:41789)
I20240503 02:00:34.778225 396 server_base.cc:1034] running on GCE node
I20240503 02:00:34.778398 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:34.778437 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:34.778453 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701634778453 us; error 0 us; skew 500 ppm
I20240503 02:00:34.778738 2925 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58074
I20240503 02:00:34.779534 396 webserver.cc:460] Webserver started at http://127.0.99.2:43861/ using document root <none> and password file <none>
I20240503 02:00:34.779670 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:34.779714 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:34.779771 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:34.780131 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-1-root/instance:
uuid: "9950702fbaca48efaa2292f75ba0ff7b"
format_stamp: "Formatted at 2024-05-03 02:00:34 on dist-test-slave-82l4"
I20240503 02:00:34.781491 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:34.782454 3066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:34.782680 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:34.782742 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-1-root
uuid: "9950702fbaca48efaa2292f75ba0ff7b"
format_stamp: "Formatted at 2024-05-03 02:00:34 on dist-test-slave-82l4"
I20240503 02:00:34.782811 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:34.818544 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:34.819087 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:34.819429 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:34.819932 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:34.819976 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:34.820026 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:34.820057 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:34.824465 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:43973
I20240503 02:00:34.824491 3129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:43973 every 8 connection(s)
I20240503 02:00:34.825295 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:34.828580 3134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:34.829289 3130 heartbeater.cc:347] Connected to a master server at 127.0.99.62:45463
I20240503 02:00:34.829386 3130 heartbeater.cc:464] Registering TS with master...
W20240503 02:00:34.829393 3136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:34.829627 3130 heartbeater.cc:511] Master 127.0.99.62:45463 requested a full tablet report, sending...
W20240503 02:00:34.830214 3133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:34.830298 2925 ts_manager.cc:194] Registered new tserver with Master: 9950702fbaca48efaa2292f75ba0ff7b (127.0.99.2:43973)
I20240503 02:00:34.830852 2925 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58084
I20240503 02:00:34.831064 396 server_base.cc:1034] running on GCE node
I20240503 02:00:34.831226 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:34.831279 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:34.831303 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701634831303 us; error 0 us; skew 500 ppm
I20240503 02:00:34.832301 396 webserver.cc:460] Webserver started at http://127.0.99.3:42329/ using document root <none> and password file <none>
I20240503 02:00:34.832444 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:34.832494 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:34.832564 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:34.832938 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-2-root/instance:
uuid: "ec470e4a3b234b6a851e2d8a102e7ebb"
format_stamp: "Formatted at 2024-05-03 02:00:34 on dist-test-slave-82l4"
I20240503 02:00:34.834365 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:34.835307 3141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:34.835546 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:34.835610 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-2-root
uuid: "ec470e4a3b234b6a851e2d8a102e7ebb"
format_stamp: "Formatted at 2024-05-03 02:00:34 on dist-test-slave-82l4"
I20240503 02:00:34.835682 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:34.844185 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:34.844640 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:34.844971 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:34.845475 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:34.845516 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:34.845566 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:34.845595 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:34.850147 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:37385
I20240503 02:00:34.850178 3204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:37385 every 8 connection(s)
I20240503 02:00:34.851027 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:34.853943 3209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:34.854853 3208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:34.855163 3205 heartbeater.cc:347] Connected to a master server at 127.0.99.62:45463
I20240503 02:00:34.855242 3205 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:34.855435 3205 heartbeater.cc:511] Master 127.0.99.62:45463 requested a full tablet report, sending...
W20240503 02:00:34.855666 3211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:34.855978 2925 ts_manager.cc:194] Registered new tserver with Master: ec470e4a3b234b6a851e2d8a102e7ebb (127.0.99.3:37385)
I20240503 02:00:34.856459 396 server_base.cc:1034] running on GCE node
I20240503 02:00:34.856623 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 02:00:34.856653 2925 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58086
W20240503 02:00:34.856660 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:34.856748 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701634856747 us; error 0 us; skew 500 ppm
I20240503 02:00:34.857739 396 webserver.cc:460] Webserver started at http://127.0.99.4:38843/ using document root <none> and password file <none>
I20240503 02:00:34.857884 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:34.857941 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:34.858021 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:34.858426 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-3-root/instance:
uuid: "09d219ecfc1846e2aae849d91a81f283"
format_stamp: "Formatted at 2024-05-03 02:00:34 on dist-test-slave-82l4"
I20240503 02:00:34.859927 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 02:00:34.860838 3216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:34.861097 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:34.861156 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-3-root
uuid: "09d219ecfc1846e2aae849d91a81f283"
format_stamp: "Formatted at 2024-05-03 02:00:34 on dist-test-slave-82l4"
I20240503 02:00:34.861212 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714701568895932-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:34.879575 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:34.879976 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:34.880257 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:34.880685 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:34.880726 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:34.880761 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:34.880797 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:34.885346 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:45685
I20240503 02:00:34.885401 3279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:45685 every 8 connection(s)
I20240503 02:00:34.889969 3280 heartbeater.cc:347] Connected to a master server at 127.0.99.62:45463
I20240503 02:00:34.890085 3280 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:34.890319 3280 heartbeater.cc:511] Master 127.0.99.62:45463 requested a full tablet report, sending...
I20240503 02:00:34.890884 2925 ts_manager.cc:194] Registered new tserver with Master: 09d219ecfc1846e2aae849d91a81f283 (127.0.99.4:45685)
I20240503 02:00:34.891412 396 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.005678599s
I20240503 02:00:34.891420 2925 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58092
I20240503 02:00:35.779853 3054 heartbeater.cc:503] Master 127.0.99.62:45463 was elected leader, sending a full tablet report...
I20240503 02:00:35.831869 3130 heartbeater.cc:503] Master 127.0.99.62:45463 was elected leader, sending a full tablet report...
I20240503 02:00:35.857578 3205 heartbeater.cc:503] Master 127.0.99.62:45463 was elected leader, sending a full tablet report...
I20240503 02:00:35.892318 3280 heartbeater.cc:503] Master 127.0.99.62:45463 was elected leader, sending a full tablet report...
I20240503 02:00:35.922498 396 test_util.cc:268] Using random seed: 367503935
I20240503 02:00:35.929746 2925 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58104:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 02:00:35.939123 3094 tablet_service.cc:1467] Processing CreateTablet for tablet 54144da42eb44f4386bc19cb7eddb87e (DEFAULT_TABLE table=test-workload [id=6162653514664970aec8f60dea2afe8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:35.939460 3094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54144da42eb44f4386bc19cb7eddb87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:35.939901 3169 tablet_service.cc:1467] Processing CreateTablet for tablet 54144da42eb44f4386bc19cb7eddb87e (DEFAULT_TABLE table=test-workload [id=6162653514664970aec8f60dea2afe8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:35.940061 3244 tablet_service.cc:1467] Processing CreateTablet for tablet 54144da42eb44f4386bc19cb7eddb87e (DEFAULT_TABLE table=test-workload [id=6162653514664970aec8f60dea2afe8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:35.940121 3169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54144da42eb44f4386bc19cb7eddb87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:35.940304 3244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54144da42eb44f4386bc19cb7eddb87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:35.941865 3301 tablet_bootstrap.cc:492] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b: Bootstrap starting.
I20240503 02:00:35.942166 3302 tablet_bootstrap.cc:492] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb: Bootstrap starting.
I20240503 02:00:35.942664 3303 tablet_bootstrap.cc:492] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283: Bootstrap starting.
I20240503 02:00:35.942884 3301 tablet_bootstrap.cc:654] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:35.943387 3302 tablet_bootstrap.cc:654] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:35.943547 3303 tablet_bootstrap.cc:654] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:35.944052 3301 tablet_bootstrap.cc:492] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b: No bootstrap required, opened a new log
I20240503 02:00:35.944141 3301 ts_tablet_manager.cc:1367] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:35.944337 3302 tablet_bootstrap.cc:492] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb: No bootstrap required, opened a new log
I20240503 02:00:35.944419 3302 ts_tablet_manager.cc:1367] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:35.944520 3303 tablet_bootstrap.cc:492] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283: No bootstrap required, opened a new log
I20240503 02:00:35.944610 3303 ts_tablet_manager.cc:1367] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:35.944576 3301 raft_consensus.cc:348] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950702fbaca48efaa2292f75ba0ff7b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43973 } } peers { permanent_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37385 } } peers { permanent_uuid: "09d219ecfc1846e2aae849d91a81f283" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45685 } }
I20240503 02:00:35.944680 3301 raft_consensus.cc:374] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:35.944720 3301 raft_consensus.cc:729] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9950702fbaca48efaa2292f75ba0ff7b, State: Initialized, Role: FOLLOWER
I20240503 02:00:35.944794 3302 raft_consensus.cc:348] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950702fbaca48efaa2292f75ba0ff7b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43973 } } peers { permanent_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37385 } } peers { permanent_uuid: "09d219ecfc1846e2aae849d91a81f283" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45685 } }
I20240503 02:00:35.944837 3301 consensus_queue.cc:260] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950702fbaca48efaa2292f75ba0ff7b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43973 } } peers { permanent_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37385 } } peers { permanent_uuid: "09d219ecfc1846e2aae849d91a81f283" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45685 } }
I20240503 02:00:35.944892 3302 raft_consensus.cc:374] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:35.944940 3302 raft_consensus.cc:729] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec470e4a3b234b6a851e2d8a102e7ebb, State: Initialized, Role: FOLLOWER
I20240503 02:00:35.944983 3303 raft_consensus.cc:348] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950702fbaca48efaa2292f75ba0ff7b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43973 } } peers { permanent_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37385 } } peers { permanent_uuid: "09d219ecfc1846e2aae849d91a81f283" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45685 } }
I20240503 02:00:35.945070 3303 raft_consensus.cc:374] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:35.945044 3302 consensus_queue.cc:260] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950702fbaca48efaa2292f75ba0ff7b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43973 } } peers { permanent_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37385 } } peers { permanent_uuid: "09d219ecfc1846e2aae849d91a81f283" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45685 } }
I20240503 02:00:35.945250 3301 ts_tablet_manager.cc:1398] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:35.945406 3302 ts_tablet_manager.cc:1398] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:35.945098 3303 raft_consensus.cc:729] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d219ecfc1846e2aae849d91a81f283, State: Initialized, Role: FOLLOWER
I20240503 02:00:35.945963 3303 consensus_queue.cc:260] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950702fbaca48efaa2292f75ba0ff7b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43973 } } peers { permanent_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37385 } } peers { permanent_uuid: "09d219ecfc1846e2aae849d91a81f283" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45685 } }
I20240503 02:00:35.946336 3303 ts_tablet_manager.cc:1398] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:35.983449 3308 raft_consensus.cc:482] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:35.983464 3307 raft_consensus.cc:482] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:35.983613 3308 raft_consensus.cc:504] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950702fbaca48efaa2292f75ba0ff7b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43973 } } peers { permanent_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37385 } } peers { permanent_uuid: "09d219ecfc1846e2aae849d91a81f283" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45685 } }
I20240503 02:00:35.983682 3307 raft_consensus.cc:504] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950702fbaca48efaa2292f75ba0ff7b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43973 } } peers { permanent_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37385 } } peers { permanent_uuid: "09d219ecfc1846e2aae849d91a81f283" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45685 } }
I20240503 02:00:35.984650 3307 leader_election.cc:288] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec470e4a3b234b6a851e2d8a102e7ebb (127.0.99.3:37385), 09d219ecfc1846e2aae849d91a81f283 (127.0.99.4:45685)
I20240503 02:00:35.984719 3308 leader_election.cc:288] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9950702fbaca48efaa2292f75ba0ff7b (127.0.99.2:43973), 09d219ecfc1846e2aae849d91a81f283 (127.0.99.4:45685)
I20240503 02:00:35.988931 3254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54144da42eb44f4386bc19cb7eddb87e" candidate_uuid: "9950702fbaca48efaa2292f75ba0ff7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09d219ecfc1846e2aae849d91a81f283" is_pre_election: true
I20240503 02:00:35.989104 3254 raft_consensus.cc:2454] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9950702fbaca48efaa2292f75ba0ff7b in term 0.
I20240503 02:00:35.988931 3253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54144da42eb44f4386bc19cb7eddb87e" candidate_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09d219ecfc1846e2aae849d91a81f283" is_pre_election: true
I20240503 02:00:35.989310 3253 raft_consensus.cc:2454] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec470e4a3b234b6a851e2d8a102e7ebb in term 0.
I20240503 02:00:35.989419 3070 leader_election.cc:302] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d219ecfc1846e2aae849d91a81f283, 9950702fbaca48efaa2292f75ba0ff7b; no voters:
I20240503 02:00:35.989624 3307 raft_consensus.cc:2781] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:35.989689 3179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54144da42eb44f4386bc19cb7eddb87e" candidate_uuid: "9950702fbaca48efaa2292f75ba0ff7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" is_pre_election: true
I20240503 02:00:35.989629 3104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54144da42eb44f4386bc19cb7eddb87e" candidate_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9950702fbaca48efaa2292f75ba0ff7b" is_pre_election: true
I20240503 02:00:35.989776 3307 raft_consensus.cc:482] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:35.989831 3179 raft_consensus.cc:2454] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9950702fbaca48efaa2292f75ba0ff7b in term 0.
I20240503 02:00:35.989879 3307 raft_consensus.cc:3037] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.989754 3145 leader_election.cc:302] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d219ecfc1846e2aae849d91a81f283, ec470e4a3b234b6a851e2d8a102e7ebb; no voters:
I20240503 02:00:35.990084 3308 raft_consensus.cc:2781] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:35.990139 3308 raft_consensus.cc:482] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:35.990175 3308 raft_consensus.cc:3037] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.991031 3307 raft_consensus.cc:504] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950702fbaca48efaa2292f75ba0ff7b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43973 } } peers { permanent_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37385 } } peers { permanent_uuid: "09d219ecfc1846e2aae849d91a81f283" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45685 } }
I20240503 02:00:35.991031 3308 raft_consensus.cc:504] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950702fbaca48efaa2292f75ba0ff7b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43973 } } peers { permanent_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37385 } } peers { permanent_uuid: "09d219ecfc1846e2aae849d91a81f283" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45685 } }
I20240503 02:00:35.991281 3104 raft_consensus.cc:2379] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec470e4a3b234b6a851e2d8a102e7ebb in current term 1: Already voted for candidate 9950702fbaca48efaa2292f75ba0ff7b in this term.
I20240503 02:00:35.991549 3308 leader_election.cc:288] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [CANDIDATE]: Term 1 election: Requested vote from peers 9950702fbaca48efaa2292f75ba0ff7b (127.0.99.2:43973), 09d219ecfc1846e2aae849d91a81f283 (127.0.99.4:45685)
I20240503 02:00:35.991722 3307 leader_election.cc:288] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [CANDIDATE]: Term 1 election: Requested vote from peers ec470e4a3b234b6a851e2d8a102e7ebb (127.0.99.3:37385), 09d219ecfc1846e2aae849d91a81f283 (127.0.99.4:45685)
I20240503 02:00:35.991742 3253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54144da42eb44f4386bc19cb7eddb87e" candidate_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09d219ecfc1846e2aae849d91a81f283"
I20240503 02:00:35.991804 3103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54144da42eb44f4386bc19cb7eddb87e" candidate_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9950702fbaca48efaa2292f75ba0ff7b"
I20240503 02:00:35.991842 3253 raft_consensus.cc:3037] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:35.991909 3103 raft_consensus.cc:2379] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ec470e4a3b234b6a851e2d8a102e7ebb in current term 1: Already voted for candidate 9950702fbaca48efaa2292f75ba0ff7b in this term.
I20240503 02:00:35.992287 3179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54144da42eb44f4386bc19cb7eddb87e" candidate_uuid: "9950702fbaca48efaa2292f75ba0ff7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb"
I20240503 02:00:35.992388 3179 raft_consensus.cc:2379] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9950702fbaca48efaa2292f75ba0ff7b in current term 1: Already voted for candidate ec470e4a3b234b6a851e2d8a102e7ebb in this term.
I20240503 02:00:35.992609 3253 raft_consensus.cc:2454] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec470e4a3b234b6a851e2d8a102e7ebb in term 1.
I20240503 02:00:35.992785 3254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54144da42eb44f4386bc19cb7eddb87e" candidate_uuid: "9950702fbaca48efaa2292f75ba0ff7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09d219ecfc1846e2aae849d91a81f283"
I20240503 02:00:35.992816 3145 leader_election.cc:302] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09d219ecfc1846e2aae849d91a81f283, ec470e4a3b234b6a851e2d8a102e7ebb; no voters: 9950702fbaca48efaa2292f75ba0ff7b
I20240503 02:00:35.992894 3254 raft_consensus.cc:2379] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9950702fbaca48efaa2292f75ba0ff7b in current term 1: Already voted for candidate ec470e4a3b234b6a851e2d8a102e7ebb in this term.
I20240503 02:00:35.992949 3308 raft_consensus.cc:2781] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:35.993116 3070 leader_election.cc:302] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9950702fbaca48efaa2292f75ba0ff7b; no voters: 09d219ecfc1846e2aae849d91a81f283, ec470e4a3b234b6a851e2d8a102e7ebb
I20240503 02:00:35.993336 3307 raft_consensus.cc:2726] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:00:35.993131 3308 raft_consensus.cc:686] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 1 LEADER]: Becoming Leader. State: Replica: ec470e4a3b234b6a851e2d8a102e7ebb, State: Running, Role: LEADER
I20240503 02:00:35.993564 3308 consensus_queue.cc:237] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950702fbaca48efaa2292f75ba0ff7b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43973 } } peers { permanent_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37385 } } peers { permanent_uuid: "09d219ecfc1846e2aae849d91a81f283" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45685 } }
I20240503 02:00:35.995085 2924 catalog_manager.cc:5484] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb reported cstate change: term changed from 0 to 1, leader changed from <none> to ec470e4a3b234b6a851e2d8a102e7ebb (127.0.99.3). New cstate: current_term: 1 leader_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950702fbaca48efaa2292f75ba0ff7b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec470e4a3b234b6a851e2d8a102e7ebb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 37385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09d219ecfc1846e2aae849d91a81f283" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45685 } health_report { overall_health: UNKNOWN } } }
W20240503 02:00:36.076335 3131 tablet.cc:2369] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:36.424585 3308 consensus_queue.cc:1035] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d219ecfc1846e2aae849d91a81f283" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.428356 3319 consensus_queue.cc:1035] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9950702fbaca48efaa2292f75ba0ff7b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:36.716133 2979 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:36.716497 2979 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:36.716558 2979 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:38.716892 2979 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:38.717271 2979 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:38.717329 2979 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:40.717607 2979 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:40.717922 2979 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:40.717983 2979 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:42.718252 2979 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:42.718582 2979 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:42.718648 2979 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:44.719014 2979 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:44.719354 2979 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:44.719425 2979 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:46.719796 2979 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:46.720113 2979 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:46.720176 2979 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:47.049680 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 02:00:47.053151 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:47.065213 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 02:00:47.067037 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 02:00:47.070605 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:47.070783 396 tablet_replica.cc:330] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b: stopping tablet replica
I20240503 02:00:47.071061 396 raft_consensus.cc:2229] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:47.071206 396 raft_consensus.cc:2258] T 54144da42eb44f4386bc19cb7eddb87e P 9950702fbaca48efaa2292f75ba0ff7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:47.083791 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 02:00:47.085800 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 02:00:47.089008 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:47.089254 396 tablet_replica.cc:330] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb: stopping tablet replica
I20240503 02:00:47.089368 396 raft_consensus.cc:2229] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:47.089591 396 raft_consensus.cc:2258] T 54144da42eb44f4386bc19cb7eddb87e P ec470e4a3b234b6a851e2d8a102e7ebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:47.102324 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 02:00:47.104362 396 tablet_server.cc:178] TabletServer@127.0.99.4:0 shutting down...
I20240503 02:00:47.107810 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:47.108006 396 tablet_replica.cc:330] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283: stopping tablet replica
I20240503 02:00:47.108119 396 raft_consensus.cc:2229] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:47.108240 396 raft_consensus.cc:2258] T 54144da42eb44f4386bc19cb7eddb87e P 09d219ecfc1846e2aae849d91a81f283 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:47.120640 396 tablet_server.cc:194] TabletServer@127.0.99.4:0 shutdown complete.
I20240503 02:00:47.122762 396 master.cc:537] Master@127.0.99.62:45463 shutting down...
I20240503 02:00:47.126022 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:47.126180 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:47.126261 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P a4ecf3bcbac5400f935f4ac33b05837d: stopping tablet replica
I20240503 02:00:47.138489 396 master.cc:552] Master@127.0.99.62:45463 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12469 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240503 02:00:47.145272 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:38693
I20240503 02:00:47.145598 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:47.148695 3331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:47.149541 3328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:47.150187 3329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:47.151028 396 server_base.cc:1034] running on GCE node
I20240503 02:00:47.151178 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:47.151217 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:47.151262 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701647151261 us; error 0 us; skew 500 ppm
I20240503 02:00:47.152208 396 webserver.cc:460] Webserver started at http://127.0.99.62:40301/ using document root <none> and password file <none>
I20240503 02:00:47.152362 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:47.152416 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:47.152505 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:47.152895 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/master-0-root/instance:
uuid: "fd103e337d9e4f27a603b874aa7dbd63"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-82l4"
I20240503 02:00:47.154338 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:47.155295 3336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.155575 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:47.155637 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/master-0-root
uuid: "fd103e337d9e4f27a603b874aa7dbd63"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-82l4"
I20240503 02:00:47.155696 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:47.170522 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:47.170874 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:47.174744 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:38693
I20240503 02:00:47.176995 3388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:38693 every 8 connection(s)
I20240503 02:00:47.177467 3389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:47.179311 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63: Bootstrap starting.
I20240503 02:00:47.180058 3389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:47.180999 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63: No bootstrap required, opened a new log
I20240503 02:00:47.181361 3389 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd103e337d9e4f27a603b874aa7dbd63" member_type: VOTER }
I20240503 02:00:47.181448 3389 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:47.181470 3389 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd103e337d9e4f27a603b874aa7dbd63, State: Initialized, Role: FOLLOWER
I20240503 02:00:47.181610 3389 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd103e337d9e4f27a603b874aa7dbd63" member_type: VOTER }
I20240503 02:00:47.181697 3389 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:47.181736 3389 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:47.181794 3389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:47.182444 3389 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd103e337d9e4f27a603b874aa7dbd63" member_type: VOTER }
I20240503 02:00:47.182554 3389 leader_election.cc:302] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd103e337d9e4f27a603b874aa7dbd63; no voters:
I20240503 02:00:47.182742 3389 leader_election.cc:288] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:47.182885 3392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:47.183094 3392 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [term 1 LEADER]: Becoming Leader. State: Replica: fd103e337d9e4f27a603b874aa7dbd63, State: Running, Role: LEADER
I20240503 02:00:47.183262 3389 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:00:47.183231 3392 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd103e337d9e4f27a603b874aa7dbd63" member_type: VOTER }
I20240503 02:00:47.183661 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd103e337d9e4f27a603b874aa7dbd63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd103e337d9e4f27a603b874aa7dbd63" member_type: VOTER } }
I20240503 02:00:47.183769 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:47.183676 3394 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd103e337d9e4f27a603b874aa7dbd63. Latest consensus state: current_term: 1 leader_uuid: "fd103e337d9e4f27a603b874aa7dbd63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd103e337d9e4f27a603b874aa7dbd63" member_type: VOTER } }
I20240503 02:00:47.183902 3394 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:47.184393 3397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:00:47.185138 3397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:00:47.185389 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:00:47.186929 3397 catalog_manager.cc:1347] Generated new cluster ID: 237abd4cccc44c94ab6950151dee5ced
I20240503 02:00:47.187009 3397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:00:47.198398 3397 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:00:47.198928 3397 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:00:47.203393 3397 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63: Generated new TSK 0
I20240503 02:00:47.203549 3397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:00:47.217775 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:47.220969 3411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:47.221836 3413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:47.222708 396 server_base.cc:1034] running on GCE node
W20240503 02:00:47.223397 3410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:47.223665 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:47.223718 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:47.223734 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701647223734 us; error 0 us; skew 500 ppm
I20240503 02:00:47.224798 396 webserver.cc:460] Webserver started at http://127.0.99.1:36285/ using document root <none> and password file <none>
I20240503 02:00:47.224951 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:47.225006 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:47.225097 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:47.225531 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-0-root/instance:
uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-82l4"
I20240503 02:00:47.227030 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:47.227905 3418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.228174 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:47.228238 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-0-root
uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-82l4"
I20240503 02:00:47.228293 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:47.241007 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:47.241353 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:47.241623 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:47.242064 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:47.242101 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.242136 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:47.242168 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.246192 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:35539
I20240503 02:00:47.246234 3481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:35539 every 8 connection(s)
I20240503 02:00:47.247088 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:47.249814 3486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:47.250630 3487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:47.251483 3489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:47.251513 3482 heartbeater.cc:347] Connected to a master server at 127.0.99.62:38693
I20240503 02:00:47.251618 3482 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:47.251852 3482 heartbeater.cc:511] Master 127.0.99.62:38693 requested a full tablet report, sending...
I20240503 02:00:47.252353 396 server_base.cc:1034] running on GCE node
I20240503 02:00:47.252465 3353 ts_manager.cc:194] Registered new tserver with Master: 6733de6a1c8e42a5814fcfdd64f8d6d4 (127.0.99.1:35539)
I20240503 02:00:47.252511 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:47.252548 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:47.252578 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701647252577 us; error 0 us; skew 500 ppm
I20240503 02:00:47.253407 3353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41232
I20240503 02:00:47.253552 396 webserver.cc:460] Webserver started at http://127.0.99.2:34105/ using document root <none> and password file <none>
I20240503 02:00:47.253701 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:47.253764 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:47.253841 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:47.254336 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-1-root/instance:
uuid: "a328068b7924469eb6c52d9d7a7edb78"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-82l4"
I20240503 02:00:47.255755 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 02:00:47.256595 3494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.256903 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:47.256969 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-1-root
uuid: "a328068b7924469eb6c52d9d7a7edb78"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-82l4"
I20240503 02:00:47.257040 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:47.286370 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:47.286805 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:47.287139 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:47.287606 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:47.287644 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.287680 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:47.287714 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.291827 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:45123
I20240503 02:00:47.291896 3557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:45123 every 8 connection(s)
I20240503 02:00:47.292630 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:47.295614 3561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:47.296373 3562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:47.296670 3558 heartbeater.cc:347] Connected to a master server at 127.0.99.62:38693
I20240503 02:00:47.296764 3558 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:47.296970 3558 heartbeater.cc:511] Master 127.0.99.62:38693 requested a full tablet report, sending...
I20240503 02:00:47.297139 396 server_base.cc:1034] running on GCE node
I20240503 02:00:47.297606 3353 ts_manager.cc:194] Registered new tserver with Master: a328068b7924469eb6c52d9d7a7edb78 (127.0.99.2:45123)
W20240503 02:00:47.297936 3564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:47.298127 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:47.298177 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:47.298209 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701647298209 us; error 0 us; skew 500 ppm
I20240503 02:00:47.298229 3353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41244
I20240503 02:00:47.299261 396 webserver.cc:460] Webserver started at http://127.0.99.3:42449/ using document root <none> and password file <none>
I20240503 02:00:47.299405 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:47.299453 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:47.299526 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:47.299887 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-2-root/instance:
uuid: "4c9d0ecda49d465f96151eda35a82efc"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-82l4"
I20240503 02:00:47.301357 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:47.302305 3569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.302547 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:47.302614 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-2-root
uuid: "4c9d0ecda49d465f96151eda35a82efc"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-82l4"
I20240503 02:00:47.302685 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714701568895932-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:47.317196 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:47.317610 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:47.317909 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:47.318362 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:47.318404 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.318450 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:47.318483 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.322683 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:35347
I20240503 02:00:47.322742 3632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:35347 every 8 connection(s)
I20240503 02:00:47.327283 3633 heartbeater.cc:347] Connected to a master server at 127.0.99.62:38693
I20240503 02:00:47.327392 3633 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:47.327616 3633 heartbeater.cc:511] Master 127.0.99.62:38693 requested a full tablet report, sending...
I20240503 02:00:47.328230 3353 ts_manager.cc:194] Registered new tserver with Master: 4c9d0ecda49d465f96151eda35a82efc (127.0.99.3:35347)
I20240503 02:00:47.328725 3353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41254
I20240503 02:00:47.328831 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005691949s
I20240503 02:00:48.254426 3482 heartbeater.cc:503] Master 127.0.99.62:38693 was elected leader, sending a full tablet report...
I20240503 02:00:48.299154 3558 heartbeater.cc:503] Master 127.0.99.62:38693 was elected leader, sending a full tablet report...
I20240503 02:00:48.329676 3633 heartbeater.cc:503] Master 127.0.99.62:38693 was elected leader, sending a full tablet report...
I20240503 02:00:48.365168 396 test_util.cc:268] Using random seed: 379946606
I20240503 02:00:48.372467 3353 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41278:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:00:48.372865 3353 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:00:48.382139 3522 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf3ad52005b48c7a33cc99a00ef1056 (DEFAULT_TABLE table=test-workload [id=353fbbfa87fa4565820799180a1abc3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:48.382440 3522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf3ad52005b48c7a33cc99a00ef1056. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:48.382658 3597 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf3ad52005b48c7a33cc99a00ef1056 (DEFAULT_TABLE table=test-workload [id=353fbbfa87fa4565820799180a1abc3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:48.382899 3597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf3ad52005b48c7a33cc99a00ef1056. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:48.384923 3655 tablet_bootstrap.cc:492] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc: Bootstrap starting.
I20240503 02:00:48.385401 3654 tablet_bootstrap.cc:492] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78: Bootstrap starting.
I20240503 02:00:48.385716 3446 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf3ad52005b48c7a33cc99a00ef1056 (DEFAULT_TABLE table=test-workload [id=353fbbfa87fa4565820799180a1abc3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:48.385946 3446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf3ad52005b48c7a33cc99a00ef1056. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:48.386137 3655 tablet_bootstrap.cc:654] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:48.386243 3654 tablet_bootstrap.cc:654] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:48.387319 3654 tablet_bootstrap.cc:492] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78: No bootstrap required, opened a new log
I20240503 02:00:48.387423 3654 ts_tablet_manager.cc:1367] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:48.387434 3655 tablet_bootstrap.cc:492] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc: No bootstrap required, opened a new log
I20240503 02:00:48.387545 3655 ts_tablet_manager.cc:1367] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:48.387917 3655 raft_consensus.cc:348] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a328068b7924469eb6c52d9d7a7edb78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45123 } } peers { permanent_uuid: "4c9d0ecda49d465f96151eda35a82efc" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35347 } } peers { permanent_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35539 } }
I20240503 02:00:48.387917 3654 raft_consensus.cc:348] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a328068b7924469eb6c52d9d7a7edb78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45123 } } peers { permanent_uuid: "4c9d0ecda49d465f96151eda35a82efc" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35347 } } peers { permanent_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35539 } }
I20240503 02:00:48.388118 3654 raft_consensus.cc:374] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:48.388167 3654 raft_consensus.cc:729] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a328068b7924469eb6c52d9d7a7edb78, State: Initialized, Role: FOLLOWER
I20240503 02:00:48.388067 3655 raft_consensus.cc:374] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:48.388239 3655 raft_consensus.cc:729] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9d0ecda49d465f96151eda35a82efc, State: Initialized, Role: FOLLOWER
I20240503 02:00:48.388322 3654 consensus_queue.cc:260] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a328068b7924469eb6c52d9d7a7edb78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45123 } } peers { permanent_uuid: "4c9d0ecda49d465f96151eda35a82efc" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35347 } } peers { permanent_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35539 } }
I20240503 02:00:48.388332 3655 consensus_queue.cc:260] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a328068b7924469eb6c52d9d7a7edb78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45123 } } peers { permanent_uuid: "4c9d0ecda49d465f96151eda35a82efc" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35347 } } peers { permanent_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35539 } }
I20240503 02:00:48.388767 3655 ts_tablet_manager.cc:1398] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:48.388767 3654 ts_tablet_manager.cc:1398] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:48.389034 3658 tablet_bootstrap.cc:492] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4: Bootstrap starting.
I20240503 02:00:48.389935 3658 tablet_bootstrap.cc:654] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:48.390929 3658 tablet_bootstrap.cc:492] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4: No bootstrap required, opened a new log
I20240503 02:00:48.391031 3658 ts_tablet_manager.cc:1367] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:48.391469 3658 raft_consensus.cc:348] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a328068b7924469eb6c52d9d7a7edb78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45123 } } peers { permanent_uuid: "4c9d0ecda49d465f96151eda35a82efc" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35347 } } peers { permanent_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35539 } }
I20240503 02:00:48.391608 3658 raft_consensus.cc:374] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:48.391656 3658 raft_consensus.cc:729] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6733de6a1c8e42a5814fcfdd64f8d6d4, State: Initialized, Role: FOLLOWER
I20240503 02:00:48.391825 3658 consensus_queue.cc:260] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a328068b7924469eb6c52d9d7a7edb78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45123 } } peers { permanent_uuid: "4c9d0ecda49d465f96151eda35a82efc" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35347 } } peers { permanent_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35539 } }
I20240503 02:00:48.392129 3658 ts_tablet_manager.cc:1398] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:48.425786 3662 raft_consensus.cc:482] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:48.425925 3662 raft_consensus.cc:504] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a328068b7924469eb6c52d9d7a7edb78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45123 } } peers { permanent_uuid: "4c9d0ecda49d465f96151eda35a82efc" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35347 } } peers { permanent_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35539 } }
I20240503 02:00:48.426525 3662 leader_election.cc:288] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a328068b7924469eb6c52d9d7a7edb78 (127.0.99.2:45123), 4c9d0ecda49d465f96151eda35a82efc (127.0.99.3:35347)
I20240503 02:00:48.430483 3532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf3ad52005b48c7a33cc99a00ef1056" candidate_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a328068b7924469eb6c52d9d7a7edb78" is_pre_election: true
I20240503 02:00:48.430472 3607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf3ad52005b48c7a33cc99a00ef1056" candidate_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9d0ecda49d465f96151eda35a82efc" is_pre_election: true
I20240503 02:00:48.430662 3532 raft_consensus.cc:2454] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6733de6a1c8e42a5814fcfdd64f8d6d4 in term 0.
I20240503 02:00:48.430663 3607 raft_consensus.cc:2454] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6733de6a1c8e42a5814fcfdd64f8d6d4 in term 0.
I20240503 02:00:48.431067 3419 leader_election.cc:302] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6733de6a1c8e42a5814fcfdd64f8d6d4, a328068b7924469eb6c52d9d7a7edb78; no voters:
I20240503 02:00:48.431246 3662 raft_consensus.cc:2781] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:48.431306 3662 raft_consensus.cc:482] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:48.431349 3662 raft_consensus.cc:3037] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:48.432188 3662 raft_consensus.cc:504] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a328068b7924469eb6c52d9d7a7edb78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45123 } } peers { permanent_uuid: "4c9d0ecda49d465f96151eda35a82efc" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35347 } } peers { permanent_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35539 } }
I20240503 02:00:48.432503 3662 leader_election.cc:288] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [CANDIDATE]: Term 1 election: Requested vote from peers a328068b7924469eb6c52d9d7a7edb78 (127.0.99.2:45123), 4c9d0ecda49d465f96151eda35a82efc (127.0.99.3:35347)
I20240503 02:00:48.432816 3607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf3ad52005b48c7a33cc99a00ef1056" candidate_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9d0ecda49d465f96151eda35a82efc"
I20240503 02:00:48.432817 3532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf3ad52005b48c7a33cc99a00ef1056" candidate_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a328068b7924469eb6c52d9d7a7edb78"
I20240503 02:00:48.432925 3607 raft_consensus.cc:3037] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:48.432950 3532 raft_consensus.cc:3037] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:48.433779 3532 raft_consensus.cc:2454] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6733de6a1c8e42a5814fcfdd64f8d6d4 in term 1.
I20240503 02:00:48.433784 3607 raft_consensus.cc:2454] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6733de6a1c8e42a5814fcfdd64f8d6d4 in term 1.
I20240503 02:00:48.434021 3419 leader_election.cc:302] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6733de6a1c8e42a5814fcfdd64f8d6d4, a328068b7924469eb6c52d9d7a7edb78; no voters:
I20240503 02:00:48.434221 3662 raft_consensus.cc:2781] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:48.434417 3662 raft_consensus.cc:686] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [term 1 LEADER]: Becoming Leader. State: Replica: 6733de6a1c8e42a5814fcfdd64f8d6d4, State: Running, Role: LEADER
I20240503 02:00:48.434587 3662 consensus_queue.cc:237] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a328068b7924469eb6c52d9d7a7edb78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45123 } } peers { permanent_uuid: "4c9d0ecda49d465f96151eda35a82efc" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35347 } } peers { permanent_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35539 } }
I20240503 02:00:48.435986 3353 catalog_manager.cc:5484] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6733de6a1c8e42a5814fcfdd64f8d6d4 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a328068b7924469eb6c52d9d7a7edb78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c9d0ecda49d465f96151eda35a82efc" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6733de6a1c8e42a5814fcfdd64f8d6d4" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35539 } health_report { overall_health: HEALTHY } } }
I20240503 02:00:48.817698 3662 consensus_queue.cc:1035] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c9d0ecda49d465f96151eda35a82efc" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:48.821369 3665 consensus_queue.cc:1035] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a328068b7924469eb6c52d9d7a7edb78" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 45123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:49.185200 3407 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:49.185467 3401 placement_policy_util.cc:407] tablet 5cf3ad52005b48c7a33cc99a00ef1056: detected majority of replicas (2 of 3) at location L0
I20240503 02:00:49.185498 3407 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:49.185618 3407 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:50.187242 3401 placement_policy_util.cc:407] tablet 5cf3ad52005b48c7a33cc99a00ef1056: detected majority of replicas (2 of 3) at location L0
I20240503 02:00:51.185936 3407 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:51.186268 3407 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:51.186331 3407 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:51.188730 3401 placement_policy_util.cc:407] tablet 5cf3ad52005b48c7a33cc99a00ef1056: detected majority of replicas (2 of 3) at location L0
I20240503 02:00:52.190228 3401 placement_policy_util.cc:407] tablet 5cf3ad52005b48c7a33cc99a00ef1056: detected majority of replicas (2 of 3) at location L0
I20240503 02:00:53.186642 3407 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:53.187013 3407 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:53.187085 3407 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:53.191793 3401 placement_policy_util.cc:407] tablet 5cf3ad52005b48c7a33cc99a00ef1056: detected majority of replicas (2 of 3) at location L0
I20240503 02:00:54.193449 3401 placement_policy_util.cc:407] tablet 5cf3ad52005b48c7a33cc99a00ef1056: detected majority of replicas (2 of 3) at location L0
I20240503 02:00:55.187415 3407 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:55.187776 3407 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:55.187840 3407 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:55.195076 3401 placement_policy_util.cc:407] tablet 5cf3ad52005b48c7a33cc99a00ef1056: detected majority of replicas (2 of 3) at location L0
I20240503 02:00:56.196746 3401 placement_policy_util.cc:407] tablet 5cf3ad52005b48c7a33cc99a00ef1056: detected majority of replicas (2 of 3) at location L0
I20240503 02:00:57.188139 3407 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:57.188501 3407 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:57.188570 3407 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:57.198302 3401 placement_policy_util.cc:407] tablet 5cf3ad52005b48c7a33cc99a00ef1056: detected majority of replicas (2 of 3) at location L0
I20240503 02:00:58.199972 3401 placement_policy_util.cc:407] tablet 5cf3ad52005b48c7a33cc99a00ef1056: detected majority of replicas (2 of 3) at location L0
I20240503 02:00:59.188906 3407 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:00:59.189229 3407 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:00:59.189293 3407 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:00:59.201562 3401 placement_policy_util.cc:407] tablet 5cf3ad52005b48c7a33cc99a00ef1056: detected majority of replicas (2 of 3) at location L0
I20240503 02:00:59.492602 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 02:00:59.496035 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:59.496224 396 tablet_replica.cc:330] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4: stopping tablet replica
I20240503 02:00:59.496327 396 raft_consensus.cc:2229] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:59.496523 396 raft_consensus.cc:2258] T 5cf3ad52005b48c7a33cc99a00ef1056 P 6733de6a1c8e42a5814fcfdd64f8d6d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:59.509264 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 02:00:59.511195 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 02:00:59.514472 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:59.514686 396 tablet_replica.cc:330] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78: stopping tablet replica
I20240503 02:00:59.514804 396 raft_consensus.cc:2229] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:59.514945 396 raft_consensus.cc:2258] T 5cf3ad52005b48c7a33cc99a00ef1056 P a328068b7924469eb6c52d9d7a7edb78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:59.527540 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 02:00:59.529507 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 02:00:59.533126 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 02:00:59.533326 396 tablet_replica.cc:330] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc: stopping tablet replica
I20240503 02:00:59.533442 396 raft_consensus.cc:2229] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:59.533562 396 raft_consensus.cc:2258] T 5cf3ad52005b48c7a33cc99a00ef1056 P 4c9d0ecda49d465f96151eda35a82efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:59.546144 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 02:00:59.548168 396 master.cc:537] Master@127.0.99.62:38693 shutting down...
I20240503 02:00:59.551775 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:59.551915 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:59.551975 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P fd103e337d9e4f27a603b874aa7dbd63: stopping tablet replica
I20240503 02:00:59.564128 396 master.cc:552] Master@127.0.99.62:38693 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12425 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240503 02:00:59.570472 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:43601
I20240503 02:00:59.570832 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:59.573781 3676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:59.574671 396 server_base.cc:1034] running on GCE node
W20240503 02:00:59.575430 3679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:59.576191 3677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:59.576391 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:59.576435 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:59.576450 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701659576450 us; error 0 us; skew 500 ppm
I20240503 02:00:59.577396 396 webserver.cc:460] Webserver started at http://127.0.99.62:33653/ using document root <none> and password file <none>
I20240503 02:00:59.577527 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:59.577571 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:59.577629 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:59.578033 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/master-0-root/instance:
uuid: "4e94636424334729bfba037ee764eb13"
format_stamp: "Formatted at 2024-05-03 02:00:59 on dist-test-slave-82l4"
I20240503 02:00:59.579824 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:59.580809 3684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:59.581100 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:59.581163 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/master-0-root
uuid: "4e94636424334729bfba037ee764eb13"
format_stamp: "Formatted at 2024-05-03 02:00:59 on dist-test-slave-82l4"
I20240503 02:00:59.581235 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:59.586663 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:59.587041 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:59.590955 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:43601
I20240503 02:00:59.592558 3736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:43601 every 8 connection(s)
I20240503 02:00:59.593072 3737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.594870 3737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13: Bootstrap starting.
I20240503 02:00:59.595679 3737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:59.596683 3737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13: No bootstrap required, opened a new log
I20240503 02:00:59.597044 3737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e94636424334729bfba037ee764eb13" member_type: VOTER }
I20240503 02:00:59.597131 3737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:59.597162 3737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e94636424334729bfba037ee764eb13, State: Initialized, Role: FOLLOWER
I20240503 02:00:59.597265 3737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e94636424334729bfba037ee764eb13" member_type: VOTER }
I20240503 02:00:59.597326 3737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:59.597347 3737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:59.597380 3737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:59.598063 3737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e94636424334729bfba037ee764eb13" member_type: VOTER }
I20240503 02:00:59.598168 3737 leader_election.cc:302] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e94636424334729bfba037ee764eb13; no voters:
I20240503 02:00:59.598318 3737 leader_election.cc:288] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:59.598454 3740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:59.598652 3740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [term 1 LEADER]: Becoming Leader. State: Replica: 4e94636424334729bfba037ee764eb13, State: Running, Role: LEADER
I20240503 02:00:59.598804 3737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:00:59.598794 3740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e94636424334729bfba037ee764eb13" member_type: VOTER }
I20240503 02:00:59.599252 3741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e94636424334729bfba037ee764eb13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e94636424334729bfba037ee764eb13" member_type: VOTER } }
I20240503 02:00:59.599295 3742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e94636424334729bfba037ee764eb13. Latest consensus state: current_term: 1 leader_uuid: "4e94636424334729bfba037ee764eb13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e94636424334729bfba037ee764eb13" member_type: VOTER } }
I20240503 02:00:59.599412 3741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:59.599428 3742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13 [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:59.599900 3745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:00:59.600710 3745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:00:59.600895 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 02:00:59.602435 3745 catalog_manager.cc:1347] Generated new cluster ID: c3567ce7865d474098410cf2281715a1
I20240503 02:00:59.602483 3745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:00:59.617565 3745 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:00:59.618176 3745 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:00:59.624886 3745 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4e94636424334729bfba037ee764eb13: Generated new TSK 0
I20240503 02:00:59.625103 3745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:00:59.633507 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:59.636435 3758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:59.637336 3759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:59.638144 3761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:59.638909 396 server_base.cc:1034] running on GCE node
I20240503 02:00:59.639096 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:59.639137 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:59.639151 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701659639151 us; error 0 us; skew 500 ppm
I20240503 02:00:59.640060 396 webserver.cc:460] Webserver started at http://127.0.99.1:43093/ using document root <none> and password file <none>
I20240503 02:00:59.640208 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:59.640254 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:59.640338 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:59.640736 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-0-root/instance:
uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
format_stamp: "Formatted at 2024-05-03 02:00:59 on dist-test-slave-82l4"
I20240503 02:00:59.642158 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:59.643173 3766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:59.643424 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:59.643491 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-0-root
uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
format_stamp: "Formatted at 2024-05-03 02:00:59 on dist-test-slave-82l4"
I20240503 02:00:59.643565 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:59.660974 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:59.661377 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:59.661671 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:59.662142 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:59.662181 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:59.662225 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:59.662261 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:59.666373 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:39193
I20240503 02:00:59.666399 3829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:39193 every 8 connection(s)
I20240503 02:00:59.667270 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:59.670403 3835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:59.671221 3834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:59.671897 3830 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43601
I20240503 02:00:59.671985 3830 heartbeater.cc:464] Registering TS with master...
W20240503 02:00:59.672166 3837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:59.672322 3830 heartbeater.cc:511] Master 127.0.99.62:43601 requested a full tablet report, sending...
I20240503 02:00:59.672966 396 server_base.cc:1034] running on GCE node
I20240503 02:00:59.672976 3700 ts_manager.cc:194] Registered new tserver with Master: 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193)
I20240503 02:00:59.673153 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:59.673199 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:59.673225 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701659673224 us; error 0 us; skew 500 ppm
I20240503 02:00:59.674046 3700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49012
I20240503 02:00:59.674328 396 webserver.cc:460] Webserver started at http://127.0.99.2:45347/ using document root <none> and password file <none>
I20240503 02:00:59.674501 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:59.674553 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:59.674629 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:59.675068 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-1-root/instance:
uuid: "12e9ea600473443b80086df227e94dfc"
format_stamp: "Formatted at 2024-05-03 02:00:59 on dist-test-slave-82l4"
I20240503 02:00:59.676518 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:59.677448 3842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:59.677767 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:59.677829 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-1-root
uuid: "12e9ea600473443b80086df227e94dfc"
format_stamp: "Formatted at 2024-05-03 02:00:59 on dist-test-slave-82l4"
I20240503 02:00:59.677891 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:59.691846 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:59.692214 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:59.692469 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:59.692936 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:59.692981 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:59.693019 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:59.693049 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:59.697067 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:42879
I20240503 02:00:59.697124 3905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:42879 every 8 connection(s)
I20240503 02:00:59.697862 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:59.700999 3910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:59.701679 3909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:59.702277 3906 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43601
I20240503 02:00:59.702373 3906 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:59.702575 396 server_base.cc:1034] running on GCE node
I20240503 02:00:59.702600 3906 heartbeater.cc:511] Master 127.0.99.62:43601 requested a full tablet report, sending...
I20240503 02:00:59.703256 3700 ts_manager.cc:194] Registered new tserver with Master: 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
W20240503 02:00:59.703389 3912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:59.703646 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:00:59.703719 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:00:59.703763 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701659703761 us; error 0 us; skew 500 ppm
I20240503 02:00:59.703859 3700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49022
I20240503 02:00:59.704772 396 webserver.cc:460] Webserver started at http://127.0.99.3:43599/ using document root <none> and password file <none>
I20240503 02:00:59.704919 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:59.704975 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:59.705065 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:59.705436 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-2-root/instance:
uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
format_stamp: "Formatted at 2024-05-03 02:00:59 on dist-test-slave-82l4"
I20240503 02:00:59.706815 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:59.707681 3917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:59.707906 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:59.707971 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-2-root
uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
format_stamp: "Formatted at 2024-05-03 02:00:59 on dist-test-slave-82l4"
I20240503 02:00:59.708045 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:59.714569 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:59.714947 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:59.715281 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:59.715711 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:59.715749 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:59.715799 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:59.715831 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:59.720522 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:34683
I20240503 02:00:59.720578 3980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:34683 every 8 connection(s)
I20240503 02:00:59.724822 3981 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43601
I20240503 02:00:59.724910 3981 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:59.725132 3981 heartbeater.cc:511] Master 127.0.99.62:43601 requested a full tablet report, sending...
I20240503 02:00:59.725739 3700 ts_manager.cc:194] Registered new tserver with Master: 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:00:59.726236 3700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49028
I20240503 02:00:59.726680 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005763639s
I20240503 02:01:00.675107 3830 heartbeater.cc:503] Master 127.0.99.62:43601 was elected leader, sending a full tablet report...
I20240503 02:01:00.704689 3906 heartbeater.cc:503] Master 127.0.99.62:43601 was elected leader, sending a full tablet report...
I20240503 02:01:00.727118 3981 heartbeater.cc:503] Master 127.0.99.62:43601 was elected leader, sending a full tablet report...
I20240503 02:01:00.752846 396 test_util.cc:268] Using random seed: 392334282
I20240503 02:01:00.761015 3700 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49030:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 02:01:00.761649 3700 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:01:00.775038 3869 tablet_service.cc:1467] Processing CreateTablet for tablet 88bdd7e3ba2d4793907feb426e7bef9c (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 02:01:00.775226 3868 tablet_service.cc:1467] Processing CreateTablet for tablet b98f5009a3534765806bda2c95f7484c (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 02:01:00.775386 3869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bdd7e3ba2d4793907feb426e7bef9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.775426 3944 tablet_service.cc:1467] Processing CreateTablet for tablet 88bdd7e3ba2d4793907feb426e7bef9c (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 02:01:00.775599 3867 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1c79d275004dfeb03f56c07b3904af (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 02:01:00.775699 3944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bdd7e3ba2d4793907feb426e7bef9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.775822 3867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1c79d275004dfeb03f56c07b3904af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.775780 3938 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfd82671b084308af19ee0d7b151cd5 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 02:01:00.775995 3938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cfd82671b084308af19ee0d7b151cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.776305 3866 tablet_service.cc:1467] Processing CreateTablet for tablet 561921bd5097434497ea1f034f86749b (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 02:01:00.776527 3866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561921bd5097434497ea1f034f86749b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.777808 3943 tablet_service.cc:1467] Processing CreateTablet for tablet b98f5009a3534765806bda2c95f7484c (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 02:01:00.777923 3870 tablet_service.cc:1467] Processing CreateTablet for tablet eaa6c34cc53b4061abb42f32a272a67e (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 02:01:00.778033 3943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98f5009a3534765806bda2c95f7484c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.778168 3870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa6c34cc53b4061abb42f32a272a67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.778525 3865 tablet_service.cc:1467] Processing CreateTablet for tablet 226ca5ac33d14b43b90f5a57b2b80472 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 02:01:00.778759 3865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226ca5ac33d14b43b90f5a57b2b80472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.779675 3938 tablet_service.cc:1467] Processing CreateTablet for tablet cef107d87fa14a329bfb5c9b22f0d808 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 02:01:00.779903 3938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef107d87fa14a329bfb5c9b22f0d808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.780243 3942 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1c79d275004dfeb03f56c07b3904af (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 02:01:00.780464 3942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1c79d275004dfeb03f56c07b3904af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.780669 3862 tablet_service.cc:1467] Processing CreateTablet for tablet e8d5be30376a4040a650e1be3d6643f8 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 02:01:00.780858 3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d5be30376a4040a650e1be3d6643f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.775239 3945 tablet_service.cc:1467] Processing CreateTablet for tablet eaa6c34cc53b4061abb42f32a272a67e (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 02:01:00.782771 3945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa6c34cc53b4061abb42f32a272a67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.782981 3939 tablet_service.cc:1467] Processing CreateTablet for tablet e8d5be30376a4040a650e1be3d6643f8 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 02:01:00.783232 3939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d5be30376a4040a650e1be3d6643f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.783511 4002 tablet_bootstrap.cc:492] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980: Bootstrap starting.
I20240503 02:01:00.784371 3868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98f5009a3534765806bda2c95f7484c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.784463 4002 tablet_bootstrap.cc:654] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.782531 3941 tablet_service.cc:1467] Processing CreateTablet for tablet 561921bd5097434497ea1f034f86749b (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 02:01:00.785426 3941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561921bd5097434497ea1f034f86749b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.785526 4002 tablet_bootstrap.cc:492] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980: No bootstrap required, opened a new log
I20240503 02:01:00.785595 4002 ts_tablet_manager.cc:1367] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:01:00.786001 4002 raft_consensus.cc:348] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.786110 4002 raft_consensus.cc:374] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.786140 4002 raft_consensus.cc:729] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.786247 4002 consensus_queue.cc:260] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.786329 3865 tablet_service.cc:1467] Processing CreateTablet for tablet e253a96048114d0ebfa385eda5b0c7b0 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 02:01:00.786579 3865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e253a96048114d0ebfa385eda5b0c7b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.786617 4002 ts_tablet_manager.cc:1398] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.786787 4002 tablet_bootstrap.cc:492] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980: Bootstrap starting.
I20240503 02:01:00.782716 3940 tablet_service.cc:1467] Processing CreateTablet for tablet 226ca5ac33d14b43b90f5a57b2b80472 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 02:01:00.787472 3940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226ca5ac33d14b43b90f5a57b2b80472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.787680 4002 tablet_bootstrap.cc:654] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.787954 3866 tablet_service.cc:1467] Processing CreateTablet for tablet 80f0e3a5143d4bd7a132ee4b6fcb6f89 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 02:01:00.788025 3868 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfd82671b084308af19ee0d7b151cd5 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 02:01:00.788169 3866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f0e3a5143d4bd7a132ee4b6fcb6f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.788359 3862 tablet_service.cc:1467] Processing CreateTablet for tablet cef107d87fa14a329bfb5c9b22f0d808 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 02:01:00.788564 3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef107d87fa14a329bfb5c9b22f0d808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.788699 4002 tablet_bootstrap.cc:492] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980: No bootstrap required, opened a new log
I20240503 02:01:00.788774 4002 ts_tablet_manager.cc:1367] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:00.789198 4002 raft_consensus.cc:348] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.789287 4002 raft_consensus.cc:374] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.789317 4002 raft_consensus.cc:729] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.789423 4002 consensus_queue.cc:260] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.789775 4002 ts_tablet_manager.cc:1398] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.789939 4002 tablet_bootstrap.cc:492] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980: Bootstrap starting.
I20240503 02:01:00.790194 3941 tablet_service.cc:1467] Processing CreateTablet for tablet 98895d45cbd64b52b497261821d8ae53 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 02:01:00.790329 3868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cfd82671b084308af19ee0d7b151cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.790405 3941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98895d45cbd64b52b497261821d8ae53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.790753 4002 tablet_bootstrap.cc:654] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.791376 3939 tablet_service.cc:1467] Processing CreateTablet for tablet 80f0e3a5143d4bd7a132ee4b6fcb6f89 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 02:01:00.791604 3939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f0e3a5143d4bd7a132ee4b6fcb6f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.792006 4002 tablet_bootstrap.cc:492] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980: No bootstrap required, opened a new log
I20240503 02:01:00.792074 4002 ts_tablet_manager.cc:1367] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:00.792249 3870 tablet_service.cc:1467] Processing CreateTablet for tablet 98895d45cbd64b52b497261821d8ae53 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 02:01:00.792471 4002 raft_consensus.cc:348] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.792568 4002 raft_consensus.cc:374] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.792603 4002 raft_consensus.cc:729] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.792728 4002 consensus_queue.cc:260] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.792480 3870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98895d45cbd64b52b497261821d8ae53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.792971 4002 ts_tablet_manager.cc:1398] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.793119 4002 tablet_bootstrap.cc:492] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980: Bootstrap starting.
I20240503 02:01:00.793788 3940 tablet_service.cc:1467] Processing CreateTablet for tablet e253a96048114d0ebfa385eda5b0c7b0 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 02:01:00.794008 4002 tablet_bootstrap.cc:654] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.794059 3940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e253a96048114d0ebfa385eda5b0c7b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.795449 4001 tablet_bootstrap.cc:492] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc: Bootstrap starting.
I20240503 02:01:00.795708 3794 tablet_service.cc:1467] Processing CreateTablet for tablet eaa6c34cc53b4061abb42f32a272a67e (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 02:01:00.795960 3794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa6c34cc53b4061abb42f32a272a67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.796295 3785 tablet_service.cc:1467] Processing CreateTablet for tablet e253a96048114d0ebfa385eda5b0c7b0 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 02:01:00.796401 4001 tablet_bootstrap.cc:654] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.796525 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e253a96048114d0ebfa385eda5b0c7b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.796583 3786 tablet_service.cc:1467] Processing CreateTablet for tablet cef107d87fa14a329bfb5c9b22f0d808 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 02:01:00.796818 3786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef107d87fa14a329bfb5c9b22f0d808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.797458 4002 tablet_bootstrap.cc:492] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980: No bootstrap required, opened a new log
I20240503 02:01:00.797528 4002 ts_tablet_manager.cc:1367] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:01:00.797935 4002 raft_consensus.cc:348] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.798027 4002 raft_consensus.cc:374] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.798058 4002 raft_consensus.cc:729] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.798169 4002 consensus_queue.cc:260] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.798278 3794 tablet_service.cc:1467] Processing CreateTablet for tablet 98895d45cbd64b52b497261821d8ae53 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 02:01:00.798398 4002 ts_tablet_manager.cc:1398] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.798468 3785 tablet_service.cc:1467] Processing CreateTablet for tablet 80f0e3a5143d4bd7a132ee4b6fcb6f89 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 02:01:00.798475 3794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98895d45cbd64b52b497261821d8ae53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.798797 4010 tablet_bootstrap.cc:492] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9: Bootstrap starting.
I20240503 02:01:00.799000 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f0e3a5143d4bd7a132ee4b6fcb6f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.798995 3788 tablet_service.cc:1467] Processing CreateTablet for tablet e8d5be30376a4040a650e1be3d6643f8 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 02:01:00.799232 3788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d5be30376a4040a650e1be3d6643f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.799278 3793 tablet_service.cc:1467] Processing CreateTablet for tablet 88bdd7e3ba2d4793907feb426e7bef9c (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 02:01:00.799525 3793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bdd7e3ba2d4793907feb426e7bef9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.799700 4010 tablet_bootstrap.cc:654] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.800732 4010 tablet_bootstrap.cc:492] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9: No bootstrap required, opened a new log
I20240503 02:01:00.800806 4010 ts_tablet_manager.cc:1367] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:01:00.801268 4002 tablet_bootstrap.cc:492] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980: Bootstrap starting.
I20240503 02:01:00.801237 4010 raft_consensus.cc:348] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } }
I20240503 02:01:00.801332 4010 raft_consensus.cc:374] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.801362 4010 raft_consensus.cc:729] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.801491 4010 consensus_queue.cc:260] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } }
I20240503 02:01:00.801579 3792 tablet_service.cc:1467] Processing CreateTablet for tablet b98f5009a3534765806bda2c95f7484c (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 02:01:00.801759 3792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98f5009a3534765806bda2c95f7484c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.801817 4010 ts_tablet_manager.cc:1398] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.801999 4010 tablet_bootstrap.cc:492] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9: Bootstrap starting.
I20240503 02:01:00.802127 4002 tablet_bootstrap.cc:654] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.798777 3787 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfd82671b084308af19ee0d7b151cd5 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 02:01:00.802923 3787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cfd82671b084308af19ee0d7b151cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.802911 4010 tablet_bootstrap.cc:654] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.803582 4002 tablet_bootstrap.cc:492] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980: No bootstrap required, opened a new log
I20240503 02:01:00.803651 4002 ts_tablet_manager.cc:1367] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:01:00.803759 3791 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1c79d275004dfeb03f56c07b3904af (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 02:01:00.803995 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1c79d275004dfeb03f56c07b3904af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.804040 4002 raft_consensus.cc:348] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } }
I20240503 02:01:00.804136 4002 raft_consensus.cc:374] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.804167 4002 raft_consensus.cc:729] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.804275 4002 consensus_queue.cc:260] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } }
I20240503 02:01:00.804536 4002 ts_tablet_manager.cc:1398] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:01:00.804697 4002 tablet_bootstrap.cc:492] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980: Bootstrap starting.
I20240503 02:01:00.805361 4010 tablet_bootstrap.cc:492] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9: No bootstrap required, opened a new log
I20240503 02:01:00.805351 3789 tablet_service.cc:1467] Processing CreateTablet for tablet 226ca5ac33d14b43b90f5a57b2b80472 (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 02:01:00.805472 4010 ts_tablet_manager.cc:1367] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:01:00.805577 3789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226ca5ac33d14b43b90f5a57b2b80472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.805593 4002 tablet_bootstrap.cc:654] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.805853 4010 raft_consensus.cc:348] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.805974 4010 raft_consensus.cc:374] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.806005 4010 raft_consensus.cc:729] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.806105 4010 consensus_queue.cc:260] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.806211 3790 tablet_service.cc:1467] Processing CreateTablet for tablet 561921bd5097434497ea1f034f86749b (DEFAULT_TABLE table=test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 02:01:00.806331 4010 ts_tablet_manager.cc:1398] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.806422 3790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561921bd5097434497ea1f034f86749b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.806475 4010 tablet_bootstrap.cc:492] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9: Bootstrap starting.
I20240503 02:01:00.806571 4002 tablet_bootstrap.cc:492] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980: No bootstrap required, opened a new log
I20240503 02:01:00.806644 4002 ts_tablet_manager.cc:1367] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:00.807060 4002 raft_consensus.cc:348] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.807169 4002 raft_consensus.cc:374] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.807200 4002 raft_consensus.cc:729] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.807314 4002 consensus_queue.cc:260] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.807376 4010 tablet_bootstrap.cc:654] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.807570 4002 ts_tablet_manager.cc:1398] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.807723 4002 tablet_bootstrap.cc:492] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980: Bootstrap starting.
I20240503 02:01:00.808558 4002 tablet_bootstrap.cc:654] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.809975 4002 tablet_bootstrap.cc:492] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980: No bootstrap required, opened a new log
I20240503 02:01:00.810052 4002 ts_tablet_manager.cc:1367] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:00.810418 4001 tablet_bootstrap.cc:492] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc: No bootstrap required, opened a new log
I20240503 02:01:00.810438 4002 raft_consensus.cc:348] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.810494 4001 ts_tablet_manager.cc:1367] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:01:00.810513 4002 raft_consensus.cc:374] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.810544 4002 raft_consensus.cc:729] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.810650 4002 consensus_queue.cc:260] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.810724 4010 tablet_bootstrap.cc:492] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9: No bootstrap required, opened a new log
I20240503 02:01:00.810820 4010 ts_tablet_manager.cc:1367] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:01:00.810863 4002 ts_tablet_manager.cc:1398] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.810874 4001 raft_consensus.cc:348] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.811004 4001 raft_consensus.cc:374] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.811036 4001 raft_consensus.cc:729] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.811057 4002 tablet_bootstrap.cc:492] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980: Bootstrap starting.
I20240503 02:01:00.811136 4001 consensus_queue.cc:260] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.811321 4010 raft_consensus.cc:348] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.811432 4010 raft_consensus.cc:374] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.811468 4001 ts_tablet_manager.cc:1398] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.811674 4001 tablet_bootstrap.cc:492] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc: Bootstrap starting.
I20240503 02:01:00.811918 4002 tablet_bootstrap.cc:654] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.811468 4010 raft_consensus.cc:729] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.812286 4010 consensus_queue.cc:260] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.812537 4010 ts_tablet_manager.cc:1398] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:01:00.812573 4001 tablet_bootstrap.cc:654] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.812692 4010 tablet_bootstrap.cc:492] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9: Bootstrap starting.
I20240503 02:01:00.813160 4002 tablet_bootstrap.cc:492] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980: No bootstrap required, opened a new log
I20240503 02:01:00.813246 4002 ts_tablet_manager.cc:1367] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:01:00.813561 4001 tablet_bootstrap.cc:492] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc: No bootstrap required, opened a new log
I20240503 02:01:00.813629 4001 ts_tablet_manager.cc:1367] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:00.813688 4010 tablet_bootstrap.cc:654] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.813650 4002 raft_consensus.cc:348] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.813783 4002 raft_consensus.cc:374] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.813827 4002 raft_consensus.cc:729] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.813951 4002 consensus_queue.cc:260] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.814046 4001 raft_consensus.cc:348] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.814117 4001 raft_consensus.cc:374] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.814155 4001 raft_consensus.cc:729] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.814191 4002 ts_tablet_manager.cc:1398] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.814260 4001 consensus_queue.cc:260] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.814323 4002 tablet_bootstrap.cc:492] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980: Bootstrap starting.
I20240503 02:01:00.814561 4001 ts_tablet_manager.cc:1398] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.814746 4001 tablet_bootstrap.cc:492] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc: Bootstrap starting.
I20240503 02:01:00.815172 4002 tablet_bootstrap.cc:654] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.815548 4001 tablet_bootstrap.cc:654] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.816524 4001 tablet_bootstrap.cc:492] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc: No bootstrap required, opened a new log
I20240503 02:01:00.816609 4001 ts_tablet_manager.cc:1367] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:00.816982 4001 raft_consensus.cc:348] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } }
I20240503 02:01:00.817061 4001 raft_consensus.cc:374] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.817113 4001 raft_consensus.cc:729] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.817227 4001 consensus_queue.cc:260] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } }
I20240503 02:01:00.817498 4001 ts_tablet_manager.cc:1398] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.817651 4001 tablet_bootstrap.cc:492] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc: Bootstrap starting.
I20240503 02:01:00.818275 4001 tablet_bootstrap.cc:654] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.819048 4002 tablet_bootstrap.cc:492] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980: No bootstrap required, opened a new log
I20240503 02:01:00.819123 4002 ts_tablet_manager.cc:1367] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:01:00.819257 4001 tablet_bootstrap.cc:492] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc: No bootstrap required, opened a new log
I20240503 02:01:00.819340 4001 ts_tablet_manager.cc:1367] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:01:00.819516 4002 raft_consensus.cc:348] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.819581 4002 raft_consensus.cc:374] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.819600 4002 raft_consensus.cc:729] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.819623 4001 raft_consensus.cc:348] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.819676 4002 consensus_queue.cc:260] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.819708 4001 raft_consensus.cc:374] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.819738 4001 raft_consensus.cc:729] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.819814 4001 consensus_queue.cc:260] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.819888 4002 ts_tablet_manager.cc:1398] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:01:00.820035 4002 tablet_bootstrap.cc:492] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980: Bootstrap starting.
I20240503 02:01:00.820802 4002 tablet_bootstrap.cc:654] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.820050 4001 ts_tablet_manager.cc:1398] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:01:00.821422 4001 tablet_bootstrap.cc:492] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc: Bootstrap starting.
I20240503 02:01:00.822167 4001 tablet_bootstrap.cc:654] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.823275 4002 tablet_bootstrap.cc:492] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980: No bootstrap required, opened a new log
I20240503 02:01:00.823360 4002 ts_tablet_manager.cc:1367] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:01:00.823721 4002 raft_consensus.cc:348] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.823838 4002 raft_consensus.cc:374] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.823875 4002 raft_consensus.cc:729] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.823987 4002 consensus_queue.cc:260] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.824258 4002 ts_tablet_manager.cc:1398] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:01:00.824259 4018 raft_consensus.cc:482] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:00.824502 4002 tablet_bootstrap.cc:492] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980: Bootstrap starting.
I20240503 02:01:00.824476 4018 raft_consensus.cc:504] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } }
I20240503 02:01:00.825011 4018 leader_election.cc:288] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683), 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193)
I20240503 02:01:00.825582 4002 tablet_bootstrap.cc:654] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.826665 4010 tablet_bootstrap.cc:492] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9: No bootstrap required, opened a new log
I20240503 02:01:00.826761 4010 ts_tablet_manager.cc:1367] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240503 02:01:00.827291 4001 tablet_bootstrap.cc:492] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc: No bootstrap required, opened a new log
I20240503 02:01:00.827246 4010 raft_consensus.cc:348] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.827334 4010 raft_consensus.cc:374] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.827358 4001 ts_tablet_manager.cc:1367] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:01:00.827368 4010 raft_consensus.cc:729] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.827646 4010 consensus_queue.cc:260] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.827908 4010 ts_tablet_manager.cc:1398] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:01:00.828328 4010 tablet_bootstrap.cc:492] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9: Bootstrap starting.
I20240503 02:01:00.828516 4002 tablet_bootstrap.cc:492] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980: No bootstrap required, opened a new log
I20240503 02:01:00.828575 4002 ts_tablet_manager.cc:1367] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:01:00.828943 4002 raft_consensus.cc:348] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.829021 4002 raft_consensus.cc:374] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.829059 4002 raft_consensus.cc:729] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.829161 4010 tablet_bootstrap.cc:654] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.829207 4002 consensus_queue.cc:260] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.829480 4002 ts_tablet_manager.cc:1398] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:01:00.829641 4002 tablet_bootstrap.cc:492] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980: Bootstrap starting.
I20240503 02:01:00.830152 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa6c34cc53b4061abb42f32a272a67e" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" is_pre_election: true
I20240503 02:01:00.830276 3955 raft_consensus.cc:2454] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 0.
I20240503 02:01:00.830325 4010 tablet_bootstrap.cc:492] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9: No bootstrap required, opened a new log
I20240503 02:01:00.830391 4010 ts_tablet_manager.cc:1367] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:01:00.830518 4002 tablet_bootstrap.cc:654] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.830587 3844 leader_election.cc:302] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 1fe097edfada4dc3b18e7f74ff6dd980; no voters:
I20240503 02:01:00.831059 4001 raft_consensus.cc:348] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.831168 4001 raft_consensus.cc:374] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.831205 4001 raft_consensus.cc:729] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.831316 4010 raft_consensus.cc:348] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.831394 4022 raft_consensus.cc:2781] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:01:00.831486 4010 raft_consensus.cc:374] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.831319 4001 consensus_queue.cc:260] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.831548 4022 raft_consensus.cc:482] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:01:00.831784 4022 raft_consensus.cc:3037] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.832034 4002 tablet_bootstrap.cc:492] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980: No bootstrap required, opened a new log
I20240503 02:01:00.832115 4002 ts_tablet_manager.cc:1367] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:01:00.832492 4002 raft_consensus.cc:348] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.831593 4010 raft_consensus.cc:729] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.832602 4002 raft_consensus.cc:374] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.832715 4002 raft_consensus.cc:729] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.832769 4010 consensus_queue.cc:260] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.832855 4006 raft_consensus.cc:482] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:00.832820 4002 consensus_queue.cc:260] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.832926 4006 raft_consensus.cc:504] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.833011 4010 ts_tablet_manager.cc:1398] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:00.832993 4022 raft_consensus.cc:504] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } }
I20240503 02:01:00.833101 4002 ts_tablet_manager.cc:1398] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:01:00.833158 4010 tablet_bootstrap.cc:492] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9: Bootstrap starting.
I20240503 02:01:00.833313 4022 leader_election.cc:288] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 election: Requested vote from peers 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683), 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193)
I20240503 02:01:00.833770 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa6c34cc53b4061abb42f32a272a67e" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:00.833858 3955 raft_consensus.cc:3037] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.834051 4010 tablet_bootstrap.cc:654] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.832589 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa6c34cc53b4061abb42f32a272a67e" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" is_pre_election: true
I20240503 02:01:00.834661 3804 raft_consensus.cc:2454] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 0.
I20240503 02:01:00.834700 4001 ts_tablet_manager.cc:1398] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 02:01:00.834846 3955 raft_consensus.cc:2454] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 1.
I20240503 02:01:00.835054 3803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa6c34cc53b4061abb42f32a272a67e" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:00.835150 3844 leader_election.cc:302] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 1fe097edfada4dc3b18e7f74ff6dd980; no voters:
I20240503 02:01:00.835280 4022 raft_consensus.cc:2781] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:01:00.833326 4006 leader_election.cc:288] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:00.835161 3803 raft_consensus.cc:3037] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.835475 4022 raft_consensus.cc:686] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Becoming Leader. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:00.835772 4022 consensus_queue.cc:237] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } }
I20240503 02:01:00.835814 4006 raft_consensus.cc:482] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:00.834877 4001 tablet_bootstrap.cc:492] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc: Bootstrap starting.
I20240503 02:01:00.836424 4006 raft_consensus.cc:504] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.836846 3803 raft_consensus.cc:2454] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 1.
I20240503 02:01:00.836928 4006 leader_election.cc:288] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:00.837327 4001 tablet_bootstrap.cc:654] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.838402 4001 tablet_bootstrap.cc:492] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc: No bootstrap required, opened a new log
I20240503 02:01:00.838526 4001 ts_tablet_manager.cc:1367] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:01:00.839015 4001 raft_consensus.cc:348] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.839187 4001 raft_consensus.cc:374] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.839026 3701 catalog_manager.cc:5484] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e9ea600473443b80086df227e94dfc (127.0.99.2). New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:00.839869 4010 tablet_bootstrap.cc:492] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9: No bootstrap required, opened a new log
I20240503 02:01:00.839962 4010 ts_tablet_manager.cc:1367] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 02:01:00.840335 4010 raft_consensus.cc:348] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.840446 4010 raft_consensus.cc:374] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.840479 4010 raft_consensus.cc:729] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.839269 4001 raft_consensus.cc:729] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.840585 4010 consensus_queue.cc:260] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.840847 4010 ts_tablet_manager.cc:1398] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.841078 3803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef107d87fa14a329bfb5c9b22f0d808" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" is_pre_election: true
I20240503 02:01:00.841145 4010 tablet_bootstrap.cc:492] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9: Bootstrap starting.
I20240503 02:01:00.841171 3803 raft_consensus.cc:2454] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 0.
I20240503 02:01:00.841406 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" is_pre_election: true
I20240503 02:01:00.842010 4010 tablet_bootstrap.cc:654] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.842178 3920 leader_election.cc:302] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:00.842314 4006 raft_consensus.cc:2781] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 02:01:00.842341 3920 leader_election.cc:341] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:01:00.842361 4006 raft_consensus.cc:482] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:01:00.842398 4006 raft_consensus.cc:3037] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.842577 4001 consensus_queue.cc:260] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.842924 4001 ts_tablet_manager.cc:1398] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:01:00.843029 4010 tablet_bootstrap.cc:492] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9: No bootstrap required, opened a new log
I20240503 02:01:00.843103 4010 ts_tablet_manager.cc:1367] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:01:00.843384 4006 raft_consensus.cc:504] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.843500 4010 raft_consensus.cc:348] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.843588 4010 raft_consensus.cc:374] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.843617 4010 raft_consensus.cc:729] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.843712 4010 consensus_queue.cc:260] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.843962 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef107d87fa14a329bfb5c9b22f0d808" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:00.844018 4010 ts_tablet_manager.cc:1398] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.844058 3804 raft_consensus.cc:3037] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.844198 4010 tablet_bootstrap.cc:492] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9: Bootstrap starting.
I20240503 02:01:00.844220 4001 tablet_bootstrap.cc:492] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc: Bootstrap starting.
I20240503 02:01:00.844959 3804 raft_consensus.cc:2454] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 1.
I20240503 02:01:00.845062 4001 tablet_bootstrap.cc:654] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.843719 4006 leader_election.cc:288] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:00.845602 3920 leader_election.cc:302] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:00.845062 4010 tablet_bootstrap.cc:654] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.845755 4006 raft_consensus.cc:2781] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:01:00.846125 4006 raft_consensus.cc:686] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Becoming Leader. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Running, Role: LEADER
I20240503 02:01:00.846273 4006 consensus_queue.cc:237] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.846344 4001 tablet_bootstrap.cc:492] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc: No bootstrap required, opened a new log
I20240503 02:01:00.846423 4001 ts_tablet_manager.cc:1367] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:00.846779 4001 raft_consensus.cc:348] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.846846 4001 raft_consensus.cc:374] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.846876 4001 raft_consensus.cc:729] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.846975 4001 consensus_queue.cc:260] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.847251 4001 ts_tablet_manager.cc:1398] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.847414 4001 tablet_bootstrap.cc:492] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc: Bootstrap starting.
I20240503 02:01:00.847441 3879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9ea600473443b80086df227e94dfc" is_pre_election: true
I20240503 02:01:00.847467 3880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef107d87fa14a329bfb5c9b22f0d808" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9ea600473443b80086df227e94dfc" is_pre_election: true
I20240503 02:01:00.847703 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef107d87fa14a329bfb5c9b22f0d808" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:00.848078 3701 catalog_manager.cc:5484] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } }
I20240503 02:01:00.848263 4010 tablet_bootstrap.cc:492] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9: No bootstrap required, opened a new log
I20240503 02:01:00.848278 4001 tablet_bootstrap.cc:654] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.848362 4010 ts_tablet_manager.cc:1367] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:01:00.848771 4010 raft_consensus.cc:348] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
W20240503 02:01:00.848814 3920 leader_election.cc:341] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:01:00.848877 4010 raft_consensus.cc:374] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.848904 3920 leader_election.cc:302] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980; no voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9
I20240503 02:01:00.849048 4004 raft_consensus.cc:2726] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:01:00.849115 3920 leader_election.cc:341] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:01:00.849261 3920 leader_election.cc:341] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:01:00.848915 4010 raft_consensus.cc:729] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.849511 4010 consensus_queue.cc:260] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.849771 4010 ts_tablet_manager.cc:1398] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.849946 4010 tablet_bootstrap.cc:492] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9: Bootstrap starting.
I20240503 02:01:00.850718 4001 tablet_bootstrap.cc:492] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc: No bootstrap required, opened a new log
I20240503 02:01:00.850791 4001 ts_tablet_manager.cc:1367] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:01:00.850803 4010 tablet_bootstrap.cc:654] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.851220 4001 raft_consensus.cc:348] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.851351 4001 raft_consensus.cc:374] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.851387 4001 raft_consensus.cc:729] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.851481 4001 consensus_queue.cc:260] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.851740 4001 ts_tablet_manager.cc:1398] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.851794 4010 tablet_bootstrap.cc:492] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9: No bootstrap required, opened a new log
I20240503 02:01:00.851871 4010 ts_tablet_manager.cc:1367] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:00.852072 4001 tablet_bootstrap.cc:492] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc: Bootstrap starting.
I20240503 02:01:00.852245 4010 raft_consensus.cc:348] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.852329 4010 raft_consensus.cc:374] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.852346 4010 raft_consensus.cc:729] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.852443 4010 consensus_queue.cc:260] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.852849 4001 tablet_bootstrap.cc:654] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.853212 4010 ts_tablet_manager.cc:1398] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.853459 4010 tablet_bootstrap.cc:492] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9: Bootstrap starting.
I20240503 02:01:00.853947 4001 tablet_bootstrap.cc:492] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc: No bootstrap required, opened a new log
I20240503 02:01:00.854034 4001 ts_tablet_manager.cc:1367] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:01:00.854444 4001 raft_consensus.cc:348] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.854526 4010 tablet_bootstrap.cc:654] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.854543 4001 raft_consensus.cc:374] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.854584 4001 raft_consensus.cc:729] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.854691 4001 consensus_queue.cc:260] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.854918 4001 ts_tablet_manager.cc:1398] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.855101 4001 tablet_bootstrap.cc:492] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc: Bootstrap starting.
I20240503 02:01:00.855904 4001 tablet_bootstrap.cc:654] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.856786 4001 tablet_bootstrap.cc:492] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc: No bootstrap required, opened a new log
I20240503 02:01:00.856868 4001 ts_tablet_manager.cc:1367] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:00.857019 4010 tablet_bootstrap.cc:492] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9: No bootstrap required, opened a new log
I20240503 02:01:00.857091 4010 ts_tablet_manager.cc:1367] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:01:00.857213 4001 raft_consensus.cc:348] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.857288 4001 raft_consensus.cc:374] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.857308 4001 raft_consensus.cc:729] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.857375 4001 consensus_queue.cc:260] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.857511 4010 raft_consensus.cc:348] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.857611 4010 raft_consensus.cc:374] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.857656 4010 raft_consensus.cc:729] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.857709 4001 ts_tablet_manager.cc:1398] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.857762 4010 consensus_queue.cc:260] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.857859 4001 tablet_bootstrap.cc:492] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc: Bootstrap starting.
I20240503 02:01:00.858040 4010 ts_tablet_manager.cc:1398] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:01:00.858201 4010 tablet_bootstrap.cc:492] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9: Bootstrap starting.
I20240503 02:01:00.858614 4001 tablet_bootstrap.cc:654] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.859100 4010 tablet_bootstrap.cc:654] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.860287 4010 tablet_bootstrap.cc:492] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9: No bootstrap required, opened a new log
I20240503 02:01:00.860375 4010 ts_tablet_manager.cc:1367] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:00.860442 4001 tablet_bootstrap.cc:492] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc: No bootstrap required, opened a new log
I20240503 02:01:00.860510 4001 ts_tablet_manager.cc:1367] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:01:00.860807 4010 raft_consensus.cc:348] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.860868 4001 raft_consensus.cc:348] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.860935 4010 raft_consensus.cc:374] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.861009 4001 raft_consensus.cc:374] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.861023 4010 raft_consensus.cc:729] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.861102 4001 raft_consensus.cc:729] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.861231 4010 consensus_queue.cc:260] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.861268 4001 consensus_queue.cc:260] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.861506 4010 ts_tablet_manager.cc:1398] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.861547 4001 ts_tablet_manager.cc:1398] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.861656 4010 tablet_bootstrap.cc:492] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9: Bootstrap starting.
I20240503 02:01:00.861701 4001 tablet_bootstrap.cc:492] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc: Bootstrap starting.
I20240503 02:01:00.862403 4001 tablet_bootstrap.cc:654] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.862560 4010 tablet_bootstrap.cc:654] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:00.863529 4001 tablet_bootstrap.cc:492] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc: No bootstrap required, opened a new log
I20240503 02:01:00.863670 4001 ts_tablet_manager.cc:1367] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:01:00.863785 4010 tablet_bootstrap.cc:492] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9: No bootstrap required, opened a new log
I20240503 02:01:00.863915 4010 ts_tablet_manager.cc:1367] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:01:00.864178 4001 raft_consensus.cc:348] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.864342 4001 raft_consensus.cc:374] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.864418 4001 raft_consensus.cc:729] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.864382 4010 raft_consensus.cc:348] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.864590 4010 raft_consensus.cc:374] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:00.864590 4001 consensus_queue.cc:260] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.864703 4010 raft_consensus.cc:729] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.864835 4010 consensus_queue.cc:260] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.865069 4001 ts_tablet_manager.cc:1398] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.865096 4010 ts_tablet_manager.cc:1398] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.886696 4004 raft_consensus.cc:482] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:00.886823 4004 raft_consensus.cc:504] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.887240 4004 leader_election.cc:288] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:00.887570 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f0e3a5143d4bd7a132ee4b6fcb6f89" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" is_pre_election: true
I20240503 02:01:00.887655 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f0e3a5143d4bd7a132ee4b6fcb6f89" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9ea600473443b80086df227e94dfc" is_pre_election: true
I20240503 02:01:00.887684 3804 raft_consensus.cc:2454] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 0.
I20240503 02:01:00.887800 3878 raft_consensus.cc:2454] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 0.
I20240503 02:01:00.887992 3920 leader_election.cc:302] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:00.888162 4004 raft_consensus.cc:2781] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:01:00.888217 4004 raft_consensus.cc:482] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:01:00.888268 4004 raft_consensus.cc:3037] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.889101 4004 raft_consensus.cc:504] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.889395 4004 leader_election.cc:288] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:00.889653 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f0e3a5143d4bd7a132ee4b6fcb6f89" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:00.889696 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80f0e3a5143d4bd7a132ee4b6fcb6f89" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:00.889746 3804 raft_consensus.cc:3037] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.889779 3878 raft_consensus.cc:3037] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.890482 3804 raft_consensus.cc:2454] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 1.
I20240503 02:01:00.890733 3920 leader_election.cc:302] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:00.890798 3878 raft_consensus.cc:2454] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 1.
I20240503 02:01:00.890884 4004 raft_consensus.cc:2781] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:01:00.891021 4004 raft_consensus.cc:686] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Becoming Leader. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Running, Role: LEADER
I20240503 02:01:00.891167 4004 consensus_queue.cc:237] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.892784 3700 catalog_manager.cc:5484] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } }
I20240503 02:01:00.924072 4012 raft_consensus.cc:482] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:00.924238 4012 raft_consensus.cc:504] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.924784 4012 leader_election.cc:288] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:00.925357 4022 raft_consensus.cc:482] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:00.925460 4022 raft_consensus.cc:504] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.925856 4022 leader_election.cc:288] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:00.926223 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e253a96048114d0ebfa385eda5b0c7b0" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" is_pre_election: true
I20240503 02:01:00.926242 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e253a96048114d0ebfa385eda5b0c7b0" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" is_pre_election: true
I20240503 02:01:00.926357 3804 raft_consensus.cc:2454] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 0.
I20240503 02:01:00.926400 3955 raft_consensus.cc:2454] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 0.
I20240503 02:01:00.926617 3844 leader_election.cc:302] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 1fe097edfada4dc3b18e7f74ff6dd980; no voters:
I20240503 02:01:00.926898 4022 raft_consensus.cc:2781] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:01:00.926987 4022 raft_consensus.cc:482] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:01:00.927029 4022 raft_consensus.cc:3037] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.928076 4022 raft_consensus.cc:504] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.928447 4022 leader_election.cc:288] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:00.928618 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9ea600473443b80086df227e94dfc" is_pre_election: true
I20240503 02:01:00.928682 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e253a96048114d0ebfa385eda5b0c7b0" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:00.928735 3878 raft_consensus.cc:2454] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 0.
I20240503 02:01:00.928750 3804 raft_consensus.cc:3037] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.928781 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e253a96048114d0ebfa385eda5b0c7b0" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:00.929045 3769 leader_election.cc:302] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:00.929211 4012 raft_consensus.cc:2781] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:01:00.929275 4012 raft_consensus.cc:482] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:01:00.929296 3954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" is_pre_election: true
I20240503 02:01:00.929082 3955 raft_consensus.cc:3037] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.929409 3954 raft_consensus.cc:2454] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 0.
I20240503 02:01:00.929332 4012 raft_consensus.cc:3037] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.929695 3804 raft_consensus.cc:2454] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 1.
I20240503 02:01:00.929944 3845 leader_election.cc:302] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:00.930078 4022 raft_consensus.cc:2781] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:01:00.930145 4022 raft_consensus.cc:686] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Becoming Leader. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:00.930240 3955 raft_consensus.cc:2454] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 1.
I20240503 02:01:00.930274 4022 consensus_queue.cc:237] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.930480 4012 raft_consensus.cc:504] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.930850 4012 leader_election.cc:288] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:00.931113 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:00.931166 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:00.931201 3878 raft_consensus.cc:3037] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.931252 3955 raft_consensus.cc:3037] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.931931 3878 raft_consensus.cc:2454] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 1.
I20240503 02:01:00.931955 3955 raft_consensus.cc:2454] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 1.
I20240503 02:01:00.932193 3769 leader_election.cc:302] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:00.932207 3700 catalog_manager.cc:5484] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e9ea600473443b80086df227e94dfc (127.0.99.2). New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:00.932363 4012 raft_consensus.cc:2781] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:01:00.932538 4012 raft_consensus.cc:686] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Running, Role: LEADER
I20240503 02:01:00.932662 4012 consensus_queue.cc:237] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.933733 4012 raft_consensus.cc:482] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:00.933813 4012 raft_consensus.cc:504] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.934116 4012 leader_election.cc:288] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:00.934252 3699 catalog_manager.cc:5484] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:00.934511 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561921bd5097434497ea1f034f86749b" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" is_pre_election: true
I20240503 02:01:00.934614 3955 raft_consensus.cc:2454] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 0.
I20240503 02:01:00.934799 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561921bd5097434497ea1f034f86749b" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9ea600473443b80086df227e94dfc" is_pre_election: true
I20240503 02:01:00.934849 3768 leader_election.cc:302] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:00.934900 3878 raft_consensus.cc:2454] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 0.
I20240503 02:01:00.935030 4012 raft_consensus.cc:2781] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:01:00.935086 4012 raft_consensus.cc:482] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:01:00.935143 4012 raft_consensus.cc:3037] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.936115 4012 raft_consensus.cc:504] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.936405 4012 leader_election.cc:288] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:00.936651 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561921bd5097434497ea1f034f86749b" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:00.936676 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561921bd5097434497ea1f034f86749b" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:00.936765 3878 raft_consensus.cc:3037] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.936769 3955 raft_consensus.cc:3037] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.937551 3878 raft_consensus.cc:2454] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 1.
I20240503 02:01:00.937570 3955 raft_consensus.cc:2454] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 1.
I20240503 02:01:00.937814 3769 leader_election.cc:302] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:00.938040 4012 raft_consensus.cc:2781] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:01:00.938124 4012 raft_consensus.cc:686] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Running, Role: LEADER
I20240503 02:01:00.938288 4012 consensus_queue.cc:237] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.939682 3699 catalog_manager.cc:5484] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:00.958969 4022 raft_consensus.cc:482] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:00.959132 4022 raft_consensus.cc:504] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.959457 4022 leader_election.cc:288] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:00.959827 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bdd7e3ba2d4793907feb426e7bef9c" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" is_pre_election: true
I20240503 02:01:00.959827 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bdd7e3ba2d4793907feb426e7bef9c" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" is_pre_election: true
I20240503 02:01:00.959978 3804 raft_consensus.cc:2454] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 0.
I20240503 02:01:00.959978 3955 raft_consensus.cc:2454] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 0.
I20240503 02:01:00.960268 4022 raft_consensus.cc:482] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:00.960363 4022 raft_consensus.cc:504] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.960263 3845 leader_election.cc:302] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:00.960646 4018 raft_consensus.cc:2781] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:01:00.960680 4022 leader_election.cc:288] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:00.960703 4018 raft_consensus.cc:482] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:01:00.960754 4018 raft_consensus.cc:3037] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.960947 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" is_pre_election: true
I20240503 02:01:00.960947 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" is_pre_election: true
I20240503 02:01:00.961071 3804 raft_consensus.cc:2454] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 0.
I20240503 02:01:00.961071 3955 raft_consensus.cc:2454] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 0.
I20240503 02:01:00.961289 3844 leader_election.cc:302] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 1fe097edfada4dc3b18e7f74ff6dd980; no voters:
I20240503 02:01:00.961418 4022 raft_consensus.cc:2781] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:01:00.961462 4022 raft_consensus.cc:482] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:01:00.961495 4022 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.961704 4018 raft_consensus.cc:504] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.961978 4018 leader_election.cc:288] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:00.962215 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bdd7e3ba2d4793907feb426e7bef9c" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:00.962252 4022 raft_consensus.cc:504] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.962311 3804 raft_consensus.cc:3037] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.962289 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bdd7e3ba2d4793907feb426e7bef9c" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:00.962394 3955 raft_consensus.cc:3037] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.962616 4022 leader_election.cc:288] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:00.963013 3954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:00.963120 3954 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.963176 3955 raft_consensus.cc:2454] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 1.
I20240503 02:01:00.963188 3804 raft_consensus.cc:2454] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 1.
I20240503 02:01:00.963420 3845 leader_election.cc:302] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:00.963547 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:00.963639 3804 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.964071 3954 raft_consensus.cc:2454] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 1.
I20240503 02:01:00.964206 3804 raft_consensus.cc:2454] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 1.
I20240503 02:01:00.964361 4022 raft_consensus.cc:2781] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:01:00.964395 3844 leader_election.cc:302] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 1fe097edfada4dc3b18e7f74ff6dd980; no voters:
I20240503 02:01:00.964442 4022 raft_consensus.cc:686] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Becoming Leader. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:00.964572 4022 consensus_queue.cc:237] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.964959 4058 raft_consensus.cc:2781] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:01:00.965023 4058 raft_consensus.cc:686] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Becoming Leader. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:00.965155 4058 consensus_queue.cc:237] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.966034 3699 catalog_manager.cc:5484] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e9ea600473443b80086df227e94dfc (127.0.99.2). New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:00.968713 3699 catalog_manager.cc:5484] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e9ea600473443b80086df227e94dfc (127.0.99.2). New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:00.970052 4004 raft_consensus.cc:482] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:00.970134 4004 raft_consensus.cc:504] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.970429 4004 leader_election.cc:288] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:00.970705 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98f5009a3534765806bda2c95f7484c" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" is_pre_election: true
I20240503 02:01:00.970734 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98f5009a3534765806bda2c95f7484c" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9ea600473443b80086df227e94dfc" is_pre_election: true
I20240503 02:01:00.970817 3878 raft_consensus.cc:2454] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 0.
I20240503 02:01:00.970804 3804 raft_consensus.cc:2454] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 0.
I20240503 02:01:00.971081 3920 leader_election.cc:302] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:00.971244 4004 raft_consensus.cc:2781] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:01:00.971300 4004 raft_consensus.cc:482] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:01:00.971330 4004 raft_consensus.cc:3037] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.971971 4004 raft_consensus.cc:504] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.972260 4004 leader_election.cc:288] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:00.972537 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98f5009a3534765806bda2c95f7484c" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:00.972537 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98f5009a3534765806bda2c95f7484c" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:00.972678 3804 raft_consensus.cc:3037] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.972719 3878 raft_consensus.cc:3037] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:00.973274 3804 raft_consensus.cc:2454] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 1.
I20240503 02:01:00.973312 3878 raft_consensus.cc:2454] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 1.
I20240503 02:01:00.973577 3920 leader_election.cc:302] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 1fe097edfada4dc3b18e7f74ff6dd980; no voters:
I20240503 02:01:00.973732 4004 raft_consensus.cc:2781] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:01:00.973814 4004 raft_consensus.cc:686] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Becoming Leader. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Running, Role: LEADER
I20240503 02:01:00.973984 4004 consensus_queue.cc:237] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:00.975421 3699 catalog_manager.cc:5484] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } }
I20240503 02:01:00.999989 4058 raft_consensus.cc:482] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:01.000147 4058 raft_consensus.cc:504] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.000631 4058 leader_election.cc:288] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:01.001041 3954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef107d87fa14a329bfb5c9b22f0d808" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" is_pre_election: true
I20240503 02:01:01.001041 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef107d87fa14a329bfb5c9b22f0d808" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" is_pre_election: true
I20240503 02:01:01.001235 3804 raft_consensus.cc:2379] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 12e9ea600473443b80086df227e94dfc in current term 1: Already voted for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in this term.
I20240503 02:01:01.001480 3845 leader_election.cc:302] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12e9ea600473443b80086df227e94dfc; no voters: 1fe097edfada4dc3b18e7f74ff6dd980, 94da6cefbea94c72a5d91b8d10f31bc9
I20240503 02:01:01.001636 4058 raft_consensus.cc:3037] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:01.002543 4058 raft_consensus.cc:2726] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:01:01.004994 4058 raft_consensus.cc:482] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:01.005082 4058 raft_consensus.cc:504] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.005396 4058 leader_election.cc:288] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:01.005731 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" is_pre_election: true
I20240503 02:01:01.005731 3954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" is_pre_election: true
I20240503 02:01:01.005874 3954 raft_consensus.cc:2454] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 0.
I20240503 02:01:01.005874 3804 raft_consensus.cc:2454] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 0.
I20240503 02:01:01.006155 3844 leader_election.cc:302] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 1fe097edfada4dc3b18e7f74ff6dd980; no voters:
I20240503 02:01:01.006299 4058 raft_consensus.cc:2781] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:01:01.006340 4058 raft_consensus.cc:482] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:01:01.006379 4058 raft_consensus.cc:3037] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:01.007145 4058 raft_consensus.cc:504] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.007437 4058 leader_election.cc:288] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:01.007774 4012 raft_consensus.cc:482] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:01.007740 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:01.007740 3954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:01.007896 4012 raft_consensus.cc:504] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.007999 3804 raft_consensus.cc:3037] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:01.008023 3954 raft_consensus.cc:3037] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:01.008298 4012 leader_election.cc:288] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:01.008579 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9ea600473443b80086df227e94dfc" is_pre_election: true
I20240503 02:01:01.008679 3878 raft_consensus.cc:2454] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 0.
I20240503 02:01:01.008814 3804 raft_consensus.cc:2454] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 1.
I20240503 02:01:01.008814 3954 raft_consensus.cc:2454] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 1.
I20240503 02:01:01.008975 3769 leader_election.cc:302] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:01.009034 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" is_pre_election: true
I20240503 02:01:01.009074 3844 leader_election.cc:302] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 1fe097edfada4dc3b18e7f74ff6dd980; no voters:
I20240503 02:01:01.009114 4012 raft_consensus.cc:2781] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:01:01.009135 3955 raft_consensus.cc:2454] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 0.
I20240503 02:01:01.009158 4012 raft_consensus.cc:482] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:01:01.009196 4012 raft_consensus.cc:3037] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:01.009196 4058 raft_consensus.cc:2781] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:01:01.009390 4058 raft_consensus.cc:686] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Becoming Leader. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:01.009516 4058 consensus_queue.cc:237] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.009989 4012 raft_consensus.cc:504] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.010318 4012 leader_election.cc:288] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:01.010756 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:01.010841 3878 raft_consensus.cc:3037] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:01.011034 3699 catalog_manager.cc:5484] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e9ea600473443b80086df227e94dfc (127.0.99.2). New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:01.011086 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:01.011214 3955 raft_consensus.cc:3037] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:01.011581 3878 raft_consensus.cc:2454] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 1.
I20240503 02:01:01.011806 3769 leader_election.cc:302] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:01.011942 4012 raft_consensus.cc:2781] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:01:01.011962 3955 raft_consensus.cc:2454] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 1.
I20240503 02:01:01.012018 4012 raft_consensus.cc:686] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Running, Role: LEADER
I20240503 02:01:01.012143 4012 consensus_queue.cc:237] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.013602 3699 catalog_manager.cc:5484] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:01.171497 4058 raft_consensus.cc:482] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:01.171671 4058 raft_consensus.cc:504] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.172096 4058 leader_election.cc:288] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:01.172519 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d5be30376a4040a650e1be3d6643f8" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" is_pre_election: true
I20240503 02:01:01.172515 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d5be30376a4040a650e1be3d6643f8" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" is_pre_election: true
I20240503 02:01:01.172713 3804 raft_consensus.cc:2454] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 0.
I20240503 02:01:01.172724 3955 raft_consensus.cc:2454] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 0.
I20240503 02:01:01.173092 3845 leader_election.cc:302] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:01.173275 4058 raft_consensus.cc:2781] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:01:01.173321 4058 raft_consensus.cc:482] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:01:01.173365 4058 raft_consensus.cc:3037] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:01.174182 4058 raft_consensus.cc:504] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.174511 4058 leader_election.cc:288] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:01.174845 3955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d5be30376a4040a650e1be3d6643f8" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:01.174847 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d5be30376a4040a650e1be3d6643f8" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:01.175011 3955 raft_consensus.cc:3037] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:01.175011 3804 raft_consensus.cc:3037] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:01.175773 3804 raft_consensus.cc:2454] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 1.
I20240503 02:01:01.175774 3955 raft_consensus.cc:2454] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 1.
I20240503 02:01:01.176081 3845 leader_election.cc:302] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:01.176227 4058 raft_consensus.cc:2781] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:01:01.176321 4058 raft_consensus.cc:686] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Becoming Leader. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:01.176465 4058 consensus_queue.cc:237] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.178097 3699 catalog_manager.cc:5484] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e9ea600473443b80086df227e94dfc (127.0.99.2). New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:01.245574 4004 consensus_queue.cc:1035] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.249846 4058 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.251384 4004 consensus_queue.cc:1035] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.254384 4058 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.271026 3955 raft_consensus.cc:1261] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.272716 3804 raft_consensus.cc:1261] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.274652 4018 consensus_queue.cc:1035] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.275156 4018 consensus_queue.cc:1035] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.275879 3803 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:01:01.275988 3955 raft_consensus.cc:1261] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.276154 3802 raft_consensus.cc:1261] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.276222 3953 raft_consensus.cc:1261] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.276428 3954 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:01:01.276690 3801 raft_consensus.cc:1261] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.277082 3804 raft_consensus.cc:1261] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.277467 3951 raft_consensus.cc:1261] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.278807 4022 consensus_queue.cc:1035] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.279131 4022 consensus_queue.cc:1035] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.279381 4022 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.279611 4022 consensus_queue.cc:1035] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.281944 3802 raft_consensus.cc:1261] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:01:01.282307 3803 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.283299 4004 consensus_queue.cc:1035] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.283579 4004 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.283926 3878 raft_consensus.cc:1261] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:01:01.283926 3880 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.286383 3949 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.287411 4006 consensus_queue.cc:1035] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.287683 4006 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.287756 4012 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.288326 3951 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.287689 3949 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.288833 4058 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.289100 4058 consensus_queue.cc:1035] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.289350 4058 consensus_queue.cc:1035] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.289389 4055 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.289564 4058 consensus_queue.cc:1035] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.289641 4055 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.290257 3880 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.290346 3879 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.290592 3878 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:01:01.307646 4108 mvcc.cc:205] Tried to move back new op lower bound from 7023418004594806784 to 7023418003312177152. Current Snapshot: MvccSnapshot[applied={T|T < 7023418004594806784 or (T in {7023418004594806784})}]
I20240503 02:01:01.307823 4120 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.308585 4120 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.311600 4120 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:01:01.334662 4116 mvcc.cc:205] Tried to move back new op lower bound from 7023418004572971008 to 7023418004180226048. Current Snapshot: MvccSnapshot[applied={T|T < 7023418004572971008 or (T in {7023418004572971008})}]
I20240503 02:01:01.355165 4107 mvcc.cc:205] Tried to move back new op lower bound from 7023418004574769152 to 7023418003496239104. Current Snapshot: MvccSnapshot[applied={T|T < 7023418004565581824}]
I20240503 02:01:01.603121 3755 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:01:01.637499 3875 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
mode: GRACEFUL
new_leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
from {username='slave'} at 127.0.0.1:56270
I20240503 02:01:01.637665 3875 raft_consensus.cc:595] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Received request to transfer leadership to TS 94da6cefbea94c72a5d91b8d10f31bc9
I20240503 02:01:01.640359 3875 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
mode: GRACEFUL
new_leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
from {username='slave'} at 127.0.0.1:56270
I20240503 02:01:01.640507 3875 raft_consensus.cc:595] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Received request to transfer leadership to TS 1fe097edfada4dc3b18e7f74ff6dd980
I20240503 02:01:01.640780 3755 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 02:01:01.640908 3755 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 02:01:01.642637 3857 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56254: Service unavailable: leader transfer in progress
W20240503 02:01:01.653205 3857 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56254: Service unavailable: leader transfer in progress
I20240503 02:01:01.653828 4084 raft_consensus.cc:979] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc: : Instructing follower 1fe097edfada4dc3b18e7f74ff6dd980 to start an election
I20240503 02:01:01.653923 4084 raft_consensus.cc:1067] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Signalling peer 1fe097edfada4dc3b18e7f74ff6dd980 to start an election
I20240503 02:01:01.657545 4089 raft_consensus.cc:979] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc: : Instructing follower 94da6cefbea94c72a5d91b8d10f31bc9 to start an election
I20240503 02:01:01.657645 4089 raft_consensus.cc:1067] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Signalling peer 94da6cefbea94c72a5d91b8d10f31bc9 to start an election
I20240503 02:01:01.660372 3801 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472"
dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
from {username='slave'} at 127.0.0.1:35300
I20240503 02:01:01.660511 3801 raft_consensus.cc:482] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:01.660575 3801 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:01:01.660845 3852 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56254: Service unavailable: leader transfer in progress
I20240503 02:01:01.661777 3801 raft_consensus.cc:504] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.662545 3801 leader_election.cc:288] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 2 election: Requested vote from peers 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:01.664007 3947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: true dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:01.664122 3947 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:01.664705 3877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: true dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:01.664808 3877 raft_consensus.cc:3032] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:01.664848 3877 raft_consensus.cc:729] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:01.664969 3877 consensus_queue.cc:260] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 20, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.665078 3877 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:01.665633 3947 raft_consensus.cc:2454] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 2.
I20240503 02:01:01.666589 3877 raft_consensus.cc:2454] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 2.
I20240503 02:01:01.667356 3769 leader_election.cc:302] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:01.667424 3948 raft_consensus.cc:1226] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Rejecting Update request from peer 12e9ea600473443b80086df227e94dfc for earlier term 1. Current term is 2. Ops: []
I20240503 02:01:01.667526 3955 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53"
dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
from {username='slave'} at 127.0.0.1:50770
I20240503 02:01:01.667624 3955 raft_consensus.cc:482] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:01.667675 3955 raft_consensus.cc:3037] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:01.668570 3955 raft_consensus.cc:504] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.668922 3955 leader_election.cc:288] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 2 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:01.669615 4120 raft_consensus.cc:2781] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:01.669714 4120 raft_consensus.cc:686] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 LEADER]: Becoming Leader. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Running, Role: LEADER
I20240503 02:01:01.669844 4120 consensus_queue.cc:237] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 20, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.672348 3701 catalog_manager.cc:5484] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: term changed from 1 to 2, leader changed from 12e9ea600473443b80086df227e94dfc (127.0.99.2) to 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:01.673261 3797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: true dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:01.673379 3797 raft_consensus.cc:3037] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:01.674528 3797 raft_consensus.cc:2454] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 2.
I20240503 02:01:01.675695 3920 leader_election.cc:302] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:01.675874 4152 raft_consensus.cc:2781] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:01.675969 4152 raft_consensus.cc:686] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Becoming Leader. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Running, Role: LEADER
I20240503 02:01:01.676101 4152 consensus_queue.cc:237] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 19, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.677750 3701 catalog_manager.cc:5484] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: term changed from 1 to 2, leader changed from 12e9ea600473443b80086df227e94dfc (127.0.99.2) to 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3). New cstate: current_term: 2 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } }
I20240503 02:01:01.682294 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: true dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:01.682511 3878 raft_consensus.cc:3032] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:01.682616 3878 raft_consensus.cc:729] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:01.682786 3878 consensus_queue.cc:260] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 19, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:01.683035 3878 raft_consensus.cc:3037] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:01.684235 3878 raft_consensus.cc:2454] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 2.
W20240503 02:01:01.686201 3851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56254: Illegal state: replica 12e9ea600473443b80086df227e94dfc is not leader of this config: current role FOLLOWER
W20240503 02:01:01.690021 3851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56254: Illegal state: replica 12e9ea600473443b80086df227e94dfc is not leader of this config: current role FOLLOWER
W20240503 02:01:01.694475 3851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56254: Illegal state: replica 12e9ea600473443b80086df227e94dfc is not leader of this config: current role FOLLOWER
W20240503 02:01:01.700181 3852 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56254: Illegal state: replica 12e9ea600473443b80086df227e94dfc is not leader of this config: current role FOLLOWER
W20240503 02:01:01.703732 3852 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56254: Illegal state: replica 12e9ea600473443b80086df227e94dfc is not leader of this config: current role FOLLOWER
W20240503 02:01:01.707381 3852 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56254: Illegal state: replica 12e9ea600473443b80086df227e94dfc is not leader of this config: current role FOLLOWER
W20240503 02:01:01.708003 3852 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56254: Illegal state: replica 12e9ea600473443b80086df227e94dfc is not leader of this config: current role FOLLOWER
W20240503 02:01:01.715400 3782 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35314: Illegal state: replica 94da6cefbea94c72a5d91b8d10f31bc9 is not leader of this config: current role FOLLOWER
W20240503 02:01:01.715524 3779 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35314: Illegal state: replica 94da6cefbea94c72a5d91b8d10f31bc9 is not leader of this config: current role FOLLOWER
W20240503 02:01:01.715781 3783 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35314: Illegal state: replica 94da6cefbea94c72a5d91b8d10f31bc9 is not leader of this config: current role FOLLOWER
W20240503 02:01:01.716089 3779 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35314: Illegal state: replica 94da6cefbea94c72a5d91b8d10f31bc9 is not leader of this config: current role FOLLOWER
I20240503 02:01:01.716207 3955 raft_consensus.cc:1261] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 2 index: 22. (index mismatch)
I20240503 02:01:01.716612 4012 consensus_queue.cc:1035] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20240503 02:01:01.718075 3878 raft_consensus.cc:1261] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 2 index: 22. (index mismatch)
I20240503 02:01:01.722623 4012 consensus_queue.cc:1035] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20240503 02:01:01.723644 3799 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 2 index: 21. (index mismatch)
I20240503 02:01:01.724298 3875 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 2 index: 21. (index mismatch)
I20240503 02:01:01.726119 4152 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20240503 02:01:01.726258 4151 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20240503 02:01:01.778802 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:01:01.783198 4158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:01.784225 4161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:01.784960 4159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:01:01.785833 396 server_base.cc:1034] running on GCE node
I20240503 02:01:01.786046 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:01:01.786087 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:01:01.786103 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701661786103 us; error 0 us; skew 500 ppm
I20240503 02:01:01.787222 396 webserver.cc:460] Webserver started at http://127.0.99.4:35189/ using document root <none> and password file <none>
I20240503 02:01:01.787367 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:01:01.787413 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:01:01.787472 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:01:01.787856 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-3-root/instance:
uuid: "38e9ca6a49c4402abdd585ffd8f435ee"
format_stamp: "Formatted at 2024-05-03 02:01:01 on dist-test-slave-82l4"
I20240503 02:01:01.789283 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:01:01.790266 4166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:01.790517 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:01.790582 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-3-root
uuid: "38e9ca6a49c4402abdd585ffd8f435ee"
format_stamp: "Formatted at 2024-05-03 02:01:01 on dist-test-slave-82l4"
I20240503 02:01:01.790650 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:01:01.805886 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:01:01.806318 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:01:01.806633 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:01:01.807179 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:01:01.807220 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:01.807253 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:01:01.807276 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:01.812218 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:34437
I20240503 02:01:01.812314 4229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:34437 every 8 connection(s)
I20240503 02:01:01.813162 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:01:01.816244 4235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:01.817113 4234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:01:01.817943 396 server_base.cc:1034] running on GCE node
I20240503 02:01:01.818352 4230 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43601
I20240503 02:01:01.818452 4230 heartbeater.cc:464] Registering TS with master...
W20240503 02:01:01.818663 4237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:01:01.818706 4230 heartbeater.cc:511] Master 127.0.99.62:43601 requested a full tablet report, sending...
I20240503 02:01:01.818892 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:01:01.818951 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:01:01.819000 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701661819000 us; error 0 us; skew 500 ppm
I20240503 02:01:01.819447 3701 ts_manager.cc:194] Registered new tserver with Master: 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4:34437)
I20240503 02:01:01.820143 3701 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49044
I20240503 02:01:01.820205 396 webserver.cc:460] Webserver started at http://127.0.99.5:38835/ using document root <none> and password file <none>
I20240503 02:01:01.820351 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:01:01.820415 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:01:01.820485 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:01:01.820950 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-4-root/instance:
uuid: "dc881d7ec7fb47a2897eef434cf18767"
format_stamp: "Formatted at 2024-05-03 02:01:01 on dist-test-slave-82l4"
I20240503 02:01:01.822422 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 02:01:01.823334 4242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:01.823570 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 02:01:01.823629 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-4-root
uuid: "dc881d7ec7fb47a2897eef434cf18767"
format_stamp: "Formatted at 2024-05-03 02:01:01 on dist-test-slave-82l4"
I20240503 02:01:01.823685 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:01:01.834306 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:01:01.835230 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:01:01.835520 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:01:01.835978 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:01:01.836015 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:01.836055 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:01:01.836083 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:01.840968 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.5:42349
I20240503 02:01:01.841063 4305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.5:42349 every 8 connection(s)
I20240503 02:01:01.841939 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:01:01.844970 4310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:01.845822 4309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:01:01.846369 4306 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43601
I20240503 02:01:01.846462 4306 heartbeater.cc:464] Registering TS with master...
W20240503 02:01:01.846623 4312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:01:01.846757 4306 heartbeater.cc:511] Master 127.0.99.62:43601 requested a full tablet report, sending...
I20240503 02:01:01.847425 3701 ts_manager.cc:194] Registered new tserver with Master: dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5:42349)
I20240503 02:01:01.847476 396 server_base.cc:1034] running on GCE node
I20240503 02:01:01.847644 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 02:01:01.847692 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 02:01:01.847714 396 hybrid_clock.cc:648] HybridClock initialized: now 1714701661847714 us; error 0 us; skew 500 ppm
I20240503 02:01:01.848176 3701 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49050
I20240503 02:01:01.848923 396 webserver.cc:460] Webserver started at http://127.0.99.6:39713/ using document root <none> and password file <none>
I20240503 02:01:01.849081 396 fs_manager.cc:359] Metadata directory not provided
I20240503 02:01:01.849131 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:01:01.849192 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:01:01.849570 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-5-root/instance:
uuid: "cfdcf4a7a9734e82bddf13e25872e410"
format_stamp: "Formatted at 2024-05-03 02:01:01 on dist-test-slave-82l4"
I20240503 02:01:01.851014 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 02:01:01.851949 4317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:01.852269 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:01.852362 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-5-root
uuid: "cfdcf4a7a9734e82bddf13e25872e410"
format_stamp: "Formatted at 2024-05-03 02:01:01 on dist-test-slave-82l4"
I20240503 02:01:01.852448 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:01:01.864041 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:01:01.864487 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:01:01.864756 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:01:01.865238 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:01:01.865278 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:01.865315 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:01:01.865346 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:01.870755 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.6:38521
I20240503 02:01:01.870863 4380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.6:38521 every 8 connection(s)
I20240503 02:01:01.875423 4381 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43601
I20240503 02:01:01.875525 4381 heartbeater.cc:464] Registering TS with master...
I20240503 02:01:01.875746 4381 heartbeater.cc:511] Master 127.0.99.62:43601 requested a full tablet report, sending...
I20240503 02:01:01.876349 3701 ts_manager.cc:194] Registered new tserver with Master: cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6:38521)
I20240503 02:01:01.876885 3701 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49064
I20240503 02:01:02.613425 3950 consensus_queue.cc:237] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 19, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:02.615015 3799 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 02:01:02.615015 3875 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 02:01:02.615417 4152 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:02.615626 4387 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:02.617172 4387 raft_consensus.cc:2932] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:02.617635 3875 raft_consensus.cc:2932] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:02.617702 3799 raft_consensus.cc:2932] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:02.620256 3701 catalog_manager.cc:5484] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc reported cstate change: config changed from index -1 to 27, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New cstate: current_term: 2 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
W20240503 02:01:02.620720 3919 consensus_peers.cc:485] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 -> Peer dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5:42349): Couldn't send request to peer dc881d7ec7fb47a2897eef434cf18767. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98895d45cbd64b52b497261821d8ae53. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:02.625595 3875 consensus_queue.cc:237] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:02.627094 3799 raft_consensus.cc:1261] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:02.627213 3950 raft_consensus.cc:1261] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:02.627508 4395 consensus_queue.cc:1035] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:02.627734 4394 consensus_queue.cc:1035] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:02.629186 4091 raft_consensus.cc:2932] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:02.631126 3799 raft_consensus.cc:2932] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:02.631186 3950 raft_consensus.cc:2932] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
W20240503 02:01:02.631632 3844 consensus_peers.cc:485] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc -> Peer 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4:34437): Couldn't send request to peer 38e9ca6a49c4402abdd585ffd8f435ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88bdd7e3ba2d4793907feb426e7bef9c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:02.632982 3697 catalog_manager.cc:5484] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc reported cstate change: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:02.637991 3799 consensus_queue.cc:237] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:02.639360 3950 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:02.639546 3875 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:02.639732 4402 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:02.639911 4399 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:02.641319 4155 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:02.641737 3875 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:02.643442 3950 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:02.643937 3695 catalog_manager.cc:5484] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc reported cstate change: config changed from index -1 to 27, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New cstate: current_term: 1 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
W20240503 02:01:02.645119 3769 consensus_peers.cc:485] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 -> Peer cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6:38521): Couldn't send request to peer cfdcf4a7a9734e82bddf13e25872e410. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 561921bd5097434497ea1f034f86749b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:02.645130 3947 consensus_queue.cc:237] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:02.647161 3799 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 02:01:02.647384 3875 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 02:01:02.647679 4387 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 02:01:02.647991 4387 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 02:01:02.649448 4387 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:02.649875 3799 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
W20240503 02:01:02.651608 3920 consensus_peers.cc:485] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 -> Peer cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6:38521): Couldn't send request to peer cfdcf4a7a9734e82bddf13e25872e410. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b98f5009a3534765806bda2c95f7484c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:02.651775 3875 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:02.651932 3878 consensus_queue.cc:237] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:02.652660 3701 catalog_manager.cc:5484] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index -1 to 26, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New cstate: current_term: 1 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:02.653481 3799 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:02.654021 4394 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20240503 02:01:02.654434 3844 consensus_peers.cc:485] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc -> Peer 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4:34437): Couldn't send request to peer 38e9ca6a49c4402abdd585ffd8f435ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eaa6c34cc53b4061abb42f32a272a67e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:02.654048 3947 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:02.655697 4091 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:02.655915 4395 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:02.656277 3799 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:02.656635 3947 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:02.658360 3799 consensus_queue.cc:237] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:02.658860 3697 catalog_manager.cc:5484] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:02.660051 3947 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 02:01:02.660410 4399 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 02:01:02.660509 3875 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 02:01:02.660835 4399 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 02:01:02.662128 4399 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
W20240503 02:01:02.663290 3768 consensus_peers.cc:485] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 -> Peer dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5:42349): Couldn't send request to peer dc881d7ec7fb47a2897eef434cf18767. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f1c79d275004dfeb03f56c07b3904af. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:02.664317 3947 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:02.664548 3695 catalog_manager.cc:5484] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index -1 to 25, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New cstate: current_term: 1 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:02.664599 3875 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:02.821096 4230 heartbeater.cc:503] Master 127.0.99.62:43601 was elected leader, sending a full tablet report...
I20240503 02:01:02.849108 4306 heartbeater.cc:503] Master 127.0.99.62:43601 was elected leader, sending a full tablet report...
I20240503 02:01:02.877720 4381 heartbeater.cc:503] Master 127.0.99.62:43601 was elected leader, sending a full tablet report...
I20240503 02:01:03.037303 4405 ts_tablet_manager.cc:902] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767: Initiating tablet copy from peer 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:03.037798 4405 tablet_copy_client.cc:308] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:34683
I20240503 02:01:03.041225 3965 tablet_copy_service.cc:140] P 1fe097edfada4dc3b18e7f74ff6dd980: Received BeginTabletCopySession request for tablet 98895d45cbd64b52b497261821d8ae53 from peer dc881d7ec7fb47a2897eef434cf18767 ({username='slave'} at 127.0.0.1:50804)
I20240503 02:01:03.041347 3965 tablet_copy_service.cc:161] P 1fe097edfada4dc3b18e7f74ff6dd980: Beginning new tablet copy session on tablet 98895d45cbd64b52b497261821d8ae53 from peer dc881d7ec7fb47a2897eef434cf18767 at {username='slave'} at 127.0.0.1:50804: session id = dc881d7ec7fb47a2897eef434cf18767-98895d45cbd64b52b497261821d8ae53
I20240503 02:01:03.042495 3965 tablet_copy_source_session.cc:205] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:03.043411 4405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98895d45cbd64b52b497261821d8ae53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:03.045753 4405 tablet_copy_client.cc:783] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:03.045910 4405 tablet_copy_client.cc:647] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:03.046876 4409 tablet_copy_client.cc:1002] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:03.131630 4411 ts_tablet_manager.cc:902] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee: Initiating tablet copy from peer 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:03.132203 4411 tablet_copy_client.cc:308] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:42879
I20240503 02:01:03.135831 3890 tablet_copy_service.cc:140] P 12e9ea600473443b80086df227e94dfc: Received BeginTabletCopySession request for tablet eaa6c34cc53b4061abb42f32a272a67e from peer 38e9ca6a49c4402abdd585ffd8f435ee ({username='slave'} at 127.0.0.1:56290)
I20240503 02:01:03.135938 3890 tablet_copy_service.cc:161] P 12e9ea600473443b80086df227e94dfc: Beginning new tablet copy session on tablet eaa6c34cc53b4061abb42f32a272a67e from peer 38e9ca6a49c4402abdd585ffd8f435ee at {username='slave'} at 127.0.0.1:56290: session id = 38e9ca6a49c4402abdd585ffd8f435ee-eaa6c34cc53b4061abb42f32a272a67e
I20240503 02:01:03.136958 3890 tablet_copy_source_session.cc:205] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:03.137825 4411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa6c34cc53b4061abb42f32a272a67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:03.140116 4411 tablet_copy_client.cc:783] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:03.140323 4411 tablet_copy_client.cc:647] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:03.141253 4415 tablet_copy_client.cc:1002] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:03.186959 4419 ts_tablet_manager.cc:902] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410: Initiating tablet copy from peer 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:03.187587 4419 tablet_copy_client.cc:308] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:34683
I20240503 02:01:03.191309 3965 tablet_copy_service.cc:140] P 1fe097edfada4dc3b18e7f74ff6dd980: Received BeginTabletCopySession request for tablet b98f5009a3534765806bda2c95f7484c from peer cfdcf4a7a9734e82bddf13e25872e410 ({username='slave'} at 127.0.0.1:50812)
I20240503 02:01:03.191465 3965 tablet_copy_service.cc:161] P 1fe097edfada4dc3b18e7f74ff6dd980: Beginning new tablet copy session on tablet b98f5009a3534765806bda2c95f7484c from peer cfdcf4a7a9734e82bddf13e25872e410 at {username='slave'} at 127.0.0.1:50812: session id = cfdcf4a7a9734e82bddf13e25872e410-b98f5009a3534765806bda2c95f7484c
I20240503 02:01:03.192543 3965 tablet_copy_source_session.cc:205] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:03.193316 4419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98f5009a3534765806bda2c95f7484c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:03.195719 4419 tablet_copy_client.cc:783] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:03.195873 4419 tablet_copy_client.cc:647] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:03.196859 4423 tablet_copy_client.cc:1002] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:03.204643 4425 ts_tablet_manager.cc:902] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee: Initiating tablet copy from peer 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:03.205293 4425 tablet_copy_client.cc:308] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:42879
I20240503 02:01:03.205404 4430 ts_tablet_manager.cc:902] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410: Initiating tablet copy from peer 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193)
I20240503 02:01:03.205799 3890 tablet_copy_service.cc:140] P 12e9ea600473443b80086df227e94dfc: Received BeginTabletCopySession request for tablet 88bdd7e3ba2d4793907feb426e7bef9c from peer 38e9ca6a49c4402abdd585ffd8f435ee ({username='slave'} at 127.0.0.1:56290)
I20240503 02:01:03.205893 3890 tablet_copy_service.cc:161] P 12e9ea600473443b80086df227e94dfc: Beginning new tablet copy session on tablet 88bdd7e3ba2d4793907feb426e7bef9c from peer 38e9ca6a49c4402abdd585ffd8f435ee at {username='slave'} at 127.0.0.1:56290: session id = 38e9ca6a49c4402abdd585ffd8f435ee-88bdd7e3ba2d4793907feb426e7bef9c
I20240503 02:01:03.205929 4430 tablet_copy_client.cc:308] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:39193
I20240503 02:01:03.206888 3890 tablet_copy_source_session.cc:205] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:03.207732 4425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bdd7e3ba2d4793907feb426e7bef9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:03.209962 4425 tablet_copy_client.cc:783] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:03.210109 4425 tablet_copy_client.cc:647] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:03.211390 4427 tablet_copy_client.cc:1002] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:03.213472 3814 tablet_copy_service.cc:140] P 94da6cefbea94c72a5d91b8d10f31bc9: Received BeginTabletCopySession request for tablet 561921bd5097434497ea1f034f86749b from peer cfdcf4a7a9734e82bddf13e25872e410 ({username='slave'} at 127.0.0.1:35326)
I20240503 02:01:03.213559 3814 tablet_copy_service.cc:161] P 94da6cefbea94c72a5d91b8d10f31bc9: Beginning new tablet copy session on tablet 561921bd5097434497ea1f034f86749b from peer cfdcf4a7a9734e82bddf13e25872e410 at {username='slave'} at 127.0.0.1:35326: session id = cfdcf4a7a9734e82bddf13e25872e410-561921bd5097434497ea1f034f86749b
I20240503 02:01:03.214459 3814 tablet_copy_source_session.cc:205] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:03.215206 4430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561921bd5097434497ea1f034f86749b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:03.217224 4430 tablet_copy_client.cc:783] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:03.217378 4430 tablet_copy_client.cc:647] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:03.218408 4431 tablet_copy_client.cc:1002] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:03.225320 4436 ts_tablet_manager.cc:902] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767: Initiating tablet copy from peer 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193)
I20240503 02:01:03.225971 4436 tablet_copy_client.cc:308] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:39193
I20240503 02:01:03.229562 3814 tablet_copy_service.cc:140] P 94da6cefbea94c72a5d91b8d10f31bc9: Received BeginTabletCopySession request for tablet 4f1c79d275004dfeb03f56c07b3904af from peer dc881d7ec7fb47a2897eef434cf18767 ({username='slave'} at 127.0.0.1:35332)
I20240503 02:01:03.229677 3814 tablet_copy_service.cc:161] P 94da6cefbea94c72a5d91b8d10f31bc9: Beginning new tablet copy session on tablet 4f1c79d275004dfeb03f56c07b3904af from peer dc881d7ec7fb47a2897eef434cf18767 at {username='slave'} at 127.0.0.1:35332: session id = dc881d7ec7fb47a2897eef434cf18767-4f1c79d275004dfeb03f56c07b3904af
I20240503 02:01:03.230597 3814 tablet_copy_source_session.cc:205] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:03.231431 4436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1c79d275004dfeb03f56c07b3904af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:03.233549 4436 tablet_copy_client.cc:783] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:03.233708 4436 tablet_copy_client.cc:647] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:03.235155 4437 tablet_copy_client.cc:1002] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:03.641161 3755 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:01:03.642366 3755 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:01:03.642500 3755 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:01:05.047456 4405 tablet_copy_client.cc:523] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:05.048940 4405 tablet_bootstrap.cc:492] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767: Bootstrap starting.
I20240503 02:01:05.069851 4405 tablet_bootstrap.cc:492] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:05.070201 4405 tablet_bootstrap.cc:492] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767: Bootstrap complete.
I20240503 02:01:05.070428 4405 ts_tablet_manager.cc:1367] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.001s
I20240503 02:01:05.070926 4405 raft_consensus.cc:348] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:05.071086 4405 raft_consensus.cc:729] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: dc881d7ec7fb47a2897eef434cf18767, State: Initialized, Role: LEARNER
I20240503 02:01:05.071205 4405 consensus_queue.cc:260] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:05.071547 4405 ts_tablet_manager.cc:1398] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:01:05.071985 3965 tablet_copy_service.cc:342] P 1fe097edfada4dc3b18e7f74ff6dd980: Request end of tablet copy session dc881d7ec7fb47a2897eef434cf18767-98895d45cbd64b52b497261821d8ae53 received from {username='slave'} at 127.0.0.1:50804
I20240503 02:01:05.072060 3965 tablet_copy_service.cc:433] P 1fe097edfada4dc3b18e7f74ff6dd980: ending tablet copy session dc881d7ec7fb47a2897eef434cf18767-98895d45cbd64b52b497261821d8ae53 on tablet 98895d45cbd64b52b497261821d8ae53 with peer dc881d7ec7fb47a2897eef434cf18767
I20240503 02:01:05.142678 4411 tablet_copy_client.cc:523] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:05.144347 4411 tablet_bootstrap.cc:492] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee: Bootstrap starting.
I20240503 02:01:05.165457 4411 tablet_bootstrap.cc:492] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:05.165865 4411 tablet_bootstrap.cc:492] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee: Bootstrap complete.
I20240503 02:01:05.166087 4411 ts_tablet_manager.cc:1367] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240503 02:01:05.166611 4411 raft_consensus.cc:348] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:05.166759 4411 raft_consensus.cc:729] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38e9ca6a49c4402abdd585ffd8f435ee, State: Initialized, Role: LEARNER
I20240503 02:01:05.166872 4411 consensus_queue.cc:260] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:05.167582 4411 ts_tablet_manager.cc:1398] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:01:05.168064 3890 tablet_copy_service.cc:342] P 12e9ea600473443b80086df227e94dfc: Request end of tablet copy session 38e9ca6a49c4402abdd585ffd8f435ee-eaa6c34cc53b4061abb42f32a272a67e received from {username='slave'} at 127.0.0.1:56290
I20240503 02:01:05.168152 3890 tablet_copy_service.cc:433] P 12e9ea600473443b80086df227e94dfc: ending tablet copy session 38e9ca6a49c4402abdd585ffd8f435ee-eaa6c34cc53b4061abb42f32a272a67e on tablet eaa6c34cc53b4061abb42f32a272a67e with peer 38e9ca6a49c4402abdd585ffd8f435ee
I20240503 02:01:05.197913 4419 tablet_copy_client.cc:523] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:05.199587 4419 tablet_bootstrap.cc:492] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410: Bootstrap starting.
I20240503 02:01:05.211829 4425 tablet_copy_client.cc:523] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:05.213548 4425 tablet_bootstrap.cc:492] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee: Bootstrap starting.
I20240503 02:01:05.218940 4430 tablet_copy_client.cc:523] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:05.220469 4430 tablet_bootstrap.cc:492] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410: Bootstrap starting.
I20240503 02:01:05.224901 4419 tablet_bootstrap.cc:492] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:05.225456 4419 tablet_bootstrap.cc:492] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410: Bootstrap complete.
I20240503 02:01:05.225685 4419 ts_tablet_manager.cc:1367] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.003s
I20240503 02:01:05.226428 4419 raft_consensus.cc:348] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:05.226918 4419 raft_consensus.cc:729] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cfdcf4a7a9734e82bddf13e25872e410, State: Initialized, Role: LEARNER
I20240503 02:01:05.227115 4419 consensus_queue.cc:260] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:05.227782 4419 ts_tablet_manager.cc:1398] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:05.231485 3965 tablet_copy_service.cc:342] P 1fe097edfada4dc3b18e7f74ff6dd980: Request end of tablet copy session cfdcf4a7a9734e82bddf13e25872e410-b98f5009a3534765806bda2c95f7484c received from {username='slave'} at 127.0.0.1:50812
I20240503 02:01:05.231581 3965 tablet_copy_service.cc:433] P 1fe097edfada4dc3b18e7f74ff6dd980: ending tablet copy session cfdcf4a7a9734e82bddf13e25872e410-b98f5009a3534765806bda2c95f7484c on tablet b98f5009a3534765806bda2c95f7484c with peer cfdcf4a7a9734e82bddf13e25872e410
I20240503 02:01:05.235584 4436 tablet_copy_client.cc:523] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:05.237109 4436 tablet_bootstrap.cc:492] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767: Bootstrap starting.
I20240503 02:01:05.238224 4425 tablet_bootstrap.cc:492] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:05.238476 4425 tablet_bootstrap.cc:492] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee: Bootstrap complete.
I20240503 02:01:05.238631 4425 ts_tablet_manager.cc:1367] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.000s
I20240503 02:01:05.239140 4425 raft_consensus.cc:348] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:05.239261 4425 raft_consensus.cc:729] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38e9ca6a49c4402abdd585ffd8f435ee, State: Initialized, Role: LEARNER
I20240503 02:01:05.239358 4425 consensus_queue.cc:260] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:05.239650 4425 ts_tablet_manager.cc:1398] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:01:05.240314 3890 tablet_copy_service.cc:342] P 12e9ea600473443b80086df227e94dfc: Request end of tablet copy session 38e9ca6a49c4402abdd585ffd8f435ee-88bdd7e3ba2d4793907feb426e7bef9c received from {username='slave'} at 127.0.0.1:56290
I20240503 02:01:05.240386 3890 tablet_copy_service.cc:433] P 12e9ea600473443b80086df227e94dfc: ending tablet copy session 38e9ca6a49c4402abdd585ffd8f435ee-88bdd7e3ba2d4793907feb426e7bef9c on tablet 88bdd7e3ba2d4793907feb426e7bef9c with peer 38e9ca6a49c4402abdd585ffd8f435ee
I20240503 02:01:05.245774 4430 tablet_bootstrap.cc:492] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:05.246037 4430 tablet_bootstrap.cc:492] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410: Bootstrap complete.
I20240503 02:01:05.246225 4430 ts_tablet_manager.cc:1367] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.006s
I20240503 02:01:05.246757 4430 raft_consensus.cc:348] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:05.246986 4430 raft_consensus.cc:729] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cfdcf4a7a9734e82bddf13e25872e410, State: Initialized, Role: LEARNER
I20240503 02:01:05.247133 4430 consensus_queue.cc:260] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:05.247468 4430 ts_tablet_manager.cc:1398] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:05.247866 3814 tablet_copy_service.cc:342] P 94da6cefbea94c72a5d91b8d10f31bc9: Request end of tablet copy session cfdcf4a7a9734e82bddf13e25872e410-561921bd5097434497ea1f034f86749b received from {username='slave'} at 127.0.0.1:35326
I20240503 02:01:05.247946 3814 tablet_copy_service.cc:433] P 94da6cefbea94c72a5d91b8d10f31bc9: ending tablet copy session cfdcf4a7a9734e82bddf13e25872e410-561921bd5097434497ea1f034f86749b on tablet 561921bd5097434497ea1f034f86749b with peer cfdcf4a7a9734e82bddf13e25872e410
I20240503 02:01:05.259872 4436 tablet_bootstrap.cc:492] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:05.260186 4436 tablet_bootstrap.cc:492] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767: Bootstrap complete.
I20240503 02:01:05.260387 4436 ts_tablet_manager.cc:1367] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.005s
I20240503 02:01:05.260898 4436 raft_consensus.cc:348] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:05.261016 4436 raft_consensus.cc:729] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc881d7ec7fb47a2897eef434cf18767, State: Initialized, Role: LEARNER
I20240503 02:01:05.261111 4436 consensus_queue.cc:260] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:05.261405 4436 ts_tablet_manager.cc:1398] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:01:05.261857 3814 tablet_copy_service.cc:342] P 94da6cefbea94c72a5d91b8d10f31bc9: Request end of tablet copy session dc881d7ec7fb47a2897eef434cf18767-4f1c79d275004dfeb03f56c07b3904af received from {username='slave'} at 127.0.0.1:35332
I20240503 02:01:05.261934 3814 tablet_copy_service.cc:433] P 94da6cefbea94c72a5d91b8d10f31bc9: ending tablet copy session dc881d7ec7fb47a2897eef434cf18767-4f1c79d275004dfeb03f56c07b3904af on tablet 4f1c79d275004dfeb03f56c07b3904af with peer dc881d7ec7fb47a2897eef434cf18767
I20240503 02:01:05.525485 4204 raft_consensus.cc:1203] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 02:01:05.536291 4204 raft_consensus.cc:1203] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 02:01:05.562042 4280 raft_consensus.cc:1203] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.26->[2.27-2.27] Dedup: 2.27->[]
I20240503 02:01:05.581835 4355 raft_consensus.cc:1203] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240503 02:01:05.628297 4355 raft_consensus.cc:1203] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 02:01:05.642776 3755 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:01:05.644517 3755 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 02:01:05.644658 3755 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:01:05.670912 4280 raft_consensus.cc:1203] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20240503 02:01:05.973496 4441 raft_consensus.cc:1050] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980: attempting to promote NON_VOTER dc881d7ec7fb47a2897eef434cf18767 to VOTER
I20240503 02:01:05.973969 4441 consensus_queue.cc:237] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 19, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:05.975301 4280 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [term 2 LEARNER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:01:05.975373 3875 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:01:05.975675 3801 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:01:05.975759 4456 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20240503 02:01:05.976615 4459 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:05.976822 4441 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:05.978809 4459 raft_consensus.cc:2932] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:05.979207 3801 raft_consensus.cc:2932] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:05.979604 3875 raft_consensus.cc:2932] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:05.981736 3701 catalog_manager.cc:5484] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index 27 to 28, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:05.982720 4280 raft_consensus.cc:2932] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:05.986439 3950 consensus_queue.cc:237] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 19, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:05.987648 4280 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [term 2 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:01:05.988039 3875 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:01:05.988468 4458 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:05.988835 4458 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:05.989859 4441 raft_consensus.cc:2932] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:05.990298 3878 raft_consensus.cc:2932] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:05.992293 3686 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98895d45cbd64b52b497261821d8ae53 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:01:05.992482 3697 catalog_manager.cc:5484] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: config changed from index 28 to 29, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New cstate: current_term: 2 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:05.992779 4467 raft_consensus.cc:1050] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc: attempting to promote NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee to VOTER
I20240503 02:01:05.993141 4467 consensus_queue.cc:237] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } }
I20240503 02:01:05.993314 4280 raft_consensus.cc:2932] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:05.994419 4204 raft_consensus.cc:1261] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:05.994915 3950 raft_consensus.cc:1261] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:05.995011 3801 raft_consensus.cc:1261] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:05.995831 4469 consensus_queue.cc:1035] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:05.996397 3794 tablet_service.cc:1514] Processing DeleteTablet for tablet 98895d45cbd64b52b497261821d8ae53 with delete_type TABLET_DATA_TOMBSTONED (TS 94da6cefbea94c72a5d91b8d10f31bc9 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:35292
I20240503 02:01:05.996424 4467 consensus_queue.cc:1035] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:05.996742 4467 consensus_queue.cc:1035] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:05.999135 4471 tablet_replica.cc:330] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9: stopping tablet replica
I20240503 02:01:05.999128 4469 raft_consensus.cc:2932] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:05.999348 4471 raft_consensus.cc:2229] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:05.999503 4471 raft_consensus.cc:2258] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:05.999444 4204 raft_consensus.cc:2932] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.000218 4471 ts_tablet_manager.cc:1875] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:06.001442 3697 catalog_manager.cc:5484] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee reported cstate change: config changed from index 27 to 28, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.001490 3950 raft_consensus.cc:2932] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.002511 3801 raft_consensus.cc:2932] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.004436 4471 ts_tablet_manager.cc:1888] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.28
I20240503 02:01:06.004503 4471 log.cc:1196] T 98895d45cbd64b52b497261821d8ae53 P 94da6cefbea94c72a5d91b8d10f31bc9: Deleting WAL directory at /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-0-root/wals/98895d45cbd64b52b497261821d8ae53
I20240503 02:01:06.005072 3687 catalog_manager.cc:4830] TS 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193): tablet 98895d45cbd64b52b497261821d8ae53 (table test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]) successfully deleted
I20240503 02:01:06.007091 3878 consensus_queue.cc:237] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } }
I20240503 02:01:06.008141 4204 raft_consensus.cc:1261] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:01:06.008435 4467 consensus_queue.cc:1035] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:06.008726 3801 raft_consensus.cc:1261] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:01:06.008998 4470 consensus_queue.cc:1035] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:06.009979 4467 raft_consensus.cc:2932] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.010653 3802 raft_consensus.cc:2932] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.010819 4204 raft_consensus.cc:2932] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.011520 3687 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 88bdd7e3ba2d4793907feb426e7bef9c with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:01:06.012550 3696 catalog_manager.cc:5484] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc reported cstate change: config changed from index 28 to 29, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:06.014926 3940 tablet_service.cc:1514] Processing DeleteTablet for tablet 88bdd7e3ba2d4793907feb426e7bef9c with delete_type TABLET_DATA_TOMBSTONED (TS 1fe097edfada4dc3b18e7f74ff6dd980 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:50756
I20240503 02:01:06.015399 4475 tablet_replica.cc:330] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980: stopping tablet replica
I20240503 02:01:06.015583 4475 raft_consensus.cc:2229] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:06.015795 4475 raft_consensus.cc:2258] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:06.016399 4475 ts_tablet_manager.cc:1875] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:06.019115 4475 ts_tablet_manager.cc:1888] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 02:01:06.019210 4475 log.cc:1196] T 88bdd7e3ba2d4793907feb426e7bef9c P 1fe097edfada4dc3b18e7f74ff6dd980: Deleting WAL directory at /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-2-root/wals/88bdd7e3ba2d4793907feb426e7bef9c
I20240503 02:01:06.019855 3686 catalog_manager.cc:4830] TS 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683): tablet 88bdd7e3ba2d4793907feb426e7bef9c (table test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]) successfully deleted
I20240503 02:01:06.097092 4444 raft_consensus.cc:1050] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9: attempting to promote NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 to VOTER
I20240503 02:01:06.097527 4444 consensus_queue.cc:237] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } }
I20240503 02:01:06.098780 4355 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:06.098909 3878 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:06.099200 3950 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:06.099308 4444 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:06.099634 4478 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:06.099996 4479 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:06.101857 4444 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.102458 4355 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.103152 3950 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.104663 3701 catalog_manager.cc:5484] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index 27 to 28, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:06.105407 3878 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.108723 3801 consensus_queue.cc:237] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } }
I20240503 02:01:06.109697 4355 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:01:06.109697 3950 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:01:06.110054 4476 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:06.110229 4478 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:06.111860 4476 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.112327 3947 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.113582 3687 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 561921bd5097434497ea1f034f86749b with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:01:06.113884 4355 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.114562 3696 catalog_manager.cc:5484] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: config changed from index 28 to 29, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.116778 3870 tablet_service.cc:1514] Processing DeleteTablet for tablet 561921bd5097434497ea1f034f86749b with delete_type TABLET_DATA_TOMBSTONED (TS 12e9ea600473443b80086df227e94dfc not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:56232
I20240503 02:01:06.117201 4483 tablet_replica.cc:330] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc: stopping tablet replica
I20240503 02:01:06.117365 4483 raft_consensus.cc:2229] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:06.117550 4483 raft_consensus.cc:2258] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:06.118224 4483 ts_tablet_manager.cc:1875] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:06.120409 4483 ts_tablet_manager.cc:1888] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 02:01:06.120496 4483 log.cc:1196] T 561921bd5097434497ea1f034f86749b P 12e9ea600473443b80086df227e94dfc: Deleting WAL directory at /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-1-root/wals/561921bd5097434497ea1f034f86749b
I20240503 02:01:06.121132 3687 catalog_manager.cc:4830] TS 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879): tablet 561921bd5097434497ea1f034f86749b (table test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]) successfully deleted
I20240503 02:01:06.148617 4469 raft_consensus.cc:1050] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc: attempting to promote NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee to VOTER
I20240503 02:01:06.149036 4469 consensus_queue.cc:237] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } }
I20240503 02:01:06.150429 3802 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:06.150429 3947 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:06.150429 4204 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:06.150827 4469 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:06.151093 4467 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:06.151278 4470 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:06.153141 4467 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.153597 3802 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.153807 3947 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.155390 4204 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.156188 3695 catalog_manager.cc:5484] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index 27 to 28, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.159340 3878 consensus_queue.cc:237] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } }
I20240503 02:01:06.160274 3802 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:01:06.160632 4469 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:06.160883 4204 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:01:06.161293 4469 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:06.162262 4470 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.162642 3801 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.163839 3687 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eaa6c34cc53b4061abb42f32a272a67e with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:01:06.164744 3701 catalog_manager.cc:5484] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc reported cstate change: config changed from index 28 to 29, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:06.165267 4204 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:06.167217 3940 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa6c34cc53b4061abb42f32a272a67e with delete_type TABLET_DATA_TOMBSTONED (TS 1fe097edfada4dc3b18e7f74ff6dd980 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:50756
I20240503 02:01:06.167438 4475 tablet_replica.cc:330] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980: stopping tablet replica
I20240503 02:01:06.167591 4475 raft_consensus.cc:2229] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:06.167717 4475 raft_consensus.cc:2258] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:06.168381 4475 ts_tablet_manager.cc:1875] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:06.170011 4475 ts_tablet_manager.cc:1888] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 02:01:06.170083 4475 log.cc:1196] T eaa6c34cc53b4061abb42f32a272a67e P 1fe097edfada4dc3b18e7f74ff6dd980: Deleting WAL directory at /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-2-root/wals/eaa6c34cc53b4061abb42f32a272a67e
I20240503 02:01:06.170593 3686 catalog_manager.cc:4830] TS 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683): tablet eaa6c34cc53b4061abb42f32a272a67e (table test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]) successfully deleted
I20240503 02:01:06.179386 4458 raft_consensus.cc:1050] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980: attempting to promote NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 to VOTER
I20240503 02:01:06.179926 4458 consensus_queue.cc:237] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } }
I20240503 02:01:06.181443 4355 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:06.181495 3878 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:06.181871 4459 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:06.181948 3802 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:06.182060 4441 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:06.182611 4441 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:06.184330 4456 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.184649 3802 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.186262 4355 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.186746 3878 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.187863 3701 catalog_manager.cc:5484] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: config changed from index 26 to 27, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:06.191356 3947 consensus_queue.cc:237] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } }
I20240503 02:01:06.192469 4355 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:06.192503 3878 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:06.192849 4441 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:06.193024 4458 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:06.194432 4441 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.195259 4355 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.196223 3686 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b98f5009a3534765806bda2c95f7484c with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:01:06.196588 3875 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:06.196873 3695 catalog_manager.cc:5484] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: config changed from index 27 to 28, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:06.199795 3794 tablet_service.cc:1514] Processing DeleteTablet for tablet b98f5009a3534765806bda2c95f7484c with delete_type TABLET_DATA_TOMBSTONED (TS 94da6cefbea94c72a5d91b8d10f31bc9 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:35292
I20240503 02:01:06.200054 4471 tablet_replica.cc:330] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9: stopping tablet replica
I20240503 02:01:06.200212 4471 raft_consensus.cc:2229] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:06.200368 4471 raft_consensus.cc:2258] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:06.201467 4471 ts_tablet_manager.cc:1875] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:06.203426 4471 ts_tablet_manager.cc:1888] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240503 02:01:06.203513 4471 log.cc:1196] T b98f5009a3534765806bda2c95f7484c P 94da6cefbea94c72a5d91b8d10f31bc9: Deleting WAL directory at /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-0-root/wals/b98f5009a3534765806bda2c95f7484c
I20240503 02:01:06.203953 3687 catalog_manager.cc:4830] TS 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193): tablet b98f5009a3534765806bda2c95f7484c (table test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]) successfully deleted
I20240503 02:01:06.252303 4476 raft_consensus.cc:1050] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9: attempting to promote NON_VOTER dc881d7ec7fb47a2897eef434cf18767 to VOTER
I20240503 02:01:06.252799 4476 consensus_queue.cc:237] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:06.254235 3878 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 02:01:06.254235 3947 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 02:01:06.254235 4280 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 1 LEARNER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 02:01:06.254663 4444 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 02:01:06.254858 4476 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 02:01:06.255141 4479 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 02:01:06.256747 4476 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:06.257314 3878 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:06.257354 3947 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:06.258663 4280 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:06.259824 3697 catalog_manager.cc:5484] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc reported cstate change: config changed from index 25 to 26, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:06.263175 3801 consensus_queue.cc:237] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:06.264143 4280 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:06.264192 3947 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:06.264510 4479 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:06.264650 4444 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:06.266100 4478 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:06.266510 3947 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:06.267506 4280 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:06.268159 3696 catalog_manager.cc:5484] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index 26 to 27, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:06.268815 3687 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4f1c79d275004dfeb03f56c07b3904af with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:01:06.270604 3870 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1c79d275004dfeb03f56c07b3904af with delete_type TABLET_DATA_TOMBSTONED (TS 12e9ea600473443b80086df227e94dfc not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:56232
I20240503 02:01:06.270820 4483 tablet_replica.cc:330] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc: stopping tablet replica
I20240503 02:01:06.270952 4483 raft_consensus.cc:2229] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:06.271107 4483 raft_consensus.cc:2258] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:06.271736 4483 ts_tablet_manager.cc:1875] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:06.273381 4483 ts_tablet_manager.cc:1888] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20240503 02:01:06.273440 4483 log.cc:1196] T 4f1c79d275004dfeb03f56c07b3904af P 12e9ea600473443b80086df227e94dfc: Deleting WAL directory at /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-1-root/wals/4f1c79d275004dfeb03f56c07b3904af
I20240503 02:01:06.273881 3687 catalog_manager.cc:4830] TS 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879): tablet 4f1c79d275004dfeb03f56c07b3904af (table test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]) successfully deleted
I20240503 02:01:07.273613 3802 consensus_queue.cc:237] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:07.275082 3878 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:07.275353 3947 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:07.275457 4479 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:07.275751 4498 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:07.277464 4479 raft_consensus.cc:2932] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:07.277909 3947 raft_consensus.cc:2932] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:07.278012 3878 raft_consensus.cc:2932] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
W20240503 02:01:07.279353 3768 consensus_peers.cc:485] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 -> Peer 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4:34437): Couldn't send request to peer 38e9ca6a49c4402abdd585ffd8f435ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cfd82671b084308af19ee0d7b151cd5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:07.280416 3701 catalog_manager.cc:5484] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc reported cstate change: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:07.281960 3802 consensus_queue.cc:237] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
W20240503 02:01:07.283429 3768 consensus_peers.cc:485] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 -> Peer dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5:42349): Couldn't send request to peer dc881d7ec7fb47a2897eef434cf18767. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 561921bd5097434497ea1f034f86749b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:07.283429 4355 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 02:01:07.283818 4496 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:01:07.283879 3947 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 02:01:07.284340 4496 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:01:07.285694 4498 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:07.286250 4355 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:07.287004 3947 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:07.288388 3696 catalog_manager.cc:5484] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 reported cstate change: config changed from index 29 to 30, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New cstate: current_term: 1 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:07.289368 3950 consensus_queue.cc:237] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:07.290828 3878 raft_consensus.cc:1261] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20240503 02:01:07.290855 3920 consensus_peers.cc:485] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 -> Peer cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6:38521): Couldn't send request to peer cfdcf4a7a9734e82bddf13e25872e410. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80f0e3a5143d4bd7a132ee4b6fcb6f89. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:07.291276 4512 consensus_queue.cc:1035] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:07.291386 3802 raft_consensus.cc:1261] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:07.291755 4512 consensus_queue.cc:1035] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:07.293452 4513 raft_consensus.cc:2932] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:07.293779 3878 raft_consensus.cc:2932] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:07.294158 3802 raft_consensus.cc:2932] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:07.296248 3695 catalog_manager.cc:5484] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc reported cstate change: config changed from index -1 to 27, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New cstate: current_term: 1 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:07.296854 3878 consensus_queue.cc:237] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:07.298790 3802 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 02:01:07.299160 4204 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240503 02:01:07.299489 4091 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:01:07.299932 4517 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:01:07.301275 4517 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:07.303525 3696 catalog_manager.cc:5484] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc reported cstate change: config changed from index 29 to 30, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:07.303526 3801 consensus_queue.cc:237] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:07.303851 4204 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
W20240503 02:01:07.303695 3844 consensus_peers.cc:485] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc -> Peer dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5:42349): Couldn't send request to peer dc881d7ec7fb47a2897eef434cf18767. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eaa6c34cc53b4061abb42f32a272a67e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:07.304427 3802 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:07.305487 3950 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
W20240503 02:01:07.305850 3769 consensus_peers.cc:485] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 -> Peer cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6:38521): Couldn't send request to peer cfdcf4a7a9734e82bddf13e25872e410. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f1c79d275004dfeb03f56c07b3904af. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:07.306154 4280 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 1 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:07.306458 4479 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:07.306632 4496 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:07.308665 4479 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:07.309460 3950 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:07.310628 4280 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:07.311722 3701 catalog_manager.cc:5484] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: config changed from index 27 to 28, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New cstate: current_term: 1 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:07.313035 3950 consensus_queue.cc:237] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:07.315248 4355 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:01:07.315294 3878 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:01:07.315757 4456 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:07.315904 4512 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:07.317569 4456 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:07.317870 4355 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:07.318004 3878 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
W20240503 02:01:07.320293 3919 consensus_peers.cc:485] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 -> Peer 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4:34437): Couldn't send request to peer 38e9ca6a49c4402abdd585ffd8f435ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b98f5009a3534765806bda2c95f7484c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:07.320441 3695 catalog_manager.cc:5484] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc reported cstate change: config changed from index 28 to 29, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:07.644934 3755 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:01:07.655475 3802 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472"
dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
mode: GRACEFUL
new_leader_uuid: "12e9ea600473443b80086df227e94dfc"
from {username='slave'} at 127.0.0.1:54964
I20240503 02:01:07.655651 3802 raft_consensus.cc:595] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 LEADER]: Received request to transfer leadership to TS 12e9ea600473443b80086df227e94dfc
I20240503 02:01:07.656385 3801 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "561921bd5097434497ea1f034f86749b"
dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
mode: GRACEFUL
new_leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
from {username='slave'} at 127.0.0.1:54964
I20240503 02:01:07.656515 3801 raft_consensus.cc:595] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Received request to transfer leadership to TS 1fe097edfada4dc3b18e7f74ff6dd980
I20240503 02:01:07.657176 3878 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "88bdd7e3ba2d4793907feb426e7bef9c"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
mode: GRACEFUL
new_leader_uuid: "38e9ca6a49c4402abdd585ffd8f435ee"
from {username='slave'} at 127.0.0.1:56270
I20240503 02:01:07.657289 3878 raft_consensus.cc:595] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Received request to transfer leadership to TS 38e9ca6a49c4402abdd585ffd8f435ee
I20240503 02:01:07.662346 3947 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53"
dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
mode: GRACEFUL
new_leader_uuid: "dc881d7ec7fb47a2897eef434cf18767"
from {username='slave'} at 127.0.0.1:54834
I20240503 02:01:07.662499 3947 raft_consensus.cc:595] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Received request to transfer leadership to TS dc881d7ec7fb47a2897eef434cf18767
I20240503 02:01:07.663268 3947 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b98f5009a3534765806bda2c95f7484c"
dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
mode: GRACEFUL
new_leader_uuid: "cfdcf4a7a9734e82bddf13e25872e410"
from {username='slave'} at 127.0.0.1:54834
I20240503 02:01:07.663388 3947 raft_consensus.cc:595] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Received request to transfer leadership to TS cfdcf4a7a9734e82bddf13e25872e410
I20240503 02:01:07.664057 3878 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8d5be30376a4040a650e1be3d6643f8"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
mode: GRACEFUL
new_leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
from {username='slave'} at 127.0.0.1:56270
I20240503 02:01:07.664160 3878 raft_consensus.cc:595] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Received request to transfer leadership to TS 94da6cefbea94c72a5d91b8d10f31bc9
I20240503 02:01:07.664434 3755 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240503 02:01:07.664573 3755 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:01:07.687274 4537 ts_tablet_manager.cc:902] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767: Initiating tablet copy from peer 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193)
I20240503 02:01:07.687935 4537 tablet_copy_client.cc:308] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:39193
I20240503 02:01:07.688318 3814 tablet_copy_service.cc:140] P 94da6cefbea94c72a5d91b8d10f31bc9: Received BeginTabletCopySession request for tablet 561921bd5097434497ea1f034f86749b from peer dc881d7ec7fb47a2897eef434cf18767 ({username='slave'} at 127.0.0.1:35332)
I20240503 02:01:07.688449 3814 tablet_copy_service.cc:161] P 94da6cefbea94c72a5d91b8d10f31bc9: Beginning new tablet copy session on tablet 561921bd5097434497ea1f034f86749b from peer dc881d7ec7fb47a2897eef434cf18767 at {username='slave'} at 127.0.0.1:35332: session id = dc881d7ec7fb47a2897eef434cf18767-561921bd5097434497ea1f034f86749b
I20240503 02:01:07.689253 3814 tablet_copy_source_session.cc:205] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:07.690085 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561921bd5097434497ea1f034f86749b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:07.692421 4537 tablet_copy_client.cc:783] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:07.692574 4537 tablet_copy_client.cc:647] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:07.693742 4541 tablet_copy_client.cc:1002] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:07.707579 4542 ts_tablet_manager.cc:902] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee: Initiating tablet copy from peer 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:07.708202 4542 tablet_copy_client.cc:308] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:34683
I20240503 02:01:07.711972 3965 tablet_copy_service.cc:140] P 1fe097edfada4dc3b18e7f74ff6dd980: Received BeginTabletCopySession request for tablet b98f5009a3534765806bda2c95f7484c from peer 38e9ca6a49c4402abdd585ffd8f435ee ({username='slave'} at 127.0.0.1:54842)
I20240503 02:01:07.712108 3965 tablet_copy_service.cc:161] P 1fe097edfada4dc3b18e7f74ff6dd980: Beginning new tablet copy session on tablet b98f5009a3534765806bda2c95f7484c from peer 38e9ca6a49c4402abdd585ffd8f435ee at {username='slave'} at 127.0.0.1:54842: session id = 38e9ca6a49c4402abdd585ffd8f435ee-b98f5009a3534765806bda2c95f7484c
I20240503 02:01:07.712425 4548 ts_tablet_manager.cc:902] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410: Initiating tablet copy from peer 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:07.713030 4548 tablet_copy_client.cc:308] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:34683
I20240503 02:01:07.713102 3965 tablet_copy_source_session.cc:205] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:07.713398 3964 tablet_copy_service.cc:140] P 1fe097edfada4dc3b18e7f74ff6dd980: Received BeginTabletCopySession request for tablet 80f0e3a5143d4bd7a132ee4b6fcb6f89 from peer cfdcf4a7a9734e82bddf13e25872e410 ({username='slave'} at 127.0.0.1:50812)
I20240503 02:01:07.713490 3964 tablet_copy_service.cc:161] P 1fe097edfada4dc3b18e7f74ff6dd980: Beginning new tablet copy session on tablet 80f0e3a5143d4bd7a132ee4b6fcb6f89 from peer cfdcf4a7a9734e82bddf13e25872e410 at {username='slave'} at 127.0.0.1:50812: session id = cfdcf4a7a9734e82bddf13e25872e410-80f0e3a5143d4bd7a132ee4b6fcb6f89
I20240503 02:01:07.713855 4542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98f5009a3534765806bda2c95f7484c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:07.714500 3964 tablet_copy_source_session.cc:205] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:07.715194 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80f0e3a5143d4bd7a132ee4b6fcb6f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:07.716104 4542 tablet_copy_client.cc:783] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:07.716260 4542 tablet_copy_client.cc:647] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:07.717350 4546 tablet_copy_client.cc:1002] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:07.717371 4548 tablet_copy_client.cc:783] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:07.717530 4548 tablet_copy_client.cc:647] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:07.717701 4553 ts_tablet_manager.cc:902] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410: Initiating tablet copy from peer 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193)
I20240503 02:01:07.718379 4553 tablet_copy_client.cc:308] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:39193
I20240503 02:01:07.718559 4552 tablet_copy_client.cc:1002] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:07.718772 3814 tablet_copy_service.cc:140] P 94da6cefbea94c72a5d91b8d10f31bc9: Received BeginTabletCopySession request for tablet 4f1c79d275004dfeb03f56c07b3904af from peer cfdcf4a7a9734e82bddf13e25872e410 ({username='slave'} at 127.0.0.1:35326)
I20240503 02:01:07.718854 3814 tablet_copy_service.cc:161] P 94da6cefbea94c72a5d91b8d10f31bc9: Beginning new tablet copy session on tablet 4f1c79d275004dfeb03f56c07b3904af from peer cfdcf4a7a9734e82bddf13e25872e410 at {username='slave'} at 127.0.0.1:35326: session id = cfdcf4a7a9734e82bddf13e25872e410-4f1c79d275004dfeb03f56c07b3904af
I20240503 02:01:07.719609 3814 tablet_copy_source_session.cc:205] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:07.720376 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1c79d275004dfeb03f56c07b3904af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:07.722334 4553 tablet_copy_client.cc:783] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:07.722491 4553 tablet_copy_client.cc:647] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:07.723532 4557 tablet_copy_client.cc:1002] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:07.759532 4558 ts_tablet_manager.cc:902] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee: Initiating tablet copy from peer 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193)
I20240503 02:01:07.760195 4558 tablet_copy_client.cc:308] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:39193
I20240503 02:01:07.763654 3814 tablet_copy_service.cc:140] P 94da6cefbea94c72a5d91b8d10f31bc9: Received BeginTabletCopySession request for tablet 5cfd82671b084308af19ee0d7b151cd5 from peer 38e9ca6a49c4402abdd585ffd8f435ee ({username='slave'} at 127.0.0.1:54966)
I20240503 02:01:07.763767 3814 tablet_copy_service.cc:161] P 94da6cefbea94c72a5d91b8d10f31bc9: Beginning new tablet copy session on tablet 5cfd82671b084308af19ee0d7b151cd5 from peer 38e9ca6a49c4402abdd585ffd8f435ee at {username='slave'} at 127.0.0.1:54966: session id = 38e9ca6a49c4402abdd585ffd8f435ee-5cfd82671b084308af19ee0d7b151cd5
I20240503 02:01:07.764779 3814 tablet_copy_source_session.cc:205] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:07.765605 4558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cfd82671b084308af19ee0d7b151cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:07.766276 4091 raft_consensus.cc:979] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc: : Instructing follower 94da6cefbea94c72a5d91b8d10f31bc9 to start an election
I20240503 02:01:07.766376 4091 raft_consensus.cc:1067] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Signalling peer 94da6cefbea94c72a5d91b8d10f31bc9 to start an election
I20240503 02:01:07.766937 3801 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8d5be30376a4040a650e1be3d6643f8"
dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
from {username='slave'} at 127.0.0.1:35300
I20240503 02:01:07.767102 3801 raft_consensus.cc:482] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:07.767167 3801 raft_consensus.cc:3037] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.768186 4558 tablet_copy_client.cc:783] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:07.768167 3801 raft_consensus.cc:504] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:07.768350 4558 tablet_copy_client.cc:647] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:07.768548 3801 leader_election.cc:288] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 2 election: Requested vote from peers 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:07.768749 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d5be30376a4040a650e1be3d6643f8" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:07.768857 3878 raft_consensus.cc:3032] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:07.768929 3878 raft_consensus.cc:729] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:07.769068 3878 consensus_queue.cc:260] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:07.769168 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8d5be30376a4040a650e1be3d6643f8" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:07.769222 3878 raft_consensus.cc:3037] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.769268 3950 raft_consensus.cc:3037] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.769963 3878 raft_consensus.cc:2454] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 2.
I20240503 02:01:07.769963 3950 raft_consensus.cc:2454] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 2.
I20240503 02:01:07.770248 3768 leader_election.cc:302] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:07.770483 4528 raft_consensus.cc:2781] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:07.770577 4528 raft_consensus.cc:686] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 LEADER]: Becoming Leader. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Running, Role: LEADER
I20240503 02:01:07.770735 4528 consensus_queue.cc:237] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:07.771674 4562 tablet_copy_client.cc:1002] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:07.772432 3701 catalog_manager.cc:5484] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: term changed from 1 to 2, leader changed from 12e9ea600473443b80086df227e94dfc (127.0.99.2) to 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:07.775635 4512 raft_consensus.cc:979] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980: : Instructing follower cfdcf4a7a9734e82bddf13e25872e410 to start an election
I20240503 02:01:07.775738 4456 raft_consensus.cc:1067] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Signalling peer cfdcf4a7a9734e82bddf13e25872e410 to start an election
I20240503 02:01:07.776170 4355 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b98f5009a3534765806bda2c95f7484c"
dest_uuid: "cfdcf4a7a9734e82bddf13e25872e410"
from {username='slave'} at 127.0.0.1:34700
I20240503 02:01:07.776288 4355 raft_consensus.cc:482] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:07.776355 4355 raft_consensus.cc:3037] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.777271 4355 raft_consensus.cc:504] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:07.777796 4355 leader_election.cc:288] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [CANDIDATE]: Term 2 election: Requested vote from peers 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:07.779039 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98f5009a3534765806bda2c95f7484c" candidate_uuid: "cfdcf4a7a9734e82bddf13e25872e410" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:07.779183 3950 raft_consensus.cc:3032] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:07.779340 3950 raft_consensus.cc:729] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Running, Role: LEADER
I20240503 02:01:07.779479 3950 consensus_queue.cc:260] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:07.779668 3950 raft_consensus.cc:3037] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.780750 3950 raft_consensus.cc:2454] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfdcf4a7a9734e82bddf13e25872e410 in term 2.
I20240503 02:01:07.781098 4319 leader_election.cc:302] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980, cfdcf4a7a9734e82bddf13e25872e410; no voters:
I20240503 02:01:07.781337 4510 raft_consensus.cc:2781] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:07.781529 4510 raft_consensus.cc:686] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 2 LEADER]: Becoming Leader. State: Replica: cfdcf4a7a9734e82bddf13e25872e410, State: Running, Role: LEADER
I20240503 02:01:07.781937 4510 consensus_queue.cc:237] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:07.784031 3701 catalog_manager.cc:5484] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 reported cstate change: term changed from 1 to 2, leader changed from 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) to cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6). New cstate: current_term: 2 leader_uuid: "cfdcf4a7a9734e82bddf13e25872e410" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:07.788055 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98f5009a3534765806bda2c95f7484c" candidate_uuid: "cfdcf4a7a9734e82bddf13e25872e410" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:07.788184 3878 raft_consensus.cc:3037] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.789157 3878 raft_consensus.cc:2454] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfdcf4a7a9734e82bddf13e25872e410 in term 2.
I20240503 02:01:07.863368 4527 raft_consensus.cc:979] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9: : Instructing follower 1fe097edfada4dc3b18e7f74ff6dd980 to start an election
I20240503 02:01:07.863494 4527 raft_consensus.cc:1067] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Signalling peer 1fe097edfada4dc3b18e7f74ff6dd980 to start an election
I20240503 02:01:07.863859 3950 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "561921bd5097434497ea1f034f86749b"
dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
from {username='slave'} at 127.0.0.1:50784
I20240503 02:01:07.863979 3950 raft_consensus.cc:482] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:07.864058 3950 raft_consensus.cc:3037] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.864905 3950 raft_consensus.cc:504] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:07.865236 3950 leader_election.cc:288] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 2 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6:38521)
I20240503 02:01:07.865564 3801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561921bd5097434497ea1f034f86749b" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:07.865564 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561921bd5097434497ea1f034f86749b" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "cfdcf4a7a9734e82bddf13e25872e410"
I20240503 02:01:07.865703 4355 raft_consensus.cc:3037] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.865703 3801 raft_consensus.cc:3032] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:07.865787 3801 raft_consensus.cc:729] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Running, Role: LEADER
I20240503 02:01:07.865937 3801 consensus_queue.cc:260] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:07.866128 3801 raft_consensus.cc:3037] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.866739 4355 raft_consensus.cc:2454] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 2.
I20240503 02:01:07.867025 3920 leader_election.cc:302] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980, cfdcf4a7a9734e82bddf13e25872e410; no voters:
I20240503 02:01:07.867127 3801 raft_consensus.cc:2454] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 2.
I20240503 02:01:07.867204 4456 raft_consensus.cc:2781] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:07.867305 4456 raft_consensus.cc:686] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Becoming Leader. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Running, Role: LEADER
I20240503 02:01:07.867442 4456 consensus_queue.cc:237] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:07.868544 4569 ts_tablet_manager.cc:902] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767: Initiating tablet copy from peer 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:07.869302 3701 catalog_manager.cc:5484] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: term changed from 1 to 2, leader changed from 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) to 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3). New cstate: current_term: 2 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:07.870028 4569 tablet_copy_client.cc:308] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:42879
I20240503 02:01:07.873404 3950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "561921bd5097434497ea1f034f86749b"
dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50756
I20240503 02:01:07.873543 3950 raft_consensus.cc:595] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Received request to transfer leadership
I20240503 02:01:07.874073 3890 tablet_copy_service.cc:140] P 12e9ea600473443b80086df227e94dfc: Received BeginTabletCopySession request for tablet eaa6c34cc53b4061abb42f32a272a67e from peer dc881d7ec7fb47a2897eef434cf18767 ({username='slave'} at 127.0.0.1:50994)
I20240503 02:01:07.874156 3890 tablet_copy_service.cc:161] P 12e9ea600473443b80086df227e94dfc: Beginning new tablet copy session on tablet eaa6c34cc53b4061abb42f32a272a67e from peer dc881d7ec7fb47a2897eef434cf18767 at {username='slave'} at 127.0.0.1:50994: session id = dc881d7ec7fb47a2897eef434cf18767-eaa6c34cc53b4061abb42f32a272a67e
I20240503 02:01:07.875152 3890 tablet_copy_source_session.cc:205] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:07.875880 4569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa6c34cc53b4061abb42f32a272a67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:07.877884 4569 tablet_copy_client.cc:783] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:07.878057 4569 tablet_copy_client.cc:647] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:07.879155 4573 tablet_copy_client.cc:1002] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:07.879388 3950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "561921bd5097434497ea1f034f86749b"
dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50756
I20240503 02:01:07.879490 3950 raft_consensus.cc:595] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Received request to transfer leadership
W20240503 02:01:07.879742 3686 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 561921bd5097434497ea1f034f86749b already in progress [suppressed 11 similar messages]
I20240503 02:01:07.880384 3950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "561921bd5097434497ea1f034f86749b"
dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50756
I20240503 02:01:07.880479 3950 raft_consensus.cc:595] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Received request to transfer leadership
W20240503 02:01:07.880816 3750 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 561921bd5097434497ea1f034f86749b already in progress
W20240503 02:01:07.880883 3750 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 561921bd5097434497ea1f034f86749b already in progress
I20240503 02:01:07.930946 4528 raft_consensus.cc:979] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9: : Instructing follower 12e9ea600473443b80086df227e94dfc to start an election
I20240503 02:01:07.931095 4528 raft_consensus.cc:1067] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 LEADER]: Signalling peer 12e9ea600473443b80086df227e94dfc to start an election
I20240503 02:01:07.931480 3878 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
from {username='slave'} at 127.0.0.1:56248
I20240503 02:01:07.931608 3878 raft_consensus.cc:482] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:07.931679 3878 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:07.932721 3878 raft_consensus.cc:504] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:07.933112 3878 leader_election.cc:288] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 3 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:07.933341 3802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:07.933439 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:07.933470 3802 raft_consensus.cc:3032] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:01:07.933539 3802 raft_consensus.cc:729] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Running, Role: LEADER
I20240503 02:01:07.933542 3950 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:07.933707 3802 consensus_queue.cc:260] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:07.933889 3802 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:07.934644 3950 raft_consensus.cc:2454] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 3.
I20240503 02:01:07.934806 3802 raft_consensus.cc:2454] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 3.
I20240503 02:01:07.934931 3844 leader_election.cc:302] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 1fe097edfada4dc3b18e7f74ff6dd980; no voters:
I20240503 02:01:07.935161 4518 raft_consensus.cc:2781] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:01:07.935254 4518 raft_consensus.cc:686] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 3 LEADER]: Becoming Leader. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:07.935422 4518 consensus_queue.cc:237] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:07.937150 3701 catalog_manager.cc:5484] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc reported cstate change: term changed from 2 to 3, leader changed from 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) to 12e9ea600473443b80086df227e94dfc (127.0.99.2). New cstate: current_term: 3 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:07.943531 4518 raft_consensus.cc:979] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc: : Instructing follower 38e9ca6a49c4402abdd585ffd8f435ee to start an election
I20240503 02:01:07.943661 4518 raft_consensus.cc:1067] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Signalling peer 38e9ca6a49c4402abdd585ffd8f435ee to start an election
I20240503 02:01:07.944048 4204 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "88bdd7e3ba2d4793907feb426e7bef9c"
dest_uuid: "38e9ca6a49c4402abdd585ffd8f435ee"
from {username='slave'} at 127.0.0.1:46510
I20240503 02:01:07.944197 4204 raft_consensus.cc:482] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:07.944273 4204 raft_consensus.cc:3037] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.945223 4204 raft_consensus.cc:504] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } }
I20240503 02:01:07.945662 4204 leader_election.cc:288] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [CANDIDATE]: Term 2 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:07.945902 3802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bdd7e3ba2d4793907feb426e7bef9c" candidate_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:07.946046 3802 raft_consensus.cc:3037] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.946022 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bdd7e3ba2d4793907feb426e7bef9c" candidate_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:07.946197 3878 raft_consensus.cc:3032] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:07.946251 3878 raft_consensus.cc:729] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:07.946375 3878 consensus_queue.cc:260] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } }
I20240503 02:01:07.946540 3878 raft_consensus.cc:3037] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.947149 3802 raft_consensus.cc:2454] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e9ca6a49c4402abdd585ffd8f435ee in term 2.
I20240503 02:01:07.947422 3878 raft_consensus.cc:2454] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e9ca6a49c4402abdd585ffd8f435ee in term 2.
I20240503 02:01:07.947429 4169 leader_election.cc:302] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e9ca6a49c4402abdd585ffd8f435ee, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:07.947826 4577 raft_consensus.cc:2781] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:07.948046 4577 raft_consensus.cc:686] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [term 2 LEADER]: Becoming Leader. State: Replica: 38e9ca6a49c4402abdd585ffd8f435ee, State: Running, Role: LEADER
I20240503 02:01:07.948192 4577 consensus_queue.cc:237] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } }
I20240503 02:01:07.949647 3701 catalog_manager.cc:5484] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee reported cstate change: term changed from 1 to 2, leader changed from 12e9ea600473443b80086df227e94dfc (127.0.99.2) to 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4). New cstate: current_term: 2 leader_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:08.038421 4512 raft_consensus.cc:979] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980: : Instructing follower dc881d7ec7fb47a2897eef434cf18767 to start an election
I20240503 02:01:08.038578 4456 raft_consensus.cc:1067] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Signalling peer dc881d7ec7fb47a2897eef434cf18767 to start an election
I20240503 02:01:08.039011 4280 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53"
dest_uuid: "dc881d7ec7fb47a2897eef434cf18767"
from {username='slave'} at 127.0.0.1:45868
I20240503 02:01:08.039134 4280 raft_consensus.cc:482] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:08.039201 4280 raft_consensus.cc:3037] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:08.040123 4280 raft_consensus.cc:504] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:08.040541 4280 leader_election.cc:288] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [CANDIDATE]: Term 3 election: Requested vote from peers 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:08.040961 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53" candidate_uuid: "dc881d7ec7fb47a2897eef434cf18767" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:08.041054 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98895d45cbd64b52b497261821d8ae53" candidate_uuid: "dc881d7ec7fb47a2897eef434cf18767" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:08.041087 3878 raft_consensus.cc:3037] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:08.041142 3950 raft_consensus.cc:3032] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:01:08.041180 3950 raft_consensus.cc:729] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Running, Role: LEADER
I20240503 02:01:08.041312 3950 consensus_queue.cc:260] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:08.041479 3950 raft_consensus.cc:3037] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:08.042091 3878 raft_consensus.cc:2454] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc881d7ec7fb47a2897eef434cf18767 in term 3.
I20240503 02:01:08.042393 4245 leader_election.cc:302] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, dc881d7ec7fb47a2897eef434cf18767; no voters:
I20240503 02:01:08.042399 3950 raft_consensus.cc:2454] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc881d7ec7fb47a2897eef434cf18767 in term 3.
I20240503 02:01:08.042909 4580 raft_consensus.cc:2781] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:01:08.043145 4580 raft_consensus.cc:686] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [term 3 LEADER]: Becoming Leader. State: Replica: dc881d7ec7fb47a2897eef434cf18767, State: Running, Role: LEADER
I20240503 02:01:08.043468 4580 consensus_queue.cc:237] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:08.044865 3701 catalog_manager.cc:5484] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 reported cstate change: term changed from 2 to 3, leader changed from 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) to dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5). New cstate: current_term: 3 leader_uuid: "dc881d7ec7fb47a2897eef434cf18767" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:08.187034 3950 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Refusing update from remote peer cfdcf4a7a9734e82bddf13e25872e410: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:01:08.187400 4510 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:01:08.190135 3878 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Refusing update from remote peer cfdcf4a7a9734e82bddf13e25872e410: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:01:08.191358 4585 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240503 02:01:08.195066 4319 consensus_peers.cc:485] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 -> Peer 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4:34437): Couldn't send request to peer 38e9ca6a49c4402abdd585ffd8f435ee. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:08.323721 3947 raft_consensus.cc:1261] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240503 02:01:08.324096 4091 consensus_queue.cc:1035] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:08.326294 3801 raft_consensus.cc:1261] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240503 02:01:08.326797 4091 consensus_queue.cc:1035] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:08.339211 3801 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:01:08.339596 4512 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:08.341298 4456 raft_consensus.cc:979] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980: : Instructing follower 94da6cefbea94c72a5d91b8d10f31bc9 to start an election
I20240503 02:01:08.341428 4512 raft_consensus.cc:1067] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Signalling peer 94da6cefbea94c72a5d91b8d10f31bc9 to start an election
I20240503 02:01:08.342422 4355 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 2 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:01:08.342645 3802 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "561921bd5097434497ea1f034f86749b"
dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
from {username='slave'} at 127.0.0.1:35310
W20240503 02:01:08.342934 3919 consensus_peers.cc:485] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 -> Peer dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5:42349): Couldn't send request to peer dc881d7ec7fb47a2897eef434cf18767. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:08.343073 3802 raft_consensus.cc:482] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:08.343154 3802 raft_consensus.cc:3037] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:08.343446 4512 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:08.344153 3802 raft_consensus.cc:504] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:08.344552 3802 leader_election.cc:288] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 3 election: Requested vote from peers 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683), cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6:38521)
I20240503 02:01:08.344921 4355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561921bd5097434497ea1f034f86749b" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "cfdcf4a7a9734e82bddf13e25872e410"
I20240503 02:01:08.345017 4355 raft_consensus.cc:3037] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:08.345288 3947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561921bd5097434497ea1f034f86749b" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:08.345368 3947 raft_consensus.cc:3032] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:01:08.345400 3947 raft_consensus.cc:729] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Running, Role: LEADER
I20240503 02:01:08.345525 3947 consensus_queue.cc:260] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:08.345677 3947 raft_consensus.cc:3037] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:08.346019 4355 raft_consensus.cc:2454] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 3.
I20240503 02:01:08.346292 3769 leader_election.cc:302] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94da6cefbea94c72a5d91b8d10f31bc9, cfdcf4a7a9734e82bddf13e25872e410; no voters:
I20240503 02:01:08.346637 3947 raft_consensus.cc:2454] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 3.
I20240503 02:01:08.347203 4527 raft_consensus.cc:2781] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:01:08.347314 4527 raft_consensus.cc:686] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 LEADER]: Becoming Leader. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Running, Role: LEADER
I20240503 02:01:08.347457 4527 consensus_queue.cc:237] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:08.349001 3696 catalog_manager.cc:5484] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: term changed from 2 to 3, leader changed from 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) to 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1). New cstate: current_term: 3 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:08.371583 3878 raft_consensus.cc:1261] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 02:01:08.372005 4583 consensus_queue.cc:1035] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 02:01:08.375124 3950 raft_consensus.cc:1261] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 02:01:08.375447 4528 consensus_queue.cc:1035] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 02:01:08.422652 3875 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 3 FOLLOWER]: Refusing update from remote peer dc881d7ec7fb47a2897eef434cf18767: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 02:01:08.423087 4580 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:01:08.425546 3950 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Refusing update from remote peer dc881d7ec7fb47a2897eef434cf18767: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 02:01:08.425923 4580 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:01:08.432093 3875 raft_consensus.cc:1261] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Refusing update from remote peer 38e9ca6a49c4402abdd585ffd8f435ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:01:08.432446 4577 consensus_queue.cc:1035] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:01:08.435168 3801 raft_consensus.cc:1261] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Refusing update from remote peer 38e9ca6a49c4402abdd585ffd8f435ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 02:01:08.435622 4578 consensus_queue.cc:1035] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:01:08.920504 4354 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 3 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 02:01:08.920884 4528 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20240503 02:01:08.923691 3768 consensus_peers.cc:485] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 -> Peer dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5:42349): Couldn't send request to peer dc881d7ec7fb47a2897eef434cf18767. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:08.923943 3950 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 02:01:08.924249 4528 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:01:09.664841 3755 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:01:09.666802 3878 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
mode: GRACEFUL
new_leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
from {username='slave'} at 127.0.0.1:56270
I20240503 02:01:09.667001 3878 raft_consensus.cc:595] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 3 LEADER]: Received request to transfer leadership to TS 1fe097edfada4dc3b18e7f74ff6dd980
I20240503 02:01:09.667659 3802 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af"
dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
mode: GRACEFUL
new_leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
from {username='slave'} at 127.0.0.1:54964
I20240503 02:01:09.667773 3802 raft_consensus.cc:595] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Received request to transfer leadership to TS 1fe097edfada4dc3b18e7f74ff6dd980
I20240503 02:01:09.668457 3802 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5"
dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
mode: GRACEFUL
new_leader_uuid: "12e9ea600473443b80086df227e94dfc"
from {username='slave'} at 127.0.0.1:54964
I20240503 02:01:09.668557 3802 raft_consensus.cc:595] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Received request to transfer leadership to TS 12e9ea600473443b80086df227e94dfc
I20240503 02:01:09.668859 3755 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 02:01:09.668994 3755 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:01:09.694067 4537 tablet_copy_client.cc:523] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:09.695945 4537 tablet_bootstrap.cc:492] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767: Bootstrap starting.
I20240503 02:01:09.717694 4542 tablet_copy_client.cc:523] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:09.718130 4537 tablet_bootstrap.cc:492] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:09.718523 4537 tablet_bootstrap.cc:492] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767: Bootstrap complete.
I20240503 02:01:09.718946 4537 ts_tablet_manager.cc:1367] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767: Time spent bootstrapping tablet: real 0.023s user 0.023s sys 0.000s
I20240503 02:01:09.719215 4548 tablet_copy_client.cc:523] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:09.719389 4542 tablet_bootstrap.cc:492] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee: Bootstrap starting.
I20240503 02:01:09.719504 4537 raft_consensus.cc:348] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:09.719628 4537 raft_consensus.cc:729] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc881d7ec7fb47a2897eef434cf18767, State: Initialized, Role: LEARNER
I20240503 02:01:09.719782 4537 consensus_queue.cc:260] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:09.720189 4537 ts_tablet_manager.cc:1398] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:01:09.720436 4548 tablet_bootstrap.cc:492] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410: Bootstrap starting.
I20240503 02:01:09.720813 3814 tablet_copy_service.cc:342] P 94da6cefbea94c72a5d91b8d10f31bc9: Request end of tablet copy session dc881d7ec7fb47a2897eef434cf18767-561921bd5097434497ea1f034f86749b received from {username='slave'} at 127.0.0.1:35332
I20240503 02:01:09.720881 3814 tablet_copy_service.cc:433] P 94da6cefbea94c72a5d91b8d10f31bc9: ending tablet copy session dc881d7ec7fb47a2897eef434cf18767-561921bd5097434497ea1f034f86749b on tablet 561921bd5097434497ea1f034f86749b with peer dc881d7ec7fb47a2897eef434cf18767
I20240503 02:01:09.723785 4553 tablet_copy_client.cc:523] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:09.725291 4553 tablet_bootstrap.cc:492] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410: Bootstrap starting.
I20240503 02:01:09.739490 4542 tablet_bootstrap.cc:492] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:09.739825 4542 tablet_bootstrap.cc:492] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee: Bootstrap complete.
I20240503 02:01:09.740082 4542 ts_tablet_manager.cc:1367] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.007s
I20240503 02:01:09.740619 4542 raft_consensus.cc:348] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:09.740752 4542 raft_consensus.cc:729] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38e9ca6a49c4402abdd585ffd8f435ee, State: Initialized, Role: LEARNER
I20240503 02:01:09.740882 4542 consensus_queue.cc:260] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:09.741271 4542 ts_tablet_manager.cc:1398] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:09.741731 3964 tablet_copy_service.cc:342] P 1fe097edfada4dc3b18e7f74ff6dd980: Request end of tablet copy session 38e9ca6a49c4402abdd585ffd8f435ee-b98f5009a3534765806bda2c95f7484c received from {username='slave'} at 127.0.0.1:54842
I20240503 02:01:09.741807 3964 tablet_copy_service.cc:433] P 1fe097edfada4dc3b18e7f74ff6dd980: ending tablet copy session 38e9ca6a49c4402abdd585ffd8f435ee-b98f5009a3534765806bda2c95f7484c on tablet b98f5009a3534765806bda2c95f7484c with peer 38e9ca6a49c4402abdd585ffd8f435ee
I20240503 02:01:09.746425 4548 tablet_bootstrap.cc:492] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:09.746727 4548 tablet_bootstrap.cc:492] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410: Bootstrap complete.
I20240503 02:01:09.747105 4548 ts_tablet_manager.cc:1367] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410: Time spent bootstrapping tablet: real 0.027s user 0.024s sys 0.001s
I20240503 02:01:09.747632 4548 raft_consensus.cc:348] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:09.747766 4548 raft_consensus.cc:729] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cfdcf4a7a9734e82bddf13e25872e410, State: Initialized, Role: LEARNER
I20240503 02:01:09.747881 4548 consensus_queue.cc:260] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:09.748564 4548 ts_tablet_manager.cc:1398] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:01:09.748993 3964 tablet_copy_service.cc:342] P 1fe097edfada4dc3b18e7f74ff6dd980: Request end of tablet copy session cfdcf4a7a9734e82bddf13e25872e410-80f0e3a5143d4bd7a132ee4b6fcb6f89 received from {username='slave'} at 127.0.0.1:50812
I20240503 02:01:09.749066 3964 tablet_copy_service.cc:433] P 1fe097edfada4dc3b18e7f74ff6dd980: ending tablet copy session cfdcf4a7a9734e82bddf13e25872e410-80f0e3a5143d4bd7a132ee4b6fcb6f89 on tablet 80f0e3a5143d4bd7a132ee4b6fcb6f89 with peer cfdcf4a7a9734e82bddf13e25872e410
I20240503 02:01:09.749491 4553 tablet_bootstrap.cc:492] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:09.749800 4553 tablet_bootstrap.cc:492] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410: Bootstrap complete.
I20240503 02:01:09.749996 4553 ts_tablet_manager.cc:1367] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.000s
I20240503 02:01:09.750439 4553 raft_consensus.cc:348] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:09.750561 4553 raft_consensus.cc:729] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cfdcf4a7a9734e82bddf13e25872e410, State: Initialized, Role: LEARNER
I20240503 02:01:09.750661 4553 consensus_queue.cc:260] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:09.750933 4553 ts_tablet_manager.cc:1398] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:01:09.751338 3814 tablet_copy_service.cc:342] P 94da6cefbea94c72a5d91b8d10f31bc9: Request end of tablet copy session cfdcf4a7a9734e82bddf13e25872e410-4f1c79d275004dfeb03f56c07b3904af received from {username='slave'} at 127.0.0.1:35326
I20240503 02:01:09.751416 3814 tablet_copy_service.cc:433] P 94da6cefbea94c72a5d91b8d10f31bc9: ending tablet copy session cfdcf4a7a9734e82bddf13e25872e410-4f1c79d275004dfeb03f56c07b3904af on tablet 4f1c79d275004dfeb03f56c07b3904af with peer cfdcf4a7a9734e82bddf13e25872e410
I20240503 02:01:09.764233 4354 raft_consensus.cc:1203] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240503 02:01:09.771942 4558 tablet_copy_client.cc:523] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:09.773568 4558 tablet_bootstrap.cc:492] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee: Bootstrap starting.
I20240503 02:01:09.796324 4558 tablet_bootstrap.cc:492] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=99 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:09.796665 4558 tablet_bootstrap.cc:492] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee: Bootstrap complete.
I20240503 02:01:09.796856 4558 ts_tablet_manager.cc:1367] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.005s
I20240503 02:01:09.797369 4558 raft_consensus.cc:348] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:09.797482 4558 raft_consensus.cc:729] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38e9ca6a49c4402abdd585ffd8f435ee, State: Initialized, Role: LEARNER
I20240503 02:01:09.797565 4558 consensus_queue.cc:260] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:09.797857 4558 ts_tablet_manager.cc:1398] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:01:09.798352 3814 tablet_copy_service.cc:342] P 94da6cefbea94c72a5d91b8d10f31bc9: Request end of tablet copy session 38e9ca6a49c4402abdd585ffd8f435ee-5cfd82671b084308af19ee0d7b151cd5 received from {username='slave'} at 127.0.0.1:54966
I20240503 02:01:09.798427 3814 tablet_copy_service.cc:433] P 94da6cefbea94c72a5d91b8d10f31bc9: ending tablet copy session 38e9ca6a49c4402abdd585ffd8f435ee-5cfd82671b084308af19ee0d7b151cd5 on tablet 5cfd82671b084308af19ee0d7b151cd5 with peer 38e9ca6a49c4402abdd585ffd8f435ee
I20240503 02:01:09.801882 4608 raft_consensus.cc:979] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9: : Instructing follower 12e9ea600473443b80086df227e94dfc to start an election
I20240503 02:01:09.801995 4608 raft_consensus.cc:1067] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Signalling peer 12e9ea600473443b80086df227e94dfc to start an election
I20240503 02:01:09.802383 3878 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
from {username='slave'} at 127.0.0.1:56248
I20240503 02:01:09.802503 3878 raft_consensus.cc:482] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:09.802560 3878 raft_consensus.cc:3037] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:09.803490 3878 raft_consensus.cc:504] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:09.803869 3878 leader_election.cc:288] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 2 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:09.804145 3801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:09.804268 3801 raft_consensus.cc:3032] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:09.804327 3801 raft_consensus.cc:729] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Running, Role: LEADER
I20240503 02:01:09.804265 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:09.804405 3950 raft_consensus.cc:3037] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:09.804466 3801 consensus_queue.cc:260] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:09.804670 3801 raft_consensus.cc:3037] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:09.805220 3950 raft_consensus.cc:2454] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 2.
I20240503 02:01:09.805495 3844 leader_election.cc:302] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 1fe097edfada4dc3b18e7f74ff6dd980; no voters:
I20240503 02:01:09.805670 4518 raft_consensus.cc:2781] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:09.805856 4518 raft_consensus.cc:686] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 2 LEADER]: Becoming Leader. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:09.806037 4518 consensus_queue.cc:237] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:09.807353 3801 raft_consensus.cc:2454] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 2.
I20240503 02:01:09.808075 3701 catalog_manager.cc:5484] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc reported cstate change: term changed from 1 to 2, leader changed from 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) to 12e9ea600473443b80086df227e94dfc (127.0.99.2). New cstate: current_term: 2 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:09.813293 3878 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56232
I20240503 02:01:09.813437 3878 raft_consensus.cc:595] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 2 LEADER]: Received request to transfer leadership
I20240503 02:01:09.817792 3878 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56232
I20240503 02:01:09.817960 3878 raft_consensus.cc:595] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 2 LEADER]: Received request to transfer leadership
I20240503 02:01:09.818766 3878 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56232
I20240503 02:01:09.818876 3878 raft_consensus.cc:595] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 2 LEADER]: Received request to transfer leadership
W20240503 02:01:09.819195 3750 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 5cfd82671b084308af19ee0d7b151cd5 already in progress
W20240503 02:01:09.819254 3750 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 5cfd82671b084308af19ee0d7b151cd5 already in progress
I20240503 02:01:09.834995 4354 raft_consensus.cc:1203] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 02:01:09.859486 4204 raft_consensus.cc:3037] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Advancing to term 2
I20240503 02:01:09.879442 4569 tablet_copy_client.cc:523] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:09.880964 4569 tablet_bootstrap.cc:492] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767: Bootstrap starting.
I20240503 02:01:09.904289 4569 tablet_bootstrap.cc:492] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:09.904624 4569 tablet_bootstrap.cc:492] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767: Bootstrap complete.
I20240503 02:01:09.904824 4569 ts_tablet_manager.cc:1367] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20240503 02:01:09.905321 4569 raft_consensus.cc:348] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:09.905442 4569 raft_consensus.cc:729] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc881d7ec7fb47a2897eef434cf18767, State: Initialized, Role: LEARNER
I20240503 02:01:09.905527 4569 consensus_queue.cc:260] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:09.905831 4569 ts_tablet_manager.cc:1398] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:01:09.906275 3890 tablet_copy_service.cc:342] P 12e9ea600473443b80086df227e94dfc: Request end of tablet copy session dc881d7ec7fb47a2897eef434cf18767-eaa6c34cc53b4061abb42f32a272a67e received from {username='slave'} at 127.0.0.1:50994
I20240503 02:01:09.906358 3890 tablet_copy_service.cc:433] P 12e9ea600473443b80086df227e94dfc: ending tablet copy session dc881d7ec7fb47a2897eef434cf18767-eaa6c34cc53b4061abb42f32a272a67e on tablet eaa6c34cc53b4061abb42f32a272a67e with peer dc881d7ec7fb47a2897eef434cf18767
I20240503 02:01:09.986794 4608 raft_consensus.cc:979] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9: : Instructing follower 1fe097edfada4dc3b18e7f74ff6dd980 to start an election
I20240503 02:01:09.986924 4608 raft_consensus.cc:1067] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Signalling peer 1fe097edfada4dc3b18e7f74ff6dd980 to start an election
I20240503 02:01:09.987423 3947 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af"
dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
from {username='slave'} at 127.0.0.1:50784
I20240503 02:01:09.987550 3947 raft_consensus.cc:482] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:09.987607 3947 raft_consensus.cc:3037] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:09.988471 3947 raft_consensus.cc:504] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:09.988818 3947 leader_election.cc:288] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 2 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5:42349)
I20240503 02:01:09.989099 3802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:09.989135 4280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "dc881d7ec7fb47a2897eef434cf18767"
I20240503 02:01:09.989198 3802 raft_consensus.cc:3032] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:09.989238 4280 raft_consensus.cc:3037] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:09.989269 3802 raft_consensus.cc:729] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Running, Role: LEADER
I20240503 02:01:09.989413 3802 consensus_queue.cc:260] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:09.989601 3802 raft_consensus.cc:3037] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:09.990048 4280 raft_consensus.cc:2454] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 2.
I20240503 02:01:09.990319 3919 leader_election.cc:302] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980, dc881d7ec7fb47a2897eef434cf18767; no voters:
I20240503 02:01:09.990475 4594 raft_consensus.cc:2781] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:09.990609 3802 raft_consensus.cc:2454] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 2.
I20240503 02:01:09.990622 4594 raft_consensus.cc:686] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Becoming Leader. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Running, Role: LEADER
I20240503 02:01:09.990764 4594 consensus_queue.cc:237] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:09.992568 3701 catalog_manager.cc:5484] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: term changed from 1 to 2, leader changed from 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) to 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3). New cstate: current_term: 2 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:09.996336 3947 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af"
dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50756
I20240503 02:01:09.996464 3947 raft_consensus.cc:595] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Received request to transfer leadership
I20240503 02:01:10.000108 3947 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af"
dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50756
I20240503 02:01:10.000236 3947 raft_consensus.cc:595] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Received request to transfer leadership
I20240503 02:01:10.000877 3947 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af"
dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50756
I20240503 02:01:10.000983 3947 raft_consensus.cc:595] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Received request to transfer leadership
W20240503 02:01:10.001307 3750 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 4f1c79d275004dfeb03f56c07b3904af already in progress
W20240503 02:01:10.001346 3750 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 4f1c79d275004dfeb03f56c07b3904af already in progress
I20240503 02:01:10.029317 4280 raft_consensus.cc:3037] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767 [term 1 LEARNER]: Advancing to term 3
I20240503 02:01:10.030442 4280 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767 [term 3 LEARNER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:01:10.030846 4608 consensus_queue.cc:1037] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:10.032588 4608 raft_consensus.cc:1050] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9: attempting to promote NON_VOTER dc881d7ec7fb47a2897eef434cf18767 to VOTER
I20240503 02:01:10.033015 4608 consensus_queue.cc:237] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 31, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:10.034633 4280 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767 [term 3 LEARNER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240503 02:01:10.034679 3947 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240503 02:01:10.035035 4354 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 3 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240503 02:01:10.035248 4631 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:01:10.035578 4608 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:01:10.035712 4630 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 02:01:10.037382 4608 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 30 to 33, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.037806 4354 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 30 to 33, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.039430 4280 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 30 to 33, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.040114 3701 catalog_manager.cc:5484] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index 30 to 33, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:10.040267 3947 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 30 to 33, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.043493 3802 consensus_queue.cc:237] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:10.044539 4280 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767 [term 3 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240503 02:01:10.045042 4354 raft_consensus.cc:1261] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 3 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240503 02:01:10.045017 4630 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:01:10.045347 4607 consensus_queue.cc:1035] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 02:01:10.046489 4608 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.046909 4280 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.047503 4354 raft_consensus.cc:2932] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.048550 3687 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 561921bd5097434497ea1f034f86749b with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:01:10.048950 3695 catalog_manager.cc:5484] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index 33 to 34, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New cstate: current_term: 3 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:10.051139 3940 tablet_service.cc:1514] Processing DeleteTablet for tablet 561921bd5097434497ea1f034f86749b with delete_type TABLET_DATA_TOMBSTONED (TS 1fe097edfada4dc3b18e7f74ff6dd980 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:50756
I20240503 02:01:10.051540 4634 tablet_replica.cc:330] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980: stopping tablet replica
I20240503 02:01:10.051759 4634 raft_consensus.cc:2229] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:10.052028 4634 raft_consensus.cc:2258] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:10.052652 4634 ts_tablet_manager.cc:1875] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:10.054944 4634 ts_tablet_manager.cc:1888] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.33
I20240503 02:01:10.055063 4634 log.cc:1196] T 561921bd5097434497ea1f034f86749b P 1fe097edfada4dc3b18e7f74ff6dd980: Deleting WAL directory at /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-2-root/wals/561921bd5097434497ea1f034f86749b
I20240503 02:01:10.055559 3686 catalog_manager.cc:4830] TS 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683): tablet 561921bd5097434497ea1f034f86749b (table test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]) successfully deleted
I20240503 02:01:10.116314 4280 raft_consensus.cc:1203] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.29->[1.30-1.30] Dedup: 1.30->[]
I20240503 02:01:10.120537 4614 raft_consensus.cc:979] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc: : Instructing follower 1fe097edfada4dc3b18e7f74ff6dd980 to start an election
I20240503 02:01:10.120666 4614 raft_consensus.cc:1067] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 3 LEADER]: Signalling peer 1fe097edfada4dc3b18e7f74ff6dd980 to start an election
I20240503 02:01:10.121032 3947 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472"
dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
from {username='slave'} at 127.0.0.1:50770
I20240503 02:01:10.121166 3947 raft_consensus.cc:482] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:10.121239 3947 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:01:10.122262 3947 raft_consensus.cc:504] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:10.122619 3947 leader_election.cc:288] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 4 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:10.122907 3802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:10.123067 3802 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:01:10.124004 3802 raft_consensus.cc:2454] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 4.
I20240503 02:01:10.122907 3875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472" candidate_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:10.124246 3875 raft_consensus.cc:3032] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:01:10.124310 3875 raft_consensus.cc:729] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:10.124258 3920 leader_election.cc:302] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:10.124550 4594 raft_consensus.cc:2781] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:01:10.124640 4594 raft_consensus.cc:686] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 4 LEADER]: Becoming Leader. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Running, Role: LEADER
I20240503 02:01:10.124766 4594 consensus_queue.cc:237] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:10.125797 3875 consensus_queue.cc:260] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } }
I20240503 02:01:10.125981 3875 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:01:10.126348 3701 catalog_manager.cc:5484] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: term changed from 3 to 4, leader changed from 12e9ea600473443b80086df227e94dfc (127.0.99.2) to 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3). New cstate: current_term: 4 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } }
I20240503 02:01:10.127036 3875 raft_consensus.cc:2454] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe097edfada4dc3b18e7f74ff6dd980 in term 4.
I20240503 02:01:10.303069 3802 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:01:10.303501 4614 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:10.305276 4614 raft_consensus.cc:979] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc: : Instructing follower 94da6cefbea94c72a5d91b8d10f31bc9 to start an election
I20240503 02:01:10.305358 4614 raft_consensus.cc:1067] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 2 LEADER]: Signalling peer 94da6cefbea94c72a5d91b8d10f31bc9 to start an election
I20240503 02:01:10.306159 4204 raft_consensus.cc:3037] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 LEARNER]: Advancing to term 2
I20240503 02:01:10.307075 3801 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5"
dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
from {username='slave'} at 127.0.0.1:35300
I20240503 02:01:10.307226 4204 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee [term 2 LEARNER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:01:10.307482 3801 raft_consensus.cc:482] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:10.307552 3801 raft_consensus.cc:3037] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:10.307734 4638 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:10.307989 3947 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 02:01:10.308532 3801 raft_consensus.cc:504] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:10.308945 3801 leader_election.cc:288] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 3 election: Requested vote from peers 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:10.309388 3947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:10.309481 4518 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:10.309388 3875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfd82671b084308af19ee0d7b151cd5" candidate_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "12e9ea600473443b80086df227e94dfc"
I20240503 02:01:10.309841 3875 raft_consensus.cc:3032] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:01:10.309881 3875 raft_consensus.cc:729] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:10.310014 3875 consensus_queue.cc:260] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:10.310184 3875 raft_consensus.cc:3037] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:10.311309 3875 raft_consensus.cc:2454] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 3.
I20240503 02:01:10.309499 3947 raft_consensus.cc:3037] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:10.311960 3769 leader_election.cc:302] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:10.312139 4631 raft_consensus.cc:2781] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:01:10.312223 4631 raft_consensus.cc:686] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 LEADER]: Becoming Leader. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Running, Role: LEADER
I20240503 02:01:10.312315 3947 raft_consensus.cc:2454] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94da6cefbea94c72a5d91b8d10f31bc9 in term 3.
I20240503 02:01:10.312361 4631 consensus_queue.cc:237] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:10.312515 3950 raft_consensus.cc:1226] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Rejecting Update request from peer 12e9ea600473443b80086df227e94dfc for earlier term 2. Current term is 3. Ops: [2.28-2.28]
I20240503 02:01:10.313850 3701 catalog_manager.cc:5484] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: term changed from 2 to 3, leader changed from 12e9ea600473443b80086df227e94dfc (127.0.99.2) to 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1). New cstate: current_term: 3 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:10.396996 4280 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 2 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:01:10.397464 4594 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:10.399230 4624 raft_consensus.cc:979] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980: : Instructing follower dc881d7ec7fb47a2897eef434cf18767 to start an election
I20240503 02:01:10.399415 4594 raft_consensus.cc:1067] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Signalling peer dc881d7ec7fb47a2897eef434cf18767 to start an election
I20240503 02:01:10.399914 4279 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af"
dest_uuid: "dc881d7ec7fb47a2897eef434cf18767"
from {username='slave'} at 127.0.0.1:45868
I20240503 02:01:10.400053 4279 raft_consensus.cc:482] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:10.400133 4279 raft_consensus.cc:3037] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:10.401432 4279 raft_consensus.cc:504] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:10.401813 4279 leader_election.cc:288] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [CANDIDATE]: Term 3 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:10.402220 3947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af" candidate_uuid: "dc881d7ec7fb47a2897eef434cf18767" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:10.403052 3947 raft_consensus.cc:3032] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:01:10.403115 3947 raft_consensus.cc:729] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Running, Role: LEADER
I20240503 02:01:10.403118 4354 raft_consensus.cc:3037] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Advancing to term 2
I20240503 02:01:10.403273 3947 consensus_queue.cc:260] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:10.403405 3947 raft_consensus.cc:3037] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:10.404232 4354 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410 [term 2 LEARNER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:01:10.404273 3947 raft_consensus.cc:2454] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc881d7ec7fb47a2897eef434cf18767 in term 3.
I20240503 02:01:10.404582 3801 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 02:01:10.404934 4244 leader_election.cc:302] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe097edfada4dc3b18e7f74ff6dd980, dc881d7ec7fb47a2897eef434cf18767; no voters:
I20240503 02:01:10.405121 4628 raft_consensus.cc:2781] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:01:10.405218 4628 raft_consensus.cc:686] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 3 LEADER]: Becoming Leader. State: Replica: dc881d7ec7fb47a2897eef434cf18767, State: Running, Role: LEADER
I20240503 02:01:10.405357 4628 consensus_queue.cc:237] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:10.405840 3801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1c79d275004dfeb03f56c07b3904af" candidate_uuid: "dc881d7ec7fb47a2897eef434cf18767" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:10.405953 3801 raft_consensus.cc:3037] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:10.406526 4585 raft_consensus.cc:1050] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410: attempting to promote NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee to VOTER
I20240503 02:01:10.406914 4585 consensus_queue.cc:237] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } }
I20240503 02:01:10.407012 3801 raft_consensus.cc:2454] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc881d7ec7fb47a2897eef434cf18767 in term 3.
I20240503 02:01:10.407347 3696 catalog_manager.cc:5484] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 reported cstate change: term changed from 2 to 3, leader changed from 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) to dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5). New cstate: current_term: 3 leader_uuid: "dc881d7ec7fb47a2897eef434cf18767" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:10.408447 3875 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Refusing update from remote peer cfdcf4a7a9734e82bddf13e25872e410: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:01:10.408613 4204 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee [term 2 LEARNER]: Refusing update from remote peer cfdcf4a7a9734e82bddf13e25872e410: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:01:10.408828 4585 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:10.408996 4648 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:10.410241 3947 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Refusing update from remote peer cfdcf4a7a9734e82bddf13e25872e410: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:01:10.410573 4585 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:10.411731 4648 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.412107 3875 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.412855 3947 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.413885 4204 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.415959 3696 catalog_manager.cc:5484] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 reported cstate change: config changed from index 29 to 31, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "cfdcf4a7a9734e82bddf13e25872e410" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:10.419605 4354 consensus_queue.cc:237] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } }
I20240503 02:01:10.420888 4204 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee [term 2 FOLLOWER]: Refusing update from remote peer cfdcf4a7a9734e82bddf13e25872e410: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:01:10.420928 3947 raft_consensus.cc:1261] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Refusing update from remote peer cfdcf4a7a9734e82bddf13e25872e410: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 02:01:10.421358 4648 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:01:10.421519 4585 consensus_queue.cc:1035] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:01:10.423125 4585 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P cfdcf4a7a9734e82bddf13e25872e410 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.423583 4204 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 38e9ca6a49c4402abdd585ffd8f435ee [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.423676 3950 raft_consensus.cc:2932] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.425776 3747 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b98f5009a3534765806bda2c95f7484c with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:01:10.425804 3697 catalog_manager.cc:5484] T b98f5009a3534765806bda2c95f7484c P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: config changed from index 31 to 32, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New cstate: current_term: 2 leader_uuid: "cfdcf4a7a9734e82bddf13e25872e410" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.428303 3870 tablet_service.cc:1514] Processing DeleteTablet for tablet b98f5009a3534765806bda2c95f7484c with delete_type TABLET_DATA_TOMBSTONED (TS 12e9ea600473443b80086df227e94dfc not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:56232
I20240503 02:01:10.428695 4651 tablet_replica.cc:330] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc: stopping tablet replica
I20240503 02:01:10.428843 4651 raft_consensus.cc:2229] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:10.429066 4651 raft_consensus.cc:2258] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:10.430184 4651 ts_tablet_manager.cc:1875] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:10.431102 4594 raft_consensus.cc:1050] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980: attempting to promote NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 to VOTER
I20240503 02:01:10.431450 4594 consensus_queue.cc:237] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } }
I20240503 02:01:10.432766 4355 raft_consensus.cc:1261] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410 [term 1 LEARNER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:10.432829 3875 raft_consensus.cc:1261] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:10.433037 3801 raft_consensus.cc:1261] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 02:01:10.433267 4651 ts_tablet_manager.cc:1888] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240503 02:01:10.433351 4651 log.cc:1196] T b98f5009a3534765806bda2c95f7484c P 12e9ea600473443b80086df227e94dfc: Deleting WAL directory at /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-1-root/wals/b98f5009a3534765806bda2c95f7484c
I20240503 02:01:10.433460 4624 consensus_queue.cc:1035] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:10.433786 4624 consensus_queue.cc:1035] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:10.433936 4653 consensus_queue.cc:1035] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:10.433954 3687 catalog_manager.cc:4830] TS 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879): tablet b98f5009a3534765806bda2c95f7484c (table test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]) successfully deleted
I20240503 02:01:10.436760 4624 raft_consensus.cc:2932] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.437543 3878 raft_consensus.cc:2932] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.437628 4355 raft_consensus.cc:2932] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.439047 3801 raft_consensus.cc:2932] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.440289 3696 catalog_manager.cc:5484] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 reported cstate change: config changed from index 27 to 28, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:10.444188 3947 consensus_queue.cc:237] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } }
I20240503 02:01:10.445227 3878 raft_consensus.cc:1261] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:01:10.445228 4355 raft_consensus.cc:1261] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:01:10.445574 4653 consensus_queue.cc:1035] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:10.445757 4594 consensus_queue.cc:1035] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:10.447165 4643 raft_consensus.cc:2932] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.447525 4355 raft_consensus.cc:2932] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.447863 3878 raft_consensus.cc:2932] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.449479 3686 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80f0e3a5143d4bd7a132ee4b6fcb6f89 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:01:10.449568 3701 catalog_manager.cc:5484] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P cfdcf4a7a9734e82bddf13e25872e410 reported cstate change: config changed from index 28 to 29, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.452512 3794 tablet_service.cc:1514] Processing DeleteTablet for tablet 80f0e3a5143d4bd7a132ee4b6fcb6f89 with delete_type TABLET_DATA_TOMBSTONED (TS 94da6cefbea94c72a5d91b8d10f31bc9 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:35292
I20240503 02:01:10.453011 4657 tablet_replica.cc:330] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9: stopping tablet replica
I20240503 02:01:10.453133 4657 raft_consensus.cc:2229] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:10.453301 4657 raft_consensus.cc:2258] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:10.454156 4657 ts_tablet_manager.cc:1875] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:10.456713 4657 ts_tablet_manager.cc:1888] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 02:01:10.456801 4657 log.cc:1196] T 80f0e3a5143d4bd7a132ee4b6fcb6f89 P 94da6cefbea94c72a5d91b8d10f31bc9: Deleting WAL directory at /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-0-root/wals/80f0e3a5143d4bd7a132ee4b6fcb6f89
I20240503 02:01:10.457314 3687 catalog_manager.cc:4830] TS 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193): tablet 80f0e3a5143d4bd7a132ee4b6fcb6f89 (table test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]) successfully deleted
I20240503 02:01:10.567750 3878 raft_consensus.cc:1261] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 4 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240503 02:01:10.568092 4624 consensus_queue.cc:1035] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:10.570551 3801 raft_consensus.cc:1261] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 4 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240503 02:01:10.571161 4643 consensus_queue.cc:1035] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:10.628813 4638 raft_consensus.cc:1050] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc: attempting to promote NON_VOTER dc881d7ec7fb47a2897eef434cf18767 to VOTER
I20240503 02:01:10.629336 4638 consensus_queue.cc:237] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:10.630662 4279 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767 [term 1 LEARNER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 02:01:10.630733 3801 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 02:01:10.631096 4204 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240503 02:01:10.631103 4661 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:10.631480 4614 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:10.631649 4518 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:10.633723 4614 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.634296 3801 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.634538 4204 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.637101 3697 catalog_manager.cc:5484] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index 30 to 31, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.637076 4279 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.640594 3878 consensus_queue.cc:237] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:10.641911 4279 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240503 02:01:10.641911 4204 raft_consensus.cc:1261] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240503 02:01:10.642652 4518 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:01:10.642833 4638 consensus_queue.cc:1035] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:01:10.644155 4661 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.645536 3687 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eaa6c34cc53b4061abb42f32a272a67e with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:01:10.645709 4204 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P 38e9ca6a49c4402abdd585ffd8f435ee [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.646205 4279 raft_consensus.cc:2932] T eaa6c34cc53b4061abb42f32a272a67e P dc881d7ec7fb47a2897eef434cf18767 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } }
I20240503 02:01:10.647516 3695 catalog_manager.cc:5484] T eaa6c34cc53b4061abb42f32a272a67e P 12e9ea600473443b80086df227e94dfc reported cstate change: config changed from index 31 to 32, VOTER 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:10.649703 3794 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa6c34cc53b4061abb42f32a272a67e with delete_type TABLET_DATA_TOMBSTONED (TS 94da6cefbea94c72a5d91b8d10f31bc9 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:35292
I20240503 02:01:10.649914 4657 tablet_replica.cc:330] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9: stopping tablet replica
I20240503 02:01:10.650065 4657 raft_consensus.cc:2229] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:10.650269 4657 raft_consensus.cc:2258] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:10.650807 4657 ts_tablet_manager.cc:1875] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:10.652601 4657 ts_tablet_manager.cc:1888] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.31
I20240503 02:01:10.652671 4657 log.cc:1196] T eaa6c34cc53b4061abb42f32a272a67e P 94da6cefbea94c72a5d91b8d10f31bc9: Deleting WAL directory at /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-0-root/wals/eaa6c34cc53b4061abb42f32a272a67e
I20240503 02:01:10.653105 3687 catalog_manager.cc:4830] TS 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193): tablet eaa6c34cc53b4061abb42f32a272a67e (table test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]) successfully deleted
I20240503 02:01:10.791628 3950 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 02:01:10.792272 4608 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:01:10.794736 4204 raft_consensus.cc:3037] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee [term 2 LEARNER]: Advancing to term 3
I20240503 02:01:10.795058 3877 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 3 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 02:01:10.795748 4666 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:10.796021 4204 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee [term 3 LEARNER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 02:01:10.796406 4666 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:10.798044 4631 raft_consensus.cc:1050] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9: attempting to promote NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee to VOTER
I20240503 02:01:10.798470 4631 consensus_queue.cc:237] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } }
I20240503 02:01:10.799798 4204 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee [term 3 LEARNER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 02:01:10.799818 3950 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 02:01:10.800263 4667 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:01:10.800392 4631 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:01:10.800209 3877 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 3 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 02:01:10.801302 4667 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:01:10.802685 4666 raft_consensus.cc:2932] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 27 to 30, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.803118 3877 raft_consensus.cc:2932] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.803148 4204 raft_consensus.cc:2932] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.804673 3950 raft_consensus.cc:2932] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.805094 3695 catalog_manager.cc:5484] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index 27 to 30, 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:10.808224 3801 consensus_queue.cc:237] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } }
I20240503 02:01:10.809343 4204 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee [term 3 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 02:01:10.809391 3950 raft_consensus.cc:1261] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 02:01:10.809657 4667 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:10.809816 4631 consensus_queue.cc:1035] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:10.811213 4608 raft_consensus.cc:2932] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.811640 3950 raft_consensus.cc:2932] T 5cfd82671b084308af19ee0d7b151cd5 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.812716 3687 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5cfd82671b084308af19ee0d7b151cd5 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:01:10.813546 4204 raft_consensus.cc:2932] T 5cfd82671b084308af19ee0d7b151cd5 P 38e9ca6a49c4402abdd585ffd8f435ee [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } }
I20240503 02:01:10.813778 3697 catalog_manager.cc:5484] T 5cfd82671b084308af19ee0d7b151cd5 P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index 30 to 31, VOTER 12e9ea600473443b80086df227e94dfc (127.0.99.2) evicted. New cstate: current_term: 3 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:10.816270 3870 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cfd82671b084308af19ee0d7b151cd5 with delete_type TABLET_DATA_TOMBSTONED (TS 12e9ea600473443b80086df227e94dfc not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:56232
I20240503 02:01:10.816489 4651 tablet_replica.cc:330] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc: stopping tablet replica
I20240503 02:01:10.816629 4651 raft_consensus.cc:2229] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:10.816790 4651 raft_consensus.cc:2258] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:10.817469 4651 ts_tablet_manager.cc:1875] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:10.819164 4651 ts_tablet_manager.cc:1888] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.30
I20240503 02:01:10.819226 4651 log.cc:1196] T 5cfd82671b084308af19ee0d7b151cd5 P 12e9ea600473443b80086df227e94dfc: Deleting WAL directory at /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-1-root/wals/5cfd82671b084308af19ee0d7b151cd5
I20240503 02:01:10.819607 3687 catalog_manager.cc:4830] TS 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879): tablet 5cfd82671b084308af19ee0d7b151cd5 (table test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]) successfully deleted
I20240503 02:01:10.841150 3801 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Refusing update from remote peer dc881d7ec7fb47a2897eef434cf18767: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 02:01:10.841620 4628 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:10.844578 3950 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Refusing update from remote peer dc881d7ec7fb47a2897eef434cf18767: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 02:01:10.845274 4580 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 02:01:10.849416 4355 raft_consensus.cc:3037] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410 [term 2 LEARNER]: Advancing to term 3
I20240503 02:01:10.850282 4355 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410 [term 3 LEARNER]: Refusing update from remote peer dc881d7ec7fb47a2897eef434cf18767: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 02:01:10.850664 4669 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:10.852560 4669 raft_consensus.cc:1050] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767: attempting to promote NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 to VOTER
I20240503 02:01:10.852944 4669 consensus_queue.cc:237] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } }
I20240503 02:01:10.854190 3950 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Refusing update from remote peer dc881d7ec7fb47a2897eef434cf18767: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 02:01:10.854224 3801 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Refusing update from remote peer dc881d7ec7fb47a2897eef434cf18767: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 02:01:10.854224 4355 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410 [term 3 LEARNER]: Refusing update from remote peer dc881d7ec7fb47a2897eef434cf18767: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 02:01:10.854593 4669 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:10.854733 4674 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:10.854840 4580 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:10.856339 4669 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 28 to 31, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.856755 3801 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.856873 3950 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.858589 4355 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.860157 3696 catalog_manager.cc:5484] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 reported cstate change: config changed from index 28 to 31, cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "dc881d7ec7fb47a2897eef434cf18767" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:01:10.866878 4279 consensus_queue.cc:237] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } }
I20240503 02:01:10.868007 4355 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410 [term 3 FOLLOWER]: Refusing update from remote peer dc881d7ec7fb47a2897eef434cf18767: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 02:01:10.868053 3801 raft_consensus.cc:1261] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Refusing update from remote peer dc881d7ec7fb47a2897eef434cf18767: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 02:01:10.868386 4674 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:01:10.868587 4669 consensus_queue.cc:1035] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:01:10.869918 4580 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P dc881d7ec7fb47a2897eef434cf18767 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.870325 3801 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.870347 4355 raft_consensus.cc:2932] T 4f1c79d275004dfeb03f56c07b3904af P cfdcf4a7a9734e82bddf13e25872e410 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.871536 3686 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4f1c79d275004dfeb03f56c07b3904af with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:01:10.872363 3695 catalog_manager.cc:5484] T 4f1c79d275004dfeb03f56c07b3904af P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index 31 to 32, VOTER 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) evicted. New cstate: current_term: 3 leader_uuid: "dc881d7ec7fb47a2897eef434cf18767" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } }
I20240503 02:01:10.874527 3940 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1c79d275004dfeb03f56c07b3904af with delete_type TABLET_DATA_TOMBSTONED (TS 1fe097edfada4dc3b18e7f74ff6dd980 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:50756
I20240503 02:01:10.874989 4677 tablet_replica.cc:330] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980: stopping tablet replica
I20240503 02:01:10.875200 4677 raft_consensus.cc:2229] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:10.875433 4677 raft_consensus.cc:2258] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:10.875958 4677 ts_tablet_manager.cc:1875] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:10.877899 4677 ts_tablet_manager.cc:1888] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.31
I20240503 02:01:10.877969 4677 log.cc:1196] T 4f1c79d275004dfeb03f56c07b3904af P 1fe097edfada4dc3b18e7f74ff6dd980: Deleting WAL directory at /tmp/dist-test-taskgvbJg4/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714701568895932-396-0/minicluster-data/ts-2-root/wals/4f1c79d275004dfeb03f56c07b3904af
I20240503 02:01:10.878510 3686 catalog_manager.cc:4830] TS 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683): tablet 4f1c79d275004dfeb03f56c07b3904af (table test-workload [id=a6cb6474ff9e413abe2ee4de3de219b4]) successfully deleted
I20240503 02:01:11.653693 4279 consensus_queue.cc:237] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:11.655184 3950 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Refusing update from remote peer dc881d7ec7fb47a2897eef434cf18767: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
W20240503 02:01:11.655325 4245 consensus_peers.cc:485] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 -> Peer cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6:38521): Couldn't send request to peer cfdcf4a7a9734e82bddf13e25872e410. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98895d45cbd64b52b497261821d8ae53. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:11.655184 3878 raft_consensus.cc:1261] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 3 FOLLOWER]: Refusing update from remote peer dc881d7ec7fb47a2897eef434cf18767: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 02:01:11.655761 4674 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:11.655990 4684 consensus_queue.cc:1035] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:11.657745 4674 raft_consensus.cc:2932] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:11.658205 3950 raft_consensus.cc:2932] T 98895d45cbd64b52b497261821d8ae53 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:11.658365 3878 raft_consensus.cc:2932] T 98895d45cbd64b52b497261821d8ae53 P 12e9ea600473443b80086df227e94dfc [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:11.660526 3701 catalog_manager.cc:5484] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767 reported cstate change: config changed from index 29 to 31, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New cstate: current_term: 3 leader_uuid: "dc881d7ec7fb47a2897eef434cf18767" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:11.661806 3950 consensus_queue.cc:237] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:11.663450 3878 raft_consensus.cc:1261] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 4 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240503 02:01:11.663450 3801 raft_consensus.cc:1261] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 4 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
W20240503 02:01:11.664077 3919 consensus_peers.cc:485] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 -> Peer dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5:42349): Couldn't send request to peer dc881d7ec7fb47a2897eef434cf18767. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 226ca5ac33d14b43b90f5a57b2b80472. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:11.664639 4692 consensus_queue.cc:1035] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:11.664809 4691 consensus_queue.cc:1035] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 02:01:11.666406 4679 raft_consensus.cc:2932] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 4 LEADER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:11.666832 3801 raft_consensus.cc:2932] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 4 FOLLOWER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:11.666859 3878 raft_consensus.cc:2932] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 4 FOLLOWER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:11.669296 3755 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 02:01:11.669263 3695 catalog_manager.cc:5484] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index -1 to 29, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New cstate: current_term: 4 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:11.670343 3878 consensus_queue.cc:237] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:11.671322 3950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472"
dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
mode: GRACEFUL
new_leader_uuid: "12e9ea600473443b80086df227e94dfc"
from {username='slave'} at 127.0.0.1:54834
I20240503 02:01:11.671438 3950 raft_consensus.cc:595] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 4 LEADER]: Received request to transfer leadership to TS 12e9ea600473443b80086df227e94dfc
I20240503 02:01:11.671977 3950 raft_consensus.cc:1261] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20240503 02:01:11.672015 3844 consensus_peers.cc:485] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc -> Peer 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4:34437): Couldn't send request to peer 38e9ca6a49c4402abdd585ffd8f435ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e253a96048114d0ebfa385eda5b0c7b0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:11.672061 3801 raft_consensus.cc:1261] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 12e9ea600473443b80086df227e94dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 02:01:11.672302 4697 consensus_queue.cc:1035] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:11.672441 3801 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "561921bd5097434497ea1f034f86749b"
dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
mode: GRACEFUL
new_leader_uuid: "cfdcf4a7a9734e82bddf13e25872e410"
from {username='slave'} at 127.0.0.1:54964
I20240503 02:01:11.672464 4518 consensus_queue.cc:1035] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:11.672531 3801 raft_consensus.cc:595] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 LEADER]: Received request to transfer leadership to TS cfdcf4a7a9734e82bddf13e25872e410
I20240503 02:01:11.672875 3755 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 02:01:11.673020 3755 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 02:01:11.674793 4698 raft_consensus.cc:2932] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:11.675184 3801 raft_consensus.cc:2932] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:11.675566 3950 raft_consensus.cc:2932] T e253a96048114d0ebfa385eda5b0c7b0 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:11.677323 3696 catalog_manager.cc:5484] T e253a96048114d0ebfa385eda5b0c7b0 P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index -1 to 27, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:11.677672 3950 consensus_queue.cc:237] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } }
I20240503 02:01:11.679481 3801 raft_consensus.cc:1261] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 02:01:11.679600 3878 raft_consensus.cc:1261] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Refusing update from remote peer 1fe097edfada4dc3b18e7f74ff6dd980: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
W20240503 02:01:11.680114 3919 consensus_peers.cc:485] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 -> Peer 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4:34437): Couldn't send request to peer 38e9ca6a49c4402abdd585ffd8f435ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cef107d87fa14a329bfb5c9b22f0d808. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:11.680217 4691 consensus_queue.cc:1035] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 02:01:11.680567 4679 consensus_queue.cc:1035] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 02:01:11.682512 4691 raft_consensus.cc:2932] T cef107d87fa14a329bfb5c9b22f0d808 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:11.682904 3878 raft_consensus.cc:2932] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:11.683158 3801 raft_consensus.cc:2932] T cef107d87fa14a329bfb5c9b22f0d808 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:11.685405 3801 consensus_queue.cc:237] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } }
I20240503 02:01:11.685474 3695 catalog_manager.cc:5484] T cef107d87fa14a329bfb5c9b22f0d808 P 12e9ea600473443b80086df227e94dfc reported cstate change: config changed from index -1 to 26, NON_VOTER 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: true } } }
I20240503 02:01:11.686834 3950 raft_consensus.cc:1261] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 02:01:11.687109 3878 raft_consensus.cc:1261] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Refusing update from remote peer 94da6cefbea94c72a5d91b8d10f31bc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
W20240503 02:01:11.687383 3769 consensus_peers.cc:485] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 -> Peer cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6:38521): Couldn't send request to peer cfdcf4a7a9734e82bddf13e25872e410. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8d5be30376a4040a650e1be3d6643f8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:11.687652 4707 consensus_queue.cc:1035] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:11.687860 4667 consensus_queue.cc:1035] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 02:01:11.689709 4667 raft_consensus.cc:2932] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:11.690171 3950 raft_consensus.cc:2932] T e8d5be30376a4040a650e1be3d6643f8 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:11.692417 3701 catalog_manager.cc:5484] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9 reported cstate change: config changed from index -1 to 27, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New cstate: current_term: 2 leader_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:11.692474 3878 raft_consensus.cc:2932] T e8d5be30376a4040a650e1be3d6643f8 P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } attrs { replace: true } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: true } } }
I20240503 02:01:11.698118 4204 consensus_queue.cc:237] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:11.699954 3801 raft_consensus.cc:1261] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Refusing update from remote peer 38e9ca6a49c4402abdd585ffd8f435ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:01:11.700493 3878 raft_consensus.cc:1261] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Refusing update from remote peer 38e9ca6a49c4402abdd585ffd8f435ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 02:01:11.700634 4712 consensus_queue.cc:1035] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:11.700891 4713 consensus_queue.cc:1035] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 02:01:11.702584 4712 raft_consensus.cc:2932] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:11.702970 3878 raft_consensus.cc:2932] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
W20240503 02:01:11.704280 4168 consensus_peers.cc:485] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee -> Peer dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5:42349): Couldn't send request to peer dc881d7ec7fb47a2897eef434cf18767. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88bdd7e3ba2d4793907feb426e7bef9c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:11.704435 3801 raft_consensus.cc:2932] T 88bdd7e3ba2d4793907feb426e7bef9c P 94da6cefbea94c72a5d91b8d10f31bc9 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:11.705348 3696 catalog_manager.cc:5484] T 88bdd7e3ba2d4793907feb426e7bef9c P 12e9ea600473443b80086df227e94dfc reported cstate change: config changed from index 29 to 31, NON_VOTER dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5) added. New cstate: current_term: 2 leader_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } attrs { replace: true } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } } peers { permanent_uuid: "38e9ca6a49c4402abdd585ffd8f435ee" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34437 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } } }
I20240503 02:01:12.022392 4706 raft_consensus.cc:979] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9: : Instructing follower cfdcf4a7a9734e82bddf13e25872e410 to start an election
I20240503 02:01:12.022534 4706 raft_consensus.cc:1067] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 LEADER]: Signalling peer cfdcf4a7a9734e82bddf13e25872e410 to start an election
I20240503 02:01:12.022907 4355 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "561921bd5097434497ea1f034f86749b"
dest_uuid: "cfdcf4a7a9734e82bddf13e25872e410"
from {username='slave'} at 127.0.0.1:34696
I20240503 02:01:12.023061 4355 raft_consensus.cc:482] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:12.023138 4355 raft_consensus.cc:3037] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:01:12.024188 4355 raft_consensus.cc:504] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:12.024688 4355 leader_election.cc:288] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [CANDIDATE]: Term 4 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5:42349)
I20240503 02:01:12.025125 3801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561921bd5097434497ea1f034f86749b" candidate_uuid: "cfdcf4a7a9734e82bddf13e25872e410" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: true dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:12.025238 3801 raft_consensus.cc:3032] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:01:12.025281 3801 raft_consensus.cc:729] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 94da6cefbea94c72a5d91b8d10f31bc9, State: Running, Role: LEADER
I20240503 02:01:12.025447 3801 consensus_queue.cc:260] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 3.34, Last appended by leader: 34, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:12.025615 3801 raft_consensus.cc:3037] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:01:12.026592 3801 raft_consensus.cc:2454] T 561921bd5097434497ea1f034f86749b P 94da6cefbea94c72a5d91b8d10f31bc9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfdcf4a7a9734e82bddf13e25872e410 in term 4.
I20240503 02:01:12.026883 4320 leader_election.cc:302] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94da6cefbea94c72a5d91b8d10f31bc9, cfdcf4a7a9734e82bddf13e25872e410; no voters:
I20240503 02:01:12.027096 4680 raft_consensus.cc:2781] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:01:12.027302 4680 raft_consensus.cc:686] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [term 4 LEADER]: Becoming Leader. State: Replica: cfdcf4a7a9734e82bddf13e25872e410, State: Running, Role: LEADER
I20240503 02:01:12.027437 4680 consensus_queue.cc:237] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 3.34, Last appended by leader: 34, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } }
I20240503 02:01:12.028679 4279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561921bd5097434497ea1f034f86749b" candidate_uuid: "cfdcf4a7a9734e82bddf13e25872e410" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: true dest_uuid: "dc881d7ec7fb47a2897eef434cf18767"
I20240503 02:01:12.028792 4279 raft_consensus.cc:3037] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:01:12.028936 3695 catalog_manager.cc:5484] T 561921bd5097434497ea1f034f86749b P cfdcf4a7a9734e82bddf13e25872e410 reported cstate change: term changed from 3 to 4, leader changed from 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1) to cfdcf4a7a9734e82bddf13e25872e410 (127.0.99.6). New cstate: current_term: 4 leader_uuid: "cfdcf4a7a9734e82bddf13e25872e410" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfdcf4a7a9734e82bddf13e25872e410" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 38521 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:12.029807 4279 raft_consensus.cc:2454] T 561921bd5097434497ea1f034f86749b P dc881d7ec7fb47a2897eef434cf18767 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfdcf4a7a9734e82bddf13e25872e410 in term 4.
I20240503 02:01:12.063599 4720 ts_tablet_manager.cc:902] T e8d5be30376a4040a650e1be3d6643f8 P cfdcf4a7a9734e82bddf13e25872e410: Initiating tablet copy from peer 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193)
I20240503 02:01:12.064211 4720 tablet_copy_client.cc:308] T e8d5be30376a4040a650e1be3d6643f8 P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:39193
I20240503 02:01:12.064657 3814 tablet_copy_service.cc:140] P 94da6cefbea94c72a5d91b8d10f31bc9: Received BeginTabletCopySession request for tablet e8d5be30376a4040a650e1be3d6643f8 from peer cfdcf4a7a9734e82bddf13e25872e410 ({username='slave'} at 127.0.0.1:35326)
I20240503 02:01:12.064767 3814 tablet_copy_service.cc:161] P 94da6cefbea94c72a5d91b8d10f31bc9: Beginning new tablet copy session on tablet e8d5be30376a4040a650e1be3d6643f8 from peer cfdcf4a7a9734e82bddf13e25872e410 at {username='slave'} at 127.0.0.1:35326: session id = cfdcf4a7a9734e82bddf13e25872e410-e8d5be30376a4040a650e1be3d6643f8
I20240503 02:01:12.065798 3814 tablet_copy_source_session.cc:205] T e8d5be30376a4040a650e1be3d6643f8 P 94da6cefbea94c72a5d91b8d10f31bc9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:12.066642 4720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8d5be30376a4040a650e1be3d6643f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:12.069033 4720 tablet_copy_client.cc:783] T e8d5be30376a4040a650e1be3d6643f8 P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:12.069324 4720 tablet_copy_client.cc:647] T e8d5be30376a4040a650e1be3d6643f8 P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:12.070302 4724 tablet_copy_client.cc:1002] T e8d5be30376a4040a650e1be3d6643f8 P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:12.114434 4725 ts_tablet_manager.cc:902] T 226ca5ac33d14b43b90f5a57b2b80472 P dc881d7ec7fb47a2897eef434cf18767: Initiating tablet copy from peer 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:12.115033 4725 tablet_copy_client.cc:308] T 226ca5ac33d14b43b90f5a57b2b80472 P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:34683
I20240503 02:01:12.115569 3964 tablet_copy_service.cc:140] P 1fe097edfada4dc3b18e7f74ff6dd980: Received BeginTabletCopySession request for tablet 226ca5ac33d14b43b90f5a57b2b80472 from peer dc881d7ec7fb47a2897eef434cf18767 ({username='slave'} at 127.0.0.1:50804)
I20240503 02:01:12.115669 3964 tablet_copy_service.cc:161] P 1fe097edfada4dc3b18e7f74ff6dd980: Beginning new tablet copy session on tablet 226ca5ac33d14b43b90f5a57b2b80472 from peer dc881d7ec7fb47a2897eef434cf18767 at {username='slave'} at 127.0.0.1:50804: session id = dc881d7ec7fb47a2897eef434cf18767-226ca5ac33d14b43b90f5a57b2b80472
I20240503 02:01:12.116825 3964 tablet_copy_source_session.cc:205] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:12.117640 4725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226ca5ac33d14b43b90f5a57b2b80472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:12.119818 4725 tablet_copy_client.cc:783] T 226ca5ac33d14b43b90f5a57b2b80472 P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:12.119982 4725 tablet_copy_client.cc:647] T 226ca5ac33d14b43b90f5a57b2b80472 P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:12.120977 4729 tablet_copy_client.cc:1002] T 226ca5ac33d14b43b90f5a57b2b80472 P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:12.128523 4730 ts_tablet_manager.cc:902] T e253a96048114d0ebfa385eda5b0c7b0 P 38e9ca6a49c4402abdd585ffd8f435ee: Initiating tablet copy from peer 12e9ea600473443b80086df227e94dfc (127.0.99.2:42879)
I20240503 02:01:12.129132 4730 tablet_copy_client.cc:308] T e253a96048114d0ebfa385eda5b0c7b0 P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:42879
I20240503 02:01:12.129575 3890 tablet_copy_service.cc:140] P 12e9ea600473443b80086df227e94dfc: Received BeginTabletCopySession request for tablet e253a96048114d0ebfa385eda5b0c7b0 from peer 38e9ca6a49c4402abdd585ffd8f435ee ({username='slave'} at 127.0.0.1:56290)
I20240503 02:01:12.129673 3890 tablet_copy_service.cc:161] P 12e9ea600473443b80086df227e94dfc: Beginning new tablet copy session on tablet e253a96048114d0ebfa385eda5b0c7b0 from peer 38e9ca6a49c4402abdd585ffd8f435ee at {username='slave'} at 127.0.0.1:56290: session id = 38e9ca6a49c4402abdd585ffd8f435ee-e253a96048114d0ebfa385eda5b0c7b0
I20240503 02:01:12.130672 3890 tablet_copy_source_session.cc:205] T e253a96048114d0ebfa385eda5b0c7b0 P 12e9ea600473443b80086df227e94dfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:12.131448 4730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e253a96048114d0ebfa385eda5b0c7b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:12.133769 4730 tablet_copy_client.cc:783] T e253a96048114d0ebfa385eda5b0c7b0 P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:12.134011 4730 tablet_copy_client.cc:647] T e253a96048114d0ebfa385eda5b0c7b0 P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:12.135265 4732 tablet_copy_client.cc:1002] T e253a96048114d0ebfa385eda5b0c7b0 P 38e9ca6a49c4402abdd585ffd8f435ee: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:12.143144 4735 ts_tablet_manager.cc:902] T 88bdd7e3ba2d4793907feb426e7bef9c P dc881d7ec7fb47a2897eef434cf18767: Initiating tablet copy from peer 38e9ca6a49c4402abdd585ffd8f435ee (127.0.99.4:34437)
I20240503 02:01:12.143782 4735 tablet_copy_client.cc:308] T 88bdd7e3ba2d4793907feb426e7bef9c P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.4:34437
I20240503 02:01:12.147431 4214 tablet_copy_service.cc:140] P 38e9ca6a49c4402abdd585ffd8f435ee: Received BeginTabletCopySession request for tablet 88bdd7e3ba2d4793907feb426e7bef9c from peer dc881d7ec7fb47a2897eef434cf18767 ({username='slave'} at 127.0.0.1:54260)
I20240503 02:01:12.147558 4214 tablet_copy_service.cc:161] P 38e9ca6a49c4402abdd585ffd8f435ee: Beginning new tablet copy session on tablet 88bdd7e3ba2d4793907feb426e7bef9c from peer dc881d7ec7fb47a2897eef434cf18767 at {username='slave'} at 127.0.0.1:54260: session id = dc881d7ec7fb47a2897eef434cf18767-88bdd7e3ba2d4793907feb426e7bef9c
I20240503 02:01:12.148604 4214 tablet_copy_source_session.cc:205] T 88bdd7e3ba2d4793907feb426e7bef9c P 38e9ca6a49c4402abdd585ffd8f435ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:12.149442 4735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bdd7e3ba2d4793907feb426e7bef9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:12.151697 4735 tablet_copy_client.cc:783] T 88bdd7e3ba2d4793907feb426e7bef9c P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:12.151954 4735 tablet_copy_client.cc:647] T 88bdd7e3ba2d4793907feb426e7bef9c P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:12.153082 4739 tablet_copy_client.cc:1002] T 88bdd7e3ba2d4793907feb426e7bef9c P dc881d7ec7fb47a2897eef434cf18767: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:12.244311 4741 ts_tablet_manager.cc:902] T 98895d45cbd64b52b497261821d8ae53 P cfdcf4a7a9734e82bddf13e25872e410: Initiating tablet copy from peer dc881d7ec7fb47a2897eef434cf18767 (127.0.99.5:42349)
I20240503 02:01:12.245041 4741 tablet_copy_client.cc:308] T 98895d45cbd64b52b497261821d8ae53 P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.5:42349
I20240503 02:01:12.245543 4290 tablet_copy_service.cc:140] P dc881d7ec7fb47a2897eef434cf18767: Received BeginTabletCopySession request for tablet 98895d45cbd64b52b497261821d8ae53 from peer cfdcf4a7a9734e82bddf13e25872e410 ({username='slave'} at 127.0.0.1:51478)
I20240503 02:01:12.245661 4290 tablet_copy_service.cc:161] P dc881d7ec7fb47a2897eef434cf18767: Beginning new tablet copy session on tablet 98895d45cbd64b52b497261821d8ae53 from peer cfdcf4a7a9734e82bddf13e25872e410 at {username='slave'} at 127.0.0.1:51478: session id = cfdcf4a7a9734e82bddf13e25872e410-98895d45cbd64b52b497261821d8ae53
I20240503 02:01:12.245754 4691 raft_consensus.cc:979] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980: : Instructing follower 12e9ea600473443b80086df227e94dfc to start an election
I20240503 02:01:12.245815 4691 raft_consensus.cc:1067] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 4 LEADER]: Signalling peer 12e9ea600473443b80086df227e94dfc to start an election
I20240503 02:01:12.246178 3878 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
from {username='slave'} at 127.0.0.1:56244
I20240503 02:01:12.246294 3878 raft_consensus.cc:482] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:12.246335 3878 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:01:12.246884 4290 tablet_copy_source_session.cc:205] T 98895d45cbd64b52b497261821d8ae53 P dc881d7ec7fb47a2897eef434cf18767: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:12.247486 3878 raft_consensus.cc:504] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:12.247815 4741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98895d45cbd64b52b497261821d8ae53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:12.247889 3878 leader_election.cc:288] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 5 election: Requested vote from peers 94da6cefbea94c72a5d91b8d10f31bc9 (127.0.99.1:39193), 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
I20240503 02:01:12.248126 3801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "94da6cefbea94c72a5d91b8d10f31bc9"
I20240503 02:01:12.248242 3801 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:01:12.248376 3950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472" candidate_uuid: "12e9ea600473443b80086df227e94dfc" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "1fe097edfada4dc3b18e7f74ff6dd980"
I20240503 02:01:12.248481 3950 raft_consensus.cc:3032] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 02:01:12.248526 3950 raft_consensus.cc:729] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1fe097edfada4dc3b18e7f74ff6dd980, State: Running, Role: LEADER
I20240503 02:01:12.248661 3950 consensus_queue.cc:260] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:12.248836 3950 raft_consensus.cc:3037] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 4 FOLLOWER]: Advancing to term 5
I20240503 02:01:12.249212 3801 raft_consensus.cc:2454] T 226ca5ac33d14b43b90f5a57b2b80472 P 94da6cefbea94c72a5d91b8d10f31bc9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 5.
I20240503 02:01:12.249553 3950 raft_consensus.cc:2454] T 226ca5ac33d14b43b90f5a57b2b80472 P 1fe097edfada4dc3b18e7f74ff6dd980 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e9ea600473443b80086df227e94dfc in term 5.
I20240503 02:01:12.249550 3845 leader_election.cc:302] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e9ea600473443b80086df227e94dfc, 94da6cefbea94c72a5d91b8d10f31bc9; no voters:
I20240503 02:01:12.249755 4697 raft_consensus.cc:2781] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 5 FOLLOWER]: Leader election won for term 5
I20240503 02:01:12.249940 4697 raft_consensus.cc:686] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 5 LEADER]: Becoming Leader. State: Replica: 12e9ea600473443b80086df227e94dfc, State: Running, Role: LEADER
I20240503 02:01:12.250084 4697 consensus_queue.cc:237] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } }
I20240503 02:01:12.250470 4741 tablet_copy_client.cc:783] T 98895d45cbd64b52b497261821d8ae53 P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:12.250612 4741 tablet_copy_client.cc:647] T 98895d45cbd64b52b497261821d8ae53 P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:12.251667 4744 tablet_copy_client.cc:1002] T 98895d45cbd64b52b497261821d8ae53 P cfdcf4a7a9734e82bddf13e25872e410: tablet copy: Injecting latency into file download: 2000
I20240503 02:01:12.251989 3695 catalog_manager.cc:5484] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc reported cstate change: term changed from 4 to 5, leader changed from 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3) to 12e9ea600473443b80086df227e94dfc (127.0.99.2). New cstate: current_term: 5 leader_uuid: "12e9ea600473443b80086df227e94dfc" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94da6cefbea94c72a5d91b8d10f31bc9" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 39193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e9ea600473443b80086df227e94dfc" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42879 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe097edfada4dc3b18e7f74ff6dd980" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 34683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc881d7ec7fb47a2897eef434cf18767" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 42349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:01:12.253132 4748 ts_tablet_manager.cc:902] T cef107d87fa14a329bfb5c9b22f0d808 P 38e9ca6a49c4402abdd585ffd8f435ee: Initiating tablet copy from peer 1fe097edfada4dc3b18e7f74ff6dd980 (127.0.99.3:34683)
F20240503 02:01:12.253319 4748 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1841194d17d64058acfe79adc5d647a0.queue_time_us, tablet-download-pool-1841194d17d64058acfe79adc5d647a0.queue_time_us) (tablet-download-pool-1841194d17d64058acfe79adc5d647a0.run_cpu_time_us, tablet-download-pool-1841194d17d64058acfe79adc5d647a0.run_cpu_time_us) (tablet-download-pool-1841194d17d64058acfe79adc5d647a0.run_wall_time_us, tablet-download-pool-1841194d17d64058acfe79adc5d647a0.run_wall_time_us) (tablet-download-pool-226ca5ac33d14b43b90f5a57b2b80472.queue_time_us, tablet-download-pool-226ca5ac33d14b43b90f5a57b2b80472.queue_time_us) (tablet-download-pool-226ca5ac33d14b43b90f5a57b2b80472.run_cpu_time_us, tablet-download-pool-226ca5ac33d14b43b90f5a57b2b80472.run_cpu_time_us) (tablet-download-pool-226ca5ac33d14b43b90f5a57b2b80472.run_wall_time_us, tablet-download-pool-226ca5ac33d14b43b90f5a57b2b80472.run_wall_time_us) (tablet-download-pool-4f1c79d275004dfeb03f56c07b3904af.queue_time_us, tablet-download-pool-4f1c79d275004dfeb03f56c07b3904af.queue_time_us) (tablet-download-pool-4f1c79d275004dfeb03f56c07b3904af.run_cpu_time_us, tablet-download-pool-4f1c79d275004dfeb03f56c07b3904af.run_cpu_time_us) (tablet-download-pool-4f1c79d275004dfeb03f56c07b3904af.run_wall_time_us, tablet-download-pool-4f1c79d275004dfeb03f56c07b3904af.run_wall_time_us) (tablet-download-pool-561921bd5097434497ea1f034f86749b.queue_time_us, tablet-download-pool-561921bd5097434497ea1f034f86749b.queue_time_us) (tablet-download-pool-561921bd5097434497ea1f034f86749b.run_cpu_time_us, tablet-download-pool-561921bd5097434497ea1f034f86749b.run_cpu_time_us) (tablet-download-pool-561921bd5097434497ea1f034f86749b.run_wall_time_us, tablet-download-pool-561921bd5097434497ea1f034f86749b.run_wall_time_us) (tablet-download-pool-5cfd82671b084308af19ee0d7b151cd5.queue_time_us, tablet-download-pool-5cfd82671b084308af19ee0d7b151cd5.queue_time_us) (tablet-download-pool-5cfd82671b084308af19ee0d7b151cd5.run_cpu_time_us, tablet-download-pool-5cfd82671b084308af19ee0d7b151cd5.run_cpu_time_us) (tablet-download-pool-5cfd82671b084308af19ee0d7b151cd5.run_wall_time_us, tablet-download-pool-5cfd82671b084308af19ee0d7b151cd5.run_wall_time_us) (tablet-download-pool-80f0e3a5143d4bd7a132ee4b6fcb6f89.queue_time_us, tablet-download-pool-80f0e3a5143d4bd7a132ee4b6fcb6f89.queue_time_us) (tablet-download-pool-80f0e3a5143d4bd7a132ee4b6fcb6f89.run_cpu_time_us, tablet-download-pool-80f0e3a5143d4bd7a132ee4b6fcb6f89.run_cpu_time_us) (tablet-download-pool-80f0e3a5143d4bd7a132ee4b6fcb6f89.run_wall_time_us, tablet-download-pool-80f0e3a5143d4bd7a132ee4b6fcb6f89.run_wall_time_us) (tablet-download-pool-88bdd7e3ba2d4793907feb426e7bef9c.queue_time_us, tablet-download-pool-88bdd7e3ba2d4793907feb426e7bef9c.queue_time_us) (tablet-download-pool-88bdd7e3ba2d4793907feb426e7bef9c.run_cpu_time_us, tablet-download-pool-88bdd7e3ba2d4793907feb426e7bef9c.run_cpu_time_us) (tablet-download-pool-88bdd7e3ba2d4793907feb426e7bef9c.run_wall_time_us, tablet-download-pool-88bdd7e3ba2d4793907feb426e7bef9c.run_wall_time_us) (tablet-download-pool-98895d45cbd64b52b497261821d8ae53.queue_time_us, tablet-download-pool-98895d45cbd64b52b497261821d8ae53.queue_time_us) (tablet-download-pool-98895d45cbd64b52b497261821d8ae53.run_cpu_time_us, tablet-download-pool-98895d45cbd64b52b497261821d8ae53.run_cpu_time_us) (tablet-download-pool-98895d45cbd64b52b497261821d8ae53.run_wall_time_us, tablet-download-pool-98895d45cbd64b52b497261821d8ae53.run_wall_time_us) (tablet-download-pool-b98f5009a3534765806bda2c95f7484c.queue_time_us, tablet-download-pool-b98f5009a3534765806bda2c95f7484c.queue_time_us) (tablet-download-pool-b98f5009a3534765806bda2c95f7484c.run_cpu_time_us, tablet-download-pool-b98f5009a3534765806bda2c95f7484c.run_cpu_time_us) (tablet-download-pool-b98f5009a3534765806bda2c95f7484c.run_wall_time_us, tablet-download-pool-b98f5009a3534765806bda2c95f7484c.run_wall_time_us) (tablet-download-pool-cef107d87fa14a329bfb5c9b22f0d808.queue_time_us, tablet-download-pool-cef107d87fa14a329bfb5c9b22f0d808.queue_time_us) (tablet-download-pool-e253a96048114d0ebfa385eda5b0c7b0.queue_time_us, tablet-download-pool-e253a96048114d0ebfa385eda5b0c7b0.queue_time_us) (tablet-download-pool-e253a96048114d0ebfa385eda5b0c7b0.run_cpu_time_us, tablet-download-pool-e253a96048114d0ebfa385eda5b0c7b0.run_cpu_time_us) (tablet-download-pool-e253a96048114d0ebfa385eda5b0c7b0.run_wall_time_us, tablet-download-pool-e253a96048114d0ebfa385eda5b0c7b0.run_wall_time_us) (tablet-download-pool-e8d5be30376a4040a650e1be3d6643f8.queue_time_us, tablet-download-pool-e8d5be30376a4040a650e1be3d6643f8.queue_time_us) (tablet-download-pool-e8d5be30376a4040a650e1be3d6643f8.run_cpu_time_us, tablet-download-pool-e8d5be30376a4040a650e1be3d6643f8.run_cpu_time_us) (tablet-download-pool-e8d5be30376a4040a650e1be3d6643f8.run_wall_time_us, tablet-download-pool-e8d5be30376a4040a650e1be3d6643f8.run_wall_time_us) (tablet-download-pool-eaa6c34cc53b4061abb42f32a272a67e.queue_time_us, tablet-download-pool-eaa6c34cc53b4061abb42f32a272a67e.queue_time_us) (tablet-download-pool-eaa6c34cc53b4061abb42f32a272a67e.run_cpu_time_us, tablet-download-pool-eaa6c34cc53b4061abb42f32a272a67e.run_cpu_time_us) (tablet-download-pool-eaa6c34cc53b4061abb42f32a272a67e.run_wall_time_us, tablet-download-pool-eaa6c34cc53b4061abb42f32a272a67e.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714701672 (unix time) try "date -d @1714701672" if you are using GNU date ***
I20240503 02:01:12.255175 3878 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56232
I20240503 02:01:12.255306 3878 raft_consensus.cc:595] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 5 LEADER]: Received request to transfer leadership
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000018c) received by PID 396 (TID 0x7f92544e8700) from PID 396; stack trace: ***
I20240503 02:01:12.260358 3878 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56232
I20240503 02:01:12.260493 3878 raft_consensus.cc:595] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 5 LEADER]: Received request to transfer leadership
I20240503 02:01:12.261456 3878 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "226ca5ac33d14b43b90f5a57b2b80472"
dest_uuid: "12e9ea600473443b80086df227e94dfc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56232
I20240503 02:01:12.261575 3878 raft_consensus.cc:595] T 226ca5ac33d14b43b90f5a57b2b80472 P 12e9ea600473443b80086df227e94dfc [term 5 LEADER]: Received request to transfer leadership
W20240503 02:01:12.261893 3750 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 226ca5ac33d14b43b90f5a57b2b80472 already in progress
W20240503 02:01:12.261947 3750 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 226ca5ac33d14b43b90f5a57b2b80472 already in progress
@ 0x7f92f8776040 (unknown) at ??:0
@ 0x7f92f8775fb7 gsignal at ??:0
@ 0x7f92f8777921 abort at ??:0
@ 0x7f92f947adcd google::LogMessage::Fail() at ??:0
@ 0x7f92f947eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f92f947a7cc google::LogMessage::Flush() at ??:0
@ 0x7f92f947bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f92f9fd2044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f92f9fb6f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f92f9fb4efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f92faf3177a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f92faf3aa45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f92fafbb76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f92fafcf4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f92fafba226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f92fafc9c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f92fb5243cc std::function<>::operator()() at ??:0
@ 0x7f92f9fb9e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f92f9fba6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f92f9fbc03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f92fb5243cc std::function<>::operator()() at ??:0
@ 0x7f92f9fa9807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f92f67ab6db start_thread at ??:0
@ 0x7f92f885871f clone at ??:0