[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:48:35.224565 1958 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.233.190:37275,127.1.233.189:44091,127.1.233.188:45033
I20240502 01:48:35.226315 1958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:48:35.227416 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:35.246465 1967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:35.246474 1965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:35.246524 1964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:35.248436 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:36.380894 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:36.381181 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:36.381394 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614516381368 us; error 0 us; skew 500 ppm
I20240502 01:48:36.382232 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:36.388094 1958 webserver.cc:460] Webserver started at http://127.1.233.190:41849/ using document root <none> and password file <none>
I20240502 01:48:36.389122 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:36.389374 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:36.389845 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:36.398556 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-0-root/instance:
uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9"
format_stamp: "Formatted at 2024-05-02 01:48:36 on dist-test-slave-2090"
I20240502 01:48:36.406931 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20240502 01:48:36.412499 1973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:36.413727 1958 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:48:36.414098 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-0-root
uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9"
format_stamp: "Formatted at 2024-05-02 01:48:36 on dist-test-slave-2090"
I20240502 01:48:36.414469 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:36.480226 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:36.481894 1958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:48:36.482406 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:36.559185 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.190:37275
I20240502 01:48:36.559266 2025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.190:37275 every 8 connection(s)
I20240502 01:48:36.565877 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:48:36.566898 2026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:48:36.572731 2028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:36.574352 2029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:36.578419 2031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:36.583302 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:36.584116 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:36.584381 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:36.584545 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614516584529 us; error 0 us; skew 500 ppm
I20240502 01:48:36.585089 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:36.589985 1958 webserver.cc:460] Webserver started at http://127.1.233.189:36457/ using document root <none> and password file <none>
I20240502 01:48:36.590544 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:36.590768 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:36.591044 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:36.592276 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-1-root/instance:
uuid: "123815c8f3774e03893479171b19b0d7"
format_stamp: "Formatted at 2024-05-02 01:48:36 on dist-test-slave-2090"
I20240502 01:48:36.590595 2026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:36.599952 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20240502 01:48:36.605604 2038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:36.606551 1958 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20240502 01:48:36.606849 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-1-root
uuid: "123815c8f3774e03893479171b19b0d7"
format_stamp: "Formatted at 2024-05-02 01:48:36 on dist-test-slave-2090"
I20240502 01:48:36.607169 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:36.619230 2026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:48:36.621801 1976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.233.189:44091: connect: Connection refused (error 111)
W20240502 01:48:36.626145 2026 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.1.233.189:44091: Network error: Client connection negotiation failed: client connection to 127.1.233.189:44091: connect: Connection refused (error 111)
I20240502 01:48:36.630867 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:36.632393 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:36.675905 2026 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } attempt: 1
W20240502 01:48:36.681239 2026 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.1.233.189:44091: Network error: Client connection negotiation failed: client connection to 127.1.233.189:44091: connect: Connection refused (error 111)
I20240502 01:48:36.711223 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.189:44091
I20240502 01:48:36.711413 2091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.189:44091 every 8 connection(s)
I20240502 01:48:36.716499 2092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:36.717448 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:48:36.722880 2092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:48:36.725080 2094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:36.727839 2095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:36.732461 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:36.734081 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:36.734882 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:36.735118 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:36.735316 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614516735295 us; error 0 us; skew 500 ppm
I20240502 01:48:36.735859 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:36.739792 1958 webserver.cc:460] Webserver started at http://127.1.233.188:43801/ using document root <none> and password file <none>
I20240502 01:48:36.740550 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:36.740850 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:36.740854 2092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:36.741293 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:36.742933 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-2-root/instance:
uuid: "ae926fe6fe99435e94ebebd607e34fd6"
format_stamp: "Formatted at 2024-05-02 01:48:36 on dist-test-slave-2090"
I20240502 01:48:36.749383 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20240502 01:48:36.749920 2026 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } attempt: 2
I20240502 01:48:36.760385 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:36.762141 1958 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.005s sys 0.000s
I20240502 01:48:36.762594 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-2-root
uuid: "ae926fe6fe99435e94ebebd607e34fd6"
format_stamp: "Formatted at 2024-05-02 01:48:36 on dist-test-slave-2090"
I20240502 01:48:36.763026 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:36.764889 2092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:36.769330 2026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:48:36.775462 2092 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.1.233.188:45033: Network error: Client connection negotiation failed: client connection to 127.1.233.188:45033: connect: Connection refused (error 111)
W20240502 01:48:36.777849 2026 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.1.233.188:45033: Network error: Client connection negotiation failed: client connection to 127.1.233.188:45033: connect: Connection refused (error 111)
I20240502 01:48:36.783308 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:36.784862 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:36.809382 2026 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } attempt: 1
W20240502 01:48:36.814301 2026 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.1.233.188:45033: Network error: Client connection negotiation failed: client connection to 127.1.233.188:45033: connect: Connection refused (error 111)
I20240502 01:48:36.820385 2092 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } attempt: 1
W20240502 01:48:36.827384 2092 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.1.233.188:45033: Network error: Client connection negotiation failed: client connection to 127.1.233.188:45033: connect: Connection refused (error 111)
I20240502 01:48:36.866848 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.188:45033
I20240502 01:48:36.866953 2159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.188:45033 every 8 connection(s)
I20240502 01:48:36.871001 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:48:36.872022 2160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:36.877714 2160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:36.890018 2026 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } attempt: 2
I20240502 01:48:36.891458 2160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:36.895056 2092 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } attempt: 2
I20240502 01:48:36.903597 2160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:36.917582 2026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9: Bootstrap starting.
I20240502 01:48:36.917852 2092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7: Bootstrap starting.
I20240502 01:48:36.923949 2092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:36.924307 2026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:36.926273 2160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6: Bootstrap starting.
I20240502 01:48:36.926595 2092 log.cc:824] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7: Log is configured to *not* fsync() on all Append() calls
I20240502 01:48:36.932873 2092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7: No bootstrap required, opened a new log
I20240502 01:48:36.932876 2026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9: No bootstrap required, opened a new log
I20240502 01:48:36.933578 2160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:36.940173 2160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6: No bootstrap required, opened a new log
I20240502 01:48:36.956611 2026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } }
I20240502 01:48:36.956508 2092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } }
I20240502 01:48:36.956508 2160 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } }
I20240502 01:48:36.957535 2026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:36.957686 2092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:36.957778 2160 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:36.957958 2026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8f1331aa6a47d7b586d7e7cbbe53b9, State: Initialized, Role: FOLLOWER
I20240502 01:48:36.958042 2092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123815c8f3774e03893479171b19b0d7, State: Initialized, Role: FOLLOWER
I20240502 01:48:36.958151 2160 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae926fe6fe99435e94ebebd607e34fd6, State: Initialized, Role: FOLLOWER
I20240502 01:48:36.959165 2026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } }
I20240502 01:48:36.959159 2092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } }
I20240502 01:48:36.959476 2160 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } }
I20240502 01:48:36.965342 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } } }
I20240502 01:48:36.967396 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:36.968434 2092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:48:36.969962 2160 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:48:36.970072 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } } }
I20240502 01:48:36.971463 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:36.971643 2026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:48:36.970163 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } } }
I20240502 01:48:36.972334 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [sys.catalog]: This master's current role is: FOLLOWER
W20240502 01:48:36.989135 1979 tablet.cc:2369] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:48:37.033964 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240502 01:48:37.034265 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 01:48:37.035869 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:37.040339 2205 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:48:37.040851 2205 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 01:48:37.043713 2193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:48:37.044106 2193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 01:48:37.047641 2207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:37.049198 2206 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:48:37.049584 2206 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 01:48:37.059058 2208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:37.061010 2210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:37.062100 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:37.063225 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:37.063511 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:37.063720 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614517063704 us; error 0 us; skew 500 ppm
I20240502 01:48:37.064582 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:37.067942 1958 webserver.cc:460] Webserver started at http://127.1.233.129:46423/ using document root <none> and password file <none>
I20240502 01:48:37.068666 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:37.068917 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:37.069283 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:37.071115 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-0-root/instance:
uuid: "e2708f2fd3a94638a4a33c8a951a2e1e"
format_stamp: "Formatted at 2024-05-02 01:48:37 on dist-test-slave-2090"
I20240502 01:48:37.077980 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.005s
I20240502 01:48:37.082780 2215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:37.083741 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:48:37.084184 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "e2708f2fd3a94638a4a33c8a951a2e1e"
format_stamp: "Formatted at 2024-05-02 01:48:37 on dist-test-slave-2090"
I20240502 01:48:37.084589 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:37.128289 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:37.130209 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:37.132207 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:37.137778 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:37.138149 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:37.138473 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:37.138695 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:37.227304 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:37.228123 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } }
I20240502 01:48:37.232779 2001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "123815c8f3774e03893479171b19b0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" is_pre_election: true
I20240502 01:48:37.233690 2001 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123815c8f3774e03893479171b19b0d7 in term 0.
I20240502 01:48:37.233879 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c8f1331aa6a47d7b586d7e7cbbe53b9 (127.1.233.190:37275), ae926fe6fe99435e94ebebd607e34fd6 (127.1.233.188:45033)
I20240502 01:48:37.234712 2170 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:37.235564 2042 leader_election.cc:302] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123815c8f3774e03893479171b19b0d7, 1c8f1331aa6a47d7b586d7e7cbbe53b9; no voters:
I20240502 01:48:37.235319 2170 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } }
I20240502 01:48:37.235980 2134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "123815c8f3774e03893479171b19b0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae926fe6fe99435e94ebebd607e34fd6" is_pre_election: true
I20240502 01:48:37.236804 2134 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123815c8f3774e03893479171b19b0d7 in term 0.
I20240502 01:48:37.236881 2168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:37.237722 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:37.238102 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:37.239243 2001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae926fe6fe99435e94ebebd607e34fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" is_pre_election: true
I20240502 01:48:37.240092 2001 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae926fe6fe99435e94ebebd607e34fd6 in term 0.
I20240502 01:48:37.241407 2170 leader_election.cc:288] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c8f1331aa6a47d7b586d7e7cbbe53b9 (127.1.233.190:37275), 123815c8f3774e03893479171b19b0d7 (127.1.233.189:44091)
I20240502 01:48:37.242478 2110 leader_election.cc:302] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8f1331aa6a47d7b586d7e7cbbe53b9, ae926fe6fe99435e94ebebd607e34fd6; no voters:
I20240502 01:48:37.242673 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae926fe6fe99435e94ebebd607e34fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123815c8f3774e03893479171b19b0d7" is_pre_election: true
I20240502 01:48:37.243844 2170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:37.244402 2170 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:37.246634 2170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:37.255141 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:34059
I20240502 01:48:37.255617 2279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:34059 every 8 connection(s)
I20240502 01:48:37.287206 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:48:37.302533 2280 heartbeater.cc:347] Connected to a master server at 127.1.233.188:45033
I20240502 01:48:37.303109 2280 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:37.307016 2280 heartbeater.cc:511] Master 127.1.233.188:45033 requested a full tablet report, sending...
W20240502 01:48:37.314482 2287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:37.316108 2281 heartbeater.cc:347] Connected to a master server at 127.1.233.189:44091
I20240502 01:48:37.316764 2281 heartbeater.cc:464] Registering TS with master...
W20240502 01:48:37.317554 2288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:37.318223 2281 heartbeater.cc:511] Master 127.1.233.189:44091 requested a full tablet report, sending...
I20240502 01:48:37.319124 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } }
I20240502 01:48:37.321338 2066 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae926fe6fe99435e94ebebd607e34fd6 in current term 1: Already voted for candidate 123815c8f3774e03893479171b19b0d7 in this term.
I20240502 01:48:37.322688 2170 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } }
I20240502 01:48:37.325222 2170 leader_election.cc:288] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [CANDIDATE]: Term 1 election: Requested vote from peers 1c8f1331aa6a47d7b586d7e7cbbe53b9 (127.1.233.190:37275), 123815c8f3774e03893479171b19b0d7 (127.1.233.189:44091)
I20240502 01:48:37.326211 2056 ts_manager.cc:194] Registered new tserver with Master: e2708f2fd3a94638a4a33c8a951a2e1e (127.1.233.129:34059)
I20240502 01:48:37.328086 2001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae926fe6fe99435e94ebebd607e34fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9"
I20240502 01:48:37.328295 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae926fe6fe99435e94ebebd607e34fd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123815c8f3774e03893479171b19b0d7"
I20240502 01:48:37.328766 2001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:37.331328 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [CANDIDATE]: Term 1 election: Requested vote from peers 1c8f1331aa6a47d7b586d7e7cbbe53b9 (127.1.233.190:37275), ae926fe6fe99435e94ebebd607e34fd6 (127.1.233.188:45033)
I20240502 01:48:37.333084 2000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "123815c8f3774e03893479171b19b0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9"
I20240502 01:48:37.334669 2134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "123815c8f3774e03893479171b19b0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae926fe6fe99435e94ebebd607e34fd6"
I20240502 01:48:37.335877 2134 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 123815c8f3774e03893479171b19b0d7 in current term 1: Already voted for candidate ae926fe6fe99435e94ebebd607e34fd6 in this term.
W20240502 01:48:37.337579 2290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:37.338162 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:37.339782 2283 heartbeater.cc:347] Connected to a master server at 127.1.233.190:37275
I20240502 01:48:37.340226 2283 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:37.340013 2042 leader_election.cc:302] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123815c8f3774e03893479171b19b0d7; no voters: 1c8f1331aa6a47d7b586d7e7cbbe53b9, ae926fe6fe99435e94ebebd607e34fd6
I20240502 01:48:37.341773 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 01:48:37.341857 2283 heartbeater.cc:511] Master 127.1.233.190:37275 requested a full tablet report, sending...
W20240502 01:48:37.342113 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:37.342325 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614517342304 us; error 0 us; skew 500 ppm
I20240502 01:48:37.342905 2168 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:48:37.343107 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:37.349462 1958 webserver.cc:460] Webserver started at http://127.1.233.130:40355/ using document root <none> and password file <none>
I20240502 01:48:37.349629 2124 ts_manager.cc:194] Registered new tserver with Master: e2708f2fd3a94638a4a33c8a951a2e1e (127.1.233.129:34059)
I20240502 01:48:37.350304 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:37.350646 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:37.350951 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:37.352236 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-1-root/instance:
uuid: "b4740bccea484b20909e00061797a44d"
format_stamp: "Formatted at 2024-05-02 01:48:37 on dist-test-slave-2090"
I20240502 01:48:37.357812 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20240502 01:48:37.361603 2295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:37.362633 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 01:48:37.363039 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-1-root
uuid: "b4740bccea484b20909e00061797a44d"
format_stamp: "Formatted at 2024-05-02 01:48:37 on dist-test-slave-2090"
I20240502 01:48:37.363471 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:37.383071 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:37.384555 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:37.387308 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:37.391388 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:37.391705 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:37.392114 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:37.392357 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:37.420596 2001 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae926fe6fe99435e94ebebd607e34fd6 in term 1.
I20240502 01:48:37.423662 1991 ts_manager.cc:194] Registered new tserver with Master: e2708f2fd3a94638a4a33c8a951a2e1e (127.1.233.129:34059)
I20240502 01:48:37.423564 2110 leader_election.cc:302] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c8f1331aa6a47d7b586d7e7cbbe53b9, ae926fe6fe99435e94ebebd607e34fd6; no voters: 123815c8f3774e03893479171b19b0d7
I20240502 01:48:37.425678 2170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:37.430531 2170 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 1 LEADER]: Becoming Leader. State: Replica: ae926fe6fe99435e94ebebd607e34fd6, State: Running, Role: LEADER
I20240502 01:48:37.432828 2170 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } }
I20240502 01:48:37.444720 2315 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae926fe6fe99435e94ebebd607e34fd6. Latest consensus state: current_term: 1 leader_uuid: "ae926fe6fe99435e94ebebd607e34fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } } }
I20240502 01:48:37.445794 2315 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:37.454057 2323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:48:37.464691 2323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:48:37.489845 2066 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 1 FOLLOWER]: Refusing update from remote peer ae926fe6fe99435e94ebebd607e34fd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:37.491982 2315 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:37.498037 2001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [term 1 FOLLOWER]: Refusing update from remote peer ae926fe6fe99435e94ebebd607e34fd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:37.500072 2315 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:37.536070 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae926fe6fe99435e94ebebd607e34fd6. Latest consensus state: current_term: 1 leader_uuid: "ae926fe6fe99435e94ebebd607e34fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } } }
I20240502 01:48:37.537377 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:37.549077 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae926fe6fe99435e94ebebd607e34fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } } }
I20240502 01:48:37.550132 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:37.550797 2323 catalog_manager.cc:1347] Generated new cluster ID: 70dac25a92e84f88b94e36f20cae30dd
I20240502 01:48:37.551249 2323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:48:37.582190 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae926fe6fe99435e94ebebd607e34fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } } }
I20240502 01:48:37.583160 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:37.584868 2345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae926fe6fe99435e94ebebd607e34fd6. Latest consensus state: current_term: 1 leader_uuid: "ae926fe6fe99435e94ebebd607e34fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } } }
I20240502 01:48:37.585798 2345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:37.600122 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae926fe6fe99435e94ebebd607e34fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } } }
I20240502 01:48:37.601449 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:37.618487 2323 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:48:37.620981 2323 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:48:37.652326 2345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae926fe6fe99435e94ebebd607e34fd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8f1331aa6a47d7b586d7e7cbbe53b9" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 37275 } } peers { permanent_uuid: "123815c8f3774e03893479171b19b0d7" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 44091 } } peers { permanent_uuid: "ae926fe6fe99435e94ebebd607e34fd6" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 45033 } } }
I20240502 01:48:37.653256 2345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:37.687999 2323 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6: Generated new TSK 0
I20240502 01:48:37.689518 2323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:48:37.691617 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.130:41359
I20240502 01:48:37.691783 2371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.130:41359 every 8 connection(s)
I20240502 01:48:37.716948 2372 heartbeater.cc:347] Connected to a master server at 127.1.233.188:45033
I20240502 01:48:37.717429 2372 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:37.718485 2372 heartbeater.cc:511] Master 127.1.233.188:45033 requested a full tablet report, sending...
I20240502 01:48:37.720904 2124 ts_manager.cc:194] Registered new tserver with Master: b4740bccea484b20909e00061797a44d (127.1.233.130:41359)
I20240502 01:48:37.723919 2124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53396
I20240502 01:48:37.727362 2373 heartbeater.cc:347] Connected to a master server at 127.1.233.189:44091
I20240502 01:48:37.727797 2373 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:37.728571 2373 heartbeater.cc:511] Master 127.1.233.189:44091 requested a full tablet report, sending...
I20240502 01:48:37.731482 2056 ts_manager.cc:194] Registered new tserver with Master: b4740bccea484b20909e00061797a44d (127.1.233.130:41359)
I20240502 01:48:37.734767 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:48:37.738355 2375 heartbeater.cc:347] Connected to a master server at 127.1.233.190:37275
I20240502 01:48:37.738751 2375 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:37.739362 2375 heartbeater.cc:511] Master 127.1.233.190:37275 requested a full tablet report, sending...
I20240502 01:48:37.742492 1991 ts_manager.cc:194] Registered new tserver with Master: b4740bccea484b20909e00061797a44d (127.1.233.130:41359)
W20240502 01:48:37.744508 2378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:37.748005 2379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:37.752233 1958 server_base.cc:1034] running on GCE node
W20240502 01:48:37.753615 2381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:37.754796 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:37.755052 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:37.755270 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614517755251 us; error 0 us; skew 500 ppm
I20240502 01:48:37.755998 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:37.758852 1958 webserver.cc:460] Webserver started at http://127.1.233.131:40089/ using document root <none> and password file <none>
I20240502 01:48:37.759316 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:37.759519 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:37.759799 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:37.760977 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-2-root/instance:
uuid: "710c666b39404d5abefb60e0df99cb58"
format_stamp: "Formatted at 2024-05-02 01:48:37 on dist-test-slave-2090"
I20240502 01:48:37.766083 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20240502 01:48:37.770071 2386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:37.771118 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240502 01:48:37.771436 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-2-root
uuid: "710c666b39404d5abefb60e0df99cb58"
format_stamp: "Formatted at 2024-05-02 01:48:37 on dist-test-slave-2090"
I20240502 01:48:37.771751 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:37.802644 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:37.804018 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:37.805773 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:37.809865 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:37.810111 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:37.810405 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:37.810631 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:37.922915 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.131:34151
I20240502 01:48:37.923234 2449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.131:34151 every 8 connection(s)
I20240502 01:48:37.950841 2451 heartbeater.cc:347] Connected to a master server at 127.1.233.189:44091
I20240502 01:48:37.951386 2451 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:37.952523 2451 heartbeater.cc:511] Master 127.1.233.189:44091 requested a full tablet report, sending...
I20240502 01:48:37.955900 2056 ts_manager.cc:194] Registered new tserver with Master: 710c666b39404d5abefb60e0df99cb58 (127.1.233.131:34151)
I20240502 01:48:37.957633 2450 heartbeater.cc:347] Connected to a master server at 127.1.233.188:45033
I20240502 01:48:37.958104 2450 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:37.959131 2450 heartbeater.cc:511] Master 127.1.233.188:45033 requested a full tablet report, sending...
I20240502 01:48:37.960719 2452 heartbeater.cc:347] Connected to a master server at 127.1.233.190:37275
I20240502 01:48:37.961133 2452 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:37.961753 2124 ts_manager.cc:194] Registered new tserver with Master: 710c666b39404d5abefb60e0df99cb58 (127.1.233.131:34151)
I20240502 01:48:37.961947 2452 heartbeater.cc:511] Master 127.1.233.190:37275 requested a full tablet report, sending...
I20240502 01:48:37.963754 2124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53400
I20240502 01:48:37.964704 1991 ts_manager.cc:194] Registered new tserver with Master: 710c666b39404d5abefb60e0df99cb58 (127.1.233.131:34151)
I20240502 01:48:37.965759 1958 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019902286s
I20240502 01:48:38.045971 2205 catalog_manager.cc:1259] Loaded cluster ID: 70dac25a92e84f88b94e36f20cae30dd
I20240502 01:48:38.046293 2205 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7: loading cluster ID for follower catalog manager: success
I20240502 01:48:38.052696 2205 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7: acquiring CA information for follower catalog manager: success
I20240502 01:48:38.055567 2206 catalog_manager.cc:1259] Loaded cluster ID: 70dac25a92e84f88b94e36f20cae30dd
I20240502 01:48:38.055915 2206 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9: loading cluster ID for follower catalog manager: success
I20240502 01:48:38.058683 2205 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 01:48:38.061861 2206 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9: acquiring CA information for follower catalog manager: success
I20240502 01:48:38.065954 2206 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 01:48:38.356874 2124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53390
I20240502 01:48:38.729049 2372 heartbeater.cc:503] Master 127.1.233.188:45033 was elected leader, sending a full tablet report...
I20240502 01:48:38.967972 2450 heartbeater.cc:503] Master 127.1.233.188:45033 was elected leader, sending a full tablet report...
I20240502 01:48:39.010950 2192 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:39.035074 1958 test_util.cc:268] Using random seed: -850037602
I20240502 01:48:39.115459 2124 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53412:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 01:48:39.250130 2243 tablet_service.cc:1467] Processing CreateTablet for tablet dfb6fe78fae24ff080b55dcb628a5e62 (DEFAULT_TABLE table=test-workload [id=0ea547ec621d40d692ec0fe99fbfe6c4]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:48:39.253104 2243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb6fe78fae24ff080b55dcb628a5e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:39.253983 2415 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4e8abbc236446a9558373c64f6d984 (DEFAULT_TABLE table=test-workload [id=0ea547ec621d40d692ec0fe99fbfe6c4]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:48:39.254730 2326 tablet_service.cc:1467] Processing CreateTablet for tablet 7beb894850264531a87f9d1af24f32fd (DEFAULT_TABLE table=test-workload [id=0ea547ec621d40d692ec0fe99fbfe6c4]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:48:39.255689 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4e8abbc236446a9558373c64f6d984. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:39.255942 2326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7beb894850264531a87f9d1af24f32fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:39.257475 2242 tablet_service.cc:1467] Processing CreateTablet for tablet bc22bf07e280496bb906f534082fb400 (DEFAULT_TABLE table=test-workload [id=0ea547ec621d40d692ec0fe99fbfe6c4]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:48:39.258947 2242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc22bf07e280496bb906f534082fb400. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:39.290230 2486 tablet_bootstrap.cc:492] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d: Bootstrap starting.
I20240502 01:48:39.294440 2487 tablet_bootstrap.cc:492] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e: Bootstrap starting.
I20240502 01:48:39.299258 2488 tablet_bootstrap.cc:492] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58: Bootstrap starting.
I20240502 01:48:39.301126 2486 tablet_bootstrap.cc:654] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:39.308933 2488 tablet_bootstrap.cc:654] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:39.309067 2487 tablet_bootstrap.cc:654] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:39.317842 2486 tablet_bootstrap.cc:492] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d: No bootstrap required, opened a new log
I20240502 01:48:39.318447 2486 ts_tablet_manager.cc:1367] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.007s
I20240502 01:48:39.319861 2487 tablet_bootstrap.cc:492] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e: No bootstrap required, opened a new log
I20240502 01:48:39.320596 2487 ts_tablet_manager.cc:1367] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.004s
I20240502 01:48:39.321547 2486 raft_consensus.cc:348] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4740bccea484b20909e00061797a44d" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41359 } }
I20240502 01:48:39.322377 2486 raft_consensus.cc:374] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:39.322755 2486 raft_consensus.cc:729] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4740bccea484b20909e00061797a44d, State: Initialized, Role: FOLLOWER
I20240502 01:48:39.323630 2486 consensus_queue.cc:260] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4740bccea484b20909e00061797a44d" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41359 } }
I20240502 01:48:39.323933 2487 raft_consensus.cc:348] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2708f2fd3a94638a4a33c8a951a2e1e" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 34059 } }
I20240502 01:48:39.324411 2486 raft_consensus.cc:388] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:48:39.324602 2487 raft_consensus.cc:374] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:39.324811 2486 raft_consensus.cc:482] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:48:39.324851 2488 tablet_bootstrap.cc:492] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58: No bootstrap required, opened a new log
I20240502 01:48:39.325026 2487 raft_consensus.cc:729] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2708f2fd3a94638a4a33c8a951a2e1e, State: Initialized, Role: FOLLOWER
I20240502 01:48:39.325500 2486 raft_consensus.cc:3037] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:39.325919 2488 ts_tablet_manager.cc:1367] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.005s
I20240502 01:48:39.326139 2487 consensus_queue.cc:260] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2708f2fd3a94638a4a33c8a951a2e1e" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 34059 } }
I20240502 01:48:39.326939 2487 raft_consensus.cc:388] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:48:39.327279 2487 raft_consensus.cc:482] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:48:39.327647 2487 raft_consensus.cc:3037] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:39.329357 2488 raft_consensus.cc:348] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710c666b39404d5abefb60e0df99cb58" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 34151 } }
I20240502 01:48:39.330093 2488 raft_consensus.cc:374] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:39.330466 2488 raft_consensus.cc:729] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710c666b39404d5abefb60e0df99cb58, State: Initialized, Role: FOLLOWER
I20240502 01:48:39.331367 2488 consensus_queue.cc:260] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710c666b39404d5abefb60e0df99cb58" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 34151 } }
I20240502 01:48:39.332262 2488 raft_consensus.cc:388] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:48:39.332693 2488 raft_consensus.cc:482] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:48:39.333173 2488 raft_consensus.cc:3037] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:39.340126 2488 raft_consensus.cc:504] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710c666b39404d5abefb60e0df99cb58" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 34151 } }
I20240502 01:48:39.340337 2487 raft_consensus.cc:504] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2708f2fd3a94638a4a33c8a951a2e1e" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 34059 } }
I20240502 01:48:39.341221 2487 leader_election.cc:302] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2708f2fd3a94638a4a33c8a951a2e1e; no voters:
I20240502 01:48:39.341960 2486 raft_consensus.cc:504] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4740bccea484b20909e00061797a44d" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41359 } }
I20240502 01:48:39.341187 2488 leader_election.cc:302] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 710c666b39404d5abefb60e0df99cb58; no voters:
I20240502 01:48:39.343626 2486 leader_election.cc:302] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4740bccea484b20909e00061797a44d; no voters:
I20240502 01:48:39.345268 2487 leader_election.cc:288] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:48:39.345744 2492 raft_consensus.cc:2781] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:39.349069 2488 leader_election.cc:288] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:48:39.350716 2493 raft_consensus.cc:2781] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:39.359011 2492 raft_consensus.cc:686] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [term 1 LEADER]: Becoming Leader. State: Replica: e2708f2fd3a94638a4a33c8a951a2e1e, State: Running, Role: LEADER
I20240502 01:48:39.360385 2492 consensus_queue.cc:237] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2708f2fd3a94638a4a33c8a951a2e1e" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 34059 } }
I20240502 01:48:39.364301 2493 raft_consensus.cc:686] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [term 1 LEADER]: Becoming Leader. State: Replica: 710c666b39404d5abefb60e0df99cb58, State: Running, Role: LEADER
I20240502 01:48:39.365883 2493 consensus_queue.cc:237] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710c666b39404d5abefb60e0df99cb58" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 34151 } }
I20240502 01:48:39.375032 2280 heartbeater.cc:503] Master 127.1.233.188:45033 was elected leader, sending a full tablet report...
I20240502 01:48:39.386209 2488 ts_tablet_manager.cc:1398] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58: Time spent starting tablet: real 0.060s user 0.022s sys 0.014s
I20240502 01:48:39.388689 2487 ts_tablet_manager.cc:1398] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e: Time spent starting tablet: real 0.068s user 0.028s sys 0.009s
I20240502 01:48:39.389990 2487 tablet_bootstrap.cc:492] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e: Bootstrap starting.
I20240502 01:48:39.397814 2487 tablet_bootstrap.cc:654] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:39.414824 2487 tablet_bootstrap.cc:492] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e: No bootstrap required, opened a new log
I20240502 01:48:39.415428 2487 ts_tablet_manager.cc:1367] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.008s
I20240502 01:48:39.418350 2487 raft_consensus.cc:348] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2708f2fd3a94638a4a33c8a951a2e1e" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 34059 } }
I20240502 01:48:39.418951 2487 raft_consensus.cc:374] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:39.419256 2487 raft_consensus.cc:729] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2708f2fd3a94638a4a33c8a951a2e1e, State: Initialized, Role: FOLLOWER
I20240502 01:48:39.419937 2487 consensus_queue.cc:260] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2708f2fd3a94638a4a33c8a951a2e1e" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 34059 } }
I20240502 01:48:39.420912 2487 raft_consensus.cc:388] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:48:39.421255 2487 raft_consensus.cc:482] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:48:39.421610 2487 raft_consensus.cc:3037] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:39.415899 2124 catalog_manager.cc:5484] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e reported cstate change: term changed from 0 to 1, leader changed from <none> to e2708f2fd3a94638a4a33c8a951a2e1e (127.1.233.129). New cstate: current_term: 1 leader_uuid: "e2708f2fd3a94638a4a33c8a951a2e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2708f2fd3a94638a4a33c8a951a2e1e" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 34059 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:39.435978 2487 raft_consensus.cc:504] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2708f2fd3a94638a4a33c8a951a2e1e" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 34059 } }
I20240502 01:48:39.436952 2487 leader_election.cc:302] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2708f2fd3a94638a4a33c8a951a2e1e; no voters:
I20240502 01:48:39.437745 2487 leader_election.cc:288] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:48:39.440725 2487 ts_tablet_manager.cc:1398] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e: Time spent starting tablet: real 0.025s user 0.010s sys 0.008s
I20240502 01:48:39.453523 2496 raft_consensus.cc:2781] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:39.454183 2496 raft_consensus.cc:686] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [term 1 LEADER]: Becoming Leader. State: Replica: e2708f2fd3a94638a4a33c8a951a2e1e, State: Running, Role: LEADER
I20240502 01:48:39.444978 2123 catalog_manager.cc:5484] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 710c666b39404d5abefb60e0df99cb58 (127.1.233.131). New cstate: current_term: 1 leader_uuid: "710c666b39404d5abefb60e0df99cb58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710c666b39404d5abefb60e0df99cb58" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 34151 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:39.454996 2496 consensus_queue.cc:237] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2708f2fd3a94638a4a33c8a951a2e1e" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 34059 } }
I20240502 01:48:39.495185 2486 leader_election.cc:288] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:48:39.520411 2494 raft_consensus.cc:2781] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:39.555598 2494 raft_consensus.cc:686] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [term 1 LEADER]: Becoming Leader. State: Replica: b4740bccea484b20909e00061797a44d, State: Running, Role: LEADER
I20240502 01:48:39.563755 2494 consensus_queue.cc:237] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4740bccea484b20909e00061797a44d" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41359 } }
I20240502 01:48:39.569610 2486 ts_tablet_manager.cc:1398] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d: Time spent starting tablet: real 0.251s user 0.075s sys 0.079s
I20240502 01:48:39.566789 2124 catalog_manager.cc:5484] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e reported cstate change: term changed from 0 to 1, leader changed from <none> to e2708f2fd3a94638a4a33c8a951a2e1e (127.1.233.129). New cstate: current_term: 1 leader_uuid: "e2708f2fd3a94638a4a33c8a951a2e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2708f2fd3a94638a4a33c8a951a2e1e" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 34059 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:39.596886 2123 catalog_manager.cc:5484] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d reported cstate change: term changed from 0 to 1, leader changed from <none> to b4740bccea484b20909e00061797a44d (127.1.233.130). New cstate: current_term: 1 leader_uuid: "b4740bccea484b20909e00061797a44d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4740bccea484b20909e00061797a44d" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41359 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:39.727416 1958 tablet_server.cc:178] TabletServer@127.1.233.129:0 shutting down...
I20240502 01:48:39.750322 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:48:39.751226 1958 tablet_replica.cc:330] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e: stopping tablet replica
I20240502 01:48:39.753408 1958 raft_consensus.cc:2229] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:39.754099 1958 raft_consensus.cc:2258] T dfb6fe78fae24ff080b55dcb628a5e62 P e2708f2fd3a94638a4a33c8a951a2e1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:39.757035 1958 tablet_replica.cc:330] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e: stopping tablet replica
I20240502 01:48:39.757531 1958 raft_consensus.cc:2229] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:39.757905 1958 raft_consensus.cc:2258] T bc22bf07e280496bb906f534082fb400 P e2708f2fd3a94638a4a33c8a951a2e1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:39.912974 1958 tablet_server.cc:194] TabletServer@127.1.233.129:0 shutdown complete.
I20240502 01:48:39.923058 1958 tablet_server.cc:178] TabletServer@127.1.233.130:0 shutting down...
I20240502 01:48:39.941890 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:48:39.942615 1958 tablet_replica.cc:330] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d: stopping tablet replica
I20240502 01:48:39.943140 1958 raft_consensus.cc:2229] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:39.943555 1958 raft_consensus.cc:2258] T 7beb894850264531a87f9d1af24f32fd P b4740bccea484b20909e00061797a44d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:39.963184 1958 tablet_server.cc:194] TabletServer@127.1.233.130:0 shutdown complete.
I20240502 01:48:39.971918 1958 tablet_server.cc:178] TabletServer@127.1.233.131:0 shutting down...
I20240502 01:48:39.989037 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:48:39.989667 1958 tablet_replica.cc:330] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58: stopping tablet replica
I20240502 01:48:39.990193 1958 raft_consensus.cc:2229] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:39.990628 1958 raft_consensus.cc:2258] T 6a4e8abbc236446a9558373c64f6d984 P 710c666b39404d5abefb60e0df99cb58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:40.001345 1958 tablet_server.cc:194] TabletServer@127.1.233.131:0 shutdown complete.
I20240502 01:48:40.011574 1958 master.cc:537] Master@127.1.233.190:37275 shutting down...
I20240502 01:48:40.068112 2206 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20240502 01:48:40.120283 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:40.120878 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:40.121223 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1c8f1331aa6a47d7b586d7e7cbbe53b9: stopping tablet replica
I20240502 01:48:40.155907 1958 master.cc:552] Master@127.1.233.190:37275 shutdown complete.
I20240502 01:48:40.167392 1958 master.cc:537] Master@127.1.233.189:44091 shutting down...
I20240502 01:48:40.182317 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:40.182858 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:40.183161 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P 123815c8f3774e03893479171b19b0d7: stopping tablet replica
I20240502 01:48:40.232926 1958 master.cc:552] Master@127.1.233.189:44091 shutdown complete.
I20240502 01:48:40.268857 1958 master.cc:537] Master@127.1.233.188:45033 shutting down...
W20240502 01:48:40.269511 2110 consensus_peers.cc:485] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 -> Peer 1c8f1331aa6a47d7b586d7e7cbbe53b9 (127.1.233.190:37275): Couldn't send request to peer 1c8f1331aa6a47d7b586d7e7cbbe53b9. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.190:37275: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:48:40.284240 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:40.285153 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:40.285486 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P ae926fe6fe99435e94ebebd607e34fd6: stopping tablet replica
I20240502 01:48:40.303884 1958 master.cc:552] Master@127.1.233.188:45033 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (5113 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240502 01:48:40.334376 1958 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.233.190:46231
I20240502 01:48:40.335464 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:40.341886 2511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:40.342795 2512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:40.343408 2514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:40.344195 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:40.344956 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:40.345155 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:40.345314 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614520345298 us; error 0 us; skew 500 ppm
I20240502 01:48:40.345824 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:40.354430 1958 webserver.cc:460] Webserver started at http://127.1.233.190:37819/ using document root <none> and password file <none>
I20240502 01:48:40.354969 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:40.355154 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:40.355417 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:40.356632 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/master-0-root/instance:
uuid: "4c31c0b9054c4032bcd8a49381eade59"
format_stamp: "Formatted at 2024-05-02 01:48:40 on dist-test-slave-2090"
I20240502 01:48:40.361326 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:48:40.364729 2519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:40.365515 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20240502 01:48:40.365834 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/master-0-root
uuid: "4c31c0b9054c4032bcd8a49381eade59"
format_stamp: "Formatted at 2024-05-02 01:48:40 on dist-test-slave-2090"
I20240502 01:48:40.366124 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:40.387964 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:40.389067 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:40.425127 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.190:46231
I20240502 01:48:40.425256 2570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.190:46231 every 8 connection(s)
I20240502 01:48:40.430238 2571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:40.441525 2571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59: Bootstrap starting.
I20240502 01:48:40.445458 2571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:40.449471 2571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59: No bootstrap required, opened a new log
I20240502 01:48:40.451316 2571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c31c0b9054c4032bcd8a49381eade59" member_type: VOTER }
I20240502 01:48:40.451726 2571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:40.451982 2571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c31c0b9054c4032bcd8a49381eade59, State: Initialized, Role: FOLLOWER
I20240502 01:48:40.452654 2571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c31c0b9054c4032bcd8a49381eade59" member_type: VOTER }
I20240502 01:48:40.453133 2571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:48:40.453349 2571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:48:40.453593 2571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:40.457731 2571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c31c0b9054c4032bcd8a49381eade59" member_type: VOTER }
I20240502 01:48:40.458302 2571 leader_election.cc:302] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c31c0b9054c4032bcd8a49381eade59; no voters:
I20240502 01:48:40.459499 2571 leader_election.cc:288] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:48:40.459854 2574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:40.461351 2574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [term 1 LEADER]: Becoming Leader. State: Replica: 4c31c0b9054c4032bcd8a49381eade59, State: Running, Role: LEADER
I20240502 01:48:40.461949 2574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c31c0b9054c4032bcd8a49381eade59" member_type: VOTER }
I20240502 01:48:40.462421 2571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:48:40.465844 2575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c31c0b9054c4032bcd8a49381eade59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c31c0b9054c4032bcd8a49381eade59" member_type: VOTER } }
I20240502 01:48:40.466918 2576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c31c0b9054c4032bcd8a49381eade59. Latest consensus state: current_term: 1 leader_uuid: "4c31c0b9054c4032bcd8a49381eade59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c31c0b9054c4032bcd8a49381eade59" member_type: VOTER } }
I20240502 01:48:40.467429 2575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:40.467718 2576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:40.471832 2582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:48:40.478250 2582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:48:40.479380 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:48:40.487439 2582 catalog_manager.cc:1347] Generated new cluster ID: aab78822e7314ea3a1365153c5968d16
I20240502 01:48:40.487742 2582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:48:40.525472 2582 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:48:40.526737 2582 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:48:40.541157 2582 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59: Generated new TSK 0
I20240502 01:48:40.541790 2582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:48:40.546609 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:40.553670 2592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:40.554363 2593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:40.555836 2595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:40.556669 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:40.557473 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:40.557682 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:40.557838 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614520557822 us; error 0 us; skew 500 ppm
I20240502 01:48:40.558393 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:40.560947 1958 webserver.cc:460] Webserver started at http://127.1.233.129:39191/ using document root <none> and password file <none>
I20240502 01:48:40.561455 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:40.561643 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:40.561898 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:40.563146 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-0-root/instance:
uuid: "a23d76521e314ecd90f2c17354771ae9"
format_stamp: "Formatted at 2024-05-02 01:48:40 on dist-test-slave-2090"
I20240502 01:48:40.568078 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20240502 01:48:40.572014 2600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:40.572904 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:48:40.573212 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "a23d76521e314ecd90f2c17354771ae9"
format_stamp: "Formatted at 2024-05-02 01:48:40 on dist-test-slave-2090"
I20240502 01:48:40.573521 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:40.588804 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:40.589879 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:40.591327 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:40.593834 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:40.594056 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:40.594303 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:40.594465 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:40.632822 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:43209
I20240502 01:48:40.632936 2665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:43209 every 8 connection(s)
I20240502 01:48:40.637387 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:40.645467 2670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:40.646564 2671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:40.649013 2673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:40.649788 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:40.650938 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:40.651247 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:40.651432 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614520651415 us; error 0 us; skew 500 ppm
I20240502 01:48:40.652170 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:40.652295 2666 heartbeater.cc:347] Connected to a master server at 127.1.233.190:46231
I20240502 01:48:40.652698 2666 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:40.653649 2666 heartbeater.cc:511] Master 127.1.233.190:46231 requested a full tablet report, sending...
I20240502 01:48:40.654913 1958 webserver.cc:460] Webserver started at http://127.1.233.130:36897/ using document root <none> and password file <none>
I20240502 01:48:40.655489 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:40.655730 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:40.656101 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:40.656064 2536 ts_manager.cc:194] Registered new tserver with Master: a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209)
I20240502 01:48:40.657769 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-1-root/instance:
uuid: "ff85372f3b344d5dbd5910c94bd87afc"
format_stamp: "Formatted at 2024-05-02 01:48:40 on dist-test-slave-2090"
I20240502 01:48:40.658615 2536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49608
I20240502 01:48:40.663694 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 01:48:40.668803 2678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:40.669584 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:48:40.669927 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-1-root
uuid: "ff85372f3b344d5dbd5910c94bd87afc"
format_stamp: "Formatted at 2024-05-02 01:48:40 on dist-test-slave-2090"
I20240502 01:48:40.670228 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:40.691941 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:40.693131 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:40.694551 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:40.696851 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:40.697059 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:40.697294 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:40.697445 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:40.734606 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.130:44255
I20240502 01:48:40.734709 2741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.130:44255 every 8 connection(s)
I20240502 01:48:40.739336 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:40.746273 2745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:40.747062 2746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:40.748927 2748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:40.749686 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:40.750535 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:40.750808 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:40.750974 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614520750960 us; error 0 us; skew 500 ppm
I20240502 01:48:40.751641 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:40.752162 2742 heartbeater.cc:347] Connected to a master server at 127.1.233.190:46231
I20240502 01:48:40.752537 2742 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:40.753528 2742 heartbeater.cc:511] Master 127.1.233.190:46231 requested a full tablet report, sending...
I20240502 01:48:40.754622 1958 webserver.cc:460] Webserver started at http://127.1.233.131:39175/ using document root <none> and password file <none>
I20240502 01:48:40.755228 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:40.755445 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:40.755704 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:40.755837 2536 ts_manager.cc:194] Registered new tserver with Master: ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255)
I20240502 01:48:40.756831 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-2-root/instance:
uuid: "664eb6b43abc4cd78b65076ff662bae6"
format_stamp: "Formatted at 2024-05-02 01:48:40 on dist-test-slave-2090"
I20240502 01:48:40.757268 2536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49622
I20240502 01:48:40.762054 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20240502 01:48:40.765694 2753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:40.766564 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:48:40.766903 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-2-root
uuid: "664eb6b43abc4cd78b65076ff662bae6"
format_stamp: "Formatted at 2024-05-02 01:48:40 on dist-test-slave-2090"
I20240502 01:48:40.767237 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:40.782744 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:40.783799 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:40.785276 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:40.787710 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:40.787900 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:40.788189 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:40.788343 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:40.825968 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.131:35541
I20240502 01:48:40.826063 2818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.131:35541 every 8 connection(s)
I20240502 01:48:40.839870 2819 heartbeater.cc:347] Connected to a master server at 127.1.233.190:46231
I20240502 01:48:40.840335 2819 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:40.841125 2819 heartbeater.cc:511] Master 127.1.233.190:46231 requested a full tablet report, sending...
I20240502 01:48:40.843071 2536 ts_manager.cc:194] Registered new tserver with Master: 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:40.843961 1958 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014834722s
I20240502 01:48:40.844491 2536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49624
I20240502 01:48:41.661998 2666 heartbeater.cc:503] Master 127.1.233.190:46231 was elected leader, sending a full tablet report...
I20240502 01:48:41.760010 2742 heartbeater.cc:503] Master 127.1.233.190:46231 was elected leader, sending a full tablet report...
I20240502 01:48:41.847435 2819 heartbeater.cc:503] Master 127.1.233.190:46231 was elected leader, sending a full tablet report...
I20240502 01:48:41.877538 1958 test_util.cc:268] Using random seed: -847195133
I20240502 01:48:41.901881 2536 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49634:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:48:41.904398 2536 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:48:41.985329 2706 tablet_service.cc:1467] Processing CreateTablet for tablet 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 01:48:41.986872 2706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1601a679ad5f4c2d8f8ad3c7c7d3ffc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:41.987859 2705 tablet_service.cc:1467] Processing CreateTablet for tablet 299f63f8961a4079ac1101429673be59 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 01:48:41.991343 2704 tablet_service.cc:1467] Processing CreateTablet for tablet d19da3a831744a649c477c48f46832ff (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 01:48:41.992074 2629 tablet_service.cc:1467] Processing CreateTablet for tablet 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 01:48:41.993216 2626 tablet_service.cc:1467] Processing CreateTablet for tablet 57e780227c194beda6e8fde8df155c6a (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 01:48:41.993439 2703 tablet_service.cc:1467] Processing CreateTablet for tablet 57e780227c194beda6e8fde8df155c6a (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 01:48:41.994979 2628 tablet_service.cc:1467] Processing CreateTablet for tablet 299f63f8961a4079ac1101429673be59 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 01:48:41.995157 2625 tablet_service.cc:1467] Processing CreateTablet for tablet 9d43ecf839ef46eaa968c5c319b22e3c (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 01:48:41.995766 2627 tablet_service.cc:1467] Processing CreateTablet for tablet d19da3a831744a649c477c48f46832ff (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 01:48:41.997289 2701 tablet_service.cc:1467] Processing CreateTablet for tablet bc46edb8342d42798bdc4d08225a16b7 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 01:48:41.998540 2623 tablet_service.cc:1467] Processing CreateTablet for tablet 06ca6c78d51049ee89f37ac332842010 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 01:48:41.993698 2629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1601a679ad5f4c2d8f8ad3c7c7d3ffc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.001358 2627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19da3a831744a649c477c48f46832ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.001547 2621 tablet_service.cc:1467] Processing CreateTablet for tablet 464f34b3a41e4e959f816a0e24aae594 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 01:48:42.002010 2628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 299f63f8961a4079ac1101429673be59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.002671 2626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e780227c194beda6e8fde8df155c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:41.997289 2624 tablet_service.cc:1467] Processing CreateTablet for tablet bc46edb8342d42798bdc4d08225a16b7 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 01:48:42.005743 2624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc46edb8342d42798bdc4d08225a16b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.004011 2699 tablet_service.cc:1467] Processing CreateTablet for tablet 464f34b3a41e4e959f816a0e24aae594 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 01:48:41.995870 2702 tablet_service.cc:1467] Processing CreateTablet for tablet 9d43ecf839ef46eaa968c5c319b22e3c (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 01:48:41.999814 2700 tablet_service.cc:1467] Processing CreateTablet for tablet 06ca6c78d51049ee89f37ac332842010 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 01:48:42.011195 2623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ca6c78d51049ee89f37ac332842010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.011278 2705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 299f63f8961a4079ac1101429673be59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.012555 2625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d43ecf839ef46eaa968c5c319b22e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.013442 2782 tablet_service.cc:1467] Processing CreateTablet for tablet 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 01:48:42.014874 2782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1601a679ad5f4c2d8f8ad3c7c7d3ffc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.016520 2702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d43ecf839ef46eaa968c5c319b22e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.017064 2621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464f34b3a41e4e959f816a0e24aae594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.028553 2701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc46edb8342d42798bdc4d08225a16b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.032546 2699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464f34b3a41e4e959f816a0e24aae594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.034358 2780 tablet_service.cc:1467] Processing CreateTablet for tablet d19da3a831744a649c477c48f46832ff (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 01:48:42.036656 2704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19da3a831744a649c477c48f46832ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.038084 2777 tablet_service.cc:1467] Processing CreateTablet for tablet bc46edb8342d42798bdc4d08225a16b7 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 01:48:42.040182 2777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc46edb8342d42798bdc4d08225a16b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.040093 2779 tablet_service.cc:1467] Processing CreateTablet for tablet 57e780227c194beda6e8fde8df155c6a (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 01:48:42.041739 2779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e780227c194beda6e8fde8df155c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.043402 2776 tablet_service.cc:1467] Processing CreateTablet for tablet 06ca6c78d51049ee89f37ac332842010 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 01:48:42.044584 2703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e780227c194beda6e8fde8df155c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.047480 2776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ca6c78d51049ee89f37ac332842010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.047667 2778 tablet_service.cc:1467] Processing CreateTablet for tablet 9d43ecf839ef46eaa968c5c319b22e3c (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 01:48:42.050604 2774 tablet_service.cc:1467] Processing CreateTablet for tablet 464f34b3a41e4e959f816a0e24aae594 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 01:48:42.051815 2781 tablet_service.cc:1467] Processing CreateTablet for tablet 299f63f8961a4079ac1101429673be59 (DEFAULT_TABLE table=test-workload [id=5aac3c1ce5fd45d4b154fe752c618f43]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 01:48:42.058538 2700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ca6c78d51049ee89f37ac332842010. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.069765 2839 tablet_bootstrap.cc:492] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc: Bootstrap starting.
I20240502 01:48:42.080317 2781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 299f63f8961a4079ac1101429673be59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.081100 2780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19da3a831744a649c477c48f46832ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.082041 2778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d43ecf839ef46eaa968c5c319b22e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.085068 2774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 464f34b3a41e4e959f816a0e24aae594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:42.097540 2840 tablet_bootstrap.cc:492] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9: Bootstrap starting.
I20240502 01:48:42.123915 2840 tablet_bootstrap.cc:654] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.138259 2841 tablet_bootstrap.cc:492] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6: Bootstrap starting.
I20240502 01:48:42.146747 2839 tablet_bootstrap.cc:654] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.166258 2841 tablet_bootstrap.cc:654] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.175176 2840 tablet_bootstrap.cc:492] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9: No bootstrap required, opened a new log
I20240502 01:48:42.175839 2840 ts_tablet_manager.cc:1367] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9: Time spent bootstrapping tablet: real 0.079s user 0.009s sys 0.029s
I20240502 01:48:42.179560 2840 raft_consensus.cc:348] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.180819 2840 raft_consensus.cc:374] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.181241 2840 raft_consensus.cc:729] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23d76521e314ecd90f2c17354771ae9, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.182189 2840 consensus_queue.cc:260] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.183250 2839 tablet_bootstrap.cc:492] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc: No bootstrap required, opened a new log
I20240502 01:48:42.183715 2839 ts_tablet_manager.cc:1367] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc: Time spent bootstrapping tablet: real 0.114s user 0.012s sys 0.000s
I20240502 01:48:42.184311 2841 tablet_bootstrap.cc:492] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6: No bootstrap required, opened a new log
I20240502 01:48:42.184824 2841 ts_tablet_manager.cc:1367] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.008s
I20240502 01:48:42.185086 2840 ts_tablet_manager.cc:1398] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 01:48:42.186131 2840 tablet_bootstrap.cc:492] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9: Bootstrap starting.
I20240502 01:48:42.187489 2841 raft_consensus.cc:348] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.187726 2839 raft_consensus.cc:348] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.188345 2841 raft_consensus.cc:374] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.188570 2839 raft_consensus.cc:374] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.188690 2841 raft_consensus.cc:729] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664eb6b43abc4cd78b65076ff662bae6, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.188999 2839 raft_consensus.cc:729] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff85372f3b344d5dbd5910c94bd87afc, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.189769 2839 consensus_queue.cc:260] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.189915 2841 consensus_queue.cc:260] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.192369 2840 tablet_bootstrap.cc:654] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.194780 2839 ts_tablet_manager.cc:1398] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20240502 01:48:42.195888 2839 tablet_bootstrap.cc:492] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc: Bootstrap starting.
I20240502 01:48:42.199630 2841 ts_tablet_manager.cc:1398] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20240502 01:48:42.200819 2841 tablet_bootstrap.cc:492] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6: Bootstrap starting.
I20240502 01:48:42.205121 2840 tablet_bootstrap.cc:492] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9: No bootstrap required, opened a new log
I20240502 01:48:42.205427 2839 tablet_bootstrap.cc:654] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.205693 2840 ts_tablet_manager.cc:1367] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.006s
I20240502 01:48:42.206532 2841 tablet_bootstrap.cc:654] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.208629 2840 raft_consensus.cc:348] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.209424 2840 raft_consensus.cc:374] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.209733 2840 raft_consensus.cc:729] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23d76521e314ecd90f2c17354771ae9, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.211745 2840 consensus_queue.cc:260] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.216485 2840 ts_tablet_manager.cc:1398] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20240502 01:48:42.217103 2847 raft_consensus.cc:482] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:42.217617 2847 raft_consensus.cc:504] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.218333 2839 tablet_bootstrap.cc:492] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc: No bootstrap required, opened a new log
I20240502 01:48:42.219076 2839 ts_tablet_manager.cc:1367] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.005s
I20240502 01:48:42.219393 2841 tablet_bootstrap.cc:492] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6: No bootstrap required, opened a new log
I20240502 01:48:42.219897 2841 ts_tablet_manager.cc:1367] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.005s
I20240502 01:48:42.222064 2839 raft_consensus.cc:348] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.223017 2839 raft_consensus.cc:374] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.222610 2841 raft_consensus.cc:348] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.223416 2839 raft_consensus.cc:729] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff85372f3b344d5dbd5910c94bd87afc, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.223578 2841 raft_consensus.cc:374] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.224241 2841 raft_consensus.cc:729] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664eb6b43abc4cd78b65076ff662bae6, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.224659 2839 consensus_queue.cc:260] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.225106 2841 consensus_queue.cc:260] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.229684 2839 ts_tablet_manager.cc:1398] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20240502 01:48:42.230409 2847 leader_election.cc:288] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255)
I20240502 01:48:42.231303 2839 tablet_bootstrap.cc:492] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc: Bootstrap starting.
I20240502 01:48:42.239655 2839 tablet_bootstrap.cc:654] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.217607 2840 tablet_bootstrap.cc:492] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9: Bootstrap starting.
I20240502 01:48:42.248391 2846 raft_consensus.cc:482] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:42.248967 2846 raft_consensus.cc:504] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.251754 2846 leader_election.cc:288] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:42.252691 2839 tablet_bootstrap.cc:492] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc: No bootstrap required, opened a new log
I20240502 01:48:42.253494 2839 ts_tablet_manager.cc:1367] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.014s
I20240502 01:48:42.257303 2839 raft_consensus.cc:348] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.258531 2839 raft_consensus.cc:374] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.258919 2839 raft_consensus.cc:729] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff85372f3b344d5dbd5910c94bd87afc, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.259872 2839 consensus_queue.cc:260] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.263593 2839 ts_tablet_manager.cc:1398] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240502 01:48:42.265147 2839 tablet_bootstrap.cc:492] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc: Bootstrap starting.
I20240502 01:48:42.273586 2839 tablet_bootstrap.cc:654] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.296900 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1601a679ad5f4c2d8f8ad3c7c7d3ffc4" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9" is_pre_election: true
I20240502 01:48:42.299073 2840 tablet_bootstrap.cc:654] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9: Neither blocks nor log segments found. Creating new log.
W20240502 01:48:42.300899 2680 leader_election.cc:341] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:42.307243 2841 ts_tablet_manager.cc:1398] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6: Time spent starting tablet: real 0.087s user 0.026s sys 0.053s
I20240502 01:48:42.307904 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc46edb8342d42798bdc4d08225a16b7" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9" is_pre_election: true
I20240502 01:48:42.308364 2841 tablet_bootstrap.cc:492] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6: Bootstrap starting.
I20240502 01:48:42.309604 2839 tablet_bootstrap.cc:492] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc: No bootstrap required, opened a new log
W20240502 01:48:42.309695 2755 leader_election.cc:341] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:42.310370 2839 ts_tablet_manager.cc:1367] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc: Time spent bootstrapping tablet: real 0.046s user 0.027s sys 0.014s
I20240502 01:48:42.316614 2840 tablet_bootstrap.cc:492] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9: No bootstrap required, opened a new log
I20240502 01:48:42.317018 2853 raft_consensus.cc:482] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:42.318024 2841 tablet_bootstrap.cc:654] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.317631 2853 raft_consensus.cc:504] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.320637 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e780227c194beda6e8fde8df155c6a" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9" is_pre_election: true
I20240502 01:48:42.321334 2640 raft_consensus.cc:2454] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 664eb6b43abc4cd78b65076ff662bae6 in term 0.
I20240502 01:48:42.313582 2839 raft_consensus.cc:348] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.323390 2839 raft_consensus.cc:374] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.323571 2755 leader_election.cc:302] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 664eb6b43abc4cd78b65076ff662bae6, a23d76521e314ecd90f2c17354771ae9; no voters:
I20240502 01:48:42.323899 2839 raft_consensus.cc:729] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff85372f3b344d5dbd5910c94bd87afc, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.325275 2847 raft_consensus.cc:2781] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:42.325693 2847 raft_consensus.cc:482] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:42.326000 2847 raft_consensus.cc:3037] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.325109 2839 consensus_queue.cc:260] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.328392 2839 ts_tablet_manager.cc:1398] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20240502 01:48:42.329739 2839 tablet_bootstrap.cc:492] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc: Bootstrap starting.
I20240502 01:48:42.333268 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1601a679ad5f4c2d8f8ad3c7c7d3ffc4" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664eb6b43abc4cd78b65076ff662bae6" is_pre_election: true
W20240502 01:48:42.335175 2682 leader_election.cc:341] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:42.335688 2682 leader_election.cc:302] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff85372f3b344d5dbd5910c94bd87afc; no voters: 664eb6b43abc4cd78b65076ff662bae6, a23d76521e314ecd90f2c17354771ae9
I20240502 01:48:42.337266 2846 raft_consensus.cc:2726] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:48:42.337927 2839 tablet_bootstrap.cc:654] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.339262 2841 tablet_bootstrap.cc:492] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6: No bootstrap required, opened a new log
I20240502 01:48:42.339864 2841 ts_tablet_manager.cc:1367] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.003s
I20240502 01:48:42.339998 2853 leader_election.cc:288] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255)
I20240502 01:48:42.341441 2716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc46edb8342d42798bdc4d08225a16b7" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff85372f3b344d5dbd5910c94bd87afc" is_pre_election: true
I20240502 01:48:42.342283 2716 raft_consensus.cc:2454] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 664eb6b43abc4cd78b65076ff662bae6 in term 0.
I20240502 01:48:42.343569 2841 raft_consensus.cc:348] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.344776 2841 raft_consensus.cc:374] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.344946 2716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e780227c194beda6e8fde8df155c6a" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff85372f3b344d5dbd5910c94bd87afc" is_pre_election: true
I20240502 01:48:42.345234 2841 raft_consensus.cc:729] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664eb6b43abc4cd78b65076ff662bae6, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.344230 2757 leader_election.cc:302] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 664eb6b43abc4cd78b65076ff662bae6, ff85372f3b344d5dbd5910c94bd87afc; no voters: a23d76521e314ecd90f2c17354771ae9
W20240502 01:48:42.346906 2757 leader_election.cc:341] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:42.347469 2853 raft_consensus.cc:2781] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:42.347867 2853 raft_consensus.cc:482] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:42.348229 2853 raft_consensus.cc:3037] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.331861 2847 raft_consensus.cc:504] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.346401 2841 consensus_queue.cc:260] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.317229 2840 ts_tablet_manager.cc:1367] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9: Time spent bootstrapping tablet: real 0.100s user 0.006s sys 0.004s
I20240502 01:48:42.353652 2853 raft_consensus.cc:504] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.354772 2847 leader_election.cc:288] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 election: Requested vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255)
I20240502 01:48:42.355868 2840 raft_consensus.cc:348] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.356760 2840 raft_consensus.cc:374] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.357059 2840 raft_consensus.cc:729] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23d76521e314ecd90f2c17354771ae9, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.357941 2840 consensus_queue.cc:260] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.358896 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e780227c194beda6e8fde8df155c6a" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9"
I20240502 01:48:42.359593 2640 raft_consensus.cc:3037] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.359741 2639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc46edb8342d42798bdc4d08225a16b7" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9"
I20240502 01:48:42.360500 2840 ts_tablet_manager.cc:1398] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 01:48:42.360858 2639 raft_consensus.cc:3037] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.362339 2840 tablet_bootstrap.cc:492] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9: Bootstrap starting.
I20240502 01:48:42.365000 2716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e780227c194beda6e8fde8df155c6a" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff85372f3b344d5dbd5910c94bd87afc"
I20240502 01:48:42.366616 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc46edb8342d42798bdc4d08225a16b7" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff85372f3b344d5dbd5910c94bd87afc"
I20240502 01:48:42.367463 2715 raft_consensus.cc:3037] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.369338 2840 tablet_bootstrap.cc:654] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.373862 2839 tablet_bootstrap.cc:492] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc: No bootstrap required, opened a new log
I20240502 01:48:42.374679 2839 ts_tablet_manager.cc:1367] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc: Time spent bootstrapping tablet: real 0.045s user 0.023s sys 0.003s
W20240502 01:48:42.377210 2757 leader_election.cc:341] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:42.378844 2839 raft_consensus.cc:348] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.380218 2839 raft_consensus.cc:374] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.380784 2839 raft_consensus.cc:729] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff85372f3b344d5dbd5910c94bd87afc, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.382099 2839 consensus_queue.cc:260] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.388463 2841 ts_tablet_manager.cc:1398] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6: Time spent starting tablet: real 0.048s user 0.024s sys 0.018s
I20240502 01:48:42.383350 2840 tablet_bootstrap.cc:492] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9: No bootstrap required, opened a new log
I20240502 01:48:42.389272 2840 ts_tablet_manager.cc:1367] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.006s
I20240502 01:48:42.385149 2640 raft_consensus.cc:2454] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 664eb6b43abc4cd78b65076ff662bae6 in term 1.
I20240502 01:48:42.389744 2841 tablet_bootstrap.cc:492] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6: Bootstrap starting.
I20240502 01:48:42.390846 2755 leader_election.cc:302] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 664eb6b43abc4cd78b65076ff662bae6, a23d76521e314ecd90f2c17354771ae9; no voters: ff85372f3b344d5dbd5910c94bd87afc
I20240502 01:48:42.386121 2715 raft_consensus.cc:2454] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 664eb6b43abc4cd78b65076ff662bae6 in term 1.
I20240502 01:48:42.386438 2639 raft_consensus.cc:2454] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 664eb6b43abc4cd78b65076ff662bae6 in term 1.
I20240502 01:48:42.355772 2853 leader_election.cc:288] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 election: Requested vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255)
I20240502 01:48:42.388744 2839 ts_tablet_manager.cc:1398] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc: Time spent starting tablet: real 0.014s user 0.006s sys 0.004s
I20240502 01:48:42.392707 2840 raft_consensus.cc:348] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.393129 2757 leader_election.cc:302] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 664eb6b43abc4cd78b65076ff662bae6, ff85372f3b344d5dbd5910c94bd87afc; no voters:
I20240502 01:48:42.394912 2840 raft_consensus.cc:374] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.395447 2853 raft_consensus.cc:2781] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:42.398993 2840 raft_consensus.cc:729] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23d76521e314ecd90f2c17354771ae9, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.399260 2853 raft_consensus.cc:686] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 LEADER]: Becoming Leader. State: Replica: 664eb6b43abc4cd78b65076ff662bae6, State: Running, Role: LEADER
I20240502 01:48:42.400022 2840 consensus_queue.cc:260] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.400247 2853 consensus_queue.cc:237] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.400880 2839 tablet_bootstrap.cc:492] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc: Bootstrap starting.
I20240502 01:48:42.400920 2846 raft_consensus.cc:482] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:42.401599 2846 raft_consensus.cc:504] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.402880 2840 ts_tablet_manager.cc:1398] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240502 01:48:42.403865 2840 tablet_bootstrap.cc:492] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9: Bootstrap starting.
I20240502 01:48:42.405397 2640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "299f63f8961a4079ac1101429673be59" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9" is_pre_election: true
I20240502 01:48:42.406134 2640 raft_consensus.cc:2454] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff85372f3b344d5dbd5910c94bd87afc in term 0.
I20240502 01:48:42.406692 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "299f63f8961a4079ac1101429673be59" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664eb6b43abc4cd78b65076ff662bae6" is_pre_election: true
I20240502 01:48:42.407940 2839 tablet_bootstrap.cc:654] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.408066 2680 leader_election.cc:302] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a23d76521e314ecd90f2c17354771ae9, ff85372f3b344d5dbd5910c94bd87afc; no voters:
I20240502 01:48:42.409653 2841 tablet_bootstrap.cc:654] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.403792 2846 leader_election.cc:288] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
W20240502 01:48:42.409973 2682 leader_election.cc:341] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:42.394188 2862 raft_consensus.cc:2781] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:42.413458 2846 raft_consensus.cc:2781] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:42.417613 2862 raft_consensus.cc:686] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 1 LEADER]: Becoming Leader. State: Replica: 664eb6b43abc4cd78b65076ff662bae6, State: Running, Role: LEADER
I20240502 01:48:42.418012 2846 raft_consensus.cc:482] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:42.418426 2846 raft_consensus.cc:3037] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.418905 2862 consensus_queue.cc:237] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.423355 2840 tablet_bootstrap.cc:654] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.432660 2533 catalog_manager.cc:5484] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131). New cstate: current_term: 1 leader_uuid: "664eb6b43abc4cd78b65076ff662bae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:42.436558 2839 tablet_bootstrap.cc:492] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc: No bootstrap required, opened a new log
I20240502 01:48:42.437326 2839 ts_tablet_manager.cc:1367] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.016s
I20240502 01:48:42.438863 2846 raft_consensus.cc:504] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.441284 2846 leader_election.cc:288] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 election: Requested vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:42.442207 2639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "299f63f8961a4079ac1101429673be59" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9"
I20240502 01:48:42.442379 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "299f63f8961a4079ac1101429673be59" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664eb6b43abc4cd78b65076ff662bae6"
I20240502 01:48:42.442911 2639 raft_consensus.cc:3037] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:48:42.443747 2682 leader_election.cc:341] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:42.443488 2839 raft_consensus.cc:348] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.444284 2839 raft_consensus.cc:374] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.444599 2839 raft_consensus.cc:729] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff85372f3b344d5dbd5910c94bd87afc, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.445299 2839 consensus_queue.cc:260] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.447484 2839 ts_tablet_manager.cc:1398] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240502 01:48:42.448529 2839 tablet_bootstrap.cc:492] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc: Bootstrap starting.
I20240502 01:48:42.451267 2639 raft_consensus.cc:2454] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff85372f3b344d5dbd5910c94bd87afc in term 1.
I20240502 01:48:42.452915 2680 leader_election.cc:302] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a23d76521e314ecd90f2c17354771ae9, ff85372f3b344d5dbd5910c94bd87afc; no voters: 664eb6b43abc4cd78b65076ff662bae6
I20240502 01:48:42.454183 2846 raft_consensus.cc:2781] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:42.454746 2839 tablet_bootstrap.cc:654] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.454699 2846 raft_consensus.cc:686] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 1 LEADER]: Becoming Leader. State: Replica: ff85372f3b344d5dbd5910c94bd87afc, State: Running, Role: LEADER
I20240502 01:48:42.455667 2846 consensus_queue.cc:237] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.460638 2840 tablet_bootstrap.cc:492] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9: No bootstrap required, opened a new log
I20240502 01:48:42.461218 2840 ts_tablet_manager.cc:1367] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.006s
I20240502 01:48:42.466888 2841 tablet_bootstrap.cc:492] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6: No bootstrap required, opened a new log
I20240502 01:48:42.466919 2839 tablet_bootstrap.cc:492] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc: No bootstrap required, opened a new log
I20240502 01:48:42.467540 2841 ts_tablet_manager.cc:1367] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6: Time spent bootstrapping tablet: real 0.078s user 0.017s sys 0.006s
I20240502 01:48:42.467638 2839 ts_tablet_manager.cc:1367] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.005s
I20240502 01:48:42.468475 2840 raft_consensus.cc:348] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.466629 2535 catalog_manager.cc:5484] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131). New cstate: current_term: 1 leader_uuid: "664eb6b43abc4cd78b65076ff662bae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:42.471069 2839 raft_consensus.cc:348] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.472191 2839 raft_consensus.cc:374] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.472023 2841 raft_consensus.cc:348] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.472590 2839 raft_consensus.cc:729] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff85372f3b344d5dbd5910c94bd87afc, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.472872 2841 raft_consensus.cc:374] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.473203 2841 raft_consensus.cc:729] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664eb6b43abc4cd78b65076ff662bae6, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.469373 2840 raft_consensus.cc:374] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 01:48:42.474150 2584 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:48:42.474308 2533 catalog_manager.cc:5484] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc reported cstate change: term changed from 0 to 1, leader changed from <none> to ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130). New cstate: current_term: 1 leader_uuid: "ff85372f3b344d5dbd5910c94bd87afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:42.475487 2840 raft_consensus.cc:729] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23d76521e314ecd90f2c17354771ae9, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.475236 2841 consensus_queue.cc:260] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.476282 2840 consensus_queue.cc:260] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.477532 2841 ts_tablet_manager.cc:1398] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20240502 01:48:42.478583 2841 tablet_bootstrap.cc:492] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6: Bootstrap starting.
I20240502 01:48:42.478603 2840 ts_tablet_manager.cc:1398] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9: Time spent starting tablet: real 0.017s user 0.002s sys 0.006s
I20240502 01:48:42.473348 2839 consensus_queue.cc:260] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.479876 2589 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:42.480894 2589 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:42.481593 2840 tablet_bootstrap.cc:492] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9: Bootstrap starting.
I20240502 01:48:42.481192 2839 ts_tablet_manager.cc:1398] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc: Time spent starting tablet: real 0.013s user 0.002s sys 0.004s
I20240502 01:48:42.483475 2839 tablet_bootstrap.cc:492] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc: Bootstrap starting.
I20240502 01:48:42.485908 2841 tablet_bootstrap.cc:654] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.486474 2865 raft_consensus.cc:482] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:42.486953 2865 raft_consensus.cc:504] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.489058 2865 leader_election.cc:288] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:42.490469 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d43ecf839ef46eaa968c5c319b22e3c" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664eb6b43abc4cd78b65076ff662bae6" is_pre_election: true
W20240502 01:48:42.492101 2682 leader_election.cc:341] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:42.492168 2639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d43ecf839ef46eaa968c5c319b22e3c" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9" is_pre_election: true
I20240502 01:48:42.494073 2839 tablet_bootstrap.cc:654] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc: Neither blocks nor log segments found. Creating new log.
W20240502 01:48:42.494161 2680 leader_election.cc:341] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:48:42.494820 2680 leader_election.cc:302] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff85372f3b344d5dbd5910c94bd87afc; no voters: 664eb6b43abc4cd78b65076ff662bae6, a23d76521e314ecd90f2c17354771ae9
I20240502 01:48:42.497751 2865 raft_consensus.cc:2726] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:48:42.503381 2840 tablet_bootstrap.cc:654] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.513306 2840 tablet_bootstrap.cc:492] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9: No bootstrap required, opened a new log
I20240502 01:48:42.513696 2839 tablet_bootstrap.cc:492] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc: No bootstrap required, opened a new log
I20240502 01:48:42.514029 2840 ts_tablet_manager.cc:1367] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.003s
I20240502 01:48:42.514194 2839 ts_tablet_manager.cc:1367] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.007s
I20240502 01:48:42.516580 2839 raft_consensus.cc:348] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.517387 2839 raft_consensus.cc:374] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.517694 2839 raft_consensus.cc:729] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff85372f3b344d5dbd5910c94bd87afc, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.517419 2840 raft_consensus.cc:348] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.518447 2840 raft_consensus.cc:374] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.518796 2840 raft_consensus.cc:729] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23d76521e314ecd90f2c17354771ae9, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.518904 2841 tablet_bootstrap.cc:492] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6: No bootstrap required, opened a new log
I20240502 01:48:42.519510 2841 ts_tablet_manager.cc:1367] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.021s
I20240502 01:48:42.518260 2839 consensus_queue.cc:260] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.519541 2840 consensus_queue.cc:260] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.521613 2839 ts_tablet_manager.cc:1398] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240502 01:48:42.522358 2841 raft_consensus.cc:348] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.523154 2841 raft_consensus.cc:374] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.523463 2841 raft_consensus.cc:729] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664eb6b43abc4cd78b65076ff662bae6, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.524237 2841 consensus_queue.cc:260] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.524856 2840 ts_tablet_manager.cc:1398] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20240502 01:48:42.525933 2840 tablet_bootstrap.cc:492] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9: Bootstrap starting.
I20240502 01:48:42.526990 2841 ts_tablet_manager.cc:1398] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:48:42.528030 2841 tablet_bootstrap.cc:492] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6: Bootstrap starting.
I20240502 01:48:42.534152 2841 tablet_bootstrap.cc:654] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.536108 2840 tablet_bootstrap.cc:654] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.542591 2840 tablet_bootstrap.cc:492] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9: No bootstrap required, opened a new log
I20240502 01:48:42.542923 2841 tablet_bootstrap.cc:492] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6: No bootstrap required, opened a new log
I20240502 01:48:42.543017 2840 ts_tablet_manager.cc:1367] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.007s
I20240502 01:48:42.543409 2841 ts_tablet_manager.cc:1367] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20240502 01:48:42.545197 2840 raft_consensus.cc:348] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.545449 2841 raft_consensus.cc:348] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.545764 2840 raft_consensus.cc:374] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.546059 2841 raft_consensus.cc:374] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.546124 2840 raft_consensus.cc:729] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23d76521e314ecd90f2c17354771ae9, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.546387 2841 raft_consensus.cc:729] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664eb6b43abc4cd78b65076ff662bae6, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.546846 2840 consensus_queue.cc:260] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.547039 2841 consensus_queue.cc:260] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.549481 2841 ts_tablet_manager.cc:1398] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 01:48:42.549582 2840 ts_tablet_manager.cc:1398] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20240502 01:48:42.550577 2841 tablet_bootstrap.cc:492] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6: Bootstrap starting.
I20240502 01:48:42.550673 2840 tablet_bootstrap.cc:492] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9: Bootstrap starting.
I20240502 01:48:42.556356 2841 tablet_bootstrap.cc:654] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.557253 2840 tablet_bootstrap.cc:654] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.566013 2845 raft_consensus.cc:482] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:42.566861 2845 raft_consensus.cc:504] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.570726 2845 leader_election.cc:288] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255), 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:42.573345 2840 tablet_bootstrap.cc:492] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9: No bootstrap required, opened a new log
I20240502 01:48:42.573345 2841 tablet_bootstrap.cc:492] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6: No bootstrap required, opened a new log
I20240502 01:48:42.574065 2840 ts_tablet_manager.cc:1367] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.007s
I20240502 01:48:42.574110 2841 ts_tablet_manager.cc:1367] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.004s
I20240502 01:48:42.576506 2840 raft_consensus.cc:348] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.577314 2840 raft_consensus.cc:374] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.577582 2840 raft_consensus.cc:729] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23d76521e314ecd90f2c17354771ae9, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.578226 2840 consensus_queue.cc:260] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.578823 2841 raft_consensus.cc:348] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.579571 2841 raft_consensus.cc:374] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.579839 2841 raft_consensus.cc:729] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664eb6b43abc4cd78b65076ff662bae6, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.580741 2840 ts_tablet_manager.cc:1398] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:48:42.580658 2841 consensus_queue.cc:260] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.582708 2841 ts_tablet_manager.cc:1398] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:48:42.586676 2841 tablet_bootstrap.cc:492] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6: Bootstrap starting.
I20240502 01:48:42.590705 2865 raft_consensus.cc:482] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:42.591318 2865 raft_consensus.cc:504] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.592756 2841 tablet_bootstrap.cc:654] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:42.593502 2865 leader_election.cc:288] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:42.595481 2639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464f34b3a41e4e959f816a0e24aae594" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9" is_pre_election: true
I20240502 01:48:42.596179 2639 raft_consensus.cc:2454] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff85372f3b344d5dbd5910c94bd87afc in term 0.
I20240502 01:48:42.596897 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464f34b3a41e4e959f816a0e24aae594" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664eb6b43abc4cd78b65076ff662bae6" is_pre_election: true
I20240502 01:48:42.597352 2680 leader_election.cc:302] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a23d76521e314ecd90f2c17354771ae9, ff85372f3b344d5dbd5910c94bd87afc; no voters:
I20240502 01:48:42.597820 2792 raft_consensus.cc:2454] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff85372f3b344d5dbd5910c94bd87afc in term 0.
I20240502 01:48:42.598250 2865 raft_consensus.cc:2781] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:42.598645 2865 raft_consensus.cc:482] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:42.598973 2865 raft_consensus.cc:3037] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.605331 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1601a679ad5f4c2d8f8ad3c7c7d3ffc4" candidate_uuid: "a23d76521e314ecd90f2c17354771ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664eb6b43abc4cd78b65076ff662bae6" is_pre_election: true
I20240502 01:48:42.605957 2841 tablet_bootstrap.cc:492] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6: No bootstrap required, opened a new log
I20240502 01:48:42.606122 2792 raft_consensus.cc:2454] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a23d76521e314ecd90f2c17354771ae9 in term 0.
I20240502 01:48:42.606560 2841 ts_tablet_manager.cc:1367] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.009s
I20240502 01:48:42.607475 2604 leader_election.cc:302] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 664eb6b43abc4cd78b65076ff662bae6, a23d76521e314ecd90f2c17354771ae9; no voters:
I20240502 01:48:42.606159 2865 raft_consensus.cc:504] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.608446 2845 raft_consensus.cc:2781] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:42.608814 2845 raft_consensus.cc:482] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:42.609135 2845 raft_consensus.cc:3037] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.610471 2639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464f34b3a41e4e959f816a0e24aae594" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9"
I20240502 01:48:42.610297 2841 raft_consensus.cc:348] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.611148 2841 raft_consensus.cc:374] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:42.611152 2639 raft_consensus.cc:3037] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.611687 2841 raft_consensus.cc:729] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664eb6b43abc4cd78b65076ff662bae6, State: Initialized, Role: FOLLOWER
I20240502 01:48:42.612365 2841 consensus_queue.cc:260] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.614266 2841 ts_tablet_manager.cc:1398] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240502 01:48:42.616518 2845 raft_consensus.cc:504] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.617226 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1601a679ad5f4c2d8f8ad3c7c7d3ffc4" candidate_uuid: "a23d76521e314ecd90f2c17354771ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff85372f3b344d5dbd5910c94bd87afc" is_pre_election: true
I20240502 01:48:42.617841 2715 raft_consensus.cc:2454] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a23d76521e314ecd90f2c17354771ae9 in term 0.
I20240502 01:48:42.618736 2845 leader_election.cc:288] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [CANDIDATE]: Term 1 election: Requested vote from peers ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255), 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:42.619576 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1601a679ad5f4c2d8f8ad3c7c7d3ffc4" candidate_uuid: "a23d76521e314ecd90f2c17354771ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff85372f3b344d5dbd5910c94bd87afc"
I20240502 01:48:42.619802 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1601a679ad5f4c2d8f8ad3c7c7d3ffc4" candidate_uuid: "a23d76521e314ecd90f2c17354771ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664eb6b43abc4cd78b65076ff662bae6"
I20240502 01:48:42.620123 2715 raft_consensus.cc:3037] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.620501 2792 raft_consensus.cc:3037] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.620977 2639 raft_consensus.cc:2454] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff85372f3b344d5dbd5910c94bd87afc in term 1.
I20240502 01:48:42.622280 2865 leader_election.cc:288] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 election: Requested vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:42.622419 2680 leader_election.cc:302] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a23d76521e314ecd90f2c17354771ae9, ff85372f3b344d5dbd5910c94bd87afc; no voters:
I20240502 01:48:42.623654 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "464f34b3a41e4e959f816a0e24aae594" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664eb6b43abc4cd78b65076ff662bae6"
I20240502 01:48:42.624423 2791 raft_consensus.cc:3037] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.624933 2865 raft_consensus.cc:2781] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:42.625479 2865 raft_consensus.cc:686] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [term 1 LEADER]: Becoming Leader. State: Replica: ff85372f3b344d5dbd5910c94bd87afc, State: Running, Role: LEADER
I20240502 01:48:42.626298 2865 consensus_queue.cc:237] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.629845 2715 raft_consensus.cc:2454] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a23d76521e314ecd90f2c17354771ae9 in term 1.
I20240502 01:48:42.631072 2604 leader_election.cc:302] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a23d76521e314ecd90f2c17354771ae9, ff85372f3b344d5dbd5910c94bd87afc; no voters:
I20240502 01:48:42.631482 2791 raft_consensus.cc:2454] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff85372f3b344d5dbd5910c94bd87afc in term 1.
I20240502 01:48:42.632300 2845 raft_consensus.cc:2781] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:42.635452 2792 raft_consensus.cc:2454] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a23d76521e314ecd90f2c17354771ae9 in term 1.
I20240502 01:48:42.635890 2845 raft_consensus.cc:686] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a23d76521e314ecd90f2c17354771ae9, State: Running, Role: LEADER
I20240502 01:48:42.636715 2845 consensus_queue.cc:237] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.641403 2534 catalog_manager.cc:5484] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc reported cstate change: term changed from 0 to 1, leader changed from <none> to ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130). New cstate: current_term: 1 leader_uuid: "ff85372f3b344d5dbd5910c94bd87afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:42.645164 2845 raft_consensus.cc:482] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:42.645712 2845 raft_consensus.cc:504] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.647640 2845 leader_election.cc:288] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255), 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:42.647692 2533 catalog_manager.cc:5484] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a23d76521e314ecd90f2c17354771ae9 (127.1.233.129). New cstate: current_term: 1 leader_uuid: "a23d76521e314ecd90f2c17354771ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:42.648840 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ca6c78d51049ee89f37ac332842010" candidate_uuid: "a23d76521e314ecd90f2c17354771ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664eb6b43abc4cd78b65076ff662bae6" is_pre_election: true
I20240502 01:48:42.649539 2792 raft_consensus.cc:2454] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a23d76521e314ecd90f2c17354771ae9 in term 0.
I20240502 01:48:42.649680 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ca6c78d51049ee89f37ac332842010" candidate_uuid: "a23d76521e314ecd90f2c17354771ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff85372f3b344d5dbd5910c94bd87afc" is_pre_election: true
I20240502 01:48:42.650367 2715 raft_consensus.cc:2454] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a23d76521e314ecd90f2c17354771ae9 in term 0.
I20240502 01:48:42.650683 2604 leader_election.cc:302] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 664eb6b43abc4cd78b65076ff662bae6, a23d76521e314ecd90f2c17354771ae9; no voters:
I20240502 01:48:42.651706 2845 raft_consensus.cc:2781] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:42.652105 2845 raft_consensus.cc:482] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:42.652496 2845 raft_consensus.cc:3037] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.657889 2845 raft_consensus.cc:504] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.660320 2845 leader_election.cc:288] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [CANDIDATE]: Term 1 election: Requested vote from peers ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255), 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:42.661289 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ca6c78d51049ee89f37ac332842010" candidate_uuid: "a23d76521e314ecd90f2c17354771ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff85372f3b344d5dbd5910c94bd87afc"
I20240502 01:48:42.661702 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ca6c78d51049ee89f37ac332842010" candidate_uuid: "a23d76521e314ecd90f2c17354771ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664eb6b43abc4cd78b65076ff662bae6"
I20240502 01:48:42.662027 2715 raft_consensus.cc:3037] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.662330 2792 raft_consensus.cc:3037] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.668882 2715 raft_consensus.cc:2454] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a23d76521e314ecd90f2c17354771ae9 in term 1.
I20240502 01:48:42.668882 2792 raft_consensus.cc:2454] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a23d76521e314ecd90f2c17354771ae9 in term 1.
I20240502 01:48:42.670258 2604 leader_election.cc:302] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 664eb6b43abc4cd78b65076ff662bae6, a23d76521e314ecd90f2c17354771ae9; no voters:
I20240502 01:48:42.670991 2845 raft_consensus.cc:2781] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:42.671530 2845 raft_consensus.cc:686] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a23d76521e314ecd90f2c17354771ae9, State: Running, Role: LEADER
I20240502 01:48:42.672411 2845 consensus_queue.cc:237] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.679270 2533 catalog_manager.cc:5484] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a23d76521e314ecd90f2c17354771ae9 (127.1.233.129). New cstate: current_term: 1 leader_uuid: "a23d76521e314ecd90f2c17354771ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:42.688859 2865 raft_consensus.cc:482] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:42.689327 2865 raft_consensus.cc:504] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.690903 2865 leader_election.cc:288] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:42.691787 2639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19da3a831744a649c477c48f46832ff" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9" is_pre_election: true
I20240502 01:48:42.692008 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19da3a831744a649c477c48f46832ff" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664eb6b43abc4cd78b65076ff662bae6" is_pre_election: true
I20240502 01:48:42.692431 2639 raft_consensus.cc:2454] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff85372f3b344d5dbd5910c94bd87afc in term 0.
I20240502 01:48:42.692637 2792 raft_consensus.cc:2454] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff85372f3b344d5dbd5910c94bd87afc in term 0.
I20240502 01:48:42.693429 2680 leader_election.cc:302] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a23d76521e314ecd90f2c17354771ae9, ff85372f3b344d5dbd5910c94bd87afc; no voters:
I20240502 01:48:42.694154 2865 raft_consensus.cc:2781] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:42.694420 2865 raft_consensus.cc:482] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:42.694674 2865 raft_consensus.cc:3037] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.699134 2865 raft_consensus.cc:504] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.700835 2865 leader_election.cc:288] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 election: Requested vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:42.701763 2639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19da3a831744a649c477c48f46832ff" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9"
I20240502 01:48:42.701956 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19da3a831744a649c477c48f46832ff" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664eb6b43abc4cd78b65076ff662bae6"
I20240502 01:48:42.702466 2792 raft_consensus.cc:3037] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.702433 2639 raft_consensus.cc:3037] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.709071 2639 raft_consensus.cc:2454] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff85372f3b344d5dbd5910c94bd87afc in term 1.
I20240502 01:48:42.709067 2792 raft_consensus.cc:2454] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff85372f3b344d5dbd5910c94bd87afc in term 1.
I20240502 01:48:42.710453 2682 leader_election.cc:302] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 664eb6b43abc4cd78b65076ff662bae6, ff85372f3b344d5dbd5910c94bd87afc; no voters:
I20240502 01:48:42.711151 2865 raft_consensus.cc:2781] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:42.711656 2865 raft_consensus.cc:686] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 1 LEADER]: Becoming Leader. State: Replica: ff85372f3b344d5dbd5910c94bd87afc, State: Running, Role: LEADER
I20240502 01:48:42.712457 2865 consensus_queue.cc:237] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.719914 2862 raft_consensus.cc:482] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:42.720553 2862 raft_consensus.cc:504] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.719903 2533 catalog_manager.cc:5484] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc reported cstate change: term changed from 0 to 1, leader changed from <none> to ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130). New cstate: current_term: 1 leader_uuid: "ff85372f3b344d5dbd5910c94bd87afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:42.722146 2862 leader_election.cc:288] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255)
I20240502 01:48:42.723322 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d43ecf839ef46eaa968c5c319b22e3c" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff85372f3b344d5dbd5910c94bd87afc" is_pre_election: true
I20240502 01:48:42.723237 2639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d43ecf839ef46eaa968c5c319b22e3c" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9" is_pre_election: true
I20240502 01:48:42.724181 2639 raft_consensus.cc:2454] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 664eb6b43abc4cd78b65076ff662bae6 in term 0.
I20240502 01:48:42.724123 2715 raft_consensus.cc:2454] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 664eb6b43abc4cd78b65076ff662bae6 in term 0.
I20240502 01:48:42.725409 2757 leader_election.cc:302] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 664eb6b43abc4cd78b65076ff662bae6, ff85372f3b344d5dbd5910c94bd87afc; no voters:
I20240502 01:48:42.726224 2862 raft_consensus.cc:2781] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:42.726595 2862 raft_consensus.cc:482] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:42.726877 2862 raft_consensus.cc:3037] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.732013 2862 raft_consensus.cc:504] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.733805 2862 leader_election.cc:288] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 election: Requested vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255)
I20240502 01:48:42.734671 2639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d43ecf839ef46eaa968c5c319b22e3c" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9"
I20240502 01:48:42.734900 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d43ecf839ef46eaa968c5c319b22e3c" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff85372f3b344d5dbd5910c94bd87afc"
I20240502 01:48:42.735428 2715 raft_consensus.cc:3037] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.735423 2639 raft_consensus.cc:3037] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.742506 2639 raft_consensus.cc:2454] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 664eb6b43abc4cd78b65076ff662bae6 in term 1.
I20240502 01:48:42.742506 2715 raft_consensus.cc:2454] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 664eb6b43abc4cd78b65076ff662bae6 in term 1.
I20240502 01:48:42.744045 2755 leader_election.cc:302] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 664eb6b43abc4cd78b65076ff662bae6, a23d76521e314ecd90f2c17354771ae9; no voters:
I20240502 01:48:42.744946 2862 raft_consensus.cc:2781] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:42.745393 2862 raft_consensus.cc:686] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 1 LEADER]: Becoming Leader. State: Replica: 664eb6b43abc4cd78b65076ff662bae6, State: Running, Role: LEADER
I20240502 01:48:42.746027 2862 consensus_queue.cc:237] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.754130 2533 catalog_manager.cc:5484] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131). New cstate: current_term: 1 leader_uuid: "664eb6b43abc4cd78b65076ff662bae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:42.785727 2865 raft_consensus.cc:482] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:42.786265 2865 raft_consensus.cc:504] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.787899 2865 leader_election.cc:288] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:42.788955 2639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e780227c194beda6e8fde8df155c6a" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9" is_pre_election: true
I20240502 01:48:42.789134 2792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e780227c194beda6e8fde8df155c6a" candidate_uuid: "ff85372f3b344d5dbd5910c94bd87afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664eb6b43abc4cd78b65076ff662bae6" is_pre_election: true
I20240502 01:48:42.789695 2639 raft_consensus.cc:2379] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff85372f3b344d5dbd5910c94bd87afc in current term 1: Already voted for candidate 664eb6b43abc4cd78b65076ff662bae6 in this term.
I20240502 01:48:42.790717 2680 leader_election.cc:302] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff85372f3b344d5dbd5910c94bd87afc; no voters: 664eb6b43abc4cd78b65076ff662bae6, a23d76521e314ecd90f2c17354771ae9
I20240502 01:48:42.791373 2865 raft_consensus.cc:3037] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.796595 2865 raft_consensus.cc:2726] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:48:42.845772 2862 raft_consensus.cc:482] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:42.846283 2862 raft_consensus.cc:504] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } }
I20240502 01:48:42.847932 2862 leader_election.cc:288] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209), ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255)
I20240502 01:48:42.848933 2639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "299f63f8961a4079ac1101429673be59" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23d76521e314ecd90f2c17354771ae9" is_pre_election: true
I20240502 01:48:42.849058 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "299f63f8961a4079ac1101429673be59" candidate_uuid: "664eb6b43abc4cd78b65076ff662bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff85372f3b344d5dbd5910c94bd87afc" is_pre_election: true
I20240502 01:48:42.849640 2639 raft_consensus.cc:2379] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 664eb6b43abc4cd78b65076ff662bae6 in current term 1: Already voted for candidate ff85372f3b344d5dbd5910c94bd87afc in this term.
I20240502 01:48:42.850592 2755 leader_election.cc:302] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 664eb6b43abc4cd78b65076ff662bae6; no voters: a23d76521e314ecd90f2c17354771ae9, ff85372f3b344d5dbd5910c94bd87afc
I20240502 01:48:42.851228 2862 raft_consensus.cc:3037] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:42.855872 2853 consensus_queue.cc:1035] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:42.856245 2862 raft_consensus.cc:2726] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:48:42.867841 2862 consensus_queue.cc:1035] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:42.886757 2862 consensus_queue.cc:1035] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:42.897146 2862 consensus_queue.cc:1035] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:42.928175 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:42.935591 2895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:42.936311 2896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:42.937561 2898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:42.938171 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:42.939181 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:42.939411 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:42.939592 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614522939578 us; error 0 us; skew 500 ppm
I20240502 01:48:42.940295 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:42.940685 2865 consensus_queue.cc:1035] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:42.943514 1958 webserver.cc:460] Webserver started at http://127.1.233.132:40151/ using document root <none> and password file <none>
I20240502 01:48:42.944193 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:42.944404 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:42.944679 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:42.946346 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-3-root/instance:
uuid: "9a67c94a14a345eda27ba563e86012ce"
format_stamp: "Formatted at 2024-05-02 01:48:42 on dist-test-slave-2090"
I20240502 01:48:42.951666 2865 consensus_queue.cc:1035] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:42.970973 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.017s sys 0.008s
I20240502 01:48:42.978327 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:42.979462 1958 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20240502 01:48:42.979840 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-3-root
uuid: "9a67c94a14a345eda27ba563e86012ce"
format_stamp: "Formatted at 2024-05-02 01:48:42 on dist-test-slave-2090"
I20240502 01:48:42.980177 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:42.991112 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:42.992367 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:42.993891 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:42.996430 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:42.996695 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:42.996938 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:42.997100 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:43.010821 2903 consensus_queue.cc:1035] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:43.027510 2865 consensus_queue.cc:1035] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:43.030227 2845 consensus_queue.cc:1035] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:43.046768 2883 consensus_queue.cc:1035] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:43.065378 2845 consensus_queue.cc:1035] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:43.085472 2883 consensus_queue.cc:1035] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:43.115317 2846 consensus_queue.cc:1035] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:43.115619 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.132:36269
I20240502 01:48:43.115708 2978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.132:36269 every 8 connection(s)
I20240502 01:48:43.132004 2865 consensus_queue.cc:1035] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:43.148128 2979 heartbeater.cc:347] Connected to a master server at 127.1.233.190:46231
I20240502 01:48:43.148564 2979 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:43.149367 2979 heartbeater.cc:511] Master 127.1.233.190:46231 requested a full tablet report, sending...
I20240502 01:48:43.151590 2536 ts_manager.cc:194] Registered new tserver with Master: 9a67c94a14a345eda27ba563e86012ce (127.1.233.132:36269)
I20240502 01:48:43.154471 2536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49644
I20240502 01:48:43.190531 2862 consensus_queue.cc:1035] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:43.201529 2853 consensus_queue.cc:1035] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:44.157397 2979 heartbeater.cc:503] Master 127.1.233.190:46231 was elected leader, sending a full tablet report...
I20240502 01:48:44.481873 2589 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:44.487977 2589 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:48:44.488543 2589 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:46.489686 2589 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:46.495609 2589 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:48:46.496222 2589 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:48.497385 2589 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:48.505113 2589 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:48:48.505643 2589 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:48.515620 2791 consensus_queue.cc:237] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } attrs { replace: true } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } }
I20240502 01:48:48.524832 2715 raft_consensus.cc:1261] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Refusing update from remote peer 664eb6b43abc4cd78b65076ff662bae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:48.527273 2853 consensus_queue.cc:1035] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:48:48.531836 2639 raft_consensus.cc:1261] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Refusing update from remote peer 664eb6b43abc4cd78b65076ff662bae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:48.533479 2853 consensus_queue.cc:1035] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:48.539347 2853 raft_consensus.cc:2932] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } attrs { replace: true } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
I20240502 01:48:48.541568 2715 raft_consensus.cc:2932] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } attrs { replace: true } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
W20240502 01:48:48.543270 2755 consensus_peers.cc:485] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 -> Peer 9a67c94a14a345eda27ba563e86012ce (127.1.233.132:36269): Couldn't send request to peer 9a67c94a14a345eda27ba563e86012ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57e780227c194beda6e8fde8df155c6a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:48:48.548739 2640 raft_consensus.cc:2932] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } attrs { replace: true } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
I20240502 01:48:48.557953 2534 catalog_manager.cc:5484] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc reported cstate change: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New cstate: current_term: 1 leader_uuid: "664eb6b43abc4cd78b65076ff662bae6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } attrs { replace: true } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
I20240502 01:48:48.577654 2715 consensus_queue.cc:237] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } attrs { replace: true } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } }
I20240502 01:48:48.587150 2791 raft_consensus.cc:1261] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Refusing update from remote peer ff85372f3b344d5dbd5910c94bd87afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:48.588493 2640 raft_consensus.cc:1261] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Refusing update from remote peer ff85372f3b344d5dbd5910c94bd87afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:48.589836 2865 consensus_queue.cc:1035] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:48:48.590998 3008 consensus_queue.cc:1035] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:48:48.596541 2680 consensus_peers.cc:485] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc -> Peer 9a67c94a14a345eda27ba563e86012ce (127.1.233.132:36269): Couldn't send request to peer 9a67c94a14a345eda27ba563e86012ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 299f63f8961a4079ac1101429673be59. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:48:48.600400 3008 raft_consensus.cc:2932] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } attrs { replace: true } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
I20240502 01:48:48.602546 2640 raft_consensus.cc:2932] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } attrs { replace: true } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
I20240502 01:48:48.605571 2791 raft_consensus.cc:2932] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } attrs { replace: true } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
I20240502 01:48:48.616986 2536 catalog_manager.cc:5484] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9 reported cstate change: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New cstate: current_term: 1 leader_uuid: "ff85372f3b344d5dbd5910c94bd87afc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } attrs { replace: true } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
I20240502 01:48:48.621438 2791 consensus_queue.cc:237] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } }
W20240502 01:48:48.627876 2755 consensus_peers.cc:485] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 -> Peer 9a67c94a14a345eda27ba563e86012ce (127.1.233.132:36269): Couldn't send request to peer 9a67c94a14a345eda27ba563e86012ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d43ecf839ef46eaa968c5c319b22e3c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:48:48.629607 2715 raft_consensus.cc:1261] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Refusing update from remote peer 664eb6b43abc4cd78b65076ff662bae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:48.629740 2640 raft_consensus.cc:1261] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Refusing update from remote peer 664eb6b43abc4cd78b65076ff662bae6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:48.632246 2995 consensus_queue.cc:1035] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:48:48.633350 3000 consensus_queue.cc:1035] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:48:48.641522 3000 raft_consensus.cc:2932] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
I20240502 01:48:48.644677 2715 raft_consensus.cc:2932] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
I20240502 01:48:48.648957 2640 raft_consensus.cc:2932] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
I20240502 01:48:48.655532 2716 consensus_queue.cc:237] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } }
I20240502 01:48:48.655103 2536 catalog_manager.cc:5484] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 reported cstate change: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New cstate: current_term: 1 leader_uuid: "664eb6b43abc4cd78b65076ff662bae6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:48.669437 2640 raft_consensus.cc:1261] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Refusing update from remote peer ff85372f3b344d5dbd5910c94bd87afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:48:48.670209 2680 consensus_peers.cc:485] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc -> Peer 9a67c94a14a345eda27ba563e86012ce (127.1.233.132:36269): Couldn't send request to peer 9a67c94a14a345eda27ba563e86012ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d19da3a831744a649c477c48f46832ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:48:48.671200 2865 consensus_queue.cc:1035] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:48.673390 2791 raft_consensus.cc:1261] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Refusing update from remote peer ff85372f3b344d5dbd5910c94bd87afc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:48.674993 2865 consensus_queue.cc:1035] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:48:48.680706 3008 raft_consensus.cc:2932] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
I20240502 01:48:48.682572 2640 raft_consensus.cc:2932] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
I20240502 01:48:48.686893 2791 raft_consensus.cc:2932] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } } }
I20240502 01:48:48.693872 2534 catalog_manager.cc:5484] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc reported cstate change: config changed from index -1 to 2, NON_VOTER 9a67c94a14a345eda27ba563e86012ce (127.1.233.132) added. New cstate: current_term: 1 leader_uuid: "ff85372f3b344d5dbd5910c94bd87afc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:48:48.978421 3018 ts_tablet_manager.cc:902] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce: Initiating tablet copy from peer 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:48.983165 3018 tablet_copy_client.cc:308] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Beginning tablet copy session from remote peer at address 127.1.233.131:35541
I20240502 01:48:48.994709 2804 tablet_copy_service.cc:140] P 664eb6b43abc4cd78b65076ff662bae6: Received BeginTabletCopySession request for tablet 57e780227c194beda6e8fde8df155c6a from peer 9a67c94a14a345eda27ba563e86012ce ({username='slave'} at 127.0.0.1:55218)
I20240502 01:48:48.995348 2804 tablet_copy_service.cc:161] P 664eb6b43abc4cd78b65076ff662bae6: Beginning new tablet copy session on tablet 57e780227c194beda6e8fde8df155c6a from peer 9a67c94a14a345eda27ba563e86012ce at {username='slave'} at 127.0.0.1:55218: session id = 9a67c94a14a345eda27ba563e86012ce-57e780227c194beda6e8fde8df155c6a
I20240502 01:48:49.003793 2804 tablet_copy_source_session.cc:205] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:48:49.007462 3018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e780227c194beda6e8fde8df155c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:49.020943 3018 tablet_copy_client.cc:783] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Starting download of 0 data blocks...
I20240502 01:48:49.021668 3018 tablet_copy_client.cc:647] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Starting download of 1 WAL segments...
I20240502 01:48:49.023226 3024 ts_tablet_manager.cc:902] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce: Initiating tablet copy from peer 664eb6b43abc4cd78b65076ff662bae6 (127.1.233.131:35541)
I20240502 01:48:49.028242 3018 tablet_copy_client.cc:523] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:48:49.030584 3024 tablet_copy_client.cc:308] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Beginning tablet copy session from remote peer at address 127.1.233.131:35541
I20240502 01:48:49.032306 2804 tablet_copy_service.cc:140] P 664eb6b43abc4cd78b65076ff662bae6: Received BeginTabletCopySession request for tablet 9d43ecf839ef46eaa968c5c319b22e3c from peer 9a67c94a14a345eda27ba563e86012ce ({username='slave'} at 127.0.0.1:55218)
I20240502 01:48:49.032848 2804 tablet_copy_service.cc:161] P 664eb6b43abc4cd78b65076ff662bae6: Beginning new tablet copy session on tablet 9d43ecf839ef46eaa968c5c319b22e3c from peer 9a67c94a14a345eda27ba563e86012ce at {username='slave'} at 127.0.0.1:55218: session id = 9a67c94a14a345eda27ba563e86012ce-9d43ecf839ef46eaa968c5c319b22e3c
I20240502 01:48:49.038796 3018 tablet_bootstrap.cc:492] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce: Bootstrap starting.
I20240502 01:48:49.041344 2804 tablet_copy_source_session.cc:205] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:48:49.044210 3024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d43ecf839ef46eaa968c5c319b22e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:49.057631 3024 tablet_copy_client.cc:783] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Starting download of 0 data blocks...
I20240502 01:48:49.058179 3024 tablet_copy_client.cc:647] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Starting download of 1 WAL segments...
I20240502 01:48:49.061702 3024 tablet_copy_client.cc:523] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:48:49.068821 3018 tablet_bootstrap.cc:492] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:48:49.070129 3018 tablet_bootstrap.cc:492] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce: Bootstrap complete.
I20240502 01:48:49.070820 3018 ts_tablet_manager.cc:1367] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.004s
I20240502 01:48:49.070811 3024 tablet_bootstrap.cc:492] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce: Bootstrap starting.
I20240502 01:48:49.074015 3018 raft_consensus.cc:348] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } attrs { replace: true } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } }
I20240502 01:48:49.075327 3018 raft_consensus.cc:729] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a67c94a14a345eda27ba563e86012ce, State: Initialized, Role: LEARNER
I20240502 01:48:49.076119 3018 consensus_queue.cc:260] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } attrs { replace: true } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } }
I20240502 01:48:49.087164 3018 ts_tablet_manager.cc:1398] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce: Time spent starting tablet: real 0.016s user 0.009s sys 0.003s
I20240502 01:48:49.089278 2804 tablet_copy_service.cc:342] P 664eb6b43abc4cd78b65076ff662bae6: Request end of tablet copy session 9a67c94a14a345eda27ba563e86012ce-57e780227c194beda6e8fde8df155c6a received from {username='slave'} at 127.0.0.1:55218
I20240502 01:48:49.089810 2804 tablet_copy_service.cc:433] P 664eb6b43abc4cd78b65076ff662bae6: ending tablet copy session 9a67c94a14a345eda27ba563e86012ce-57e780227c194beda6e8fde8df155c6a on tablet 57e780227c194beda6e8fde8df155c6a with peer 9a67c94a14a345eda27ba563e86012ce
I20240502 01:48:49.091361 3024 tablet_bootstrap.cc:492] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:48:49.093528 3024 tablet_bootstrap.cc:492] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce: Bootstrap complete.
I20240502 01:48:49.094143 3024 ts_tablet_manager.cc:1367] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.004s
I20240502 01:48:49.097074 3024 raft_consensus.cc:348] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } }
I20240502 01:48:49.098037 3024 raft_consensus.cc:729] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a67c94a14a345eda27ba563e86012ce, State: Initialized, Role: LEARNER
I20240502 01:48:49.098744 3024 consensus_queue.cc:260] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } }
I20240502 01:48:49.100893 3024 ts_tablet_manager.cc:1398] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240502 01:48:49.102748 2804 tablet_copy_service.cc:342] P 664eb6b43abc4cd78b65076ff662bae6: Request end of tablet copy session 9a67c94a14a345eda27ba563e86012ce-9d43ecf839ef46eaa968c5c319b22e3c received from {username='slave'} at 127.0.0.1:55218
I20240502 01:48:49.103179 2804 tablet_copy_service.cc:433] P 664eb6b43abc4cd78b65076ff662bae6: ending tablet copy session 9a67c94a14a345eda27ba563e86012ce-9d43ecf839ef46eaa968c5c319b22e3c on tablet 9d43ecf839ef46eaa968c5c319b22e3c with peer 9a67c94a14a345eda27ba563e86012ce
I20240502 01:48:49.116441 3024 ts_tablet_manager.cc:902] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce: Initiating tablet copy from peer ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255)
I20240502 01:48:49.119956 3024 tablet_copy_client.cc:308] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Beginning tablet copy session from remote peer at address 127.1.233.130:44255
I20240502 01:48:49.130394 2727 tablet_copy_service.cc:140] P ff85372f3b344d5dbd5910c94bd87afc: Received BeginTabletCopySession request for tablet 299f63f8961a4079ac1101429673be59 from peer 9a67c94a14a345eda27ba563e86012ce ({username='slave'} at 127.0.0.1:43462)
I20240502 01:48:49.130930 2727 tablet_copy_service.cc:161] P ff85372f3b344d5dbd5910c94bd87afc: Beginning new tablet copy session on tablet 299f63f8961a4079ac1101429673be59 from peer 9a67c94a14a345eda27ba563e86012ce at {username='slave'} at 127.0.0.1:43462: session id = 9a67c94a14a345eda27ba563e86012ce-299f63f8961a4079ac1101429673be59
I20240502 01:48:49.137260 2727 tablet_copy_source_session.cc:205] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:48:49.140287 3024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 299f63f8961a4079ac1101429673be59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:49.154865 3024 tablet_copy_client.cc:783] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Starting download of 0 data blocks...
I20240502 01:48:49.155542 3024 tablet_copy_client.cc:647] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Starting download of 1 WAL segments...
I20240502 01:48:49.159123 3024 tablet_copy_client.cc:523] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:48:49.167531 3024 tablet_bootstrap.cc:492] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce: Bootstrap starting.
I20240502 01:48:49.169940 1958 tablet_server.cc:178] TabletServer@127.1.233.129:0 shutting down...
W20240502 01:48:49.184598 2755 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 6 similar messages]
W20240502 01:48:49.189448 2755 consensus_peers.cc:485] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 -> Peer a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209): Couldn't send request to peer a23d76521e314ecd90f2c17354771ae9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:48:49.204234 3024 tablet_bootstrap.cc:492] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:48:49.205689 3024 tablet_bootstrap.cc:492] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce: Bootstrap complete.
I20240502 01:48:49.206455 3024 ts_tablet_manager.cc:1367] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce: Time spent bootstrapping tablet: real 0.039s user 0.021s sys 0.013s
I20240502 01:48:49.209801 3024 raft_consensus.cc:348] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } attrs { replace: true } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } }
I20240502 01:48:49.210770 3024 raft_consensus.cc:729] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a67c94a14a345eda27ba563e86012ce, State: Initialized, Role: LEARNER
I20240502 01:48:49.211544 3024 consensus_queue.cc:260] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a23d76521e314ecd90f2c17354771ae9" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43209 } } peers { permanent_uuid: "ff85372f3b344d5dbd5910c94bd87afc" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 44255 } } peers { permanent_uuid: "664eb6b43abc4cd78b65076ff662bae6" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 35541 } attrs { replace: true } } peers { permanent_uuid: "9a67c94a14a345eda27ba563e86012ce" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 36269 } attrs { promote: true } }
I20240502 01:48:49.213779 3024 ts_tablet_manager.cc:1398] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 01:48:49.217351 2727 tablet_copy_service.cc:342] P ff85372f3b344d5dbd5910c94bd87afc: Request end of tablet copy session 9a67c94a14a345eda27ba563e86012ce-299f63f8961a4079ac1101429673be59 received from {username='slave'} at 127.0.0.1:43462
I20240502 01:48:49.217793 2727 tablet_copy_service.cc:433] P ff85372f3b344d5dbd5910c94bd87afc: ending tablet copy session 9a67c94a14a345eda27ba563e86012ce-299f63f8961a4079ac1101429673be59 on tablet 299f63f8961a4079ac1101429673be59 with peer 9a67c94a14a345eda27ba563e86012ce
I20240502 01:48:49.235154 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:48:49.236040 1958 tablet_replica.cc:330] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9: stopping tablet replica
I20240502 01:48:49.236945 1958 raft_consensus.cc:2229] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.237423 1958 raft_consensus.cc:2258] T 9d43ecf839ef46eaa968c5c319b22e3c P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.240012 1958 tablet_replica.cc:330] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9: stopping tablet replica
I20240502 01:48:49.240658 1958 raft_consensus.cc:2229] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:49.241487 1958 raft_consensus.cc:2258] T 06ca6c78d51049ee89f37ac332842010 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.243674 1958 tablet_replica.cc:330] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9: stopping tablet replica
I20240502 01:48:49.244307 1958 raft_consensus.cc:2229] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:49.245265 1958 raft_consensus.cc:2258] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.247349 1958 tablet_replica.cc:330] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9: stopping tablet replica
I20240502 01:48:49.247877 1958 raft_consensus.cc:2229] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.248453 1958 raft_consensus.cc:2258] T d19da3a831744a649c477c48f46832ff P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.251137 1958 tablet_replica.cc:330] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9: stopping tablet replica
I20240502 01:48:49.251868 1958 raft_consensus.cc:2229] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.252338 1958 raft_consensus.cc:2258] T 464f34b3a41e4e959f816a0e24aae594 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.254490 1958 tablet_replica.cc:330] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9: stopping tablet replica
I20240502 01:48:49.255198 1958 raft_consensus.cc:2229] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.255685 1958 raft_consensus.cc:2258] T bc46edb8342d42798bdc4d08225a16b7 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.257956 1958 tablet_replica.cc:330] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9: stopping tablet replica
I20240502 01:48:49.258560 1958 raft_consensus.cc:2229] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.258992 1958 raft_consensus.cc:2258] T 299f63f8961a4079ac1101429673be59 P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.261616 1958 tablet_replica.cc:330] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9: stopping tablet replica
I20240502 01:48:49.262156 1958 raft_consensus.cc:2229] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.262661 1958 raft_consensus.cc:2258] T 57e780227c194beda6e8fde8df155c6a P a23d76521e314ecd90f2c17354771ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.276747 3024 ts_tablet_manager.cc:902] T d19da3a831744a649c477c48f46832ff P 9a67c94a14a345eda27ba563e86012ce: Initiating tablet copy from peer ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255)
W20240502 01:48:49.277582 2680 consensus_peers.cc:485] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc -> Peer a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209): Couldn't send request to peer a23d76521e314ecd90f2c17354771ae9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:48:49.289711 1958 tablet_server.cc:194] TabletServer@127.1.233.129:0 shutdown complete.
I20240502 01:48:49.290791 3024 tablet_copy_client.cc:308] T d19da3a831744a649c477c48f46832ff P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Beginning tablet copy session from remote peer at address 127.1.233.130:44255
I20240502 01:48:49.292461 2727 tablet_copy_service.cc:140] P ff85372f3b344d5dbd5910c94bd87afc: Received BeginTabletCopySession request for tablet d19da3a831744a649c477c48f46832ff from peer 9a67c94a14a345eda27ba563e86012ce ({username='slave'} at 127.0.0.1:43462)
I20240502 01:48:49.292948 2727 tablet_copy_service.cc:161] P ff85372f3b344d5dbd5910c94bd87afc: Beginning new tablet copy session on tablet d19da3a831744a649c477c48f46832ff from peer 9a67c94a14a345eda27ba563e86012ce at {username='slave'} at 127.0.0.1:43462: session id = 9a67c94a14a345eda27ba563e86012ce-d19da3a831744a649c477c48f46832ff
I20240502 01:48:49.301183 2727 tablet_copy_source_session.cc:205] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:48:49.304719 3024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19da3a831744a649c477c48f46832ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:49.315339 1958 tablet_server.cc:178] TabletServer@127.1.233.130:0 shutting down...
I20240502 01:48:49.322011 3024 tablet_copy_client.cc:783] T d19da3a831744a649c477c48f46832ff P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Starting download of 0 data blocks...
I20240502 01:48:49.322737 3024 tablet_copy_client.cc:647] T d19da3a831744a649c477c48f46832ff P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Starting download of 1 WAL segments...
W20240502 01:48:49.323421 2584 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:48:49.323824 2584 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 01:48:49.335062 1958 tablet_copy_service.cc:362] P ff85372f3b344d5dbd5910c94bd87afc: Destroying tablet copy session 9a67c94a14a345eda27ba563e86012ce-d19da3a831744a649c477c48f46832ff due to service shutdown
I20240502 01:48:49.335582 1958 tablet_copy_service.cc:433] P ff85372f3b344d5dbd5910c94bd87afc: ending tablet copy session 9a67c94a14a345eda27ba563e86012ce-d19da3a831744a649c477c48f46832ff on tablet d19da3a831744a649c477c48f46832ff with peer 9a67c94a14a345eda27ba563e86012ce
W20240502 01:48:49.335634 2584 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:48:49.336024 2584 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:48:49.349059 2755 consensus_peers.cc:485] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 -> Peer a23d76521e314ecd90f2c17354771ae9 (127.1.233.129:43209): Couldn't send request to peer a23d76521e314ecd90f2c17354771ae9. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43209: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:48:49.360024 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:48:49.360857 1958 tablet_replica.cc:330] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc: stopping tablet replica
I20240502 01:48:49.361701 1958 raft_consensus.cc:2229] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.362191 1958 raft_consensus.cc:2258] T 06ca6c78d51049ee89f37ac332842010 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.364526 1958 tablet_replica.cc:330] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc: stopping tablet replica
I20240502 01:48:49.365064 1958 raft_consensus.cc:2229] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.365522 1958 raft_consensus.cc:2258] T 9d43ecf839ef46eaa968c5c319b22e3c P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.368172 1958 tablet_replica.cc:330] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc: stopping tablet replica
I20240502 01:48:49.368923 1958 raft_consensus.cc:2229] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:49.370182 1958 raft_consensus.cc:2258] T d19da3a831744a649c477c48f46832ff P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.374280 1958 tablet_replica.cc:330] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc: stopping tablet replica
I20240502 01:48:49.374971 1958 raft_consensus.cc:2229] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.375547 1958 raft_consensus.cc:2258] T 57e780227c194beda6e8fde8df155c6a P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.378247 1958 tablet_replica.cc:330] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc: stopping tablet replica
I20240502 01:48:49.378984 1958 raft_consensus.cc:2229] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:49.379909 1958 raft_consensus.cc:2258] T 464f34b3a41e4e959f816a0e24aae594 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.382318 1958 tablet_replica.cc:330] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc: stopping tablet replica
I20240502 01:48:49.382974 1958 raft_consensus.cc:2229] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.383493 1958 raft_consensus.cc:2258] T bc46edb8342d42798bdc4d08225a16b7 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.385521 1958 tablet_replica.cc:330] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc: stopping tablet replica
I20240502 01:48:49.386207 1958 raft_consensus.cc:2229] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:49.387133 1958 raft_consensus.cc:2258] T 299f63f8961a4079ac1101429673be59 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.389544 1958 tablet_replica.cc:330] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc: stopping tablet replica
I20240502 01:48:49.390022 1958 raft_consensus.cc:2229] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.390408 1958 raft_consensus.cc:2258] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P ff85372f3b344d5dbd5910c94bd87afc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:48:49.411329 3024 ts_tablet_manager.cc:980] T d19da3a831744a649c477c48f46832ff P 9a67c94a14a345eda27ba563e86012ce: Tablet Copy: Unable to fetch data from remote peer ff85372f3b344d5dbd5910c94bd87afc (127.1.233.130:44255): Network error: Unable to download WAL segment with seq. number 1: unable to fetch data from remote: Client connection negotiation failed: client connection to 127.1.233.130:44255: connect: Connection refused (error 111)
I20240502 01:48:49.411810 3024 tablet_replica.cc:330] T d19da3a831744a649c477c48f46832ff P 9a67c94a14a345eda27ba563e86012ce: stopping tablet replica
I20240502 01:48:49.412130 3024 raft_consensus.cc:2229] T d19da3a831744a649c477c48f46832ff P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:48:49.412974 3024 raft_consensus.cc:2258] T d19da3a831744a649c477c48f46832ff P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:48:49.416025 1958 tablet_server.cc:194] TabletServer@127.1.233.130:0 shutdown complete.
W20240502 01:48:49.416195 3024 tablet_copy_client.cc:1108] T d19da3a831744a649c477c48f46832ff P 9a67c94a14a345eda27ba563e86012ce: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.1.233.130:44255: connect: Connection refused (error 111)
W20240502 01:48:49.417654 3024 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Already present: tried to load directory group for tablet d19da3a831744a649c477c48f46832ff but one is already registered
I20240502 01:48:49.418298 3024 ts_tablet_manager.cc:1875] T d19da3a831744a649c477c48f46832ff P 9a67c94a14a345eda27ba563e86012ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:48:49.438416 3024 ts_tablet_manager.cc:1888] T d19da3a831744a649c477c48f46832ff P 9a67c94a14a345eda27ba563e86012ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.0
I20240502 01:48:49.438995 3024 log.cc:1196] T d19da3a831744a649c477c48f46832ff P 9a67c94a14a345eda27ba563e86012ce: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714614515033618-1958-0/minicluster-data/ts-3-root/wals/d19da3a831744a649c477c48f46832ff
I20240502 01:48:49.443256 1958 tablet_server.cc:178] TabletServer@127.1.233.131:0 shutting down...
W20240502 01:48:49.448946 2584 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:48:49.449314 2584 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:48:49.454427 2584 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 01:48:49.454761 2584 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 01:48:49.472604 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:48:49.473351 1958 tablet_replica.cc:330] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6: stopping tablet replica
I20240502 01:48:49.473971 1958 raft_consensus.cc:2229] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.474504 1958 raft_consensus.cc:2258] T 299f63f8961a4079ac1101429673be59 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.476819 1958 tablet_replica.cc:330] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6: stopping tablet replica
I20240502 01:48:49.477380 1958 raft_consensus.cc:2229] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.477872 1958 raft_consensus.cc:2258] T d19da3a831744a649c477c48f46832ff P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.480226 1958 tablet_replica.cc:330] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6: stopping tablet replica
I20240502 01:48:49.480731 1958 raft_consensus.cc:2229] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.481140 1958 raft_consensus.cc:2258] T 06ca6c78d51049ee89f37ac332842010 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.482936 1958 tablet_replica.cc:330] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6: stopping tablet replica
I20240502 01:48:49.483423 1958 raft_consensus.cc:2229] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:49.484377 1958 raft_consensus.cc:2258] T 9d43ecf839ef46eaa968c5c319b22e3c P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.486423 1958 tablet_replica.cc:330] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6: stopping tablet replica
I20240502 01:48:49.486877 1958 raft_consensus.cc:2229] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.487238 1958 raft_consensus.cc:2258] T 1601a679ad5f4c2d8f8ad3c7c7d3ffc4 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.488950 1958 tablet_replica.cc:330] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6: stopping tablet replica
I20240502 01:48:49.489378 1958 raft_consensus.cc:2229] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:49.490053 1958 raft_consensus.cc:2258] T bc46edb8342d42798bdc4d08225a16b7 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.491859 1958 tablet_replica.cc:330] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6: stopping tablet replica
I20240502 01:48:49.492412 1958 raft_consensus.cc:2229] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:48:49.492781 1958 raft_consensus.cc:2258] T 464f34b3a41e4e959f816a0e24aae594 P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.494593 1958 tablet_replica.cc:330] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6: stopping tablet replica
I20240502 01:48:49.495025 1958 raft_consensus.cc:2229] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:49.495929 1958 raft_consensus.cc:2258] T 57e780227c194beda6e8fde8df155c6a P 664eb6b43abc4cd78b65076ff662bae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.522920 1958 tablet_server.cc:194] TabletServer@127.1.233.131:0 shutdown complete.
I20240502 01:48:49.539106 1958 tablet_server.cc:178] TabletServer@127.1.233.132:0 shutting down...
I20240502 01:48:49.561101 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:48:49.562156 1958 tablet_replica.cc:330] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce: stopping tablet replica
I20240502 01:48:49.562815 1958 raft_consensus.cc:2229] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:48:49.563310 1958 raft_consensus.cc:2258] T 299f63f8961a4079ac1101429673be59 P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:48:49.565461 1958 tablet_replica.cc:330] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce: stopping tablet replica
I20240502 01:48:49.565971 1958 raft_consensus.cc:2229] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:48:49.566344 1958 raft_consensus.cc:2258] T 9d43ecf839ef46eaa968c5c319b22e3c P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:48:49.568307 1958 tablet_replica.cc:330] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce: stopping tablet replica
I20240502 01:48:49.568722 1958 raft_consensus.cc:2229] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:48:49.569048 1958 raft_consensus.cc:2258] T 57e780227c194beda6e8fde8df155c6a P 9a67c94a14a345eda27ba563e86012ce [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:48:49.589414 1958 tablet_server.cc:194] TabletServer@127.1.233.132:0 shutdown complete.
I20240502 01:48:49.601557 1958 master.cc:537] Master@127.1.233.190:46231 shutting down...
I20240502 01:48:49.620656 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:49.621177 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:49.621454 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4c31c0b9054c4032bcd8a49381eade59: stopping tablet replica
I20240502 01:48:49.640750 1958 master.cc:552] Master@127.1.233.190:46231 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (9347 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240502 01:48:49.681747 1958 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.233.190:41845,127.1.233.189:34797,127.1.233.188:42235
I20240502 01:48:49.683255 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:49.689567 3046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:49.690407 3047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:49.690927 3050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:49.691504 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:49.692399 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:49.692586 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:49.692701 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614529692691 us; error 0 us; skew 500 ppm
I20240502 01:48:49.693169 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:49.695624 1958 webserver.cc:460] Webserver started at http://127.1.233.190:34317/ using document root <none> and password file <none>
I20240502 01:48:49.696048 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:49.696251 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:49.696496 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:49.697566 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-0-root/instance:
uuid: "31819667ca494a8fafeb6ab2d554a41b"
format_stamp: "Formatted at 2024-05-02 01:48:49 on dist-test-slave-2090"
I20240502 01:48:49.702068 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240502 01:48:49.705390 3055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:49.706140 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:48:49.706425 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-0-root
uuid: "31819667ca494a8fafeb6ab2d554a41b"
format_stamp: "Formatted at 2024-05-02 01:48:49 on dist-test-slave-2090"
I20240502 01:48:49.706707 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:49.732841 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:49.734081 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:49.773160 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.190:41845
I20240502 01:48:49.773300 3107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.190:41845 every 8 connection(s)
I20240502 01:48:49.777168 3108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:49.777292 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:49.783875 3110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:49.784507 3111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:49.784883 3108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } has no permanent_uuid. Determining permanent_uuid...
W20240502 01:48:49.786800 3113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:49.788165 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:49.789280 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:49.789463 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:49.789645 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614529789624 us; error 0 us; skew 500 ppm
I20240502 01:48:49.790144 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:49.793032 1958 webserver.cc:460] Webserver started at http://127.1.233.189:33669/ using document root <none> and password file <none>
I20240502 01:48:49.793660 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:49.793872 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:49.794134 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:49.795476 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-1-root/instance:
uuid: "88a38fcae4944328befe752468c6892b"
format_stamp: "Formatted at 2024-05-02 01:48:49 on dist-test-slave-2090"
I20240502 01:48:49.798952 3108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:49.800817 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
W20240502 01:48:49.804477 3108 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.1.233.189:34797: Network error: Client connection negotiation failed: client connection to 127.1.233.189:34797: connect: Connection refused (error 111)
I20240502 01:48:49.805687 3121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:49.806492 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240502 01:48:49.806784 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-1-root
uuid: "88a38fcae4944328befe752468c6892b"
format_stamp: "Formatted at 2024-05-02 01:48:49 on dist-test-slave-2090"
I20240502 01:48:49.807065 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:49.831439 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:49.832859 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:49.854004 3108 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } attempt: 1
W20240502 01:48:49.860010 3108 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.1.233.189:34797: Network error: Client connection negotiation failed: client connection to 127.1.233.189:34797: connect: Connection refused (error 111)
I20240502 01:48:49.874037 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.189:34797
I20240502 01:48:49.874152 3173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.189:34797 every 8 connection(s)
I20240502 01:48:49.878170 3174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:49.878315 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:49.885010 3176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:49.885669 3177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:49.887778 3179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:49.887760 3174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:49.889447 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:49.891391 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:49.891687 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:49.891857 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614529891842 us; error 0 us; skew 500 ppm
I20240502 01:48:49.892590 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:49.895594 1958 webserver.cc:460] Webserver started at http://127.1.233.188:42847/ using document root <none> and password file <none>
I20240502 01:48:49.896287 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:49.896553 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:49.896907 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:49.897606 3108 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } attempt: 2
I20240502 01:48:49.898257 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-2-root/instance:
uuid: "c8af6c0d37394254b320e8f12d045e58"
format_stamp: "Formatted at 2024-05-02 01:48:49 on dist-test-slave-2090"
I20240502 01:48:49.902597 3174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:49.914331 3108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:49.920274 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.013s sys 0.008s
W20240502 01:48:49.921777 3108 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.1.233.188:42235: Network error: Client connection negotiation failed: client connection to 127.1.233.188:42235: connect: Connection refused (error 111)
I20240502 01:48:49.925156 3187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:49.925812 3174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:49.926168 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:48:49.926517 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-2-root
uuid: "c8af6c0d37394254b320e8f12d045e58"
format_stamp: "Formatted at 2024-05-02 01:48:49 on dist-test-slave-2090"
I20240502 01:48:49.926816 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240502 01:48:49.932535 3174 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.1.233.188:42235: Network error: Client connection negotiation failed: client connection to 127.1.233.188:42235: connect: Connection refused (error 111)
I20240502 01:48:49.953387 3108 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } attempt: 1
W20240502 01:48:49.958279 3108 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.1.233.188:42235: Network error: Client connection negotiation failed: client connection to 127.1.233.188:42235: connect: Connection refused (error 111)
I20240502 01:48:49.958668 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:49.959849 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:49.990155 3174 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } attempt: 1
W20240502 01:48:49.995581 3174 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.1.233.188:42235: Network error: Client connection negotiation failed: client connection to 127.1.233.188:42235: connect: Connection refused (error 111)
I20240502 01:48:50.001019 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.188:42235
I20240502 01:48:50.001142 3241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.188:42235 every 8 connection(s)
I20240502 01:48:50.004356 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:48:50.005326 3242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:50.011694 3242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:50.022956 3242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:50.024760 3108 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } attempt: 2
I20240502 01:48:50.034109 3242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } has no permanent_uuid. Determining permanent_uuid...
I20240502 01:48:50.042722 3108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b: Bootstrap starting.
I20240502 01:48:50.047989 3108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:50.049542 3242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58: Bootstrap starting.
I20240502 01:48:50.052497 3108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b: No bootstrap required, opened a new log
I20240502 01:48:50.054008 3242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:50.054667 3108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:50.055269 3108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:50.055487 3108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31819667ca494a8fafeb6ab2d554a41b, State: Initialized, Role: FOLLOWER
I20240502 01:48:50.056083 3108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:50.058497 3242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58: No bootstrap required, opened a new log
I20240502 01:48:50.058267 3248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:50.059113 3248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:50.059783 3108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:48:50.060750 3242 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:50.061287 3242 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:50.061501 3242 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8af6c0d37394254b320e8f12d045e58, State: Initialized, Role: FOLLOWER
I20240502 01:48:50.062220 3174 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } attempt: 2
I20240502 01:48:50.062121 3242 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:50.064630 3251 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:50.065577 3251 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:50.066220 3242 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:48:50.085304 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240502 01:48:50.088063 3174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b: Bootstrap starting.
W20240502 01:48:50.091168 3266 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:48:50.091629 3266 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 01:48:50.094027 3174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b: Neither blocks nor log segments found. Creating new log.
W20240502 01:48:50.094877 3271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:48:50.095187 3271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 01:48:50.098775 3174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b: No bootstrap required, opened a new log
I20240502 01:48:50.101012 3174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:50.101599 3174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:50.101848 3174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a38fcae4944328befe752468c6892b, State: Initialized, Role: FOLLOWER
I20240502 01:48:50.102434 3174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:50.104429 3273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:50.105100 3273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:50.105477 3174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:48:50.116595 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 01:48:50.118115 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:48:50.119822 3284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:48:50.120187 3284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 01:48:50.125010 3285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:50.126111 3286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:50.127630 3288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:50.128929 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:50.130602 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:50.130812 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:50.130947 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614530130936 us; error 0 us; skew 500 ppm
I20240502 01:48:50.131433 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:50.134061 1958 webserver.cc:460] Webserver started at http://127.1.233.129:46577/ using document root <none> and password file <none>
I20240502 01:48:50.134508 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:50.134673 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:50.134894 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:50.136065 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-0-root/instance:
uuid: "ce5cf4ab08d94c0ba8b638f4c1d7740b"
format_stamp: "Formatted at 2024-05-02 01:48:50 on dist-test-slave-2090"
I20240502 01:48:50.140913 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:48:50.144387 3293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:50.145207 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:48:50.145668 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "ce5cf4ab08d94c0ba8b638f4c1d7740b"
format_stamp: "Formatted at 2024-05-02 01:48:50 on dist-test-slave-2090"
I20240502 01:48:50.146025 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:50.166065 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:50.167481 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:50.169118 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:50.171831 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:50.172083 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:50.172361 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:50.172539 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:50.214430 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:45237
I20240502 01:48:50.214545 3355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:45237 every 8 connection(s)
I20240502 01:48:50.236981 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:48:50.238013 3356 heartbeater.cc:347] Connected to a master server at 127.1.233.188:42235
I20240502 01:48:50.238440 3356 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:50.239370 3356 heartbeater.cc:511] Master 127.1.233.188:42235 requested a full tablet report, sending...
I20240502 01:48:50.241493 3357 heartbeater.cc:347] Connected to a master server at 127.1.233.189:34797
I20240502 01:48:50.241899 3357 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:50.242614 3205 ts_manager.cc:194] Registered new tserver with Master: ce5cf4ab08d94c0ba8b638f4c1d7740b (127.1.233.129:45237)
I20240502 01:48:50.242908 3357 heartbeater.cc:511] Master 127.1.233.189:34797 requested a full tablet report, sending...
I20240502 01:48:50.244130 3358 heartbeater.cc:347] Connected to a master server at 127.1.233.190:41845
I20240502 01:48:50.244647 3358 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:50.245797 3358 heartbeater.cc:511] Master 127.1.233.190:41845 requested a full tablet report, sending...
W20240502 01:48:50.247113 3363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:48:50.247778 3364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:50.249881 3139 ts_manager.cc:194] Registered new tserver with Master: ce5cf4ab08d94c0ba8b638f4c1d7740b (127.1.233.129:45237)
W20240502 01:48:50.250286 3366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:50.251250 3072 ts_manager.cc:194] Registered new tserver with Master: ce5cf4ab08d94c0ba8b638f4c1d7740b (127.1.233.129:45237)
I20240502 01:48:50.254071 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:50.254832 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:50.255028 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:50.255178 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614530255167 us; error 0 us; skew 500 ppm
I20240502 01:48:50.255625 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:50.258193 1958 webserver.cc:460] Webserver started at http://127.1.233.130:45627/ using document root <none> and password file <none>
I20240502 01:48:50.258617 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:50.258772 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:50.258980 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:50.260099 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-1-root/instance:
uuid: "69c856ecd7204c24addb3c8c0bc9df05"
format_stamp: "Formatted at 2024-05-02 01:48:50 on dist-test-slave-2090"
I20240502 01:48:50.264853 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:48:50.268371 3371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:50.269210 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:48:50.269508 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-1-root
uuid: "69c856ecd7204c24addb3c8c0bc9df05"
format_stamp: "Formatted at 2024-05-02 01:48:50 on dist-test-slave-2090"
I20240502 01:48:50.269798 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:50.295220 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:50.296505 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:50.299211 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:50.301761 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:50.302000 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:50.302245 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:50.302402 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:50.344520 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.130:43821
I20240502 01:48:50.344614 3435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.130:43821 every 8 connection(s)
I20240502 01:48:50.347291 3248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:50.347941 3248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:50.350338 3248 leader_election.cc:288] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88a38fcae4944328befe752468c6892b (127.1.233.189:34797), c8af6c0d37394254b320e8f12d045e58 (127.1.233.188:42235)
I20240502 01:48:50.352756 3216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31819667ca494a8fafeb6ab2d554a41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8af6c0d37394254b320e8f12d045e58" is_pre_election: true
I20240502 01:48:50.352754 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31819667ca494a8fafeb6ab2d554a41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a38fcae4944328befe752468c6892b" is_pre_election: true
I20240502 01:48:50.354261 3216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31819667ca494a8fafeb6ab2d554a41b in term 0.
I20240502 01:48:50.354403 3149 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31819667ca494a8fafeb6ab2d554a41b in term 0.
I20240502 01:48:50.356835 3059 leader_election.cc:302] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31819667ca494a8fafeb6ab2d554a41b, 88a38fcae4944328befe752468c6892b; no voters:
I20240502 01:48:50.358821 3248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:48:50.359215 3248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:50.359567 3248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:50.360392 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:48:50.371542 3248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:50.375721 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31819667ca494a8fafeb6ab2d554a41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a38fcae4944328befe752468c6892b"
I20240502 01:48:50.376335 3436 heartbeater.cc:347] Connected to a master server at 127.1.233.188:42235
I20240502 01:48:50.376794 3436 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:50.376648 3149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:50.377789 3436 heartbeater.cc:511] Master 127.1.233.188:42235 requested a full tablet report, sending...
I20240502 01:48:50.380793 3205 ts_manager.cc:194] Registered new tserver with Master: 69c856ecd7204c24addb3c8c0bc9df05 (127.1.233.130:43821)
I20240502 01:48:50.389045 3438 heartbeater.cc:347] Connected to a master server at 127.1.233.190:41845
I20240502 01:48:50.389511 3438 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:50.389107 3216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31819667ca494a8fafeb6ab2d554a41b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8af6c0d37394254b320e8f12d045e58"
I20240502 01:48:50.390112 3216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:50.390312 3438 heartbeater.cc:511] Master 127.1.233.190:41845 requested a full tablet report, sending...
I20240502 01:48:50.392369 3437 heartbeater.cc:347] Connected to a master server at 127.1.233.189:34797
I20240502 01:48:50.392989 3072 ts_manager.cc:194] Registered new tserver with Master: 69c856ecd7204c24addb3c8c0bc9df05 (127.1.233.130:43821)
I20240502 01:48:50.394905 3149 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31819667ca494a8fafeb6ab2d554a41b in term 1.
I20240502 01:48:50.393198 3248 leader_election.cc:288] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [CANDIDATE]: Term 1 election: Requested vote from peers 88a38fcae4944328befe752468c6892b (127.1.233.189:34797), c8af6c0d37394254b320e8f12d045e58 (127.1.233.188:42235)
I20240502 01:48:50.396102 3059 leader_election.cc:302] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31819667ca494a8fafeb6ab2d554a41b, 88a38fcae4944328befe752468c6892b; no voters:
I20240502 01:48:50.397325 3248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:50.398774 3248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [term 1 LEADER]: Becoming Leader. State: Replica: 31819667ca494a8fafeb6ab2d554a41b, State: Running, Role: LEADER
W20240502 01:48:50.399466 3442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:50.392952 3437 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:50.399688 3248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:50.401400 3437 heartbeater.cc:511] Master 127.1.233.189:34797 requested a full tablet report, sending...
I20240502 01:48:50.404294 3445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31819667ca494a8fafeb6ab2d554a41b. Latest consensus state: current_term: 1 leader_uuid: "31819667ca494a8fafeb6ab2d554a41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:50.405125 3445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:50.405203 3139 ts_manager.cc:194] Registered new tserver with Master: 69c856ecd7204c24addb3c8c0bc9df05 (127.1.233.130:43821)
I20240502 01:48:50.407207 3216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31819667ca494a8fafeb6ab2d554a41b in term 1.
I20240502 01:48:50.413164 3447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20240502 01:48:50.413746 3444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:50.418901 3447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20240502 01:48:50.422519 3450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:48:50.423137 1958 server_base.cc:1034] running on GCE node
I20240502 01:48:50.424405 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:48:50.424726 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:48:50.424981 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614530424961 us; error 0 us; skew 500 ppm
I20240502 01:48:50.425760 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:48:50.429071 1958 webserver.cc:460] Webserver started at http://127.1.233.131:41331/ using document root <none> and password file <none>
I20240502 01:48:50.429708 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:48:50.429966 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:48:50.430080 3149 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 1 FOLLOWER]: Refusing update from remote peer 31819667ca494a8fafeb6ab2d554a41b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:50.430300 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:48:50.430119 3216 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 1 FOLLOWER]: Refusing update from remote peer 31819667ca494a8fafeb6ab2d554a41b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:48:50.431916 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-2-root/instance:
uuid: "5c3b3cbbd0334c70af8fe9b2079c9e23"
format_stamp: "Formatted at 2024-05-02 01:48:50 on dist-test-slave-2090"
I20240502 01:48:50.431977 3445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:48:50.432853 3248 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:48:50.437960 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240502 01:48:50.441283 3251 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31819667ca494a8fafeb6ab2d554a41b. Latest consensus state: current_term: 1 leader_uuid: "31819667ca494a8fafeb6ab2d554a41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:50.442423 3251 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:50.445751 3459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:50.447373 1958 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.004s
I20240502 01:48:50.448127 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-2-root
uuid: "5c3b3cbbd0334c70af8fe9b2079c9e23"
format_stamp: "Formatted at 2024-05-02 01:48:50 on dist-test-slave-2090"
I20240502 01:48:50.448710 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714614515033618-1958-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:48:50.449494 3445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31819667ca494a8fafeb6ab2d554a41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:50.450601 3445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:50.454420 3447 catalog_manager.cc:1347] Generated new cluster ID: 46dfda230726444e995f230c7dab8e9b
I20240502 01:48:50.454893 3447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:48:50.458870 3251 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31819667ca494a8fafeb6ab2d554a41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:50.460028 3251 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:50.465117 3273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31819667ca494a8fafeb6ab2d554a41b. Latest consensus state: current_term: 1 leader_uuid: "31819667ca494a8fafeb6ab2d554a41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:50.465917 3273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:50.470485 3445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31819667ca494a8fafeb6ab2d554a41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:50.471289 3445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:50.480816 3273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31819667ca494a8fafeb6ab2d554a41b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:50.481714 3273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:50.500660 3447 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:48:50.502519 3447 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:48:50.510591 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:48:50.512321 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:48:50.513732 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:48:50.516180 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:48:50.516402 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:50.516638 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:48:50.516794 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:48:50.534545 3447 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b: Generated new TSK 0
I20240502 01:48:50.535408 3447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:48:50.571732 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.131:43239
I20240502 01:48:50.571815 3526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.131:43239 every 8 connection(s)
I20240502 01:48:50.590314 3527 heartbeater.cc:347] Connected to a master server at 127.1.233.188:42235
I20240502 01:48:50.590808 3527 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:50.591707 3527 heartbeater.cc:511] Master 127.1.233.188:42235 requested a full tablet report, sending...
I20240502 01:48:50.592272 3528 heartbeater.cc:347] Connected to a master server at 127.1.233.189:34797
I20240502 01:48:50.592633 3528 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:50.593511 3528 heartbeater.cc:511] Master 127.1.233.189:34797 requested a full tablet report, sending...
I20240502 01:48:50.595273 3205 ts_manager.cc:194] Registered new tserver with Master: 5c3b3cbbd0334c70af8fe9b2079c9e23 (127.1.233.131:43239)
I20240502 01:48:50.595927 3139 ts_manager.cc:194] Registered new tserver with Master: 5c3b3cbbd0334c70af8fe9b2079c9e23 (127.1.233.131:43239)
I20240502 01:48:50.597035 3529 heartbeater.cc:347] Connected to a master server at 127.1.233.190:41845
I20240502 01:48:50.597357 3529 heartbeater.cc:464] Registering TS with master...
I20240502 01:48:50.598171 3529 heartbeater.cc:511] Master 127.1.233.190:41845 requested a full tablet report, sending...
I20240502 01:48:50.599872 3072 ts_manager.cc:194] Registered new tserver with Master: 5c3b3cbbd0334c70af8fe9b2079c9e23 (127.1.233.131:43239)
I20240502 01:48:50.600286 1958 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.020310319s
I20240502 01:48:50.601550 3072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49818
I20240502 01:48:51.099355 3271 catalog_manager.cc:1259] Loaded cluster ID: 46dfda230726444e995f230c7dab8e9b
I20240502 01:48:51.099658 3271 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58: loading cluster ID for follower catalog manager: success
I20240502 01:48:51.104357 3271 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58: acquiring CA information for follower catalog manager: success
I20240502 01:48:51.108233 3271 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 01:48:51.124038 3284 catalog_manager.cc:1259] Loaded cluster ID: 46dfda230726444e995f230c7dab8e9b
I20240502 01:48:51.124414 3284 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b: loading cluster ID for follower catalog manager: success
I20240502 01:48:51.129235 3284 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b: acquiring CA information for follower catalog manager: success
I20240502 01:48:51.132766 3284 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 01:48:51.256085 3072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49792
I20240502 01:48:51.398286 3072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49804
I20240502 01:48:51.604285 3529 heartbeater.cc:503] Master 127.1.233.190:41845 was elected leader, sending a full tablet report...
I20240502 01:48:52.084195 3265 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:52.152581 1958 test_util.cc:268] Using random seed: -836920085
I20240502 01:48:52.182165 3072 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49832:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 01:48:52.258489 3358 heartbeater.cc:503] Master 127.1.233.190:41845 was elected leader, sending a full tablet report...
I20240502 01:48:52.280304 3321 tablet_service.cc:1467] Processing CreateTablet for tablet 0614d8cbba3f4765b9c76332d3e92367 (DEFAULT_TABLE table=test-workload [id=4f07ac7e3c7744c1ac6bbde7cb5486a2]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 01:48:52.281680 3321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0614d8cbba3f4765b9c76332d3e92367. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:52.282953 3492 tablet_service.cc:1467] Processing CreateTablet for tablet 83c27b3e4bae4f3fa3d3c4f99901f7e1 (DEFAULT_TABLE table=test-workload [id=4f07ac7e3c7744c1ac6bbde7cb5486a2]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 01:48:52.284394 3492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c27b3e4bae4f3fa3d3c4f99901f7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:52.285621 3399 tablet_service.cc:1467] Processing CreateTablet for tablet 1c91f41480604b1095d4978c48bcc269 (DEFAULT_TABLE table=test-workload [id=4f07ac7e3c7744c1ac6bbde7cb5486a2]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 01:48:52.286572 3399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c91f41480604b1095d4978c48bcc269. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:48:52.300647 3565 tablet_bootstrap.cc:492] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23: Bootstrap starting.
I20240502 01:48:52.300094 3563 tablet_bootstrap.cc:492] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05: Bootstrap starting.
I20240502 01:48:52.306934 3564 tablet_bootstrap.cc:492] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b: Bootstrap starting.
I20240502 01:48:52.307152 3563 tablet_bootstrap.cc:654] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:52.307432 3565 tablet_bootstrap.cc:654] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:52.312733 3564 tablet_bootstrap.cc:654] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b: Neither blocks nor log segments found. Creating new log.
I20240502 01:48:52.313400 3563 tablet_bootstrap.cc:492] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05: No bootstrap required, opened a new log
I20240502 01:48:52.313414 3565 tablet_bootstrap.cc:492] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23: No bootstrap required, opened a new log
I20240502 01:48:52.314083 3563 ts_tablet_manager.cc:1367] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240502 01:48:52.314129 3565 ts_tablet_manager.cc:1367] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20240502 01:48:52.316990 3565 raft_consensus.cc:348] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3b3cbbd0334c70af8fe9b2079c9e23" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 43239 } }
I20240502 01:48:52.317669 3565 raft_consensus.cc:374] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:52.317591 3563 raft_consensus.cc:348] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c856ecd7204c24addb3c8c0bc9df05" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43821 } }
I20240502 01:48:52.318287 3565 raft_consensus.cc:729] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3b3cbbd0334c70af8fe9b2079c9e23, State: Initialized, Role: FOLLOWER
I20240502 01:48:52.318610 3563 raft_consensus.cc:374] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:52.319025 3565 consensus_queue.cc:260] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3b3cbbd0334c70af8fe9b2079c9e23" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 43239 } }
I20240502 01:48:52.319106 3563 raft_consensus.cc:729] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c856ecd7204c24addb3c8c0bc9df05, State: Initialized, Role: FOLLOWER
I20240502 01:48:52.319695 3565 raft_consensus.cc:388] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:48:52.320068 3565 raft_consensus.cc:482] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:48:52.320082 3563 consensus_queue.cc:260] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c856ecd7204c24addb3c8c0bc9df05" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43821 } }
I20240502 01:48:52.320652 3565 raft_consensus.cc:3037] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:52.320994 3563 raft_consensus.cc:388] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:48:52.321290 3563 raft_consensus.cc:482] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:48:52.321671 3563 raft_consensus.cc:3037] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:52.322932 3564 tablet_bootstrap.cc:492] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b: No bootstrap required, opened a new log
I20240502 01:48:52.323454 3564 ts_tablet_manager.cc:1367] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20240502 01:48:52.325654 3565 raft_consensus.cc:504] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3b3cbbd0334c70af8fe9b2079c9e23" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 43239 } }
I20240502 01:48:52.326243 3565 leader_election.cc:302] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c3b3cbbd0334c70af8fe9b2079c9e23; no voters:
I20240502 01:48:52.326115 3564 raft_consensus.cc:348] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5cf4ab08d94c0ba8b638f4c1d7740b" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 45237 } }
I20240502 01:48:52.326853 3564 raft_consensus.cc:374] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:48:52.327198 3564 raft_consensus.cc:729] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5cf4ab08d94c0ba8b638f4c1d7740b, State: Initialized, Role: FOLLOWER
I20240502 01:48:52.327772 3565 leader_election.cc:288] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:48:52.328063 3564 consensus_queue.cc:260] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5cf4ab08d94c0ba8b638f4c1d7740b" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 45237 } }
I20240502 01:48:52.328192 3563 raft_consensus.cc:504] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c856ecd7204c24addb3c8c0bc9df05" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43821 } }
I20240502 01:48:52.328418 3569 raft_consensus.cc:2781] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:52.329064 3564 raft_consensus.cc:388] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:48:52.329227 3563 leader_election.cc:302] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c856ecd7204c24addb3c8c0bc9df05; no voters:
I20240502 01:48:52.329551 3564 raft_consensus.cc:482] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:48:52.330145 3564 raft_consensus.cc:3037] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:48:52.337054 3565 ts_tablet_manager.cc:1398] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
I20240502 01:48:52.339103 3564 raft_consensus.cc:504] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5cf4ab08d94c0ba8b638f4c1d7740b" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 45237 } }
I20240502 01:48:52.340543 3563 leader_election.cc:288] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:48:52.341063 3569 raft_consensus.cc:686] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [term 1 LEADER]: Becoming Leader. State: Replica: 5c3b3cbbd0334c70af8fe9b2079c9e23, State: Running, Role: LEADER
I20240502 01:48:52.341404 3564 leader_election.cc:302] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce5cf4ab08d94c0ba8b638f4c1d7740b; no voters:
I20240502 01:48:52.341747 3569 consensus_queue.cc:237] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3b3cbbd0334c70af8fe9b2079c9e23" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 43239 } }
I20240502 01:48:52.340842 3572 raft_consensus.cc:2781] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:52.344355 3438 heartbeater.cc:503] Master 127.1.233.190:41845 was elected leader, sending a full tablet report...
I20240502 01:48:52.344594 3563 ts_tablet_manager.cc:1398] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05: Time spent starting tablet: real 0.030s user 0.017s sys 0.012s
I20240502 01:48:52.346088 3564 leader_election.cc:288] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:48:52.346664 3574 raft_consensus.cc:2781] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:48:52.347321 3572 raft_consensus.cc:686] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [term 1 LEADER]: Becoming Leader. State: Replica: 69c856ecd7204c24addb3c8c0bc9df05, State: Running, Role: LEADER
I20240502 01:48:52.352694 3572 consensus_queue.cc:237] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c856ecd7204c24addb3c8c0bc9df05" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43821 } }
I20240502 01:48:52.353053 3072 catalog_manager.cc:5484] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c3b3cbbd0334c70af8fe9b2079c9e23 (127.1.233.131). New cstate: current_term: 1 leader_uuid: "5c3b3cbbd0334c70af8fe9b2079c9e23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3b3cbbd0334c70af8fe9b2079c9e23" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 43239 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:52.398633 3072 catalog_manager.cc:5484] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69c856ecd7204c24addb3c8c0bc9df05 (127.1.233.130). New cstate: current_term: 1 leader_uuid: "69c856ecd7204c24addb3c8c0bc9df05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c856ecd7204c24addb3c8c0bc9df05" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43821 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:52.412103 3564 ts_tablet_manager.cc:1398] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b: Time spent starting tablet: real 0.088s user 0.033s sys 0.040s
I20240502 01:48:52.413447 3574 raft_consensus.cc:686] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [term 1 LEADER]: Becoming Leader. State: Replica: ce5cf4ab08d94c0ba8b638f4c1d7740b, State: Running, Role: LEADER
I20240502 01:48:52.414377 3574 consensus_queue.cc:237] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5cf4ab08d94c0ba8b638f4c1d7740b" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 45237 } }
I20240502 01:48:52.434559 3071 catalog_manager.cc:5484] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b reported cstate change: term changed from 0 to 1, leader changed from <none> to ce5cf4ab08d94c0ba8b638f4c1d7740b (127.1.233.129). New cstate: current_term: 1 leader_uuid: "ce5cf4ab08d94c0ba8b638f4c1d7740b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5cf4ab08d94c0ba8b638f4c1d7740b" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 45237 } health_report { overall_health: HEALTHY } } }
I20240502 01:48:52.479894 1958 master.cc:537] Master@127.1.233.190:41845 shutting down...
I20240502 01:48:52.494792 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:52.495708 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:52.496083 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P 31819667ca494a8fafeb6ab2d554a41b: stopping tablet replica
I20240502 01:48:52.516697 1958 master.cc:552] Master@127.1.233.190:41845 shutdown complete.
W20240502 01:48:53.421415 3529 heartbeater.cc:644] Failed to heartbeat to 127.1.233.190:41845 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.233.190:41845: connect: Connection refused (error 111)
I20240502 01:48:53.949090 3587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31819667ca494a8fafeb6ab2d554a41b)
I20240502 01:48:53.949468 3588 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31819667ca494a8fafeb6ab2d554a41b)
I20240502 01:48:53.949831 3587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:53.949996 3588 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:53.951857 3588 leader_election.cc:288] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845), 88a38fcae4944328befe752468c6892b (127.1.233.189:34797)
I20240502 01:48:53.952026 3587 leader_election.cc:288] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845), c8af6c0d37394254b320e8f12d045e58 (127.1.233.188:42235)
I20240502 01:48:53.955251 3148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8af6c0d37394254b320e8f12d045e58" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "88a38fcae4944328befe752468c6892b" is_pre_election: true
I20240502 01:48:53.955428 3216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88a38fcae4944328befe752468c6892b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c8af6c0d37394254b320e8f12d045e58" is_pre_election: true
I20240502 01:48:53.956033 3148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8af6c0d37394254b320e8f12d045e58 in term 1.
I20240502 01:48:53.956377 3216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a38fcae4944328befe752468c6892b in term 1.
I20240502 01:48:53.958977 3125 leader_election.cc:302] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a38fcae4944328befe752468c6892b, c8af6c0d37394254b320e8f12d045e58; no voters:
I20240502 01:48:53.959226 3192 leader_election.cc:302] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a38fcae4944328befe752468c6892b, c8af6c0d37394254b320e8f12d045e58; no voters:
I20240502 01:48:53.959836 3587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:48:53.960269 3587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 31819667ca494a8fafeb6ab2d554a41b)
I20240502 01:48:53.960660 3587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:48:53.960439 3588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:48:53.961139 3588 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 31819667ca494a8fafeb6ab2d554a41b)
W20240502 01:48:53.961436 3125 leader_election.cc:334] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845): Network error: Client connection negotiation failed: client connection to 127.1.233.190:41845: connect: Connection refused (error 111)
I20240502 01:48:53.961699 3588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:48:53.965950 3192 leader_election.cc:334] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845): Network error: Client connection negotiation failed: client connection to 127.1.233.190:41845: connect: Connection refused (error 111)
I20240502 01:48:53.966557 3587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:53.968264 3587 leader_election.cc:288] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [CANDIDATE]: Term 2 election: Requested vote from peers 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845), c8af6c0d37394254b320e8f12d045e58 (127.1.233.188:42235)
I20240502 01:48:53.968436 3588 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:53.970733 3588 leader_election.cc:288] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [CANDIDATE]: Term 2 election: Requested vote from peers 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845), 88a38fcae4944328befe752468c6892b (127.1.233.189:34797)
I20240502 01:48:53.970697 3216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88a38fcae4944328befe752468c6892b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c8af6c0d37394254b320e8f12d045e58"
I20240502 01:48:53.972286 3216 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a38fcae4944328befe752468c6892b in current term 2: Already voted for candidate c8af6c0d37394254b320e8f12d045e58 in this term.
I20240502 01:48:53.972121 3148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8af6c0d37394254b320e8f12d045e58" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "88a38fcae4944328befe752468c6892b"
I20240502 01:48:53.972987 3148 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c8af6c0d37394254b320e8f12d045e58 in current term 2: Already voted for candidate 88a38fcae4944328befe752468c6892b in this term.
W20240502 01:48:53.973153 3125 leader_election.cc:334] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845): Network error: Client connection negotiation failed: client connection to 127.1.233.190:41845: connect: Connection refused (error 111)
I20240502 01:48:53.974046 3125 leader_election.cc:302] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a38fcae4944328befe752468c6892b; no voters: 31819667ca494a8fafeb6ab2d554a41b, c8af6c0d37394254b320e8f12d045e58
I20240502 01:48:53.974848 3587 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240502 01:48:53.975028 3192 leader_election.cc:334] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845): Network error: Client connection negotiation failed: client connection to 127.1.233.190:41845: connect: Connection refused (error 111)
I20240502 01:48:53.975507 3192 leader_election.cc:302] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8af6c0d37394254b320e8f12d045e58; no voters: 31819667ca494a8fafeb6ab2d554a41b, 88a38fcae4944328befe752468c6892b
I20240502 01:48:53.976153 3588 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 01:48:55.521626 3600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:48:55.522343 3600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:55.524202 3600 leader_election.cc:288] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845), c8af6c0d37394254b320e8f12d045e58 (127.1.233.188:42235)
W20240502 01:48:55.526921 3125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.233.190:41845: connect: Connection refused (error 111) [suppressed 16 similar messages]
I20240502 01:48:55.527324 3216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88a38fcae4944328befe752468c6892b" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c8af6c0d37394254b320e8f12d045e58" is_pre_election: true
I20240502 01:48:55.528323 3216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a38fcae4944328befe752468c6892b in term 2.
I20240502 01:48:55.529688 3125 leader_election.cc:302] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a38fcae4944328befe752468c6892b, c8af6c0d37394254b320e8f12d045e58; no voters:
I20240502 01:48:55.530294 3600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 01:48:55.530594 3600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:48:55.530851 3600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 2 FOLLOWER]: Advancing to term 3
W20240502 01:48:55.531553 3125 leader_election.cc:334] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845): Network error: Client connection negotiation failed: client connection to 127.1.233.190:41845: connect: Connection refused (error 111)
I20240502 01:48:55.537102 3600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:55.538619 3600 leader_election.cc:288] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [CANDIDATE]: Term 3 election: Requested vote from peers 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845), c8af6c0d37394254b320e8f12d045e58 (127.1.233.188:42235)
I20240502 01:48:55.540042 3216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88a38fcae4944328befe752468c6892b" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c8af6c0d37394254b320e8f12d045e58"
I20240502 01:48:55.540591 3216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 2 FOLLOWER]: Advancing to term 3
W20240502 01:48:55.541811 3125 leader_election.cc:334] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845): Network error: Client connection negotiation failed: client connection to 127.1.233.190:41845: connect: Connection refused (error 111)
I20240502 01:48:55.546309 3216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a38fcae4944328befe752468c6892b in term 3.
I20240502 01:48:55.547326 3125 leader_election.cc:302] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a38fcae4944328befe752468c6892b, c8af6c0d37394254b320e8f12d045e58; no voters: 31819667ca494a8fafeb6ab2d554a41b
I20240502 01:48:55.547978 3600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:48:55.549101 3600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 3 LEADER]: Becoming Leader. State: Replica: 88a38fcae4944328befe752468c6892b, State: Running, Role: LEADER
I20240502 01:48:55.549917 3600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } }
I20240502 01:48:55.556048 3603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88a38fcae4944328befe752468c6892b. Latest consensus state: current_term: 3 leader_uuid: "88a38fcae4944328befe752468c6892b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:55.556844 3603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:55.558342 3605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:48:55.564927 3605 catalog_manager.cc:669] Loaded metadata for table test-workload [id=4f07ac7e3c7744c1ac6bbde7cb5486a2]
I20240502 01:48:55.569898 3605 tablet_loader.cc:96] loaded metadata for tablet 0614d8cbba3f4765b9c76332d3e92367 (table test-workload [id=4f07ac7e3c7744c1ac6bbde7cb5486a2])
I20240502 01:48:55.570945 3605 tablet_loader.cc:96] loaded metadata for tablet 1c91f41480604b1095d4978c48bcc269 (table test-workload [id=4f07ac7e3c7744c1ac6bbde7cb5486a2])
I20240502 01:48:55.571869 3605 tablet_loader.cc:96] loaded metadata for tablet 83c27b3e4bae4f3fa3d3c4f99901f7e1 (table test-workload [id=4f07ac7e3c7744c1ac6bbde7cb5486a2])
I20240502 01:48:55.572891 3605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:48:55.575942 3605 catalog_manager.cc:1259] Loaded cluster ID: 46dfda230726444e995f230c7dab8e9b
I20240502 01:48:55.576191 3605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:48:55.579741 3605 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:48:55.582726 3605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b: Loaded TSK: 0
I20240502 01:48:55.583706 3605 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240502 01:48:56.012180 3125 consensus_peers.cc:485] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b -> Peer 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845): Couldn't send request to peer 31819667ca494a8fafeb6ab2d554a41b. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.190:41845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:48:56.069026 3216 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 3 FOLLOWER]: Refusing update from remote peer 88a38fcae4944328befe752468c6892b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 01:48:56.070477 3603 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:48:56.075574 3606 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88a38fcae4944328befe752468c6892b. Latest consensus state: current_term: 3 leader_uuid: "88a38fcae4944328befe752468c6892b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:56.076282 3606 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:56.080016 3608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "88a38fcae4944328befe752468c6892b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:56.081058 3608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [sys.catalog]: This master's current role is: LEADER
I20240502 01:48:56.081787 3606 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "88a38fcae4944328befe752468c6892b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31819667ca494a8fafeb6ab2d554a41b" member_type: VOTER last_known_addr { host: "127.1.233.190" port: 41845 } } peers { permanent_uuid: "88a38fcae4944328befe752468c6892b" member_type: VOTER last_known_addr { host: "127.1.233.189" port: 34797 } } peers { permanent_uuid: "c8af6c0d37394254b320e8f12d045e58" member_type: VOTER last_known_addr { host: "127.1.233.188" port: 42235 } } }
I20240502 01:48:56.082441 3606 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 01:48:56.118094 3283 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:56.118480 3283 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:48:56.183648 1958 tablet_server.cc:178] TabletServer@127.1.233.129:0 shutting down...
I20240502 01:48:56.514848 3528 heartbeater.cc:503] Master 127.1.233.189:34797 was elected leader, sending a full tablet report...
I20240502 01:48:56.523741 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:48:56.525166 1958 tablet_replica.cc:330] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b: stopping tablet replica
I20240502 01:48:56.525969 1958 raft_consensus.cc:2229] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:56.526710 1958 raft_consensus.cc:2258] T 0614d8cbba3f4765b9c76332d3e92367 P ce5cf4ab08d94c0ba8b638f4c1d7740b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:56.548763 1958 tablet_server.cc:194] TabletServer@127.1.233.129:0 shutdown complete.
I20240502 01:48:56.560659 1958 tablet_server.cc:178] TabletServer@127.1.233.130:0 shutting down...
I20240502 01:48:56.583822 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:48:56.584511 1958 tablet_replica.cc:330] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05: stopping tablet replica
I20240502 01:48:56.585067 1958 raft_consensus.cc:2229] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:56.585582 1958 raft_consensus.cc:2258] T 1c91f41480604b1095d4978c48bcc269 P 69c856ecd7204c24addb3c8c0bc9df05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:48:56.605608 1958 tablet_server.cc:194] TabletServer@127.1.233.130:0 shutdown complete.
I20240502 01:48:56.615747 1958 tablet_server.cc:178] TabletServer@127.1.233.131:0 shutting down...
I20240502 01:48:56.635547 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:48:56.636242 1958 tablet_replica.cc:330] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23: stopping tablet replica
I20240502 01:48:56.636809 1958 raft_consensus.cc:2229] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:48:56.637262 1958 raft_consensus.cc:2258] T 83c27b3e4bae4f3fa3d3c4f99901f7e1 P 5c3b3cbbd0334c70af8fe9b2079c9e23 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:48:57.461541 3523 debug-util.cc:398] Leaking SignalData structure 0x7b080009fd20 after lost signal to thread 1961
W20240502 01:48:57.462462 3523 debug-util.cc:398] Leaking SignalData structure 0x7b08000a7440 after lost signal to thread 3173
W20240502 01:48:57.463217 3523 debug-util.cc:398] Leaking SignalData structure 0x7b08000a7b00 after lost signal to thread 3241
W20240502 01:48:57.654278 1958 thread.cc:536] Waited for 1000ms trying to join with diag-logger (tid 3523)
I20240502 01:48:58.119360 3283 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:48:58.119704 3283 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 01:48:58.654855 1958 thread.cc:536] Waited for 2000ms trying to join with diag-logger (tid 3523)
W20240502 01:48:58.700109 3125 consensus_peers.cc:485] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b -> Peer 31819667ca494a8fafeb6ab2d554a41b (127.1.233.190:41845): Couldn't send request to peer 31819667ca494a8fafeb6ab2d554a41b. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.190:41845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:48:59.655437 1958 thread.cc:536] Waited for 3000ms trying to join with diag-logger (tid 3523)
I20240502 01:49:00.120849 3283 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:00.121273 3283 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:00.469576 1958 tablet_server.cc:194] TabletServer@127.1.233.131:0 shutdown complete.
I20240502 01:49:00.480113 1958 master.cc:537] Master@127.1.233.189:34797 shutting down...
I20240502 01:49:00.499076 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:49:00.500190 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:00.500536 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P 88a38fcae4944328befe752468c6892b: stopping tablet replica
I20240502 01:49:00.519563 1958 master.cc:552] Master@127.1.233.189:34797 shutdown complete.
I20240502 01:49:00.535656 1958 master.cc:537] Master@127.1.233.188:42235 shutting down...
I20240502 01:49:00.554018 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:00.554648 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:00.554984 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P c8af6c0d37394254b320e8f12d045e58: stopping tablet replica
I20240502 01:49:00.573133 1958 master.cc:552] Master@127.1.233.188:42235 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (10923 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240502 01:49:00.605825 1958 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.233.190:42291
I20240502 01:49:00.606899 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:00.613189 3622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:00.613538 3623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:00.613976 3625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:00.614696 1958 server_base.cc:1034] running on GCE node
I20240502 01:49:00.615502 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:00.615703 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:00.615849 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614540615831 us; error 0 us; skew 500 ppm
I20240502 01:49:00.616400 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:00.623148 1958 webserver.cc:460] Webserver started at http://127.1.233.190:37963/ using document root <none> and password file <none>
I20240502 01:49:00.623664 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:00.623867 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:00.624194 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:00.625522 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/master-0-root/instance:
uuid: "10d93877c88c4078893a8ec36fd17020"
format_stamp: "Formatted at 2024-05-02 01:49:00 on dist-test-slave-2090"
I20240502 01:49:00.629979 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:49:00.633330 3630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:00.634214 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240502 01:49:00.634456 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/master-0-root
uuid: "10d93877c88c4078893a8ec36fd17020"
format_stamp: "Formatted at 2024-05-02 01:49:00 on dist-test-slave-2090"
I20240502 01:49:00.634699 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:00.646872 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:00.648206 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:00.688699 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.190:42291
I20240502 01:49:00.688792 3681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.190:42291 every 8 connection(s)
I20240502 01:49:00.692535 3682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:00.702826 3682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020: Bootstrap starting.
I20240502 01:49:00.707192 3682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:00.711597 3682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020: No bootstrap required, opened a new log
I20240502 01:49:00.713665 3682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d93877c88c4078893a8ec36fd17020" member_type: VOTER }
I20240502 01:49:00.714080 3682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:00.714327 3682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d93877c88c4078893a8ec36fd17020, State: Initialized, Role: FOLLOWER
I20240502 01:49:00.714867 3682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d93877c88c4078893a8ec36fd17020" member_type: VOTER }
I20240502 01:49:00.715315 3682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:49:00.715528 3682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:49:00.715833 3682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:00.720309 3682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d93877c88c4078893a8ec36fd17020" member_type: VOTER }
I20240502 01:49:00.720860 3682 leader_election.cc:302] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10d93877c88c4078893a8ec36fd17020; no voters:
I20240502 01:49:00.722048 3682 leader_election.cc:288] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:49:00.722455 3685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:00.723822 3685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [term 1 LEADER]: Becoming Leader. State: Replica: 10d93877c88c4078893a8ec36fd17020, State: Running, Role: LEADER
I20240502 01:49:00.724524 3685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d93877c88c4078893a8ec36fd17020" member_type: VOTER }
I20240502 01:49:00.725035 3682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:49:00.728885 3686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10d93877c88c4078893a8ec36fd17020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d93877c88c4078893a8ec36fd17020" member_type: VOTER } }
I20240502 01:49:00.728855 3687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10d93877c88c4078893a8ec36fd17020. Latest consensus state: current_term: 1 leader_uuid: "10d93877c88c4078893a8ec36fd17020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10d93877c88c4078893a8ec36fd17020" member_type: VOTER } }
I20240502 01:49:00.729681 3686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [sys.catalog]: This master's current role is: LEADER
I20240502 01:49:00.729696 3687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [sys.catalog]: This master's current role is: LEADER
I20240502 01:49:00.734632 3692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:49:00.739848 3692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:49:00.742499 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:49:00.748610 3692 catalog_manager.cc:1347] Generated new cluster ID: 0516f0c8cf794be7b2ba65f415421987
I20240502 01:49:00.748907 3692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:49:00.776849 3692 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:49:00.778205 3692 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:49:00.794839 3692 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020: Generated new TSK 0
I20240502 01:49:00.795475 3692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:49:00.809631 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:00.816095 3703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:00.816993 3704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:00.818410 3706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:00.819154 1958 server_base.cc:1034] running on GCE node
I20240502 01:49:00.819885 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:00.820082 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:00.820266 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614540820246 us; error 0 us; skew 500 ppm
I20240502 01:49:00.820793 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:00.823190 1958 webserver.cc:460] Webserver started at http://127.1.233.129:37443/ using document root <none> and password file <none>
I20240502 01:49:00.823669 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:00.823848 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:00.824110 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:00.825286 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-0-root/instance:
uuid: "629a1ca39c064c9b86201e265fd29bc4"
format_stamp: "Formatted at 2024-05-02 01:49:00 on dist-test-slave-2090"
I20240502 01:49:00.829798 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:49:00.832971 3711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:00.833680 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:49:00.833953 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "629a1ca39c064c9b86201e265fd29bc4"
format_stamp: "Formatted at 2024-05-02 01:49:00 on dist-test-slave-2090"
I20240502 01:49:00.834264 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:00.850785 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:00.851989 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:00.853504 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:00.855885 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:00.856088 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:00.856334 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:00.856499 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:00.896514 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:44953
I20240502 01:49:00.896607 3775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:44953 every 8 connection(s)
I20240502 01:49:00.901093 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:00.908926 3780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:00.910954 3781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:00.911636 3783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:00.914433 1958 server_base.cc:1034] running on GCE node
I20240502 01:49:00.916129 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:00.916464 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:00.916610 3776 heartbeater.cc:347] Connected to a master server at 127.1.233.190:42291
I20240502 01:49:00.916635 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614540916621 us; error 0 us; skew 500 ppm
I20240502 01:49:00.917122 3776 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:00.917454 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:00.917884 3776 heartbeater.cc:511] Master 127.1.233.190:42291 requested a full tablet report, sending...
I20240502 01:49:00.919752 3647 ts_manager.cc:194] Registered new tserver with Master: 629a1ca39c064c9b86201e265fd29bc4 (127.1.233.129:44953)
I20240502 01:49:00.920446 1958 webserver.cc:460] Webserver started at http://127.1.233.130:39997/ using document root <none> and password file <none>
I20240502 01:49:00.920933 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:00.921110 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:00.921371 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:00.921674 3647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43848
I20240502 01:49:00.922554 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-1-root/instance:
uuid: "2d7f66c3625b4c36b496a42302cab30e"
format_stamp: "Formatted at 2024-05-02 01:49:00 on dist-test-slave-2090"
I20240502 01:49:00.927399 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 01:49:00.930570 3788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:00.931375 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:49:00.931751 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-1-root
uuid: "2d7f66c3625b4c36b496a42302cab30e"
format_stamp: "Formatted at 2024-05-02 01:49:00 on dist-test-slave-2090"
I20240502 01:49:00.932061 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:00.940614 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:00.941787 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:00.943163 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:00.945482 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:00.945683 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:00.945897 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:00.946049 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:00.984493 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.130:43905
I20240502 01:49:00.984614 3851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.130:43905 every 8 connection(s)
I20240502 01:49:00.989276 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:00.997181 3855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:00.999135 3856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:01.001642 1958 server_base.cc:1034] running on GCE node
I20240502 01:49:01.001873 3852 heartbeater.cc:347] Connected to a master server at 127.1.233.190:42291
I20240502 01:49:01.002310 3852 heartbeater.cc:464] Registering TS with master...
W20240502 01:49:01.002331 3858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:01.003334 3852 heartbeater.cc:511] Master 127.1.233.190:42291 requested a full tablet report, sending...
I20240502 01:49:01.003432 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:01.003753 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:01.003965 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614541003948 us; error 0 us; skew 500 ppm
I20240502 01:49:01.004648 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:01.005532 3647 ts_manager.cc:194] Registered new tserver with Master: 2d7f66c3625b4c36b496a42302cab30e (127.1.233.130:43905)
I20240502 01:49:01.007054 3647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43856
I20240502 01:49:01.007526 1958 webserver.cc:460] Webserver started at http://127.1.233.131:33407/ using document root <none> and password file <none>
I20240502 01:49:01.008023 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:01.008307 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:01.008584 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:01.009739 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-2-root/instance:
uuid: "9e62f8941cf74cafb0e2185af5868dbb"
format_stamp: "Formatted at 2024-05-02 01:49:01 on dist-test-slave-2090"
I20240502 01:49:01.014312 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:49:01.017521 3863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:01.018288 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:49:01.018584 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-2-root
uuid: "9e62f8941cf74cafb0e2185af5868dbb"
format_stamp: "Formatted at 2024-05-02 01:49:01 on dist-test-slave-2090"
I20240502 01:49:01.018867 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:01.029611 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:01.030718 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:01.032227 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:01.034555 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:01.034763 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:01.035003 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:01.035184 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:01.074399 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.131:37913
I20240502 01:49:01.074488 3926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.131:37913 every 8 connection(s)
I20240502 01:49:01.087970 3927 heartbeater.cc:347] Connected to a master server at 127.1.233.190:42291
I20240502 01:49:01.088405 3927 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:01.089133 3927 heartbeater.cc:511] Master 127.1.233.190:42291 requested a full tablet report, sending...
I20240502 01:49:01.090930 3647 ts_manager.cc:194] Registered new tserver with Master: 9e62f8941cf74cafb0e2185af5868dbb (127.1.233.131:37913)
I20240502 01:49:01.091809 1958 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014406255s
I20240502 01:49:01.092190 3647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43866
I20240502 01:49:01.924691 3776 heartbeater.cc:503] Master 127.1.233.190:42291 was elected leader, sending a full tablet report...
I20240502 01:49:02.009604 3852 heartbeater.cc:503] Master 127.1.233.190:42291 was elected leader, sending a full tablet report...
I20240502 01:49:02.094626 3927 heartbeater.cc:503] Master 127.1.233.190:42291 was elected leader, sending a full tablet report...
I20240502 01:49:02.125358 1958 test_util.cc:268] Using random seed: -826947310
I20240502 01:49:02.147557 3647 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43868:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:49:02.149617 3647 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:49:02.202314 3816 tablet_service.cc:1467] Processing CreateTablet for tablet 9119e2d663834f5182f41406f021b349 (DEFAULT_TABLE table=test-workload [id=2da16a36bae7473dbe2132590b44c52a]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 01:49:02.202591 3815 tablet_service.cc:1467] Processing CreateTablet for tablet dd639b5a3260467aade1d19d4e99b1d2 (DEFAULT_TABLE table=test-workload [id=2da16a36bae7473dbe2132590b44c52a]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 01:49:02.204010 3816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9119e2d663834f5182f41406f021b349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:02.205516 3815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd639b5a3260467aade1d19d4e99b1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:02.207677 3891 tablet_service.cc:1467] Processing CreateTablet for tablet 9119e2d663834f5182f41406f021b349 (DEFAULT_TABLE table=test-workload [id=2da16a36bae7473dbe2132590b44c52a]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 01:49:02.209092 3891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9119e2d663834f5182f41406f021b349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:02.213165 3739 tablet_service.cc:1467] Processing CreateTablet for tablet 9119e2d663834f5182f41406f021b349 (DEFAULT_TABLE table=test-workload [id=2da16a36bae7473dbe2132590b44c52a]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 01:49:02.213307 3890 tablet_service.cc:1467] Processing CreateTablet for tablet dd639b5a3260467aade1d19d4e99b1d2 (DEFAULT_TABLE table=test-workload [id=2da16a36bae7473dbe2132590b44c52a]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 01:49:02.214532 3739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9119e2d663834f5182f41406f021b349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:02.214663 3890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd639b5a3260467aade1d19d4e99b1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:02.226440 3738 tablet_service.cc:1467] Processing CreateTablet for tablet dd639b5a3260467aade1d19d4e99b1d2 (DEFAULT_TABLE table=test-workload [id=2da16a36bae7473dbe2132590b44c52a]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 01:49:02.228679 3738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd639b5a3260467aade1d19d4e99b1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:02.238438 3947 tablet_bootstrap.cc:492] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e: Bootstrap starting.
I20240502 01:49:02.246623 3948 tablet_bootstrap.cc:492] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb: Bootstrap starting.
I20240502 01:49:02.248275 3949 tablet_bootstrap.cc:492] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4: Bootstrap starting.
I20240502 01:49:02.249150 3947 tablet_bootstrap.cc:654] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:02.251699 3948 tablet_bootstrap.cc:654] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:02.254575 3949 tablet_bootstrap.cc:654] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:02.255288 3947 tablet_bootstrap.cc:492] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e: No bootstrap required, opened a new log
I20240502 01:49:02.255841 3947 ts_tablet_manager.cc:1367] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.002s
I20240502 01:49:02.256762 3948 tablet_bootstrap.cc:492] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb: No bootstrap required, opened a new log
I20240502 01:49:02.257241 3948 ts_tablet_manager.cc:1367] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240502 01:49:02.260383 3948 raft_consensus.cc:348] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.261005 3949 tablet_bootstrap.cc:492] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4: No bootstrap required, opened a new log
I20240502 01:49:02.260807 3947 raft_consensus.cc:348] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.261211 3948 raft_consensus.cc:374] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:02.261518 3949 ts_tablet_manager.cc:1367] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240502 01:49:02.261586 3947 raft_consensus.cc:374] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:02.261639 3948 raft_consensus.cc:729] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e62f8941cf74cafb0e2185af5868dbb, State: Initialized, Role: FOLLOWER
I20240502 01:49:02.262142 3947 raft_consensus.cc:729] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7f66c3625b4c36b496a42302cab30e, State: Initialized, Role: FOLLOWER
I20240502 01:49:02.262773 3947 consensus_queue.cc:260] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.262756 3948 consensus_queue.cc:260] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.264413 3949 raft_consensus.cc:348] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.265522 3949 raft_consensus.cc:374] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:02.265851 3947 ts_tablet_manager.cc:1398] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e: Time spent starting tablet: real 0.010s user 0.004s sys 0.005s
I20240502 01:49:02.265910 3949 raft_consensus.cc:729] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 629a1ca39c064c9b86201e265fd29bc4, State: Initialized, Role: FOLLOWER
I20240502 01:49:02.266958 3947 tablet_bootstrap.cc:492] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e: Bootstrap starting.
I20240502 01:49:02.266978 3949 consensus_queue.cc:260] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.273537 3947 tablet_bootstrap.cc:654] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:02.276118 3948 ts_tablet_manager.cc:1398] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb: Time spent starting tablet: real 0.019s user 0.004s sys 0.012s
I20240502 01:49:02.279026 3948 tablet_bootstrap.cc:492] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb: Bootstrap starting.
I20240502 01:49:02.280108 3949 ts_tablet_manager.cc:1398] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4: Time spent starting tablet: real 0.018s user 0.010s sys 0.004s
I20240502 01:49:02.284828 3948 tablet_bootstrap.cc:654] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:02.286681 3949 tablet_bootstrap.cc:492] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4: Bootstrap starting.
I20240502 01:49:02.286754 3947 tablet_bootstrap.cc:492] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e: No bootstrap required, opened a new log
I20240502 01:49:02.287632 3947 ts_tablet_manager.cc:1367] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.008s
I20240502 01:49:02.290493 3947 raft_consensus.cc:348] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.291324 3947 raft_consensus.cc:374] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:02.291678 3947 raft_consensus.cc:729] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7f66c3625b4c36b496a42302cab30e, State: Initialized, Role: FOLLOWER
I20240502 01:49:02.292536 3947 consensus_queue.cc:260] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.293565 3949 tablet_bootstrap.cc:654] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:02.294756 3947 ts_tablet_manager.cc:1398] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:49:02.299806 3948 tablet_bootstrap.cc:492] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb: No bootstrap required, opened a new log
I20240502 01:49:02.300498 3948 ts_tablet_manager.cc:1367] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.007s
I20240502 01:49:02.303889 3948 raft_consensus.cc:348] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.304852 3948 raft_consensus.cc:374] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:02.305089 3948 raft_consensus.cc:729] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e62f8941cf74cafb0e2185af5868dbb, State: Initialized, Role: FOLLOWER
I20240502 01:49:02.305734 3948 consensus_queue.cc:260] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.306583 3949 tablet_bootstrap.cc:492] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4: No bootstrap required, opened a new log
I20240502 01:49:02.307092 3949 ts_tablet_manager.cc:1367] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.010s
I20240502 01:49:02.307744 3948 ts_tablet_manager.cc:1398] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:49:02.309243 3949 raft_consensus.cc:348] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.309752 3949 raft_consensus.cc:374] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:02.309959 3949 raft_consensus.cc:729] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 629a1ca39c064c9b86201e265fd29bc4, State: Initialized, Role: FOLLOWER
I20240502 01:49:02.310503 3949 consensus_queue.cc:260] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.312413 3949 ts_tablet_manager.cc:1398] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 01:49:02.397284 3955 raft_consensus.cc:482] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:02.397728 3955 raft_consensus.cc:504] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.399535 3955 leader_election.cc:288] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d7f66c3625b4c36b496a42302cab30e (127.1.233.130:43905), 9e62f8941cf74cafb0e2185af5868dbb (127.1.233.131:37913)
I20240502 01:49:02.411198 3901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd639b5a3260467aade1d19d4e99b1d2" candidate_uuid: "629a1ca39c064c9b86201e265fd29bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e62f8941cf74cafb0e2185af5868dbb" is_pre_election: true
I20240502 01:49:02.411813 3826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd639b5a3260467aade1d19d4e99b1d2" candidate_uuid: "629a1ca39c064c9b86201e265fd29bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7f66c3625b4c36b496a42302cab30e" is_pre_election: true
I20240502 01:49:02.412066 3901 raft_consensus.cc:2454] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 629a1ca39c064c9b86201e265fd29bc4 in term 0.
I20240502 01:49:02.412564 3826 raft_consensus.cc:2454] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 629a1ca39c064c9b86201e265fd29bc4 in term 0.
I20240502 01:49:02.413311 3714 leader_election.cc:302] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 629a1ca39c064c9b86201e265fd29bc4, 9e62f8941cf74cafb0e2185af5868dbb; no voters:
I20240502 01:49:02.414062 3955 raft_consensus.cc:2781] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:02.414366 3955 raft_consensus.cc:482] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:02.414599 3955 raft_consensus.cc:3037] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:02.419433 3955 raft_consensus.cc:504] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.421164 3955 leader_election.cc:288] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [CANDIDATE]: Term 1 election: Requested vote from peers 2d7f66c3625b4c36b496a42302cab30e (127.1.233.130:43905), 9e62f8941cf74cafb0e2185af5868dbb (127.1.233.131:37913)
I20240502 01:49:02.422029 3826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd639b5a3260467aade1d19d4e99b1d2" candidate_uuid: "629a1ca39c064c9b86201e265fd29bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7f66c3625b4c36b496a42302cab30e"
I20240502 01:49:02.422219 3901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd639b5a3260467aade1d19d4e99b1d2" candidate_uuid: "629a1ca39c064c9b86201e265fd29bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e62f8941cf74cafb0e2185af5868dbb"
I20240502 01:49:02.422524 3826 raft_consensus.cc:3037] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:02.422739 3901 raft_consensus.cc:3037] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:02.427681 3826 raft_consensus.cc:2454] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 629a1ca39c064c9b86201e265fd29bc4 in term 1.
I20240502 01:49:02.427884 3901 raft_consensus.cc:2454] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 629a1ca39c064c9b86201e265fd29bc4 in term 1.
I20240502 01:49:02.428987 3712 leader_election.cc:302] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d7f66c3625b4c36b496a42302cab30e, 629a1ca39c064c9b86201e265fd29bc4; no voters:
I20240502 01:49:02.429678 3955 raft_consensus.cc:2781] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:02.430676 3955 raft_consensus.cc:686] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [term 1 LEADER]: Becoming Leader. State: Replica: 629a1ca39c064c9b86201e265fd29bc4, State: Running, Role: LEADER
I20240502 01:49:02.431418 3955 consensus_queue.cc:237] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.439740 3647 catalog_manager.cc:5484] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 629a1ca39c064c9b86201e265fd29bc4 (127.1.233.129). New cstate: current_term: 1 leader_uuid: "629a1ca39c064c9b86201e265fd29bc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:02.515969 3953 raft_consensus.cc:482] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:02.516613 3953 raft_consensus.cc:504] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.518630 3953 leader_election.cc:288] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 629a1ca39c064c9b86201e265fd29bc4 (127.1.233.129:44953), 9e62f8941cf74cafb0e2185af5868dbb (127.1.233.131:37913)
I20240502 01:49:02.520203 3954 raft_consensus.cc:482] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:02.520728 3954 raft_consensus.cc:504] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.523494 3954 leader_election.cc:288] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d7f66c3625b4c36b496a42302cab30e (127.1.233.130:43905), 629a1ca39c064c9b86201e265fd29bc4 (127.1.233.129:44953)
I20240502 01:49:02.534663 3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9119e2d663834f5182f41406f021b349" candidate_uuid: "2d7f66c3625b4c36b496a42302cab30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "629a1ca39c064c9b86201e265fd29bc4" is_pre_election: true
I20240502 01:49:02.535581 3750 raft_consensus.cc:2454] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d7f66c3625b4c36b496a42302cab30e in term 0.
I20240502 01:49:02.536481 3901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9119e2d663834f5182f41406f021b349" candidate_uuid: "2d7f66c3625b4c36b496a42302cab30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e62f8941cf74cafb0e2185af5868dbb" is_pre_election: true
I20240502 01:49:02.537220 3789 leader_election.cc:302] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d7f66c3625b4c36b496a42302cab30e, 629a1ca39c064c9b86201e265fd29bc4; no voters:
I20240502 01:49:02.537214 3901 raft_consensus.cc:2454] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d7f66c3625b4c36b496a42302cab30e in term 0.
I20240502 01:49:02.538230 3953 raft_consensus.cc:2781] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:02.538615 3953 raft_consensus.cc:482] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:02.538924 3953 raft_consensus.cc:3037] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:02.540628 3826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9119e2d663834f5182f41406f021b349" candidate_uuid: "9e62f8941cf74cafb0e2185af5868dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7f66c3625b4c36b496a42302cab30e" is_pre_election: true
I20240502 01:49:02.544457 3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9119e2d663834f5182f41406f021b349" candidate_uuid: "9e62f8941cf74cafb0e2185af5868dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "629a1ca39c064c9b86201e265fd29bc4" is_pre_election: true
I20240502 01:49:02.544988 3750 raft_consensus.cc:2454] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e62f8941cf74cafb0e2185af5868dbb in term 0.
I20240502 01:49:02.545454 3953 raft_consensus.cc:504] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.546012 3864 leader_election.cc:302] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 629a1ca39c064c9b86201e265fd29bc4, 9e62f8941cf74cafb0e2185af5868dbb; no voters:
I20240502 01:49:02.546658 3826 raft_consensus.cc:2379] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e62f8941cf74cafb0e2185af5868dbb in current term 1: Already voted for candidate 2d7f66c3625b4c36b496a42302cab30e in this term.
I20240502 01:49:02.546864 3954 raft_consensus.cc:2781] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:02.547252 3954 raft_consensus.cc:482] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:02.547557 3954 raft_consensus.cc:3037] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:02.548578 3901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9119e2d663834f5182f41406f021b349" candidate_uuid: "2d7f66c3625b4c36b496a42302cab30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e62f8941cf74cafb0e2185af5868dbb"
I20240502 01:49:02.548566 3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9119e2d663834f5182f41406f021b349" candidate_uuid: "2d7f66c3625b4c36b496a42302cab30e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "629a1ca39c064c9b86201e265fd29bc4"
I20240502 01:49:02.549556 3750 raft_consensus.cc:3037] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:02.552896 3954 raft_consensus.cc:504] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.553496 3953 leader_election.cc:288] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [CANDIDATE]: Term 1 election: Requested vote from peers 629a1ca39c064c9b86201e265fd29bc4 (127.1.233.129:44953), 9e62f8941cf74cafb0e2185af5868dbb (127.1.233.131:37913)
I20240502 01:49:02.553939 3901 raft_consensus.cc:2379] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2d7f66c3625b4c36b496a42302cab30e in current term 1: Already voted for candidate 9e62f8941cf74cafb0e2185af5868dbb in this term.
I20240502 01:49:02.554630 3750 raft_consensus.cc:2454] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d7f66c3625b4c36b496a42302cab30e in term 1.
I20240502 01:49:02.554704 3954 leader_election.cc:288] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [CANDIDATE]: Term 1 election: Requested vote from peers 2d7f66c3625b4c36b496a42302cab30e (127.1.233.130:43905), 629a1ca39c064c9b86201e265fd29bc4 (127.1.233.129:44953)
I20240502 01:49:02.555773 3750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9119e2d663834f5182f41406f021b349" candidate_uuid: "9e62f8941cf74cafb0e2185af5868dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "629a1ca39c064c9b86201e265fd29bc4"
I20240502 01:49:02.555994 3789 leader_election.cc:302] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d7f66c3625b4c36b496a42302cab30e, 629a1ca39c064c9b86201e265fd29bc4; no voters: 9e62f8941cf74cafb0e2185af5868dbb
I20240502 01:49:02.556190 3826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9119e2d663834f5182f41406f021b349" candidate_uuid: "9e62f8941cf74cafb0e2185af5868dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7f66c3625b4c36b496a42302cab30e"
I20240502 01:49:02.556582 3750 raft_consensus.cc:2379] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e62f8941cf74cafb0e2185af5868dbb in current term 1: Already voted for candidate 2d7f66c3625b4c36b496a42302cab30e in this term.
I20240502 01:49:02.556955 3826 raft_consensus.cc:2379] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e62f8941cf74cafb0e2185af5868dbb in current term 1: Already voted for candidate 2d7f66c3625b4c36b496a42302cab30e in this term.
I20240502 01:49:02.557334 3953 raft_consensus.cc:2781] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:02.558182 3864 leader_election.cc:302] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e62f8941cf74cafb0e2185af5868dbb; no voters: 2d7f66c3625b4c36b496a42302cab30e, 629a1ca39c064c9b86201e265fd29bc4
I20240502 01:49:02.558560 3953 raft_consensus.cc:686] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 1 LEADER]: Becoming Leader. State: Replica: 2d7f66c3625b4c36b496a42302cab30e, State: Running, Role: LEADER
I20240502 01:49:02.559379 3954 raft_consensus.cc:2726] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:49:02.559415 3953 consensus_queue.cc:237] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } }
I20240502 01:49:02.566579 3647 catalog_manager.cc:5484] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d7f66c3625b4c36b496a42302cab30e (127.1.233.130). New cstate: current_term: 1 leader_uuid: "2d7f66c3625b4c36b496a42302cab30e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:02.742300 3700 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:02.744361 3700 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:02.744732 3700 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:02.849939 3955 consensus_queue.cc:1035] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:02.862753 3972 consensus_queue.cc:1035] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:49:03.042977 3953 consensus_queue.cc:1035] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:03.053701 3968 consensus_queue.cc:1035] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:04.745622 3700 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:04.747925 3700 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:04.748489 3700 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:06.749707 3700 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:06.751724 3700 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:06.752082 3700 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:08.753085 3700 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:08.754988 3700 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:08.755298 3700 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:10.756276 3700 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:10.758220 3700 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:10.758535 3700 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:12.759614 3700 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:12.761742 3700 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:12.762053 3700 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:13.727922 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:13.736187 3984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:13.736693 3985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:13.739739 3987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:13.740746 1958 server_base.cc:1034] running on GCE node
I20240502 01:49:13.741920 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:13.742149 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:13.742362 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614553742343 us; error 0 us; skew 500 ppm
I20240502 01:49:13.742929 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:13.751843 1958 webserver.cc:460] Webserver started at http://127.1.233.132:41107/ using document root <none> and password file <none>
I20240502 01:49:13.752444 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:13.752647 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:13.752962 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:13.754557 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-3-root/instance:
uuid: "abcff7e8718d4544b0b26ecc64a6f2f0"
format_stamp: "Formatted at 2024-05-02 01:49:13 on dist-test-slave-2090"
I20240502 01:49:13.759500 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:49:13.763080 3992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:13.763897 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:49:13.764247 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-3-root
uuid: "abcff7e8718d4544b0b26ecc64a6f2f0"
format_stamp: "Formatted at 2024-05-02 01:49:13 on dist-test-slave-2090"
I20240502 01:49:13.764537 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:13.787145 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:13.788807 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:13.790676 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:13.793556 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:13.793761 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:13.794046 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:13.794207 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:13.837683 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.132:35465
I20240502 01:49:13.837792 4055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.132:35465 every 8 connection(s)
I20240502 01:49:13.853323 4056 heartbeater.cc:347] Connected to a master server at 127.1.233.190:42291
I20240502 01:49:13.853696 4056 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:13.854452 4056 heartbeater.cc:511] Master 127.1.233.190:42291 requested a full tablet report, sending...
I20240502 01:49:13.856314 3647 ts_manager.cc:194] Registered new tserver with Master: abcff7e8718d4544b0b26ecc64a6f2f0 (127.1.233.132:35465)
I20240502 01:49:13.857636 3647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52920
I20240502 01:49:14.762946 3700 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:14.764977 3700 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:14.765429 3700 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:14.809566 3826 consensus_queue.cc:237] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: true } }
I20240502 01:49:14.817354 3901 raft_consensus.cc:1261] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Refusing update from remote peer 2d7f66c3625b4c36b496a42302cab30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:49:14.819080 3750 raft_consensus.cc:1261] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 1 FOLLOWER]: Refusing update from remote peer 2d7f66c3625b4c36b496a42302cab30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:49:14.819561 4063 consensus_queue.cc:1035] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:49:14.820837 3983 consensus_queue.cc:1035] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240502 01:49:14.831254 3790 consensus_peers.cc:485] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e -> Peer abcff7e8718d4544b0b26ecc64a6f2f0 (127.1.233.132:35465): Couldn't send request to peer abcff7e8718d4544b0b26ecc64a6f2f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9119e2d663834f5182f41406f021b349. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:49:14.831192 3983 raft_consensus.cc:2932] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER abcff7e8718d4544b0b26ecc64a6f2f0 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: true } } }
I20240502 01:49:14.833132 3750 raft_consensus.cc:2932] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER abcff7e8718d4544b0b26ecc64a6f2f0 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: true } } }
I20240502 01:49:14.836597 3901 raft_consensus.cc:2932] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER abcff7e8718d4544b0b26ecc64a6f2f0 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: true } } }
I20240502 01:49:14.847687 3647 catalog_manager.cc:5484] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 reported cstate change: config changed from index -1 to 2, NON_VOTER abcff7e8718d4544b0b26ecc64a6f2f0 (127.1.233.132) added. New cstate: current_term: 1 leader_uuid: "2d7f66c3625b4c36b496a42302cab30e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: true } } }
I20240502 01:49:14.859968 4056 heartbeater.cc:503] Master 127.1.233.190:42291 was elected leader, sending a full tablet report...
I20240502 01:49:15.299010 4073 ts_tablet_manager.cc:902] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0: Initiating tablet copy from peer 2d7f66c3625b4c36b496a42302cab30e (127.1.233.130:43905)
I20240502 01:49:15.303543 4073 tablet_copy_client.cc:308] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0: tablet copy: Beginning tablet copy session from remote peer at address 127.1.233.130:43905
I20240502 01:49:15.315589 3836 tablet_copy_service.cc:140] P 2d7f66c3625b4c36b496a42302cab30e: Received BeginTabletCopySession request for tablet 9119e2d663834f5182f41406f021b349 from peer abcff7e8718d4544b0b26ecc64a6f2f0 ({username='slave'} at 127.0.0.1:49244)
I20240502 01:49:15.316181 3836 tablet_copy_service.cc:161] P 2d7f66c3625b4c36b496a42302cab30e: Beginning new tablet copy session on tablet 9119e2d663834f5182f41406f021b349 from peer abcff7e8718d4544b0b26ecc64a6f2f0 at {username='slave'} at 127.0.0.1:49244: session id = abcff7e8718d4544b0b26ecc64a6f2f0-9119e2d663834f5182f41406f021b349
I20240502 01:49:15.325459 3836 tablet_copy_source_session.cc:205] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:49:15.331084 4073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9119e2d663834f5182f41406f021b349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:15.346352 4073 tablet_copy_client.cc:783] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0: tablet copy: Starting download of 0 data blocks...
I20240502 01:49:15.346971 4073 tablet_copy_client.cc:647] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0: tablet copy: Starting download of 1 WAL segments...
I20240502 01:49:15.350445 4073 tablet_copy_client.cc:523] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:49:15.358368 4073 tablet_bootstrap.cc:492] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0: Bootstrap starting.
I20240502 01:49:15.375489 4073 tablet_bootstrap.cc:492] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:49:15.376376 4073 tablet_bootstrap.cc:492] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0: Bootstrap complete.
I20240502 01:49:15.377087 4073 ts_tablet_manager.cc:1367] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.004s
I20240502 01:49:15.379588 4073 raft_consensus.cc:348] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: true } }
I20240502 01:49:15.380491 4073 raft_consensus.cc:729] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: abcff7e8718d4544b0b26ecc64a6f2f0, State: Initialized, Role: LEARNER
I20240502 01:49:15.381145 4073 consensus_queue.cc:260] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: true } }
I20240502 01:49:15.383728 4073 ts_tablet_manager.cc:1398] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 01:49:15.385592 3836 tablet_copy_service.cc:342] P 2d7f66c3625b4c36b496a42302cab30e: Request end of tablet copy session abcff7e8718d4544b0b26ecc64a6f2f0-9119e2d663834f5182f41406f021b349 received from {username='slave'} at 127.0.0.1:49244
I20240502 01:49:15.386162 3836 tablet_copy_service.cc:433] P 2d7f66c3625b4c36b496a42302cab30e: ending tablet copy session abcff7e8718d4544b0b26ecc64a6f2f0-9119e2d663834f5182f41406f021b349 on tablet 9119e2d663834f5182f41406f021b349 with peer abcff7e8718d4544b0b26ecc64a6f2f0
I20240502 01:49:15.680644 4031 raft_consensus.cc:1203] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:49:16.273026 4082 raft_consensus.cc:1050] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e: attempting to promote NON_VOTER abcff7e8718d4544b0b26ecc64a6f2f0 to VOTER
I20240502 01:49:16.275549 4082 consensus_queue.cc:237] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: false } }
I20240502 01:49:16.282505 4031 raft_consensus.cc:1261] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0 [term 1 LEARNER]: Refusing update from remote peer 2d7f66c3625b4c36b496a42302cab30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:49:16.283708 3901 raft_consensus.cc:1261] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Refusing update from remote peer 2d7f66c3625b4c36b496a42302cab30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:49:16.285998 3750 raft_consensus.cc:1261] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 1 FOLLOWER]: Refusing update from remote peer 2d7f66c3625b4c36b496a42302cab30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:49:16.286056 4085 consensus_queue.cc:1035] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:49:16.287374 4083 consensus_queue.cc:1035] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:49:16.288412 4082 consensus_queue.cc:1035] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:49:16.301195 4083 raft_consensus.cc:2932] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, abcff7e8718d4544b0b26ecc64a6f2f0 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: false } } }
I20240502 01:49:16.303200 3901 raft_consensus.cc:2932] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, abcff7e8718d4544b0b26ecc64a6f2f0 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: false } } }
I20240502 01:49:16.310668 3750 raft_consensus.cc:2932] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, abcff7e8718d4544b0b26ecc64a6f2f0 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: false } } }
I20240502 01:49:16.312280 4031 raft_consensus.cc:2932] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, abcff7e8718d4544b0b26ecc64a6f2f0 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: false } } }
I20240502 01:49:16.319156 3646 catalog_manager.cc:5484] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb reported cstate change: config changed from index 2 to 3, abcff7e8718d4544b0b26ecc64a6f2f0 (127.1.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d7f66c3625b4c36b496a42302cab30e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "629a1ca39c064c9b86201e265fd29bc4" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 44953 } attrs { replace: true } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: false } } }
I20240502 01:49:16.351374 3825 consensus_queue.cc:237] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: false } }
I20240502 01:49:16.355774 4031 raft_consensus.cc:1261] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0 [term 1 FOLLOWER]: Refusing update from remote peer 2d7f66c3625b4c36b496a42302cab30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:49:16.356667 3901 raft_consensus.cc:1261] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Refusing update from remote peer 2d7f66c3625b4c36b496a42302cab30e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:49:16.357014 4082 consensus_queue.cc:1035] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:49:16.357932 4083 consensus_queue.cc:1035] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:49:16.364845 4082 raft_consensus.cc:2932] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 629a1ca39c064c9b86201e265fd29bc4 (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: false } } }
I20240502 01:49:16.365975 3901 raft_consensus.cc:2932] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 629a1ca39c064c9b86201e265fd29bc4 (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: false } } }
I20240502 01:49:16.369377 4031 raft_consensus.cc:2932] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 629a1ca39c064c9b86201e265fd29bc4 (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: false } } }
I20240502 01:49:16.378574 3631 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9119e2d663834f5182f41406f021b349 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:49:16.379164 3647 catalog_manager.cc:5484] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e reported cstate change: config changed from index 3 to 4, VOTER 629a1ca39c064c9b86201e265fd29bc4 (127.1.233.129) evicted. New cstate: current_term: 1 leader_uuid: "2d7f66c3625b4c36b496a42302cab30e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d7f66c3625b4c36b496a42302cab30e" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e62f8941cf74cafb0e2185af5868dbb" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abcff7e8718d4544b0b26ecc64a6f2f0" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 35465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:49:16.390144 3739 tablet_service.cc:1514] Processing DeleteTablet for tablet 9119e2d663834f5182f41406f021b349 with delete_type TABLET_DATA_TOMBSTONED (TS 629a1ca39c064c9b86201e265fd29bc4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54942
I20240502 01:49:16.392748 4095 tablet_replica.cc:330] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4: stopping tablet replica
I20240502 01:49:16.393486 4095 raft_consensus.cc:2229] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:16.394157 4095 raft_consensus.cc:2258] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:16.397073 4095 ts_tablet_manager.cc:1875] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:49:16.408422 4095 ts_tablet_manager.cc:1888] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:49:16.408814 4095 log.cc:1196] T 9119e2d663834f5182f41406f021b349 P 629a1ca39c064c9b86201e265fd29bc4: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714614515033618-1958-0/minicluster-data/ts-0-root/wals/9119e2d663834f5182f41406f021b349
I20240502 01:49:16.410315 3631 catalog_manager.cc:4830] TS 629a1ca39c064c9b86201e265fd29bc4 (127.1.233.129:44953): tablet 9119e2d663834f5182f41406f021b349 (table test-workload [id=2da16a36bae7473dbe2132590b44c52a]) successfully deleted
I20240502 01:49:16.766664 3700 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:16.768821 3700 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:16.769238 3700 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:18.770241 3700 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:18.772226 3700 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:18.772595 3700 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:20.773624 3700 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:20.775640 3700 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:20.776038 3700 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:22.777180 3700 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:22.779155 3700 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:22.779526 3700 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:24.780536 3700 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:24.782521 3700 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:24.782931 3700 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:26.784094 3700 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:26.786337 3700 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:26.786731 3700 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:27.483633 1958 tablet_server.cc:178] TabletServer@127.1.233.129:0 shutting down...
I20240502 01:49:27.513823 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:49:27.514586 1958 tablet_replica.cc:330] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4: stopping tablet replica
I20240502 01:49:27.515156 1958 raft_consensus.cc:2229] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:49:27.516219 1958 raft_consensus.cc:2258] T dd639b5a3260467aade1d19d4e99b1d2 P 629a1ca39c064c9b86201e265fd29bc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:27.539376 1958 tablet_server.cc:194] TabletServer@127.1.233.129:0 shutdown complete.
I20240502 01:49:27.552111 1958 tablet_server.cc:178] TabletServer@127.1.233.130:0 shutting down...
I20240502 01:49:27.580317 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:49:27.581159 1958 tablet_replica.cc:330] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e: stopping tablet replica
I20240502 01:49:27.581851 1958 raft_consensus.cc:2229] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:27.582343 1958 raft_consensus.cc:2258] T dd639b5a3260467aade1d19d4e99b1d2 P 2d7f66c3625b4c36b496a42302cab30e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:27.584482 1958 tablet_replica.cc:330] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e: stopping tablet replica
I20240502 01:49:27.584923 1958 raft_consensus.cc:2229] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:49:27.585827 1958 raft_consensus.cc:2258] T 9119e2d663834f5182f41406f021b349 P 2d7f66c3625b4c36b496a42302cab30e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:27.609436 1958 tablet_server.cc:194] TabletServer@127.1.233.130:0 shutdown complete.
I20240502 01:49:27.621904 1958 tablet_server.cc:178] TabletServer@127.1.233.131:0 shutting down...
I20240502 01:49:27.649309 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:49:27.649940 1958 tablet_replica.cc:330] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb: stopping tablet replica
I20240502 01:49:27.650436 1958 raft_consensus.cc:2229] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:27.651201 1958 raft_consensus.cc:2258] T 9119e2d663834f5182f41406f021b349 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:27.653219 1958 tablet_replica.cc:330] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb: stopping tablet replica
I20240502 01:49:27.653617 1958 raft_consensus.cc:2229] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:27.653970 1958 raft_consensus.cc:2258] T dd639b5a3260467aade1d19d4e99b1d2 P 9e62f8941cf74cafb0e2185af5868dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:27.674855 1958 tablet_server.cc:194] TabletServer@127.1.233.131:0 shutdown complete.
I20240502 01:49:27.686543 1958 tablet_server.cc:178] TabletServer@127.1.233.132:0 shutting down...
I20240502 01:49:27.709760 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:49:27.710484 1958 tablet_replica.cc:330] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0: stopping tablet replica
I20240502 01:49:27.711099 1958 raft_consensus.cc:2229] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:49:27.711720 1958 raft_consensus.cc:2258] T 9119e2d663834f5182f41406f021b349 P abcff7e8718d4544b0b26ecc64a6f2f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:27.731101 1958 tablet_server.cc:194] TabletServer@127.1.233.132:0 shutdown complete.
I20240502 01:49:27.742257 1958 master.cc:537] Master@127.1.233.190:42291 shutting down...
I20240502 01:49:27.759500 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:49:27.760118 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:27.760495 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P 10d93877c88c4078893a8ec36fd17020: stopping tablet replica
I20240502 01:49:27.778919 1958 master.cc:552] Master@127.1.233.190:42291 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (27211 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240502 01:49:27.817917 1958 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.233.190:42545
I20240502 01:49:27.819089 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:27.826041 4100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:27.826601 4101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:27.827186 4103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:27.827814 1958 server_base.cc:1034] running on GCE node
I20240502 01:49:27.828768 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:27.828969 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:27.829123 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614567829101 us; error 0 us; skew 500 ppm
I20240502 01:49:27.829614 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:27.832219 1958 webserver.cc:460] Webserver started at http://127.1.233.190:35223/ using document root <none> and password file <none>
I20240502 01:49:27.832716 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:27.832892 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:27.833187 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:27.834470 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714614515033618-1958-0/minicluster-data/master-0-root/instance:
uuid: "1b479a3564114444a8e86bd68ea0ac2b"
format_stamp: "Formatted at 2024-05-02 01:49:27 on dist-test-slave-2090"
I20240502 01:49:27.839174 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:49:27.842491 4108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:27.843184 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:49:27.843446 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714614515033618-1958-0/minicluster-data/master-0-root
uuid: "1b479a3564114444a8e86bd68ea0ac2b"
format_stamp: "Formatted at 2024-05-02 01:49:27 on dist-test-slave-2090"
I20240502 01:49:27.843712 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714614515033618-1958-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714614515033618-1958-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714614515033618-1958-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:27.863780 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:27.865090 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:27.903226 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.190:42545
I20240502 01:49:27.903333 4159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.190:42545 every 8 connection(s)
I20240502 01:49:27.907119 4160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:27.918274 4160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b: Bootstrap starting.
I20240502 01:49:27.922674 4160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:27.927215 4160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b: No bootstrap required, opened a new log
I20240502 01:49:27.929489 4160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479a3564114444a8e86bd68ea0ac2b" member_type: VOTER }
I20240502 01:49:27.930013 4160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:27.930270 4160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b479a3564114444a8e86bd68ea0ac2b, State: Initialized, Role: FOLLOWER
I20240502 01:49:27.930836 4160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479a3564114444a8e86bd68ea0ac2b" member_type: VOTER }
I20240502 01:49:27.931331 4160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:49:27.931545 4160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:49:27.931807 4160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:27.936367 4160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479a3564114444a8e86bd68ea0ac2b" member_type: VOTER }
I20240502 01:49:27.936924 4160 leader_election.cc:302] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b479a3564114444a8e86bd68ea0ac2b; no voters:
I20240502 01:49:27.938079 4160 leader_election.cc:288] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:49:27.938489 4163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:27.939807 4163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [term 1 LEADER]: Becoming Leader. State: Replica: 1b479a3564114444a8e86bd68ea0ac2b, State: Running, Role: LEADER
I20240502 01:49:27.940593 4163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479a3564114444a8e86bd68ea0ac2b" member_type: VOTER }
I20240502 01:49:27.941097 4160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:49:27.943642 4165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b479a3564114444a8e86bd68ea0ac2b. Latest consensus state: current_term: 1 leader_uuid: "1b479a3564114444a8e86bd68ea0ac2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479a3564114444a8e86bd68ea0ac2b" member_type: VOTER } }
I20240502 01:49:27.943585 4164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b479a3564114444a8e86bd68ea0ac2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479a3564114444a8e86bd68ea0ac2b" member_type: VOTER } }
I20240502 01:49:27.944432 4165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [sys.catalog]: This master's current role is: LEADER
I20240502 01:49:27.944514 4164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [sys.catalog]: This master's current role is: LEADER
I20240502 01:49:27.948895 4168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:49:27.957621 4168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:49:27.958691 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:49:27.967461 4168 catalog_manager.cc:1347] Generated new cluster ID: c7a822cd9c7a4e0aa88843b35bf980c9
I20240502 01:49:27.967803 4168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:49:27.993753 4168 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:49:27.995136 4168 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:49:28.011714 4168 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b: Generated new TSK 0
I20240502 01:49:28.012466 4168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:49:28.024351 1958 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 0.00020378s
I20240502 01:49:40.121595 1958 master.cc:537] Master@127.1.233.190:42545 shutting down...
I20240502 01:49:40.146108 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:49:40.146945 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:40.147298 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1b479a3564114444a8e86bd68ea0ac2b: stopping tablet replica
I20240502 01:49:40.166831 1958 master.cc:552] Master@127.1.233.190:42545 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12381 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240502 01:49:40.199451 1958 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.233.190:38829
I20240502 01:49:40.200654 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:40.207499 4183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:40.208323 4184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:40.208830 4186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:40.209530 1958 server_base.cc:1034] running on GCE node
I20240502 01:49:40.210255 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:40.210454 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:40.210594 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614580210576 us; error 0 us; skew 500 ppm
I20240502 01:49:40.211081 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:40.213725 1958 webserver.cc:460] Webserver started at http://127.1.233.190:39399/ using document root <none> and password file <none>
I20240502 01:49:40.214229 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:40.214514 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:40.214838 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:40.216225 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/master-0-root/instance:
uuid: "959f56330049415b981939684ceadcd4"
format_stamp: "Formatted at 2024-05-02 01:49:40 on dist-test-slave-2090"
I20240502 01:49:40.221204 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:49:40.224783 4191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:40.225600 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 01:49:40.225881 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/master-0-root
uuid: "959f56330049415b981939684ceadcd4"
format_stamp: "Formatted at 2024-05-02 01:49:40 on dist-test-slave-2090"
I20240502 01:49:40.226372 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:40.246321 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:40.247617 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:40.287410 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.190:38829
I20240502 01:49:40.287501 4243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.190:38829 every 8 connection(s)
I20240502 01:49:40.291601 4244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:40.304639 4244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4: Bootstrap starting.
I20240502 01:49:40.309933 4244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:40.314759 4244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4: No bootstrap required, opened a new log
I20240502 01:49:40.317116 4244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "959f56330049415b981939684ceadcd4" member_type: VOTER }
I20240502 01:49:40.317664 4244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:40.317865 4244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 959f56330049415b981939684ceadcd4, State: Initialized, Role: FOLLOWER
I20240502 01:49:40.318486 4244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "959f56330049415b981939684ceadcd4" member_type: VOTER }
I20240502 01:49:40.318972 4244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:49:40.319202 4244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:49:40.319505 4244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:40.324476 4244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "959f56330049415b981939684ceadcd4" member_type: VOTER }
I20240502 01:49:40.325017 4244 leader_election.cc:302] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 959f56330049415b981939684ceadcd4; no voters:
I20240502 01:49:40.326164 4244 leader_election.cc:288] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:49:40.326655 4247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:40.328171 4247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [term 1 LEADER]: Becoming Leader. State: Replica: 959f56330049415b981939684ceadcd4, State: Running, Role: LEADER
I20240502 01:49:40.328850 4247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "959f56330049415b981939684ceadcd4" member_type: VOTER }
I20240502 01:49:40.329236 4244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:49:40.332051 4249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 959f56330049415b981939684ceadcd4. Latest consensus state: current_term: 1 leader_uuid: "959f56330049415b981939684ceadcd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "959f56330049415b981939684ceadcd4" member_type: VOTER } }
I20240502 01:49:40.332034 4248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "959f56330049415b981939684ceadcd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "959f56330049415b981939684ceadcd4" member_type: VOTER } }
I20240502 01:49:40.332948 4249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [sys.catalog]: This master's current role is: LEADER
I20240502 01:49:40.333067 4248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [sys.catalog]: This master's current role is: LEADER
I20240502 01:49:40.335788 4253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:49:40.344405 4253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:49:40.345438 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:49:40.353621 4253 catalog_manager.cc:1347] Generated new cluster ID: 6bf7392e0c984060a517afc37ad8e920
I20240502 01:49:40.353930 4253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:49:40.378470 4253 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:49:40.379845 4253 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:49:40.395272 4253 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4: Generated new TSK 0
I20240502 01:49:40.395885 4253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:49:40.412240 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:40.419553 4265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:40.420552 4266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:40.421686 4268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:40.422366 1958 server_base.cc:1034] running on GCE node
I20240502 01:49:40.423238 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:40.423444 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:40.423589 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614580423573 us; error 0 us; skew 500 ppm
I20240502 01:49:40.424121 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:40.430511 1958 webserver.cc:460] Webserver started at http://127.1.233.129:43653/ using document root <none> and password file <none>
I20240502 01:49:40.431085 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:40.431265 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:40.431524 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:40.432904 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-0-root/instance:
uuid: "5668a10187e84a12a9fb448df427bb70"
format_stamp: "Formatted at 2024-05-02 01:49:40 on dist-test-slave-2090"
I20240502 01:49:40.438030 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:49:40.441602 4273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:40.442333 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:49:40.442633 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "5668a10187e84a12a9fb448df427bb70"
format_stamp: "Formatted at 2024-05-02 01:49:40 on dist-test-slave-2090"
I20240502 01:49:40.442915 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:40.480856 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:40.482102 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:40.483634 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:40.486107 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:40.486305 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:40.486523 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:40.486694 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:40.529107 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:44181
I20240502 01:49:40.529225 4336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:44181 every 8 connection(s)
I20240502 01:49:40.534018 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:40.543689 4342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:40.543762 4341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:40.546530 1958 server_base.cc:1034] running on GCE node
W20240502 01:49:40.547190 4344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:40.548174 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 01:49:40.548211 4337 heartbeater.cc:347] Connected to a master server at 127.1.233.190:38829
W20240502 01:49:40.548460 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:40.548640 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614580548625 us; error 0 us; skew 500 ppm
I20240502 01:49:40.548652 4337 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:40.549235 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:40.549508 4337 heartbeater.cc:511] Master 127.1.233.190:38829 requested a full tablet report, sending...
I20240502 01:49:40.551630 4209 ts_manager.cc:194] Registered new tserver with Master: 5668a10187e84a12a9fb448df427bb70 (127.1.233.129:44181)
I20240502 01:49:40.551841 1958 webserver.cc:460] Webserver started at http://127.1.233.130:43707/ using document root <none> and password file <none>
I20240502 01:49:40.552395 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:40.552584 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:40.552819 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:40.553640 4209 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39072
I20240502 01:49:40.554061 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-1-root/instance:
uuid: "45d011083bb14970a1fdbaf33074c75d"
format_stamp: "Formatted at 2024-05-02 01:49:40 on dist-test-slave-2090"
I20240502 01:49:40.558743 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:49:40.562136 4349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:40.562935 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:49:40.563210 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-1-root
uuid: "45d011083bb14970a1fdbaf33074c75d"
format_stamp: "Formatted at 2024-05-02 01:49:40 on dist-test-slave-2090"
I20240502 01:49:40.563519 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:40.575798 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:40.577136 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:40.578620 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:40.580926 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:40.581130 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:40.581357 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:40.581501 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:40.622987 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.130:45957
I20240502 01:49:40.623088 4411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.130:45957 every 8 connection(s)
I20240502 01:49:40.627862 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:40.635726 4415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:40.636317 4416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:40.639330 4412 heartbeater.cc:347] Connected to a master server at 127.1.233.190:38829
I20240502 01:49:40.639683 1958 server_base.cc:1034] running on GCE node
I20240502 01:49:40.639845 4412 heartbeater.cc:464] Registering TS with master...
W20240502 01:49:40.640529 4418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:40.641017 4412 heartbeater.cc:511] Master 127.1.233.190:38829 requested a full tablet report, sending...
I20240502 01:49:40.641480 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:40.641710 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:40.641865 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614580641852 us; error 0 us; skew 500 ppm
I20240502 01:49:40.642592 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:40.642982 4209 ts_manager.cc:194] Registered new tserver with Master: 45d011083bb14970a1fdbaf33074c75d (127.1.233.130:45957)
I20240502 01:49:40.644567 4209 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39074
I20240502 01:49:40.647298 1958 webserver.cc:460] Webserver started at http://127.1.233.131:35601/ using document root <none> and password file <none>
I20240502 01:49:40.647836 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:40.648028 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:40.648310 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:40.649521 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-2-root/instance:
uuid: "51d7c40f2a73451ca52faa615ec4aa69"
format_stamp: "Formatted at 2024-05-02 01:49:40 on dist-test-slave-2090"
I20240502 01:49:40.654165 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:49:40.657475 4423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:40.658320 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:49:40.658696 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-2-root
uuid: "51d7c40f2a73451ca52faa615ec4aa69"
format_stamp: "Formatted at 2024-05-02 01:49:40 on dist-test-slave-2090"
I20240502 01:49:40.659124 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714614515033618-1958-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:40.676707 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:40.677920 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:40.679366 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:40.681697 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:40.681890 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:40.682114 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:40.682248 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:40.722841 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.131:44965
I20240502 01:49:40.723070 4485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.131:44965 every 8 connection(s)
I20240502 01:49:40.739756 4486 heartbeater.cc:347] Connected to a master server at 127.1.233.190:38829
I20240502 01:49:40.740175 4486 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:40.740937 4486 heartbeater.cc:511] Master 127.1.233.190:38829 requested a full tablet report, sending...
I20240502 01:49:40.742744 4209 ts_manager.cc:194] Registered new tserver with Master: 51d7c40f2a73451ca52faa615ec4aa69 (127.1.233.131:44965)
I20240502 01:49:40.743593 1958 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017219293s
I20240502 01:49:40.744191 4209 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39090
I20240502 01:49:41.556278 4337 heartbeater.cc:503] Master 127.1.233.190:38829 was elected leader, sending a full tablet report...
I20240502 01:49:41.647390 4412 heartbeater.cc:503] Master 127.1.233.190:38829 was elected leader, sending a full tablet report...
I20240502 01:49:41.747263 4486 heartbeater.cc:503] Master 127.1.233.190:38829 was elected leader, sending a full tablet report...
I20240502 01:49:42.345245 4262 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:44.346203 4262 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:46.347450 4262 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:48.348629 4262 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:50.349782 4262 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:52.350865 4262 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:52.840533 1958 tablet_server.cc:178] TabletServer@127.1.233.129:0 shutting down...
I20240502 01:49:52.871383 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:49:52.888926 1958 tablet_server.cc:194] TabletServer@127.1.233.129:0 shutdown complete.
I20240502 01:49:52.899302 1958 tablet_server.cc:178] TabletServer@127.1.233.130:0 shutting down...
I20240502 01:49:52.918910 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:49:52.934856 1958 tablet_server.cc:194] TabletServer@127.1.233.130:0 shutdown complete.
I20240502 01:49:52.943837 1958 tablet_server.cc:178] TabletServer@127.1.233.131:0 shutting down...
I20240502 01:49:52.964179 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:49:52.980433 1958 tablet_server.cc:194] TabletServer@127.1.233.131:0 shutdown complete.
I20240502 01:49:52.990350 1958 master.cc:537] Master@127.1.233.190:38829 shutting down...
I20240502 01:49:53.008374 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:49:53.009172 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:49:53.009536 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P 959f56330049415b981939684ceadcd4: stopping tablet replica
I20240502 01:49:53.027825 1958 master.cc:552] Master@127.1.233.190:38829 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12860 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240502 01:49:53.060122 1958 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.233.190:37349
I20240502 01:49:53.061280 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:53.067911 4491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:53.068611 4492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:53.069121 4494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:53.069540 1958 server_base.cc:1034] running on GCE node
I20240502 01:49:53.070318 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:53.070499 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:53.070608 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614593070598 us; error 0 us; skew 500 ppm
I20240502 01:49:53.071096 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:53.078591 1958 webserver.cc:460] Webserver started at http://127.1.233.190:39781/ using document root <none> and password file <none>
I20240502 01:49:53.079156 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:53.079357 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:53.079646 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:53.080976 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/master-0-root/instance:
uuid: "c58cd22edd4d4e248e75979d665b83a3"
format_stamp: "Formatted at 2024-05-02 01:49:53 on dist-test-slave-2090"
I20240502 01:49:53.085858 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:49:53.089424 4499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:53.090199 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:49:53.090472 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/master-0-root
uuid: "c58cd22edd4d4e248e75979d665b83a3"
format_stamp: "Formatted at 2024-05-02 01:49:53 on dist-test-slave-2090"
I20240502 01:49:53.090732 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:53.114681 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:53.115968 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:53.155057 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.190:37349
I20240502 01:49:53.155165 4552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.190:37349 every 8 connection(s)
I20240502 01:49:53.159088 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:53.170531 4553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3: Bootstrap starting.
I20240502 01:49:53.175163 4553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:53.179672 4553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3: No bootstrap required, opened a new log
I20240502 01:49:53.181838 4553 raft_consensus.cc:348] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58cd22edd4d4e248e75979d665b83a3" member_type: VOTER }
I20240502 01:49:53.182353 4553 raft_consensus.cc:374] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:53.182538 4553 raft_consensus.cc:729] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c58cd22edd4d4e248e75979d665b83a3, State: Initialized, Role: FOLLOWER
I20240502 01:49:53.183107 4553 consensus_queue.cc:260] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58cd22edd4d4e248e75979d665b83a3" member_type: VOTER }
I20240502 01:49:53.183547 4553 raft_consensus.cc:388] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:49:53.183729 4553 raft_consensus.cc:482] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:49:53.183948 4553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:53.188696 4553 raft_consensus.cc:504] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58cd22edd4d4e248e75979d665b83a3" member_type: VOTER }
I20240502 01:49:53.189218 4553 leader_election.cc:302] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c58cd22edd4d4e248e75979d665b83a3; no voters:
I20240502 01:49:53.190343 4553 leader_election.cc:288] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:49:53.190804 4556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:53.192183 4556 raft_consensus.cc:686] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [term 1 LEADER]: Becoming Leader. State: Replica: c58cd22edd4d4e248e75979d665b83a3, State: Running, Role: LEADER
I20240502 01:49:53.192929 4556 consensus_queue.cc:237] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58cd22edd4d4e248e75979d665b83a3" member_type: VOTER }
I20240502 01:49:53.193775 4553 sys_catalog.cc:564] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:49:53.196314 4558 sys_catalog.cc:455] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c58cd22edd4d4e248e75979d665b83a3. Latest consensus state: current_term: 1 leader_uuid: "c58cd22edd4d4e248e75979d665b83a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58cd22edd4d4e248e75979d665b83a3" member_type: VOTER } }
I20240502 01:49:53.196944 4558 sys_catalog.cc:458] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [sys.catalog]: This master's current role is: LEADER
I20240502 01:49:53.196206 4557 sys_catalog.cc:455] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c58cd22edd4d4e248e75979d665b83a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58cd22edd4d4e248e75979d665b83a3" member_type: VOTER } }
I20240502 01:49:53.197422 4557 sys_catalog.cc:458] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [sys.catalog]: This master's current role is: LEADER
I20240502 01:49:53.202724 4562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:49:53.208309 4562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:49:53.212554 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:49:53.218056 4562 catalog_manager.cc:1347] Generated new cluster ID: e595299c02b54d3480f1c5274bff7f48
I20240502 01:49:53.218327 4562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:49:53.230799 4562 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:49:53.232095 4562 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:49:53.247576 4562 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3: Generated new TSK 0
I20240502 01:49:53.248203 4562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:49:53.280021 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:53.286803 4574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:53.288012 4575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:53.289201 4577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:53.289948 1958 server_base.cc:1034] running on GCE node
I20240502 01:49:53.290810 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:53.291057 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:53.291182 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614593291171 us; error 0 us; skew 500 ppm
I20240502 01:49:53.291666 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:53.294312 1958 webserver.cc:460] Webserver started at http://127.1.233.129:39359/ using document root <none> and password file <none>
I20240502 01:49:53.294770 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:53.295053 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:53.295315 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:53.296561 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-0-root/instance:
uuid: "15c05dfb846a492cbed393517d2e2954"
format_stamp: "Formatted at 2024-05-02 01:49:53 on dist-test-slave-2090"
I20240502 01:49:53.301311 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240502 01:49:53.304812 4582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:53.305747 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 01:49:53.306068 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "15c05dfb846a492cbed393517d2e2954"
format_stamp: "Formatted at 2024-05-02 01:49:53 on dist-test-slave-2090"
I20240502 01:49:53.306362 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:53.328312 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:53.329568 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:53.331146 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:53.333735 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:53.333961 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:53.334196 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:53.334364 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:53.375985 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:43503
I20240502 01:49:53.376096 4645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:43503 every 8 connection(s)
I20240502 01:49:53.380954 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:53.389487 4650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:53.390339 4651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:53.393522 1958 server_base.cc:1034] running on GCE node
I20240502 01:49:53.394086 4646 heartbeater.cc:347] Connected to a master server at 127.1.233.190:37349
W20240502 01:49:53.394090 4653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:53.394673 4646 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:53.395284 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:53.395498 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:53.395534 4646 heartbeater.cc:511] Master 127.1.233.190:37349 requested a full tablet report, sending...
I20240502 01:49:53.395700 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614593395681 us; error 0 us; skew 500 ppm
I20240502 01:49:53.396412 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:53.397945 4516 ts_manager.cc:194] Registered new tserver with Master: 15c05dfb846a492cbed393517d2e2954 (127.1.233.129:43503)
I20240502 01:49:53.399220 1958 webserver.cc:460] Webserver started at http://127.1.233.130:38717/ using document root <none> and password file <none>
I20240502 01:49:53.399718 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:53.399893 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:53.400161 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:53.400192 4516 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45112
I20240502 01:49:53.401643 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-1-root/instance:
uuid: "ad48a8f8481040a48085802e2213990e"
format_stamp: "Formatted at 2024-05-02 01:49:53 on dist-test-slave-2090"
I20240502 01:49:53.406252 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:49:53.409790 4658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:53.410568 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:49:53.410866 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-1-root
uuid: "ad48a8f8481040a48085802e2213990e"
format_stamp: "Formatted at 2024-05-02 01:49:53 on dist-test-slave-2090"
I20240502 01:49:53.411162 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:53.449759 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:53.450955 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:53.452415 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:53.454602 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:53.454799 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:53.455009 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:53.455147 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:53.494990 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.130:45793
I20240502 01:49:53.495085 4721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.130:45793 every 8 connection(s)
I20240502 01:49:53.499751 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:53.507211 4725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:53.510090 4726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:53.511790 1958 server_base.cc:1034] running on GCE node
W20240502 01:49:53.512449 4728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:53.512821 4722 heartbeater.cc:347] Connected to a master server at 127.1.233.190:37349
I20240502 01:49:53.513332 4722 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:53.513571 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:53.513839 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:53.514045 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614593514025 us; error 0 us; skew 500 ppm
I20240502 01:49:53.514286 4722 heartbeater.cc:511] Master 127.1.233.190:37349 requested a full tablet report, sending...
I20240502 01:49:53.514809 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:53.516537 4516 ts_manager.cc:194] Registered new tserver with Master: ad48a8f8481040a48085802e2213990e (127.1.233.130:45793)
I20240502 01:49:53.517771 1958 webserver.cc:460] Webserver started at http://127.1.233.131:42009/ using document root <none> and password file <none>
I20240502 01:49:53.518457 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:53.518535 4516 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45118
I20240502 01:49:53.518754 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:53.519106 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:53.520658 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-2-root/instance:
uuid: "cd87945cfddb4238a8b5478c63811848"
format_stamp: "Formatted at 2024-05-02 01:49:53 on dist-test-slave-2090"
I20240502 01:49:53.525240 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:49:53.528564 4733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:53.529331 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:49:53.529593 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-2-root
uuid: "cd87945cfddb4238a8b5478c63811848"
format_stamp: "Formatted at 2024-05-02 01:49:53 on dist-test-slave-2090"
I20240502 01:49:53.529840 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:53.549094 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:53.550320 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:53.551748 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:53.554191 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:53.554368 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:53.554605 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:53.554735 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:53.595489 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.131:33831
I20240502 01:49:53.595588 4795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.131:33831 every 8 connection(s)
I20240502 01:49:53.600320 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:49:53.608943 4799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:49:53.609634 4800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:53.613282 4796 heartbeater.cc:347] Connected to a master server at 127.1.233.190:37349
I20240502 01:49:53.613705 4796 heartbeater.cc:464] Registering TS with master...
W20240502 01:49:53.614281 4802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:49:53.614596 4796 heartbeater.cc:511] Master 127.1.233.190:37349 requested a full tablet report, sending...
I20240502 01:49:53.615342 1958 server_base.cc:1034] running on GCE node
I20240502 01:49:53.616233 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:49:53.616441 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:49:53.616633 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614593616615 us; error 0 us; skew 500 ppm
I20240502 01:49:53.616915 4516 ts_manager.cc:194] Registered new tserver with Master: cd87945cfddb4238a8b5478c63811848 (127.1.233.131:33831)
I20240502 01:49:53.617169 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:49:53.618672 4516 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45122
I20240502 01:49:53.620338 1958 webserver.cc:460] Webserver started at http://127.1.233.132:39815/ using document root <none> and password file <none>
I20240502 01:49:53.621028 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:49:53.621281 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:49:53.621569 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:49:53.622629 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-3-root/instance:
uuid: "b6befb7c862540e38bc4829f774752ee"
format_stamp: "Formatted at 2024-05-02 01:49:53 on dist-test-slave-2090"
I20240502 01:49:53.627079 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240502 01:49:53.630708 4808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:53.631505 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 01:49:53.631822 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-3-root
uuid: "b6befb7c862540e38bc4829f774752ee"
format_stamp: "Formatted at 2024-05-02 01:49:53 on dist-test-slave-2090"
I20240502 01:49:53.632109 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714614515033618-1958-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:49:53.646162 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:49:53.647408 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:49:53.649015 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:49:53.651321 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:49:53.651510 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:53.651749 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:49:53.651909 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:49:53.691496 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.132:38347
I20240502 01:49:53.691602 4871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.132:38347 every 8 connection(s)
I20240502 01:49:53.706059 4872 heartbeater.cc:347] Connected to a master server at 127.1.233.190:37349
I20240502 01:49:53.706494 4872 heartbeater.cc:464] Registering TS with master...
I20240502 01:49:53.707356 4872 heartbeater.cc:511] Master 127.1.233.190:37349 requested a full tablet report, sending...
I20240502 01:49:53.709391 4516 ts_manager.cc:194] Registered new tserver with Master: b6befb7c862540e38bc4829f774752ee (127.1.233.132:38347)
I20240502 01:49:53.710088 1958 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.015379296s
I20240502 01:49:53.711010 4516 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45138
I20240502 01:49:54.242755 1958 test_util.cc:268] Using random seed: -774829912
I20240502 01:49:54.265301 4516 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58542:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 01:49:54.303737 4761 tablet_service.cc:1467] Processing CreateTablet for tablet 7eafc94242a14fc28b7342b38dca575b (DEFAULT_TABLE table=test-workload [id=94ddf22c7dac4a14b4f40f8271c0cb04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:49:54.305122 4761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eafc94242a14fc28b7342b38dca575b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:54.305820 4837 tablet_service.cc:1467] Processing CreateTablet for tablet 7eafc94242a14fc28b7342b38dca575b (DEFAULT_TABLE table=test-workload [id=94ddf22c7dac4a14b4f40f8271c0cb04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:49:54.308176 4837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eafc94242a14fc28b7342b38dca575b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:54.310613 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 7eafc94242a14fc28b7342b38dca575b (DEFAULT_TABLE table=test-workload [id=94ddf22c7dac4a14b4f40f8271c0cb04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:49:54.311897 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eafc94242a14fc28b7342b38dca575b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:49:54.323756 4893 tablet_bootstrap.cc:492] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee: Bootstrap starting.
I20240502 01:49:54.325521 4894 tablet_bootstrap.cc:492] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848: Bootstrap starting.
I20240502 01:49:54.329051 4895 tablet_bootstrap.cc:492] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954: Bootstrap starting.
I20240502 01:49:54.331988 4894 tablet_bootstrap.cc:654] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:54.333364 4893 tablet_bootstrap.cc:654] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:54.335481 4895 tablet_bootstrap.cc:654] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954: Neither blocks nor log segments found. Creating new log.
I20240502 01:49:54.337006 4894 tablet_bootstrap.cc:492] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848: No bootstrap required, opened a new log
I20240502 01:49:54.337514 4894 ts_tablet_manager.cc:1367] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 01:49:54.340461 4893 tablet_bootstrap.cc:492] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee: No bootstrap required, opened a new log
I20240502 01:49:54.340958 4893 ts_tablet_manager.cc:1367] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.001s
I20240502 01:49:54.340627 4894 raft_consensus.cc:348] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd87945cfddb4238a8b5478c63811848" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33831 } } peers { permanent_uuid: "b6befb7c862540e38bc4829f774752ee" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38347 } } peers { permanent_uuid: "15c05dfb846a492cbed393517d2e2954" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43503 } }
I20240502 01:49:54.341398 4894 raft_consensus.cc:374] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:54.341709 4894 raft_consensus.cc:729] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd87945cfddb4238a8b5478c63811848, State: Initialized, Role: FOLLOWER
I20240502 01:49:54.342465 4894 consensus_queue.cc:260] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd87945cfddb4238a8b5478c63811848" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33831 } } peers { permanent_uuid: "b6befb7c862540e38bc4829f774752ee" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38347 } } peers { permanent_uuid: "15c05dfb846a492cbed393517d2e2954" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43503 } }
I20240502 01:49:54.342900 4895 tablet_bootstrap.cc:492] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954: No bootstrap required, opened a new log
I20240502 01:49:54.343422 4895 ts_tablet_manager.cc:1367] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20240502 01:49:54.343606 4893 raft_consensus.cc:348] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd87945cfddb4238a8b5478c63811848" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33831 } } peers { permanent_uuid: "b6befb7c862540e38bc4829f774752ee" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38347 } } peers { permanent_uuid: "15c05dfb846a492cbed393517d2e2954" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43503 } }
I20240502 01:49:54.344451 4893 raft_consensus.cc:374] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:54.344763 4893 raft_consensus.cc:729] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6befb7c862540e38bc4829f774752ee, State: Initialized, Role: FOLLOWER
I20240502 01:49:54.345130 4796 heartbeater.cc:503] Master 127.1.233.190:37349 was elected leader, sending a full tablet report...
I20240502 01:49:54.345500 4893 consensus_queue.cc:260] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd87945cfddb4238a8b5478c63811848" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33831 } } peers { permanent_uuid: "b6befb7c862540e38bc4829f774752ee" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38347 } } peers { permanent_uuid: "15c05dfb846a492cbed393517d2e2954" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43503 } }
I20240502 01:49:54.346343 4895 raft_consensus.cc:348] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd87945cfddb4238a8b5478c63811848" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33831 } } peers { permanent_uuid: "b6befb7c862540e38bc4829f774752ee" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38347 } } peers { permanent_uuid: "15c05dfb846a492cbed393517d2e2954" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43503 } }
I20240502 01:49:54.347148 4895 raft_consensus.cc:374] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:49:54.347251 4894 ts_tablet_manager.cc:1398] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 01:49:54.347693 4895 raft_consensus.cc:729] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15c05dfb846a492cbed393517d2e2954, State: Initialized, Role: FOLLOWER
I20240502 01:49:54.349337 4872 heartbeater.cc:503] Master 127.1.233.190:37349 was elected leader, sending a full tablet report...
I20240502 01:49:54.349782 4893 ts_tablet_manager.cc:1398] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee: Time spent starting tablet: real 0.009s user 0.000s sys 0.008s
I20240502 01:49:54.348839 4895 consensus_queue.cc:260] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd87945cfddb4238a8b5478c63811848" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33831 } } peers { permanent_uuid: "b6befb7c862540e38bc4829f774752ee" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38347 } } peers { permanent_uuid: "15c05dfb846a492cbed393517d2e2954" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43503 } }
W20240502 01:49:54.350391 4797 tablet.cc:2369] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:49:54.352391 4646 heartbeater.cc:503] Master 127.1.233.190:37349 was elected leader, sending a full tablet report...
I20240502 01:49:54.352955 4895 ts_tablet_manager.cc:1398] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20240502 01:49:54.422281 4900 raft_consensus.cc:482] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:49:54.422784 4900 raft_consensus.cc:504] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd87945cfddb4238a8b5478c63811848" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33831 } } peers { permanent_uuid: "b6befb7c862540e38bc4829f774752ee" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38347 } } peers { permanent_uuid: "15c05dfb846a492cbed393517d2e2954" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43503 } }
I20240502 01:49:54.425014 4900 leader_election.cc:288] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd87945cfddb4238a8b5478c63811848 (127.1.233.131:33831), 15c05dfb846a492cbed393517d2e2954 (127.1.233.129:43503)
I20240502 01:49:54.437610 4621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eafc94242a14fc28b7342b38dca575b" candidate_uuid: "b6befb7c862540e38bc4829f774752ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c05dfb846a492cbed393517d2e2954" is_pre_election: true
I20240502 01:49:54.437670 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eafc94242a14fc28b7342b38dca575b" candidate_uuid: "b6befb7c862540e38bc4829f774752ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd87945cfddb4238a8b5478c63811848" is_pre_election: true
I20240502 01:49:54.438547 4621 raft_consensus.cc:2454] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6befb7c862540e38bc4829f774752ee in term 0.
I20240502 01:49:54.438745 4771 raft_consensus.cc:2454] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6befb7c862540e38bc4829f774752ee in term 0.
I20240502 01:49:54.439703 4810 leader_election.cc:302] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c05dfb846a492cbed393517d2e2954, b6befb7c862540e38bc4829f774752ee; no voters:
I20240502 01:49:54.440680 4900 raft_consensus.cc:2781] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:49:54.441092 4900 raft_consensus.cc:482] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:49:54.441452 4900 raft_consensus.cc:3037] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:54.447427 4900 raft_consensus.cc:504] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd87945cfddb4238a8b5478c63811848" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33831 } } peers { permanent_uuid: "b6befb7c862540e38bc4829f774752ee" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38347 } } peers { permanent_uuid: "15c05dfb846a492cbed393517d2e2954" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43503 } }
I20240502 01:49:54.449173 4900 leader_election.cc:288] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [CANDIDATE]: Term 1 election: Requested vote from peers cd87945cfddb4238a8b5478c63811848 (127.1.233.131:33831), 15c05dfb846a492cbed393517d2e2954 (127.1.233.129:43503)
I20240502 01:49:54.450122 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eafc94242a14fc28b7342b38dca575b" candidate_uuid: "b6befb7c862540e38bc4829f774752ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd87945cfddb4238a8b5478c63811848"
I20240502 01:49:54.450199 4621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eafc94242a14fc28b7342b38dca575b" candidate_uuid: "b6befb7c862540e38bc4829f774752ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c05dfb846a492cbed393517d2e2954"
I20240502 01:49:54.450680 4771 raft_consensus.cc:3037] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:54.450723 4621 raft_consensus.cc:3037] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:49:54.456096 4771 raft_consensus.cc:2454] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6befb7c862540e38bc4829f774752ee in term 1.
I20240502 01:49:54.456096 4621 raft_consensus.cc:2454] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6befb7c862540e38bc4829f774752ee in term 1.
I20240502 01:49:54.457541 4812 leader_election.cc:302] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6befb7c862540e38bc4829f774752ee, cd87945cfddb4238a8b5478c63811848; no voters:
I20240502 01:49:54.458299 4900 raft_consensus.cc:2781] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:49:54.459362 4900 raft_consensus.cc:686] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [term 1 LEADER]: Becoming Leader. State: Replica: b6befb7c862540e38bc4829f774752ee, State: Running, Role: LEADER
I20240502 01:49:54.460011 4900 consensus_queue.cc:237] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd87945cfddb4238a8b5478c63811848" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33831 } } peers { permanent_uuid: "b6befb7c862540e38bc4829f774752ee" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38347 } } peers { permanent_uuid: "15c05dfb846a492cbed393517d2e2954" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43503 } }
I20240502 01:49:54.467288 4516 catalog_manager.cc:5484] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee reported cstate change: term changed from 0 to 1, leader changed from <none> to b6befb7c862540e38bc4829f774752ee (127.1.233.132). New cstate: current_term: 1 leader_uuid: "b6befb7c862540e38bc4829f774752ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd87945cfddb4238a8b5478c63811848" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6befb7c862540e38bc4829f774752ee" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15c05dfb846a492cbed393517d2e2954" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43503 } health_report { overall_health: UNKNOWN } } }
I20240502 01:49:54.521487 4722 heartbeater.cc:503] Master 127.1.233.190:37349 was elected leader, sending a full tablet report...
I20240502 01:49:55.040820 4906 consensus_queue.cc:1035] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "15c05dfb846a492cbed393517d2e2954" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:55.054293 4906 consensus_queue.cc:1035] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd87945cfddb4238a8b5478c63811848" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:49:55.212250 4571 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:55.213761 4571 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:55.214103 4571 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:57.215420 4571 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:57.217104 4571 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:57.217484 4571 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:49:59.218490 4571 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:49:59.219841 4571 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:49:59.220191 4571 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:01.221161 4571 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:01.222512 4571 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:01.222826 4571 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 01:50:02.896675 4868 debug-util.cc:398] Leaking SignalData structure 0x7b080023bd60 after lost signal to thread 1961
W20240502 01:50:02.897678 4868 debug-util.cc:398] Leaking SignalData structure 0x7b080021c020 after lost signal to thread 4552
W20240502 01:50:02.898716 4868 debug-util.cc:398] Leaking SignalData structure 0x7b0800157700 after lost signal to thread 4645
W20240502 01:50:02.899614 4868 debug-util.cc:398] Leaking SignalData structure 0x7b0800168480 after lost signal to thread 4721
W20240502 01:50:02.900473 4868 debug-util.cc:398] Leaking SignalData structure 0x7b0800167ae0 after lost signal to thread 4795
W20240502 01:50:02.901314 4868 debug-util.cc:398] Leaking SignalData structure 0x7b0800199fc0 after lost signal to thread 4871
I20240502 01:50:03.223778 4571 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:03.225250 4571 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:03.225603 4571 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:05.226707 4571 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:05.228045 4571 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:05.228466 4571 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:05.979506 1958 tablet_server.cc:178] TabletServer@127.1.233.129:0 shutting down...
I20240502 01:50:06.006171 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:06.006982 1958 tablet_replica.cc:330] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954: stopping tablet replica
I20240502 01:50:06.007656 1958 raft_consensus.cc:2229] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:06.008374 1958 raft_consensus.cc:2258] T 7eafc94242a14fc28b7342b38dca575b P 15c05dfb846a492cbed393517d2e2954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:06.029306 1958 tablet_server.cc:194] TabletServer@127.1.233.129:0 shutdown complete.
I20240502 01:50:06.042462 1958 tablet_server.cc:178] TabletServer@127.1.233.130:0 shutting down...
I20240502 01:50:06.065621 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:06.082688 1958 tablet_server.cc:194] TabletServer@127.1.233.130:0 shutdown complete.
I20240502 01:50:06.093294 1958 tablet_server.cc:178] TabletServer@127.1.233.131:0 shutting down...
I20240502 01:50:06.117444 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:06.118230 1958 tablet_replica.cc:330] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848: stopping tablet replica
I20240502 01:50:06.118836 1958 raft_consensus.cc:2229] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:06.119338 1958 raft_consensus.cc:2258] T 7eafc94242a14fc28b7342b38dca575b P cd87945cfddb4238a8b5478c63811848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:06.140075 1958 tablet_server.cc:194] TabletServer@127.1.233.131:0 shutdown complete.
I20240502 01:50:06.152089 1958 tablet_server.cc:178] TabletServer@127.1.233.132:0 shutting down...
I20240502 01:50:06.177469 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:06.178169 1958 tablet_replica.cc:330] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee: stopping tablet replica
I20240502 01:50:06.178704 1958 raft_consensus.cc:2229] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:06.179734 1958 raft_consensus.cc:2258] T 7eafc94242a14fc28b7342b38dca575b P b6befb7c862540e38bc4829f774752ee [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:50:06.717737 4868 debug-util.cc:398] Leaking SignalData structure 0x7b08000945c0 after lost signal to thread 1961
W20240502 01:50:06.718761 4868 debug-util.cc:398] Leaking SignalData structure 0x7b08001a45e0 after lost signal to thread 4552
W20240502 01:50:06.719805 4868 debug-util.cc:398] Leaking SignalData structure 0x7b08001f4ae0 after lost signal to thread 4645
W20240502 01:50:06.720768 4868 debug-util.cc:398] Leaking SignalData structure 0x7b080023bac0 after lost signal to thread 4721
W20240502 01:50:06.721983 4868 debug-util.cc:398] Leaking SignalData structure 0x7b0800108820 after lost signal to thread 4795
W20240502 01:50:06.722821 4868 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4180 after lost signal to thread 4871
I20240502 01:50:06.896957 1958 tablet_server.cc:194] TabletServer@127.1.233.132:0 shutdown complete.
I20240502 01:50:06.908676 1958 master.cc:537] Master@127.1.233.190:37349 shutting down...
I20240502 01:50:06.930807 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:06.931478 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:06.931833 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P c58cd22edd4d4e248e75979d665b83a3: stopping tablet replica
I20240502 01:50:06.951283 1958 master.cc:552] Master@127.1.233.190:37349 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (13929 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240502 01:50:06.990621 1958 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.233.190:40609
I20240502 01:50:06.991818 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:06.999229 4926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:06.999841 4927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:07.000257 4929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:07.000919 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:07.001907 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:07.002197 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:07.002394 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614607002374 us; error 0 us; skew 500 ppm
I20240502 01:50:07.003091 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:07.006767 1958 webserver.cc:460] Webserver started at http://127.1.233.190:38389/ using document root <none> and password file <none>
I20240502 01:50:07.007447 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:07.007769 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:07.008240 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:07.009836 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/master-0-root/instance:
uuid: "650e91ab972e4cee85b47c50d240032f"
format_stamp: "Formatted at 2024-05-02 01:50:07 on dist-test-slave-2090"
I20240502 01:50:07.015812 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 01:50:07.021216 4934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:07.022045 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240502 01:50:07.022328 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/master-0-root
uuid: "650e91ab972e4cee85b47c50d240032f"
format_stamp: "Formatted at 2024-05-02 01:50:07 on dist-test-slave-2090"
I20240502 01:50:07.022614 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:07.036013 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:07.037405 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:07.078673 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.190:40609
I20240502 01:50:07.078773 4986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.190:40609 every 8 connection(s)
I20240502 01:50:07.082777 4987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:07.094519 4987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f: Bootstrap starting.
I20240502 01:50:07.099227 4987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:07.103989 4987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f: No bootstrap required, opened a new log
I20240502 01:50:07.106573 4987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650e91ab972e4cee85b47c50d240032f" member_type: VOTER }
I20240502 01:50:07.107080 4987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:07.107319 4987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 650e91ab972e4cee85b47c50d240032f, State: Initialized, Role: FOLLOWER
I20240502 01:50:07.107980 4987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650e91ab972e4cee85b47c50d240032f" member_type: VOTER }
I20240502 01:50:07.108564 4987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:50:07.108789 4987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:50:07.109052 4987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:07.114216 4987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650e91ab972e4cee85b47c50d240032f" member_type: VOTER }
I20240502 01:50:07.114804 4987 leader_election.cc:302] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 650e91ab972e4cee85b47c50d240032f; no voters:
I20240502 01:50:07.116034 4987 leader_election.cc:288] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:50:07.116494 4990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:07.117920 4990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [term 1 LEADER]: Becoming Leader. State: Replica: 650e91ab972e4cee85b47c50d240032f, State: Running, Role: LEADER
I20240502 01:50:07.118677 4990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650e91ab972e4cee85b47c50d240032f" member_type: VOTER }
I20240502 01:50:07.119364 4987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:50:07.123621 4991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "650e91ab972e4cee85b47c50d240032f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650e91ab972e4cee85b47c50d240032f" member_type: VOTER } }
I20240502 01:50:07.125428 4991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [sys.catalog]: This master's current role is: LEADER
I20240502 01:50:07.124783 4992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 650e91ab972e4cee85b47c50d240032f. Latest consensus state: current_term: 1 leader_uuid: "650e91ab972e4cee85b47c50d240032f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650e91ab972e4cee85b47c50d240032f" member_type: VOTER } }
I20240502 01:50:07.126677 4992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [sys.catalog]: This master's current role is: LEADER
I20240502 01:50:07.130184 4999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:50:07.135375 4999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:50:07.136479 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:50:07.144915 4999 catalog_manager.cc:1347] Generated new cluster ID: f09b7073cb3746b5bd7a117cda414dbd
I20240502 01:50:07.145188 4999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:50:07.157480 4999 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:50:07.158854 4999 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:50:07.174580 4999 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f: Generated new TSK 0
I20240502 01:50:07.175282 4999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:50:07.203151 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:07.210083 5008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:07.212232 5009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:07.212898 5011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:07.213515 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:07.214361 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:07.214573 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:07.214723 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614607214706 us; error 0 us; skew 500 ppm
I20240502 01:50:07.215256 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:07.217902 1958 webserver.cc:460] Webserver started at http://127.1.233.129:36207/ using document root <none> and password file <none>
I20240502 01:50:07.218422 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:07.218622 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:07.218880 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:07.220280 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-0-root/instance:
uuid: "3f00d5a8508d4a648f239f9794c5baca"
format_stamp: "Formatted at 2024-05-02 01:50:07 on dist-test-slave-2090"
I20240502 01:50:07.225519 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 01:50:07.229408 5016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:07.230288 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:50:07.230576 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "3f00d5a8508d4a648f239f9794c5baca"
format_stamp: "Formatted at 2024-05-02 01:50:07 on dist-test-slave-2090"
I20240502 01:50:07.230847 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:07.248947 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:07.250152 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:07.251737 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:07.254443 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:07.254660 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:07.254936 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:07.255106 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:07.298614 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:35265
I20240502 01:50:07.298704 5078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:35265 every 8 connection(s)
I20240502 01:50:07.303699 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:07.313036 5083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:07.313696 5084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:07.316023 5086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:07.316732 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:07.318238 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:07.318535 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:07.318739 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614607318720 us; error 0 us; skew 500 ppm
I20240502 01:50:07.318996 5079 heartbeater.cc:347] Connected to a master server at 127.1.233.190:40609
I20240502 01:50:07.319350 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:07.319355 5079 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:07.320549 5079 heartbeater.cc:511] Master 127.1.233.190:40609 requested a full tablet report, sending...
I20240502 01:50:07.322486 1958 webserver.cc:460] Webserver started at http://127.1.233.130:39311/ using document root <none> and password file <none>
I20240502 01:50:07.323163 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:07.323168 4951 ts_manager.cc:194] Registered new tserver with Master: 3f00d5a8508d4a648f239f9794c5baca (127.1.233.129:35265)
I20240502 01:50:07.323563 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:07.323957 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:07.325700 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-1-root/instance:
uuid: "6b02d550a583494684aa75c83aa51594"
format_stamp: "Formatted at 2024-05-02 01:50:07 on dist-test-slave-2090"
I20240502 01:50:07.326088 4951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41200
I20240502 01:50:07.331662 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 01:50:07.335654 5091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:07.336561 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:50:07.336829 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-1-root
uuid: "6b02d550a583494684aa75c83aa51594"
format_stamp: "Formatted at 2024-05-02 01:50:07 on dist-test-slave-2090"
I20240502 01:50:07.337100 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:07.347620 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:07.348959 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:07.350447 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:07.353030 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:07.353238 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:07.353484 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:07.353622 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:07.395751 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.130:34121
I20240502 01:50:07.395864 5154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.130:34121 every 8 connection(s)
I20240502 01:50:07.400799 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:07.409648 5158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:07.410111 5159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:07.412925 5161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:07.413875 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:07.414593 5155 heartbeater.cc:347] Connected to a master server at 127.1.233.190:40609
I20240502 01:50:07.414822 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 01:50:07.414963 5155 heartbeater.cc:464] Registering TS with master...
W20240502 01:50:07.415078 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:07.415333 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614607415314 us; error 0 us; skew 500 ppm
I20240502 01:50:07.415933 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:07.416018 5155 heartbeater.cc:511] Master 127.1.233.190:40609 requested a full tablet report, sending...
I20240502 01:50:07.418275 4951 ts_manager.cc:194] Registered new tserver with Master: 6b02d550a583494684aa75c83aa51594 (127.1.233.130:34121)
I20240502 01:50:07.418989 1958 webserver.cc:460] Webserver started at http://127.1.233.131:41567/ using document root <none> and password file <none>
I20240502 01:50:07.419517 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:07.419711 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:07.419958 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:07.419935 4951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41204
I20240502 01:50:07.421681 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-2-root/instance:
uuid: "8034065b11c548689f80d59e9ab109b2"
format_stamp: "Formatted at 2024-05-02 01:50:07 on dist-test-slave-2090"
I20240502 01:50:07.426663 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:50:07.430203 5166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:07.431093 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:50:07.431430 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-2-root
uuid: "8034065b11c548689f80d59e9ab109b2"
format_stamp: "Formatted at 2024-05-02 01:50:07 on dist-test-slave-2090"
I20240502 01:50:07.431726 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714614515033618-1958-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:07.458328 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:07.459566 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:07.461094 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:07.463519 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:07.463722 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:07.463936 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:07.464089 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:07.506085 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.131:41797
I20240502 01:50:07.506188 5228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.131:41797 every 8 connection(s)
I20240502 01:50:07.520620 5229 heartbeater.cc:347] Connected to a master server at 127.1.233.190:40609
I20240502 01:50:07.521200 5229 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:07.522017 5229 heartbeater.cc:511] Master 127.1.233.190:40609 requested a full tablet report, sending...
I20240502 01:50:07.524029 4951 ts_manager.cc:194] Registered new tserver with Master: 8034065b11c548689f80d59e9ab109b2 (127.1.233.131:41797)
I20240502 01:50:07.524255 1958 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014895978s
I20240502 01:50:07.525465 4951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41220
I20240502 01:50:08.329564 5079 heartbeater.cc:503] Master 127.1.233.190:40609 was elected leader, sending a full tablet report...
I20240502 01:50:08.423043 5155 heartbeater.cc:503] Master 127.1.233.190:40609 was elected leader, sending a full tablet report...
I20240502 01:50:08.528103 5229 heartbeater.cc:503] Master 127.1.233.190:40609 was elected leader, sending a full tablet report...
I20240502 01:50:08.568724 1958 test_util.cc:268] Using random seed: -760503942
I20240502 01:50:08.590404 4951 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41242:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:50:08.592463 4951 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:50:08.642832 5120 tablet_service.cc:1467] Processing CreateTablet for tablet e27988ff0f4c4f3b842de76bc411be0a (DEFAULT_TABLE table=test-workload [id=4099a1e1956a44be84b248564ded2cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:50:08.642992 5044 tablet_service.cc:1467] Processing CreateTablet for tablet e27988ff0f4c4f3b842de76bc411be0a (DEFAULT_TABLE table=test-workload [id=4099a1e1956a44be84b248564ded2cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:50:08.644093 5120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27988ff0f4c4f3b842de76bc411be0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:08.644230 5044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27988ff0f4c4f3b842de76bc411be0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:08.646391 5194 tablet_service.cc:1467] Processing CreateTablet for tablet e27988ff0f4c4f3b842de76bc411be0a (DEFAULT_TABLE table=test-workload [id=4099a1e1956a44be84b248564ded2cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:50:08.647730 5194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27988ff0f4c4f3b842de76bc411be0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:08.659588 5250 tablet_bootstrap.cc:492] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594: Bootstrap starting.
I20240502 01:50:08.666606 5250 tablet_bootstrap.cc:654] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:08.674494 5252 tablet_bootstrap.cc:492] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2: Bootstrap starting.
I20240502 01:50:08.674646 5251 tablet_bootstrap.cc:492] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca: Bootstrap starting.
I20240502 01:50:08.674993 5250 tablet_bootstrap.cc:492] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594: No bootstrap required, opened a new log
I20240502 01:50:08.675567 5250 ts_tablet_manager.cc:1367] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20240502 01:50:08.678555 5250 raft_consensus.cc:348] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02d550a583494684aa75c83aa51594" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 34121 } } peers { permanent_uuid: "3f00d5a8508d4a648f239f9794c5baca" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 35265 } } peers { permanent_uuid: "8034065b11c548689f80d59e9ab109b2" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41797 } }
I20240502 01:50:08.679450 5250 raft_consensus.cc:374] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:08.679678 5251 tablet_bootstrap.cc:654] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:08.679867 5250 raft_consensus.cc:729] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b02d550a583494684aa75c83aa51594, State: Initialized, Role: FOLLOWER
I20240502 01:50:08.680768 5250 consensus_queue.cc:260] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02d550a583494684aa75c83aa51594" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 34121 } } peers { permanent_uuid: "3f00d5a8508d4a648f239f9794c5baca" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 35265 } } peers { permanent_uuid: "8034065b11c548689f80d59e9ab109b2" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41797 } }
I20240502 01:50:08.682608 5252 tablet_bootstrap.cc:654] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:08.684595 5250 ts_tablet_manager.cc:1398] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594: Time spent starting tablet: real 0.009s user 0.001s sys 0.009s
I20240502 01:50:08.686776 5251 tablet_bootstrap.cc:492] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca: No bootstrap required, opened a new log
I20240502 01:50:08.687284 5251 ts_tablet_manager.cc:1367] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240502 01:50:08.688344 5252 tablet_bootstrap.cc:492] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2: No bootstrap required, opened a new log
I20240502 01:50:08.688769 5252 ts_tablet_manager.cc:1367] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20240502 01:50:08.690184 5251 raft_consensus.cc:348] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02d550a583494684aa75c83aa51594" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 34121 } } peers { permanent_uuid: "3f00d5a8508d4a648f239f9794c5baca" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 35265 } } peers { permanent_uuid: "8034065b11c548689f80d59e9ab109b2" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41797 } }
I20240502 01:50:08.690716 5252 raft_consensus.cc:348] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02d550a583494684aa75c83aa51594" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 34121 } } peers { permanent_uuid: "3f00d5a8508d4a648f239f9794c5baca" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 35265 } } peers { permanent_uuid: "8034065b11c548689f80d59e9ab109b2" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41797 } }
I20240502 01:50:08.691038 5251 raft_consensus.cc:374] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:08.691366 5252 raft_consensus.cc:374] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:08.691426 5251 raft_consensus.cc:729] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f00d5a8508d4a648f239f9794c5baca, State: Initialized, Role: FOLLOWER
I20240502 01:50:08.691687 5252 raft_consensus.cc:729] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8034065b11c548689f80d59e9ab109b2, State: Initialized, Role: FOLLOWER
I20240502 01:50:08.692773 5251 consensus_queue.cc:260] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02d550a583494684aa75c83aa51594" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 34121 } } peers { permanent_uuid: "3f00d5a8508d4a648f239f9794c5baca" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 35265 } } peers { permanent_uuid: "8034065b11c548689f80d59e9ab109b2" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41797 } }
I20240502 01:50:08.692548 5252 consensus_queue.cc:260] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02d550a583494684aa75c83aa51594" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 34121 } } peers { permanent_uuid: "3f00d5a8508d4a648f239f9794c5baca" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 35265 } } peers { permanent_uuid: "8034065b11c548689f80d59e9ab109b2" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41797 } }
I20240502 01:50:08.696385 5251 ts_tablet_manager.cc:1398] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20240502 01:50:08.698798 5252 ts_tablet_manager.cc:1398] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240502 01:50:08.711261 5258 raft_consensus.cc:482] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:50:08.711704 5258 raft_consensus.cc:504] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02d550a583494684aa75c83aa51594" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 34121 } } peers { permanent_uuid: "3f00d5a8508d4a648f239f9794c5baca" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 35265 } } peers { permanent_uuid: "8034065b11c548689f80d59e9ab109b2" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41797 } }
I20240502 01:50:08.713855 5258 leader_election.cc:288] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b02d550a583494684aa75c83aa51594 (127.1.233.130:34121), 3f00d5a8508d4a648f239f9794c5baca (127.1.233.129:35265)
I20240502 01:50:08.724295 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27988ff0f4c4f3b842de76bc411be0a" candidate_uuid: "8034065b11c548689f80d59e9ab109b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b02d550a583494684aa75c83aa51594" is_pre_election: true
I20240502 01:50:08.725291 5130 raft_consensus.cc:2454] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8034065b11c548689f80d59e9ab109b2 in term 0.
I20240502 01:50:08.724329 5054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27988ff0f4c4f3b842de76bc411be0a" candidate_uuid: "8034065b11c548689f80d59e9ab109b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f00d5a8508d4a648f239f9794c5baca" is_pre_election: true
I20240502 01:50:08.726286 5054 raft_consensus.cc:2454] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8034065b11c548689f80d59e9ab109b2 in term 0.
I20240502 01:50:08.726598 5167 leader_election.cc:302] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b02d550a583494684aa75c83aa51594, 8034065b11c548689f80d59e9ab109b2; no voters:
I20240502 01:50:08.727463 5258 raft_consensus.cc:2781] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:50:08.727763 5258 raft_consensus.cc:482] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:50:08.728062 5258 raft_consensus.cc:3037] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:08.733589 5258 raft_consensus.cc:504] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02d550a583494684aa75c83aa51594" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 34121 } } peers { permanent_uuid: "3f00d5a8508d4a648f239f9794c5baca" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 35265 } } peers { permanent_uuid: "8034065b11c548689f80d59e9ab109b2" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41797 } }
I20240502 01:50:08.735157 5258 leader_election.cc:288] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [CANDIDATE]: Term 1 election: Requested vote from peers 6b02d550a583494684aa75c83aa51594 (127.1.233.130:34121), 3f00d5a8508d4a648f239f9794c5baca (127.1.233.129:35265)
I20240502 01:50:08.736138 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27988ff0f4c4f3b842de76bc411be0a" candidate_uuid: "8034065b11c548689f80d59e9ab109b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b02d550a583494684aa75c83aa51594"
I20240502 01:50:08.736250 5054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27988ff0f4c4f3b842de76bc411be0a" candidate_uuid: "8034065b11c548689f80d59e9ab109b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f00d5a8508d4a648f239f9794c5baca"
I20240502 01:50:08.736698 5130 raft_consensus.cc:3037] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:08.736769 5054 raft_consensus.cc:3037] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:08.741725 5130 raft_consensus.cc:2454] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8034065b11c548689f80d59e9ab109b2 in term 1.
I20240502 01:50:08.741896 5054 raft_consensus.cc:2454] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8034065b11c548689f80d59e9ab109b2 in term 1.
I20240502 01:50:08.742663 5167 leader_election.cc:302] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b02d550a583494684aa75c83aa51594, 8034065b11c548689f80d59e9ab109b2; no voters:
I20240502 01:50:08.743312 5258 raft_consensus.cc:2781] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:08.744321 5258 raft_consensus.cc:686] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [term 1 LEADER]: Becoming Leader. State: Replica: 8034065b11c548689f80d59e9ab109b2, State: Running, Role: LEADER
I20240502 01:50:08.745061 5258 consensus_queue.cc:237] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02d550a583494684aa75c83aa51594" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 34121 } } peers { permanent_uuid: "3f00d5a8508d4a648f239f9794c5baca" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 35265 } } peers { permanent_uuid: "8034065b11c548689f80d59e9ab109b2" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41797 } }
I20240502 01:50:08.753098 4950 catalog_manager.cc:5484] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8034065b11c548689f80d59e9ab109b2 (127.1.233.131). New cstate: current_term: 1 leader_uuid: "8034065b11c548689f80d59e9ab109b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02d550a583494684aa75c83aa51594" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 34121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f00d5a8508d4a648f239f9794c5baca" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 35265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8034065b11c548689f80d59e9ab109b2" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41797 } health_report { overall_health: HEALTHY } } }
I20240502 01:50:09.132913 5000 placement_policy_util.cc:407] tablet e27988ff0f4c4f3b842de76bc411be0a: detected majority of replicas (2 of 3) at location L0
I20240502 01:50:09.135249 5005 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:09.136770 5005 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:09.137109 5005 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:09.193037 5258 consensus_queue.cc:1035] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f00d5a8508d4a648f239f9794c5baca" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 35265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:50:09.206912 5258 consensus_queue.cc:1035] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b02d550a583494684aa75c83aa51594" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:50:10.141959 5000 placement_policy_util.cc:407] tablet e27988ff0f4c4f3b842de76bc411be0a: detected majority of replicas (2 of 3) at location L0
I20240502 01:50:11.138131 5005 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:11.139525 5005 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:11.139825 5005 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:11.150166 5000 placement_policy_util.cc:407] tablet e27988ff0f4c4f3b842de76bc411be0a: detected majority of replicas (2 of 3) at location L0
I20240502 01:50:12.158373 5000 placement_policy_util.cc:407] tablet e27988ff0f4c4f3b842de76bc411be0a: detected majority of replicas (2 of 3) at location L0
I20240502 01:50:13.140766 5005 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:13.142171 5005 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:13.142470 5005 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:13.166505 5000 placement_policy_util.cc:407] tablet e27988ff0f4c4f3b842de76bc411be0a: detected majority of replicas (2 of 3) at location L0
I20240502 01:50:14.174518 5000 placement_policy_util.cc:407] tablet e27988ff0f4c4f3b842de76bc411be0a: detected majority of replicas (2 of 3) at location L0
I20240502 01:50:15.143446 5005 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:15.144762 5005 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:15.145062 5005 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:15.182224 5000 placement_policy_util.cc:407] tablet e27988ff0f4c4f3b842de76bc411be0a: detected majority of replicas (2 of 3) at location L0
I20240502 01:50:16.190349 5000 placement_policy_util.cc:407] tablet e27988ff0f4c4f3b842de76bc411be0a: detected majority of replicas (2 of 3) at location L0
I20240502 01:50:17.146287 5005 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:17.147806 5005 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:17.148146 5005 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:17.198468 5000 placement_policy_util.cc:407] tablet e27988ff0f4c4f3b842de76bc411be0a: detected majority of replicas (2 of 3) at location L0
I20240502 01:50:18.205410 5000 placement_policy_util.cc:407] tablet e27988ff0f4c4f3b842de76bc411be0a: detected majority of replicas (2 of 3) at location L0
I20240502 01:50:19.149333 5005 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:19.150694 5005 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:19.150983 5005 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:19.212014 5000 placement_policy_util.cc:407] tablet e27988ff0f4c4f3b842de76bc411be0a: detected majority of replicas (2 of 3) at location L0
I20240502 01:50:19.836532 1958 tablet_server.cc:178] TabletServer@127.1.233.129:0 shutting down...
I20240502 01:50:19.865391 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:19.866099 1958 tablet_replica.cc:330] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca: stopping tablet replica
I20240502 01:50:19.866642 1958 raft_consensus.cc:2229] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:19.867203 1958 raft_consensus.cc:2258] T e27988ff0f4c4f3b842de76bc411be0a P 3f00d5a8508d4a648f239f9794c5baca [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:19.888185 1958 tablet_server.cc:194] TabletServer@127.1.233.129:0 shutdown complete.
I20240502 01:50:19.900980 1958 tablet_server.cc:178] TabletServer@127.1.233.130:0 shutting down...
W20240502 01:50:19.921918 5167 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 10 similar messages]
I20240502 01:50:19.924099 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:19.924983 1958 tablet_replica.cc:330] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594: stopping tablet replica
I20240502 01:50:19.925572 1958 raft_consensus.cc:2229] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:19.926057 1958 raft_consensus.cc:2258] T e27988ff0f4c4f3b842de76bc411be0a P 6b02d550a583494684aa75c83aa51594 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:50:19.931627 5167 consensus_peers.cc:485] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 -> Peer 6b02d550a583494684aa75c83aa51594 (127.1.233.130:34121): Couldn't send request to peer 6b02d550a583494684aa75c83aa51594. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:19.946717 1958 tablet_server.cc:194] TabletServer@127.1.233.130:0 shutdown complete.
I20240502 01:50:19.957998 1958 tablet_server.cc:178] TabletServer@127.1.233.131:0 shutting down...
I20240502 01:50:19.978586 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:19.979220 1958 tablet_replica.cc:330] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2: stopping tablet replica
I20240502 01:50:19.979734 1958 raft_consensus.cc:2229] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:19.980702 1958 raft_consensus.cc:2258] T e27988ff0f4c4f3b842de76bc411be0a P 8034065b11c548689f80d59e9ab109b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:20.000101 1958 tablet_server.cc:194] TabletServer@127.1.233.131:0 shutdown complete.
I20240502 01:50:20.011528 1958 master.cc:537] Master@127.1.233.190:40609 shutting down...
I20240502 01:50:20.031143 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:20.031776 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:20.032094 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P 650e91ab972e4cee85b47c50d240032f: stopping tablet replica
I20240502 01:50:20.052392 1958 master.cc:552] Master@127.1.233.190:40609 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (13098 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:516: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] AutoRebalancerTest.TestMaxMovesPerServer (6 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240502 01:50:20.095451 1958 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.233.190:35229
I20240502 01:50:20.096650 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:20.102856 5273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:20.103533 5274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:20.103900 5276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:20.104614 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:20.105376 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:20.105559 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:20.105715 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614620105696 us; error 0 us; skew 500 ppm
I20240502 01:50:20.106211 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:20.108706 1958 webserver.cc:460] Webserver started at http://127.1.233.190:41991/ using document root <none> and password file <none>
I20240502 01:50:20.109205 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:20.109380 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:20.109650 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:20.110908 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/master-0-root/instance:
uuid: "8c00149b16dc42c283ec7c2f4f51b88d"
format_stamp: "Formatted at 2024-05-02 01:50:20 on dist-test-slave-2090"
I20240502 01:50:20.115568 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240502 01:50:20.119019 5281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:20.119825 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:50:20.120103 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/master-0-root
uuid: "8c00149b16dc42c283ec7c2f4f51b88d"
format_stamp: "Formatted at 2024-05-02 01:50:20 on dist-test-slave-2090"
I20240502 01:50:20.120441 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:20.137995 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:20.139315 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:20.177276 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.190:35229
I20240502 01:50:20.177379 5332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.190:35229 every 8 connection(s)
I20240502 01:50:20.181145 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:20.192289 5333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d: Bootstrap starting.
I20240502 01:50:20.196449 5333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:20.200830 5333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d: No bootstrap required, opened a new log
I20240502 01:50:20.202924 5333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c00149b16dc42c283ec7c2f4f51b88d" member_type: VOTER }
I20240502 01:50:20.203404 5333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:20.203629 5333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c00149b16dc42c283ec7c2f4f51b88d, State: Initialized, Role: FOLLOWER
I20240502 01:50:20.204268 5333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c00149b16dc42c283ec7c2f4f51b88d" member_type: VOTER }
I20240502 01:50:20.204747 5333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:50:20.204948 5333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:50:20.205207 5333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:20.209740 5333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c00149b16dc42c283ec7c2f4f51b88d" member_type: VOTER }
I20240502 01:50:20.210256 5333 leader_election.cc:302] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c00149b16dc42c283ec7c2f4f51b88d; no voters:
I20240502 01:50:20.211400 5333 leader_election.cc:288] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:50:20.211755 5336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:20.213224 5336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [term 1 LEADER]: Becoming Leader. State: Replica: 8c00149b16dc42c283ec7c2f4f51b88d, State: Running, Role: LEADER
I20240502 01:50:20.213992 5336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c00149b16dc42c283ec7c2f4f51b88d" member_type: VOTER }
I20240502 01:50:20.214473 5333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:50:20.217224 5338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c00149b16dc42c283ec7c2f4f51b88d. Latest consensus state: current_term: 1 leader_uuid: "8c00149b16dc42c283ec7c2f4f51b88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c00149b16dc42c283ec7c2f4f51b88d" member_type: VOTER } }
I20240502 01:50:20.217159 5337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c00149b16dc42c283ec7c2f4f51b88d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c00149b16dc42c283ec7c2f4f51b88d" member_type: VOTER } }
I20240502 01:50:20.218003 5338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [sys.catalog]: This master's current role is: LEADER
I20240502 01:50:20.218117 5337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [sys.catalog]: This master's current role is: LEADER
I20240502 01:50:20.220826 5341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:50:20.226091 5341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:50:20.231552 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:50:20.235858 5341 catalog_manager.cc:1347] Generated new cluster ID: edbfc649c5b941608ac86391829ec6f2
I20240502 01:50:20.236166 5341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:50:20.254935 5341 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:50:20.256314 5341 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:50:20.282800 5341 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d: Generated new TSK 0
I20240502 01:50:20.283421 5341 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:50:20.299222 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:20.306020 5354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:20.307045 5355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:20.308354 5357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:20.309002 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:20.309875 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:20.310070 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:20.310195 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614620310184 us; error 0 us; skew 500 ppm
I20240502 01:50:20.310676 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:20.313170 1958 webserver.cc:460] Webserver started at http://127.1.233.129:44785/ using document root <none> and password file <none>
I20240502 01:50:20.313628 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:20.313808 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:20.314026 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:20.315156 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-0-root/instance:
uuid: "913a0a493b5e4a329ab0da24c1bf5160"
format_stamp: "Formatted at 2024-05-02 01:50:20 on dist-test-slave-2090"
I20240502 01:50:20.319947 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:50:20.323495 5362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:20.324340 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:50:20.324625 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "913a0a493b5e4a329ab0da24c1bf5160"
format_stamp: "Formatted at 2024-05-02 01:50:20 on dist-test-slave-2090"
I20240502 01:50:20.324905 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:20.335496 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:20.336723 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:20.338227 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:20.340719 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:20.340911 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:20.341145 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:20.341301 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:20.381521 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:36223
I20240502 01:50:20.381636 5424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:36223 every 8 connection(s)
I20240502 01:50:20.386090 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:20.395175 5430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:20.395978 5429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:20.400516 5432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:20.400774 5425 heartbeater.cc:347] Connected to a master server at 127.1.233.190:35229
I20240502 01:50:20.401238 5425 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:20.401384 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:20.402115 5425 heartbeater.cc:511] Master 127.1.233.190:35229 requested a full tablet report, sending...
I20240502 01:50:20.402419 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:20.402662 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:20.402819 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614620402804 us; error 0 us; skew 500 ppm
I20240502 01:50:20.403398 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:20.404055 5298 ts_manager.cc:194] Registered new tserver with Master: 913a0a493b5e4a329ab0da24c1bf5160 (127.1.233.129:36223)
I20240502 01:50:20.405953 5298 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45272
I20240502 01:50:20.406278 1958 webserver.cc:460] Webserver started at http://127.1.233.130:45001/ using document root <none> and password file <none>
I20240502 01:50:20.406769 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:20.406963 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:20.407228 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:20.408588 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-1-root/instance:
uuid: "db231fb687ee46c5aa5754e285f53a5c"
format_stamp: "Formatted at 2024-05-02 01:50:20 on dist-test-slave-2090"
I20240502 01:50:20.413218 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240502 01:50:20.416762 5437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:20.417512 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 01:50:20.417799 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-1-root
uuid: "db231fb687ee46c5aa5754e285f53a5c"
format_stamp: "Formatted at 2024-05-02 01:50:20 on dist-test-slave-2090"
I20240502 01:50:20.418087 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:20.436640 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:20.437863 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:20.439339 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:20.441781 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:20.441994 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:20.442224 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:20.442375 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:20.481638 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.130:37161
I20240502 01:50:20.481729 5499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.130:37161 every 8 connection(s)
I20240502 01:50:20.486296 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:20.493674 5503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:20.496366 5504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:20.498191 1958 server_base.cc:1034] running on GCE node
W20240502 01:50:20.498809 5506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:20.499289 5500 heartbeater.cc:347] Connected to a master server at 127.1.233.190:35229
I20240502 01:50:20.499651 5500 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:20.499781 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:20.500047 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:20.500249 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614620500233 us; error 0 us; skew 500 ppm
I20240502 01:50:20.500391 5500 heartbeater.cc:511] Master 127.1.233.190:35229 requested a full tablet report, sending...
I20240502 01:50:20.500835 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:20.502501 5298 ts_manager.cc:194] Registered new tserver with Master: db231fb687ee46c5aa5754e285f53a5c (127.1.233.130:37161)
I20240502 01:50:20.503603 1958 webserver.cc:460] Webserver started at http://127.1.233.131:41677/ using document root <none> and password file <none>
I20240502 01:50:20.504283 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:20.504434 5298 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45282
I20240502 01:50:20.504525 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:20.504998 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:20.506238 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-2-root/instance:
uuid: "d826a2588e5d44269d272bfb604b7d4f"
format_stamp: "Formatted at 2024-05-02 01:50:20 on dist-test-slave-2090"
I20240502 01:50:20.510743 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:50:20.514313 5511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:20.515096 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:50:20.515488 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-2-root
uuid: "d826a2588e5d44269d272bfb604b7d4f"
format_stamp: "Formatted at 2024-05-02 01:50:20 on dist-test-slave-2090"
I20240502 01:50:20.515806 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:20.536371 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:20.537520 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:20.539016 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:20.541431 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:20.541633 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:20.541854 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:20.542003 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:20.584365 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.131:41707
I20240502 01:50:20.584483 5573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.131:41707 every 8 connection(s)
I20240502 01:50:20.597930 5574 heartbeater.cc:347] Connected to a master server at 127.1.233.190:35229
I20240502 01:50:20.598285 5574 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:20.599030 5574 heartbeater.cc:511] Master 127.1.233.190:35229 requested a full tablet report, sending...
I20240502 01:50:20.600993 5298 ts_manager.cc:194] Registered new tserver with Master: d826a2588e5d44269d272bfb604b7d4f (127.1.233.131:41707)
I20240502 01:50:20.601244 1958 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013757429s
I20240502 01:50:20.602877 5298 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45288
I20240502 01:50:21.408636 5425 heartbeater.cc:503] Master 127.1.233.190:35229 was elected leader, sending a full tablet report...
I20240502 01:50:21.507040 5500 heartbeater.cc:503] Master 127.1.233.190:35229 was elected leader, sending a full tablet report...
I20240502 01:50:21.606050 5574 heartbeater.cc:503] Master 127.1.233.190:35229 was elected leader, sending a full tablet report...
I20240502 01:50:21.634671 1958 test_util.cc:268] Using random seed: -747437998
I20240502 01:50:21.664885 5298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45302:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:50:21.667912 5298 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:50:21.713599 5465 tablet_service.cc:1467] Processing CreateTablet for tablet c54b599ba1ee4033b73975c3077abd6f (DEFAULT_TABLE table=test-workload [id=b073301217d34649a158c49039ea96a3]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 01:50:21.715166 5465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54b599ba1ee4033b73975c3077abd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:21.716830 5463 tablet_service.cc:1467] Processing CreateTablet for tablet 47805d1b4cc44fcca988db5eba2e05c8 (DEFAULT_TABLE table=test-workload [id=b073301217d34649a158c49039ea96a3]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 01:50:21.718340 5463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47805d1b4cc44fcca988db5eba2e05c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:21.719020 5390 tablet_service.cc:1467] Processing CreateTablet for tablet c54b599ba1ee4033b73975c3077abd6f (DEFAULT_TABLE table=test-workload [id=b073301217d34649a158c49039ea96a3]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 01:50:21.721299 5464 tablet_service.cc:1467] Processing CreateTablet for tablet b272117917a04efdb418b036b959cf1e (DEFAULT_TABLE table=test-workload [id=b073301217d34649a158c49039ea96a3]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 01:50:21.722805 5464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b272117917a04efdb418b036b959cf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:21.722492 5388 tablet_service.cc:1467] Processing CreateTablet for tablet 47805d1b4cc44fcca988db5eba2e05c8 (DEFAULT_TABLE table=test-workload [id=b073301217d34649a158c49039ea96a3]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 01:50:21.723865 5388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47805d1b4cc44fcca988db5eba2e05c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:21.724767 5390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54b599ba1ee4033b73975c3077abd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:21.721288 5389 tablet_service.cc:1467] Processing CreateTablet for tablet b272117917a04efdb418b036b959cf1e (DEFAULT_TABLE table=test-workload [id=b073301217d34649a158c49039ea96a3]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 01:50:21.726749 5389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b272117917a04efdb418b036b959cf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:21.743690 5594 tablet_bootstrap.cc:492] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c: Bootstrap starting.
I20240502 01:50:21.747953 5539 tablet_service.cc:1467] Processing CreateTablet for tablet c54b599ba1ee4033b73975c3077abd6f (DEFAULT_TABLE table=test-workload [id=b073301217d34649a158c49039ea96a3]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 01:50:21.749367 5539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54b599ba1ee4033b73975c3077abd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:21.752835 5538 tablet_service.cc:1467] Processing CreateTablet for tablet b272117917a04efdb418b036b959cf1e (DEFAULT_TABLE table=test-workload [id=b073301217d34649a158c49039ea96a3]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 01:50:21.754179 5538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b272117917a04efdb418b036b959cf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:21.757340 5537 tablet_service.cc:1467] Processing CreateTablet for tablet 47805d1b4cc44fcca988db5eba2e05c8 (DEFAULT_TABLE table=test-workload [id=b073301217d34649a158c49039ea96a3]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 01:50:21.757833 5594 tablet_bootstrap.cc:654] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:21.758831 5537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47805d1b4cc44fcca988db5eba2e05c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:21.765372 5595 tablet_bootstrap.cc:492] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160: Bootstrap starting.
I20240502 01:50:21.771834 5595 tablet_bootstrap.cc:654] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:21.773726 5594 tablet_bootstrap.cc:492] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c: No bootstrap required, opened a new log
I20240502 01:50:21.776758 5594 ts_tablet_manager.cc:1367] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.000s
I20240502 01:50:21.779691 5594 raft_consensus.cc:348] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.780663 5594 raft_consensus.cc:374] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:21.781019 5594 raft_consensus.cc:729] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db231fb687ee46c5aa5754e285f53a5c, State: Initialized, Role: FOLLOWER
I20240502 01:50:21.781919 5594 consensus_queue.cc:260] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.788836 5597 tablet_bootstrap.cc:492] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f: Bootstrap starting.
I20240502 01:50:21.789727 5595 tablet_bootstrap.cc:492] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160: No bootstrap required, opened a new log
I20240502 01:50:21.790151 5595 ts_tablet_manager.cc:1367] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.005s
I20240502 01:50:21.791491 5594 ts_tablet_manager.cc:1398] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240502 01:50:21.792721 5594 tablet_bootstrap.cc:492] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c: Bootstrap starting.
I20240502 01:50:21.792805 5595 raft_consensus.cc:348] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.793776 5595 raft_consensus.cc:374] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:21.794154 5595 raft_consensus.cc:729] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913a0a493b5e4a329ab0da24c1bf5160, State: Initialized, Role: FOLLOWER
I20240502 01:50:21.795485 5597 tablet_bootstrap.cc:654] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:21.795042 5595 consensus_queue.cc:260] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.805967 5595 ts_tablet_manager.cc:1398] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160: Time spent starting tablet: real 0.016s user 0.005s sys 0.011s
I20240502 01:50:21.807996 5594 tablet_bootstrap.cc:654] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:21.807983 5595 tablet_bootstrap.cc:492] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160: Bootstrap starting.
I20240502 01:50:21.813289 5597 tablet_bootstrap.cc:492] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f: No bootstrap required, opened a new log
I20240502 01:50:21.814313 5595 tablet_bootstrap.cc:654] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:21.818149 5597 ts_tablet_manager.cc:1367] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.009s
I20240502 01:50:21.819720 5595 tablet_bootstrap.cc:492] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160: No bootstrap required, opened a new log
I20240502 01:50:21.820245 5595 ts_tablet_manager.cc:1367] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.002s
I20240502 01:50:21.821738 5597 raft_consensus.cc:348] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.822628 5597 raft_consensus.cc:374] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:21.823000 5594 tablet_bootstrap.cc:492] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c: No bootstrap required, opened a new log
I20240502 01:50:21.823570 5594 ts_tablet_manager.cc:1367] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c: Time spent bootstrapping tablet: real 0.031s user 0.016s sys 0.004s
I20240502 01:50:21.823383 5595 raft_consensus.cc:348] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.823973 5595 raft_consensus.cc:374] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:21.824229 5595 raft_consensus.cc:729] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913a0a493b5e4a329ab0da24c1bf5160, State: Initialized, Role: FOLLOWER
I20240502 01:50:21.822969 5597 raft_consensus.cc:729] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d826a2588e5d44269d272bfb604b7d4f, State: Initialized, Role: FOLLOWER
I20240502 01:50:21.824999 5595 consensus_queue.cc:260] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.825311 5597 consensus_queue.cc:260] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.826316 5594 raft_consensus.cc:348] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.827240 5595 ts_tablet_manager.cc:1398] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20240502 01:50:21.827221 5594 raft_consensus.cc:374] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:21.827808 5594 raft_consensus.cc:729] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db231fb687ee46c5aa5754e285f53a5c, State: Initialized, Role: FOLLOWER
I20240502 01:50:21.828564 5594 consensus_queue.cc:260] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.830381 5594 ts_tablet_manager.cc:1398] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240502 01:50:21.831248 5594 tablet_bootstrap.cc:492] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c: Bootstrap starting.
I20240502 01:50:21.833449 5597 ts_tablet_manager.cc:1398] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f: Time spent starting tablet: real 0.015s user 0.003s sys 0.006s
I20240502 01:50:21.834489 5597 tablet_bootstrap.cc:492] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f: Bootstrap starting.
I20240502 01:50:21.836803 5594 tablet_bootstrap.cc:654] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:21.837029 5595 tablet_bootstrap.cc:492] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160: Bootstrap starting.
I20240502 01:50:21.842864 5597 tablet_bootstrap.cc:654] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:21.843680 5595 tablet_bootstrap.cc:654] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:21.849354 5594 tablet_bootstrap.cc:492] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c: No bootstrap required, opened a new log
I20240502 01:50:21.849855 5594 ts_tablet_manager.cc:1367] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.010s
I20240502 01:50:21.852114 5594 raft_consensus.cc:348] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.852636 5595 tablet_bootstrap.cc:492] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160: No bootstrap required, opened a new log
I20240502 01:50:21.852943 5594 raft_consensus.cc:374] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:21.853322 5594 raft_consensus.cc:729] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db231fb687ee46c5aa5754e285f53a5c, State: Initialized, Role: FOLLOWER
I20240502 01:50:21.853382 5595 ts_tablet_manager.cc:1367] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.005s
I20240502 01:50:21.853922 5594 consensus_queue.cc:260] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.855041 5597 tablet_bootstrap.cc:492] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f: No bootstrap required, opened a new log
I20240502 01:50:21.855561 5597 ts_tablet_manager.cc:1367] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.009s
I20240502 01:50:21.855829 5594 ts_tablet_manager.cc:1398] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20240502 01:50:21.856215 5595 raft_consensus.cc:348] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.856938 5595 raft_consensus.cc:374] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:21.857221 5595 raft_consensus.cc:729] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913a0a493b5e4a329ab0da24c1bf5160, State: Initialized, Role: FOLLOWER
I20240502 01:50:21.857890 5595 consensus_queue.cc:260] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.858326 5597 raft_consensus.cc:348] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.859086 5597 raft_consensus.cc:374] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:21.859398 5597 raft_consensus.cc:729] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d826a2588e5d44269d272bfb604b7d4f, State: Initialized, Role: FOLLOWER
I20240502 01:50:21.860255 5595 ts_tablet_manager.cc:1398] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 01:50:21.860122 5597 consensus_queue.cc:260] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.862785 5597 ts_tablet_manager.cc:1398] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:50:21.863649 5597 tablet_bootstrap.cc:492] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f: Bootstrap starting.
I20240502 01:50:21.864414 5599 raft_consensus.cc:482] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:50:21.864869 5599 raft_consensus.cc:504] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.867449 5599 leader_election.cc:288] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 913a0a493b5e4a329ab0da24c1bf5160 (127.1.233.129:36223), d826a2588e5d44269d272bfb604b7d4f (127.1.233.131:41707)
I20240502 01:50:21.869171 5597 tablet_bootstrap.cc:654] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:21.874856 5597 tablet_bootstrap.cc:492] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f: No bootstrap required, opened a new log
I20240502 01:50:21.875350 5597 ts_tablet_manager.cc:1367] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240502 01:50:21.878193 5597 raft_consensus.cc:348] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.878949 5597 raft_consensus.cc:374] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:21.879211 5597 raft_consensus.cc:729] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d826a2588e5d44269d272bfb604b7d4f, State: Initialized, Role: FOLLOWER
I20240502 01:50:21.879633 5600 raft_consensus.cc:482] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:50:21.880125 5600 raft_consensus.cc:504] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.880405 5597 consensus_queue.cc:260] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.882623 5597 ts_tablet_manager.cc:1398] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:50:21.885792 5600 leader_election.cc:288] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db231fb687ee46c5aa5754e285f53a5c (127.1.233.130:37161), d826a2588e5d44269d272bfb604b7d4f (127.1.233.131:41707)
I20240502 01:50:21.896075 5400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47805d1b4cc44fcca988db5eba2e05c8" candidate_uuid: "db231fb687ee46c5aa5754e285f53a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913a0a493b5e4a329ab0da24c1bf5160" is_pre_election: true
I20240502 01:50:21.897073 5400 raft_consensus.cc:2454] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db231fb687ee46c5aa5754e285f53a5c in term 0.
I20240502 01:50:21.899238 5440 leader_election.cc:302] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913a0a493b5e4a329ab0da24c1bf5160, db231fb687ee46c5aa5754e285f53a5c; no voters:
I20240502 01:50:21.900306 5599 raft_consensus.cc:2781] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:50:21.900691 5599 raft_consensus.cc:482] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:50:21.900976 5599 raft_consensus.cc:3037] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:21.905642 5475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47805d1b4cc44fcca988db5eba2e05c8" candidate_uuid: "913a0a493b5e4a329ab0da24c1bf5160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db231fb687ee46c5aa5754e285f53a5c" is_pre_election: true
I20240502 01:50:21.907039 5549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47805d1b4cc44fcca988db5eba2e05c8" candidate_uuid: "913a0a493b5e4a329ab0da24c1bf5160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d826a2588e5d44269d272bfb604b7d4f" is_pre_election: true
I20240502 01:50:21.907171 5599 raft_consensus.cc:504] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.907657 5549 raft_consensus.cc:2454] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913a0a493b5e4a329ab0da24c1bf5160 in term 0.
I20240502 01:50:21.908218 5475 raft_consensus.cc:2379] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 913a0a493b5e4a329ab0da24c1bf5160 in current term 1: Already voted for candidate db231fb687ee46c5aa5754e285f53a5c in this term.
I20240502 01:50:21.909107 5599 leader_election.cc:288] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [CANDIDATE]: Term 1 election: Requested vote from peers 913a0a493b5e4a329ab0da24c1bf5160 (127.1.233.129:36223), d826a2588e5d44269d272bfb604b7d4f (127.1.233.131:41707)
I20240502 01:50:21.910123 5400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47805d1b4cc44fcca988db5eba2e05c8" candidate_uuid: "db231fb687ee46c5aa5754e285f53a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913a0a493b5e4a329ab0da24c1bf5160"
I20240502 01:50:21.910897 5400 raft_consensus.cc:3037] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:21.911926 5549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47805d1b4cc44fcca988db5eba2e05c8" candidate_uuid: "db231fb687ee46c5aa5754e285f53a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d826a2588e5d44269d272bfb604b7d4f" is_pre_election: true
I20240502 01:50:21.912601 5549 raft_consensus.cc:2454] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db231fb687ee46c5aa5754e285f53a5c in term 0.
I20240502 01:50:21.912676 5548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47805d1b4cc44fcca988db5eba2e05c8" candidate_uuid: "db231fb687ee46c5aa5754e285f53a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d826a2588e5d44269d272bfb604b7d4f"
I20240502 01:50:21.913314 5548 raft_consensus.cc:3037] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:21.913471 5600 raft_consensus.cc:482] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:50:21.913995 5600 raft_consensus.cc:504] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.916033 5600 leader_election.cc:288] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db231fb687ee46c5aa5754e285f53a5c (127.1.233.130:37161), d826a2588e5d44269d272bfb604b7d4f (127.1.233.131:41707)
I20240502 01:50:21.917155 5365 leader_election.cc:302] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913a0a493b5e4a329ab0da24c1bf5160, d826a2588e5d44269d272bfb604b7d4f; no voters:
I20240502 01:50:21.917755 5549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54b599ba1ee4033b73975c3077abd6f" candidate_uuid: "913a0a493b5e4a329ab0da24c1bf5160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d826a2588e5d44269d272bfb604b7d4f" is_pre_election: true
I20240502 01:50:21.918335 5400 raft_consensus.cc:2454] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db231fb687ee46c5aa5754e285f53a5c in term 1.
I20240502 01:50:21.918488 5549 raft_consensus.cc:2454] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913a0a493b5e4a329ab0da24c1bf5160 in term 0.
I20240502 01:50:21.919179 5600 raft_consensus.cc:2741] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240502 01:50:21.919926 5365 leader_election.cc:302] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913a0a493b5e4a329ab0da24c1bf5160, d826a2588e5d44269d272bfb604b7d4f; no voters:
I20240502 01:50:21.921717 5440 leader_election.cc:302] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913a0a493b5e4a329ab0da24c1bf5160, db231fb687ee46c5aa5754e285f53a5c; no voters:
I20240502 01:50:21.922297 5475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54b599ba1ee4033b73975c3077abd6f" candidate_uuid: "913a0a493b5e4a329ab0da24c1bf5160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db231fb687ee46c5aa5754e285f53a5c" is_pre_election: true
I20240502 01:50:21.923034 5600 raft_consensus.cc:2781] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:50:21.923085 5475 raft_consensus.cc:2454] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913a0a493b5e4a329ab0da24c1bf5160 in term 0.
I20240502 01:50:21.923497 5600 raft_consensus.cc:482] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:50:21.923908 5600 raft_consensus.cc:3037] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:21.924962 5548 raft_consensus.cc:2454] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db231fb687ee46c5aa5754e285f53a5c in term 1.
I20240502 01:50:21.925806 5599 raft_consensus.cc:2781] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:21.927126 5599 raft_consensus.cc:686] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 1 LEADER]: Becoming Leader. State: Replica: db231fb687ee46c5aa5754e285f53a5c, State: Running, Role: LEADER
I20240502 01:50:21.927874 5599 consensus_queue.cc:237] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.929834 5600 raft_consensus.cc:504] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.932586 5600 leader_election.cc:288] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [CANDIDATE]: Term 1 election: Requested vote from peers db231fb687ee46c5aa5754e285f53a5c (127.1.233.130:37161), d826a2588e5d44269d272bfb604b7d4f (127.1.233.131:41707)
I20240502 01:50:21.932664 5475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54b599ba1ee4033b73975c3077abd6f" candidate_uuid: "913a0a493b5e4a329ab0da24c1bf5160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db231fb687ee46c5aa5754e285f53a5c"
I20240502 01:50:21.933382 5475 raft_consensus.cc:3037] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:21.933379 5548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54b599ba1ee4033b73975c3077abd6f" candidate_uuid: "913a0a493b5e4a329ab0da24c1bf5160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d826a2588e5d44269d272bfb604b7d4f"
I20240502 01:50:21.934123 5548 raft_consensus.cc:3037] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:21.935053 5297 catalog_manager.cc:5484] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c reported cstate change: term changed from 0 to 1, leader changed from <none> to db231fb687ee46c5aa5754e285f53a5c (127.1.233.130). New cstate: current_term: 1 leader_uuid: "db231fb687ee46c5aa5754e285f53a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:21.940886 5475 raft_consensus.cc:2454] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913a0a493b5e4a329ab0da24c1bf5160 in term 1.
I20240502 01:50:21.940886 5548 raft_consensus.cc:2454] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913a0a493b5e4a329ab0da24c1bf5160 in term 1.
I20240502 01:50:21.942178 5365 leader_election.cc:302] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913a0a493b5e4a329ab0da24c1bf5160, d826a2588e5d44269d272bfb604b7d4f; no voters:
I20240502 01:50:21.943229 5600 raft_consensus.cc:2781] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:21.943651 5600 raft_consensus.cc:686] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 LEADER]: Becoming Leader. State: Replica: 913a0a493b5e4a329ab0da24c1bf5160, State: Running, Role: LEADER
I20240502 01:50:21.944418 5600 consensus_queue.cc:237] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.950857 5297 catalog_manager.cc:5484] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 reported cstate change: term changed from 0 to 1, leader changed from <none> to 913a0a493b5e4a329ab0da24c1bf5160 (127.1.233.129). New cstate: current_term: 1 leader_uuid: "913a0a493b5e4a329ab0da24c1bf5160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:21.959689 5604 raft_consensus.cc:482] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:50:21.960101 5604 raft_consensus.cc:504] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.962041 5604 leader_election.cc:288] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 913a0a493b5e4a329ab0da24c1bf5160 (127.1.233.129:36223), db231fb687ee46c5aa5754e285f53a5c (127.1.233.130:37161)
I20240502 01:50:21.973126 5475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b272117917a04efdb418b036b959cf1e" candidate_uuid: "d826a2588e5d44269d272bfb604b7d4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db231fb687ee46c5aa5754e285f53a5c" is_pre_election: true
I20240502 01:50:21.973534 5400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b272117917a04efdb418b036b959cf1e" candidate_uuid: "d826a2588e5d44269d272bfb604b7d4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913a0a493b5e4a329ab0da24c1bf5160" is_pre_election: true
I20240502 01:50:21.973723 5475 raft_consensus.cc:2454] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d826a2588e5d44269d272bfb604b7d4f in term 0.
I20240502 01:50:21.974128 5400 raft_consensus.cc:2454] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d826a2588e5d44269d272bfb604b7d4f in term 0.
I20240502 01:50:21.974804 5512 leader_election.cc:302] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d826a2588e5d44269d272bfb604b7d4f, db231fb687ee46c5aa5754e285f53a5c; no voters:
I20240502 01:50:21.975522 5604 raft_consensus.cc:2781] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:50:21.975852 5604 raft_consensus.cc:482] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:50:21.976078 5604 raft_consensus.cc:3037] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:21.981098 5604 raft_consensus.cc:504] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.982501 5604 leader_election.cc:288] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [CANDIDATE]: Term 1 election: Requested vote from peers 913a0a493b5e4a329ab0da24c1bf5160 (127.1.233.129:36223), db231fb687ee46c5aa5754e285f53a5c (127.1.233.130:37161)
I20240502 01:50:21.983403 5400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b272117917a04efdb418b036b959cf1e" candidate_uuid: "d826a2588e5d44269d272bfb604b7d4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913a0a493b5e4a329ab0da24c1bf5160"
I20240502 01:50:21.983562 5475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b272117917a04efdb418b036b959cf1e" candidate_uuid: "d826a2588e5d44269d272bfb604b7d4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db231fb687ee46c5aa5754e285f53a5c"
I20240502 01:50:21.984108 5400 raft_consensus.cc:3037] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:21.984213 5475 raft_consensus.cc:3037] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:21.989164 5400 raft_consensus.cc:2454] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d826a2588e5d44269d272bfb604b7d4f in term 1.
I20240502 01:50:21.989477 5475 raft_consensus.cc:2454] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d826a2588e5d44269d272bfb604b7d4f in term 1.
I20240502 01:50:21.990347 5514 leader_election.cc:302] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913a0a493b5e4a329ab0da24c1bf5160, d826a2588e5d44269d272bfb604b7d4f; no voters:
I20240502 01:50:21.991145 5604 raft_consensus.cc:2781] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:21.992281 5604 raft_consensus.cc:686] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [term 1 LEADER]: Becoming Leader. State: Replica: d826a2588e5d44269d272bfb604b7d4f, State: Running, Role: LEADER
I20240502 01:50:21.993054 5604 consensus_queue.cc:237] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:21.999651 5297 catalog_manager.cc:5484] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f reported cstate change: term changed from 0 to 1, leader changed from <none> to d826a2588e5d44269d272bfb604b7d4f (127.1.233.131). New cstate: current_term: 1 leader_uuid: "d826a2588e5d44269d272bfb604b7d4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } health_report { overall_health: HEALTHY } } }
I20240502 01:50:22.028268 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:22.034894 5622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:22.035984 5623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:22.037335 5625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:22.037990 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:22.039253 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:22.039451 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:22.039592 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614622039581 us; error 0 us; skew 500 ppm
I20240502 01:50:22.040081 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:22.042593 1958 webserver.cc:460] Webserver started at http://127.1.233.132:44349/ using document root <none> and password file <none>
I20240502 01:50:22.043015 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:22.043164 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:22.043380 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:22.044530 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-3-root/instance:
uuid: "ff920fd774714a5b8810c23ebdd61892"
format_stamp: "Formatted at 2024-05-02 01:50:22 on dist-test-slave-2090"
I20240502 01:50:22.048892 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:50:22.052091 5630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:22.052927 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:50:22.053231 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-3-root
uuid: "ff920fd774714a5b8810c23ebdd61892"
format_stamp: "Formatted at 2024-05-02 01:50:22 on dist-test-slave-2090"
I20240502 01:50:22.053500 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:22.069182 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:22.070470 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:22.072090 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:22.074664 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:22.074882 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:22.075124 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:22.075280 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:22.118121 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.132:38645
I20240502 01:50:22.118235 5692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.132:38645 every 8 connection(s)
I20240502 01:50:22.132611 5693 heartbeater.cc:347] Connected to a master server at 127.1.233.190:35229
I20240502 01:50:22.133025 5693 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:22.133812 5693 heartbeater.cc:511] Master 127.1.233.190:35229 requested a full tablet report, sending...
I20240502 01:50:22.135694 5297 ts_manager.cc:194] Registered new tserver with Master: ff920fd774714a5b8810c23ebdd61892 (127.1.233.132:38645)
I20240502 01:50:22.137341 5297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45306
I20240502 01:50:22.192082 5599 consensus_queue.cc:1035] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:50:22.198410 5615 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240502 01:50:22.198653 5615 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240502 01:50:22.203933 5600 consensus_queue.cc:1035] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:50:22.231536 5351 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:22.235532 5351 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:22.236017 5351 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:22.247301 5475 consensus_queue.cc:237] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } }
I20240502 01:50:22.256541 5548 raft_consensus.cc:1261] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Refusing update from remote peer db231fb687ee46c5aa5754e285f53a5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:22.257912 5599 consensus_queue.cc:1035] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:22.261234 5400 raft_consensus.cc:1261] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Refusing update from remote peer db231fb687ee46c5aa5754e285f53a5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:22.262602 5700 consensus_queue.cc:1035] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:50:22.267194 5441 consensus_peers.cc:485] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c -> Peer ff920fd774714a5b8810c23ebdd61892 (127.1.233.132:38645): Couldn't send request to peer ff920fd774714a5b8810c23ebdd61892. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47805d1b4cc44fcca988db5eba2e05c8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:22.267693 5548 raft_consensus.cc:2932] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } } }
I20240502 01:50:22.278026 5400 raft_consensus.cc:2932] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } } }
I20240502 01:50:22.280885 5297 catalog_manager.cc:5484] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f reported cstate change: config changed from index -1 to 2, NON_VOTER ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) added. New cstate: current_term: 1 leader_uuid: "db231fb687ee46c5aa5754e285f53a5c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } } }
I20240502 01:50:22.284469 5600 consensus_queue.cc:1035] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:50:22.306661 5604 consensus_queue.cc:1035] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:50:22.325389 5604 consensus_queue.cc:1035] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:50:22.624889 5709 ts_tablet_manager.cc:902] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892: Initiating tablet copy from peer db231fb687ee46c5aa5754e285f53a5c (127.1.233.130:37161)
I20240502 01:50:22.627924 5709 tablet_copy_client.cc:308] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892: tablet copy: Beginning tablet copy session from remote peer at address 127.1.233.130:37161
I20240502 01:50:22.637562 5485 tablet_copy_service.cc:140] P db231fb687ee46c5aa5754e285f53a5c: Received BeginTabletCopySession request for tablet 47805d1b4cc44fcca988db5eba2e05c8 from peer ff920fd774714a5b8810c23ebdd61892 ({username='slave'} at 127.0.0.1:54218)
I20240502 01:50:22.638191 5485 tablet_copy_service.cc:161] P db231fb687ee46c5aa5754e285f53a5c: Beginning new tablet copy session on tablet 47805d1b4cc44fcca988db5eba2e05c8 from peer ff920fd774714a5b8810c23ebdd61892 at {username='slave'} at 127.0.0.1:54218: session id = ff920fd774714a5b8810c23ebdd61892-47805d1b4cc44fcca988db5eba2e05c8
I20240502 01:50:22.645052 5485 tablet_copy_source_session.cc:205] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:22.647742 5709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47805d1b4cc44fcca988db5eba2e05c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:22.659308 5709 tablet_copy_client.cc:783] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:22.660096 5709 tablet_copy_client.cc:647] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:22.664113 5709 tablet_copy_client.cc:523] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:22.672634 5709 tablet_bootstrap.cc:492] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892: Bootstrap starting.
I20240502 01:50:22.685189 5709 tablet_bootstrap.cc:492] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:50:22.685959 5709 tablet_bootstrap.cc:492] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892: Bootstrap complete.
I20240502 01:50:22.686522 5709 ts_tablet_manager.cc:1367] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240502 01:50:22.688593 5709 raft_consensus.cc:348] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } }
I20240502 01:50:22.689608 5709 raft_consensus.cc:729] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ff920fd774714a5b8810c23ebdd61892, State: Initialized, Role: LEARNER
I20240502 01:50:22.690122 5709 consensus_queue.cc:260] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } }
I20240502 01:50:22.692236 5693 heartbeater.cc:503] Master 127.1.233.190:35229 was elected leader, sending a full tablet report...
I20240502 01:50:22.692435 5709 ts_tablet_manager.cc:1398] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20240502 01:50:22.694099 5485 tablet_copy_service.cc:342] P db231fb687ee46c5aa5754e285f53a5c: Request end of tablet copy session ff920fd774714a5b8810c23ebdd61892-47805d1b4cc44fcca988db5eba2e05c8 received from {username='slave'} at 127.0.0.1:54218
I20240502 01:50:22.694406 5485 tablet_copy_service.cc:433] P db231fb687ee46c5aa5754e285f53a5c: ending tablet copy session ff920fd774714a5b8810c23ebdd61892-47805d1b4cc44fcca988db5eba2e05c8 on tablet 47805d1b4cc44fcca988db5eba2e05c8 with peer ff920fd774714a5b8810c23ebdd61892
I20240502 01:50:22.752429 5700 raft_consensus.cc:2932] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } } }
I20240502 01:50:22.774194 5400 consensus_queue.cc:237] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } }
I20240502 01:50:22.780671 5475 raft_consensus.cc:1261] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Refusing update from remote peer 913a0a493b5e4a329ab0da24c1bf5160: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:22.780999 5548 raft_consensus.cc:1261] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Refusing update from remote peer 913a0a493b5e4a329ab0da24c1bf5160: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:22.782542 5600 consensus_queue.cc:1035] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:22.783516 5719 consensus_queue.cc:1035] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:22.792809 5548 raft_consensus.cc:2932] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } } }
W20240502 01:50:22.794135 5366 consensus_peers.cc:485] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 -> Peer ff920fd774714a5b8810c23ebdd61892 (127.1.233.132:38645): Couldn't send request to peer ff920fd774714a5b8810c23ebdd61892. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c54b599ba1ee4033b73975c3077abd6f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:22.794193 5475 raft_consensus.cc:2932] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } } }
I20240502 01:50:22.805536 5297 catalog_manager.cc:5484] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f reported cstate change: config changed from index -1 to 2, NON_VOTER ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) added. New cstate: current_term: 1 leader_uuid: "913a0a493b5e4a329ab0da24c1bf5160" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } } }
I20240502 01:50:22.864477 5614 raft_consensus.cc:2932] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } } }
I20240502 01:50:22.884240 5668 raft_consensus.cc:1203] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:22.885090 5668 raft_consensus.cc:2932] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892 [term 1 LEARNER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } } }
I20240502 01:50:23.087440 5709 ts_tablet_manager.cc:902] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892: Initiating tablet copy from peer 913a0a493b5e4a329ab0da24c1bf5160 (127.1.233.129:36223)
I20240502 01:50:23.092126 5709 tablet_copy_client.cc:308] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892: tablet copy: Beginning tablet copy session from remote peer at address 127.1.233.129:36223
I20240502 01:50:23.101440 5410 tablet_copy_service.cc:140] P 913a0a493b5e4a329ab0da24c1bf5160: Received BeginTabletCopySession request for tablet c54b599ba1ee4033b73975c3077abd6f from peer ff920fd774714a5b8810c23ebdd61892 ({username='slave'} at 127.0.0.1:46160)
I20240502 01:50:23.102108 5410 tablet_copy_service.cc:161] P 913a0a493b5e4a329ab0da24c1bf5160: Beginning new tablet copy session on tablet c54b599ba1ee4033b73975c3077abd6f from peer ff920fd774714a5b8810c23ebdd61892 at {username='slave'} at 127.0.0.1:46160: session id = ff920fd774714a5b8810c23ebdd61892-c54b599ba1ee4033b73975c3077abd6f
I20240502 01:50:23.109936 5410 tablet_copy_source_session.cc:205] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:23.112998 5709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54b599ba1ee4033b73975c3077abd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:23.126766 5709 tablet_copy_client.cc:783] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:23.127343 5709 tablet_copy_client.cc:647] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:23.130949 5709 tablet_copy_client.cc:523] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:23.138695 5709 tablet_bootstrap.cc:492] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892: Bootstrap starting.
I20240502 01:50:23.155215 5709 tablet_bootstrap.cc:492] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:23.156025 5709 tablet_bootstrap.cc:492] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892: Bootstrap complete.
I20240502 01:50:23.156626 5709 ts_tablet_manager.cc:1367] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.007s
I20240502 01:50:23.158998 5709 raft_consensus.cc:348] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } }
I20240502 01:50:23.159921 5709 raft_consensus.cc:729] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ff920fd774714a5b8810c23ebdd61892, State: Initialized, Role: LEARNER
I20240502 01:50:23.160583 5709 consensus_queue.cc:260] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: true } }
I20240502 01:50:23.162707 5709 ts_tablet_manager.cc:1398] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240502 01:50:23.164340 5410 tablet_copy_service.cc:342] P 913a0a493b5e4a329ab0da24c1bf5160: Request end of tablet copy session ff920fd774714a5b8810c23ebdd61892-c54b599ba1ee4033b73975c3077abd6f received from {username='slave'} at 127.0.0.1:46160
I20240502 01:50:23.164724 5410 tablet_copy_service.cc:433] P 913a0a493b5e4a329ab0da24c1bf5160: ending tablet copy session ff920fd774714a5b8810c23ebdd61892-c54b599ba1ee4033b73975c3077abd6f on tablet c54b599ba1ee4033b73975c3077abd6f with peer ff920fd774714a5b8810c23ebdd61892
I20240502 01:50:23.343827 5716 raft_consensus.cc:1050] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c: attempting to promote NON_VOTER ff920fd774714a5b8810c23ebdd61892 to VOTER
I20240502 01:50:23.346187 5716 consensus_queue.cc:237] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } }
I20240502 01:50:23.351337 5548 raft_consensus.cc:1261] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Refusing update from remote peer db231fb687ee46c5aa5754e285f53a5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:23.352483 5668 raft_consensus.cc:1261] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892 [term 1 LEARNER]: Refusing update from remote peer db231fb687ee46c5aa5754e285f53a5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:23.353461 5399 raft_consensus.cc:1261] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Refusing update from remote peer db231fb687ee46c5aa5754e285f53a5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:23.353672 5667 raft_consensus.cc:1203] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:23.353484 5716 consensus_queue.cc:1035] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:50:23.355626 5700 consensus_queue.cc:1035] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:23.357853 5716 consensus_queue.cc:1035] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:23.367302 5548 raft_consensus.cc:2932] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:23.368722 5399 raft_consensus.cc:2932] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:23.380906 5298 catalog_manager.cc:5484] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f reported cstate change: config changed from index 2 to 3, ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db231fb687ee46c5aa5754e285f53a5c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:23.492435 5700 raft_consensus.cc:2932] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:23.501613 5667 raft_consensus.cc:2932] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } attrs { replace: true } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:23.690378 5475 consensus_queue.cc:237] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } }
I20240502 01:50:23.695015 5667 raft_consensus.cc:1261] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892 [term 1 FOLLOWER]: Refusing update from remote peer db231fb687ee46c5aa5754e285f53a5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:23.695397 5400 raft_consensus.cc:1261] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Refusing update from remote peer db231fb687ee46c5aa5754e285f53a5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:23.696270 5615 consensus_queue.cc:1035] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:23.697137 5734 consensus_queue.cc:1035] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:23.705380 5399 raft_consensus.cc:2932] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d826a2588e5d44269d272bfb604b7d4f (127.1.233.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:23.705690 5667 raft_consensus.cc:2932] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d826a2588e5d44269d272bfb604b7d4f (127.1.233.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:23.722065 5295 catalog_manager.cc:5484] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 reported cstate change: config changed from index 3 to 4, VOTER d826a2588e5d44269d272bfb604b7d4f (127.1.233.131) evicted. New cstate: current_term: 1 leader_uuid: "db231fb687ee46c5aa5754e285f53a5c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:23.888721 5614 raft_consensus.cc:1050] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160: attempting to promote NON_VOTER ff920fd774714a5b8810c23ebdd61892 to VOTER
I20240502 01:50:23.891247 5614 consensus_queue.cc:237] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } }
I20240502 01:50:23.897609 5667 raft_consensus.cc:1261] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892 [term 1 LEARNER]: Refusing update from remote peer 913a0a493b5e4a329ab0da24c1bf5160: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:23.897662 5548 raft_consensus.cc:1261] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Refusing update from remote peer 913a0a493b5e4a329ab0da24c1bf5160: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:23.899395 5723 consensus_queue.cc:1035] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:50:23.899853 5474 raft_consensus.cc:1261] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Refusing update from remote peer 913a0a493b5e4a329ab0da24c1bf5160: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:23.900607 5614 consensus_queue.cc:1035] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:23.901868 5600 consensus_queue.cc:1035] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:23.918888 5548 raft_consensus.cc:2932] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:23.918884 5667 raft_consensus.cc:2932] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:23.932062 5296 catalog_manager.cc:5484] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f reported cstate change: config changed from index 2 to 3, ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "913a0a493b5e4a329ab0da24c1bf5160" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:24.022012 5539 tablet_service.cc:1514] Processing DeleteTablet for tablet 47805d1b4cc44fcca988db5eba2e05c8 with delete_type TABLET_DATA_TOMBSTONED (TS d826a2588e5d44269d272bfb604b7d4f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58866
I20240502 01:50:24.028494 5740 tablet_replica.cc:330] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f: stopping tablet replica
I20240502 01:50:24.029345 5740 raft_consensus.cc:2229] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:24.030189 5740 raft_consensus.cc:2258] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:24.033059 5740 ts_tablet_manager.cc:1875] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:24.038610 5734 raft_consensus.cc:2932] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d826a2588e5d44269d272bfb604b7d4f (127.1.233.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:24.048185 5740 ts_tablet_manager.cc:1888] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:24.048743 5740 log.cc:1196] T 47805d1b4cc44fcca988db5eba2e05c8 P d826a2588e5d44269d272bfb604b7d4f: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-2-root/wals/47805d1b4cc44fcca988db5eba2e05c8
I20240502 01:50:24.049760 5282 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 47805d1b4cc44fcca988db5eba2e05c8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:24.052541 5284 catalog_manager.cc:4830] TS d826a2588e5d44269d272bfb604b7d4f (127.1.233.131:41707): tablet 47805d1b4cc44fcca988db5eba2e05c8 (table test-workload [id=b073301217d34649a158c49039ea96a3]) successfully deleted
I20240502 01:50:24.073163 5723 raft_consensus.cc:2932] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:24.082610 5475 raft_consensus.cc:2932] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ff920fd774714a5b8810c23ebdd61892 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db231fb687ee46c5aa5754e285f53a5c" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 37161 } attrs { replace: true } } peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:24.237386 5351 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:24.240309 5351 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:24.240845 5351 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:24.288767 5399 consensus_queue.cc:237] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } }
I20240502 01:50:24.293483 5667 raft_consensus.cc:1261] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892 [term 1 FOLLOWER]: Refusing update from remote peer 913a0a493b5e4a329ab0da24c1bf5160: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:24.293917 5548 raft_consensus.cc:1261] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Refusing update from remote peer 913a0a493b5e4a329ab0da24c1bf5160: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:24.294664 5741 consensus_queue.cc:1035] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:24.296401 5741 consensus_queue.cc:1035] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:24.302078 5667 raft_consensus.cc:2932] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db231fb687ee46c5aa5754e285f53a5c (127.1.233.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:24.304255 5548 raft_consensus.cc:2932] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db231fb687ee46c5aa5754e285f53a5c (127.1.233.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:24.313220 5295 catalog_manager.cc:5484] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892 reported cstate change: config changed from index 3 to 4, VOTER db231fb687ee46c5aa5754e285f53a5c (127.1.233.130) evicted. New cstate: current_term: 1 leader_uuid: "913a0a493b5e4a329ab0da24c1bf5160" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:24.361248 5464 tablet_service.cc:1514] Processing DeleteTablet for tablet c54b599ba1ee4033b73975c3077abd6f with delete_type TABLET_DATA_TOMBSTONED (TS db231fb687ee46c5aa5754e285f53a5c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54184
I20240502 01:50:24.363416 5742 tablet_replica.cc:330] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c: stopping tablet replica
I20240502 01:50:24.364241 5742 raft_consensus.cc:2229] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:24.364812 5742 raft_consensus.cc:2258] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:24.367321 5742 ts_tablet_manager.cc:1875] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:50:24.381197 5742 ts_tablet_manager.cc:1888] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:50:24.381513 5742 log.cc:1196] T c54b599ba1ee4033b73975c3077abd6f P db231fb687ee46c5aa5754e285f53a5c: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714614515033618-1958-0/minicluster-data/ts-1-root/wals/c54b599ba1ee4033b73975c3077abd6f
I20240502 01:50:24.382972 5282 catalog_manager.cc:4830] TS db231fb687ee46c5aa5754e285f53a5c (127.1.233.130:37161): tablet c54b599ba1ee4033b73975c3077abd6f (table test-workload [id=b073301217d34649a158c49039ea96a3]) successfully deleted
I20240502 01:50:24.712457 5741 raft_consensus.cc:2932] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db231fb687ee46c5aa5754e285f53a5c (127.1.233.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "913a0a493b5e4a329ab0da24c1bf5160" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36223 } } peers { permanent_uuid: "d826a2588e5d44269d272bfb604b7d4f" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 41707 } } peers { permanent_uuid: "ff920fd774714a5b8810c23ebdd61892" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 38645 } attrs { promote: false } } }
I20240502 01:50:24.719794 5284 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c54b599ba1ee4033b73975c3077abd6f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:26.241925 5351 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:26.244874 5351 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:26.245317 5351 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:28.246306 5351 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:28.248931 5351 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:28.249362 5351 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:30.250458 5351 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:30.253221 5351 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:30.253620 5351 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:31.187609 1958 tablet_server.cc:178] TabletServer@127.1.233.129:0 shutting down...
I20240502 01:50:31.215390 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:31.216120 1958 tablet_replica.cc:330] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160: stopping tablet replica
I20240502 01:50:31.216989 1958 raft_consensus.cc:2229] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:31.217576 1958 raft_consensus.cc:2258] T b272117917a04efdb418b036b959cf1e P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:31.220099 1958 tablet_replica.cc:330] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160: stopping tablet replica
I20240502 01:50:31.220687 1958 raft_consensus.cc:2229] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:31.221802 1958 raft_consensus.cc:2258] T 47805d1b4cc44fcca988db5eba2e05c8 P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:31.224021 1958 tablet_replica.cc:330] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160: stopping tablet replica
I20240502 01:50:31.224524 1958 raft_consensus.cc:2229] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:31.225514 1958 raft_consensus.cc:2258] T c54b599ba1ee4033b73975c3077abd6f P 913a0a493b5e4a329ab0da24c1bf5160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:31.250828 1958 tablet_server.cc:194] TabletServer@127.1.233.129:0 shutdown complete.
I20240502 01:50:31.266767 1958 tablet_server.cc:178] TabletServer@127.1.233.130:0 shutting down...
I20240502 01:50:31.291550 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:31.292268 1958 tablet_replica.cc:330] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c: stopping tablet replica
I20240502 01:50:31.292846 1958 raft_consensus.cc:2229] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:31.293318 1958 raft_consensus.cc:2258] T b272117917a04efdb418b036b959cf1e P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:31.295372 1958 tablet_replica.cc:330] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c: stopping tablet replica
I20240502 01:50:31.295800 1958 raft_consensus.cc:2229] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:31.296682 1958 raft_consensus.cc:2258] T 47805d1b4cc44fcca988db5eba2e05c8 P db231fb687ee46c5aa5754e285f53a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:31.320930 1958 tablet_server.cc:194] TabletServer@127.1.233.130:0 shutdown complete.
I20240502 01:50:31.336277 1958 tablet_server.cc:178] TabletServer@127.1.233.131:0 shutting down...
I20240502 01:50:31.363831 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:31.364547 1958 tablet_replica.cc:330] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f: stopping tablet replica
I20240502 01:50:31.365177 1958 raft_consensus.cc:2229] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:31.366164 1958 raft_consensus.cc:2258] T b272117917a04efdb418b036b959cf1e P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:31.368515 1958 tablet_replica.cc:330] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f: stopping tablet replica
I20240502 01:50:31.369006 1958 raft_consensus.cc:2229] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:31.369567 1958 raft_consensus.cc:2258] T c54b599ba1ee4033b73975c3077abd6f P d826a2588e5d44269d272bfb604b7d4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:31.392436 1958 tablet_server.cc:194] TabletServer@127.1.233.131:0 shutdown complete.
I20240502 01:50:31.406877 1958 tablet_server.cc:178] TabletServer@127.1.233.132:0 shutting down...
I20240502 01:50:31.438480 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:31.439229 1958 tablet_replica.cc:330] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892: stopping tablet replica
I20240502 01:50:31.439812 1958 raft_consensus.cc:2229] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:31.440601 1958 raft_consensus.cc:2258] T c54b599ba1ee4033b73975c3077abd6f P ff920fd774714a5b8810c23ebdd61892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:31.442687 1958 tablet_replica.cc:330] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892: stopping tablet replica
I20240502 01:50:31.443158 1958 raft_consensus.cc:2229] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:31.443753 1958 raft_consensus.cc:2258] T 47805d1b4cc44fcca988db5eba2e05c8 P ff920fd774714a5b8810c23ebdd61892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:31.464255 1958 tablet_server.cc:194] TabletServer@127.1.233.132:0 shutdown complete.
I20240502 01:50:31.476362 1958 master.cc:537] Master@127.1.233.190:35229 shutting down...
I20240502 01:50:31.496603 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:31.497211 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:31.497516 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8c00149b16dc42c283ec7c2f4f51b88d: stopping tablet replica
I20240502 01:50:31.515733 1958 master.cc:552] Master@127.1.233.190:35229 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (11461 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20240502 01:50:31.557934 1958 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.233.190:39593
I20240502 01:50:31.559141 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:31.565425 5747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:31.566095 5748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:31.566854 5750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:31.568030 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:31.568853 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:31.569043 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:31.569196 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614631569178 us; error 0 us; skew 500 ppm
I20240502 01:50:31.569697 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:31.577986 1958 webserver.cc:460] Webserver started at http://127.1.233.190:46731/ using document root <none> and password file <none>
I20240502 01:50:31.578514 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:31.578697 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:31.578987 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:31.580355 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/master-0-root/instance:
uuid: "ec136095488f40cb9d8ed68d084d6c9c"
format_stamp: "Formatted at 2024-05-02 01:50:31 on dist-test-slave-2090"
I20240502 01:50:31.585027 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20240502 01:50:31.588419 5755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:31.589169 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:50:31.589450 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/master-0-root
uuid: "ec136095488f40cb9d8ed68d084d6c9c"
format_stamp: "Formatted at 2024-05-02 01:50:31 on dist-test-slave-2090"
I20240502 01:50:31.589737 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:31.615559 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:31.616931 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:31.655201 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.190:39593
I20240502 01:50:31.655299 5806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.190:39593 every 8 connection(s)
I20240502 01:50:31.659417 5807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:31.671430 5807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c: Bootstrap starting.
I20240502 01:50:31.675976 5807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:31.680682 5807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c: No bootstrap required, opened a new log
I20240502 01:50:31.682819 5807 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec136095488f40cb9d8ed68d084d6c9c" member_type: VOTER }
I20240502 01:50:31.683290 5807 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:31.683455 5807 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec136095488f40cb9d8ed68d084d6c9c, State: Initialized, Role: FOLLOWER
I20240502 01:50:31.683961 5807 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec136095488f40cb9d8ed68d084d6c9c" member_type: VOTER }
I20240502 01:50:31.684433 5807 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:50:31.684623 5807 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:50:31.684854 5807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:31.689695 5807 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec136095488f40cb9d8ed68d084d6c9c" member_type: VOTER }
I20240502 01:50:31.690378 5807 leader_election.cc:302] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec136095488f40cb9d8ed68d084d6c9c; no voters:
I20240502 01:50:31.691649 5807 leader_election.cc:288] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:50:31.692095 5810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:31.693518 5810 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [term 1 LEADER]: Becoming Leader. State: Replica: ec136095488f40cb9d8ed68d084d6c9c, State: Running, Role: LEADER
I20240502 01:50:31.694123 5810 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec136095488f40cb9d8ed68d084d6c9c" member_type: VOTER }
I20240502 01:50:31.694720 5807 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:50:31.697232 5812 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec136095488f40cb9d8ed68d084d6c9c. Latest consensus state: current_term: 1 leader_uuid: "ec136095488f40cb9d8ed68d084d6c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec136095488f40cb9d8ed68d084d6c9c" member_type: VOTER } }
I20240502 01:50:31.697178 5811 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec136095488f40cb9d8ed68d084d6c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec136095488f40cb9d8ed68d084d6c9c" member_type: VOTER } }
I20240502 01:50:31.698122 5812 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [sys.catalog]: This master's current role is: LEADER
I20240502 01:50:31.698199 5811 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [sys.catalog]: This master's current role is: LEADER
I20240502 01:50:31.701918 5815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:50:31.706707 5815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:50:31.709865 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:50:31.716404 5815 catalog_manager.cc:1347] Generated new cluster ID: e1e79be2a9a748d4a4eafe01ad570f83
I20240502 01:50:31.716704 5815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:50:31.729987 5815 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:50:31.731357 5815 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:50:31.747591 5815 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c: Generated new TSK 0
I20240502 01:50:31.748250 5815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:50:31.776814 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:31.783439 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:31.785317 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:31.786041 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:31.786572 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:31.787354 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:31.787557 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:31.787701 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614631787687 us; error 0 us; skew 500 ppm
I20240502 01:50:31.788273 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:31.790848 1958 webserver.cc:460] Webserver started at http://127.1.233.129:39921/ using document root <none> and password file <none>
I20240502 01:50:31.791349 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:31.791532 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:31.791796 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:31.793013 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-0-root/instance:
uuid: "745b7571454f42a9bf74e97c1739c7e1"
format_stamp: "Formatted at 2024-05-02 01:50:31 on dist-test-slave-2090"
I20240502 01:50:31.797869 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:50:31.801282 5836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:31.802064 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240502 01:50:31.802342 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "745b7571454f42a9bf74e97c1739c7e1"
format_stamp: "Formatted at 2024-05-02 01:50:31 on dist-test-slave-2090"
I20240502 01:50:31.802621 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:31.817543 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:31.818717 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:31.820355 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:31.822741 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:31.822934 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:31.823172 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:31.823339 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:31.863613 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:40253
I20240502 01:50:31.863719 5898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:40253 every 8 connection(s)
I20240502 01:50:31.868492 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:31.876675 5903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:31.877408 5904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:31.879467 5906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:31.880225 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:31.881242 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:31.881531 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:31.881748 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614631881729 us; error 0 us; skew 500 ppm
I20240502 01:50:31.882542 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:31.882778 5899 heartbeater.cc:347] Connected to a master server at 127.1.233.190:39593
I20240502 01:50:31.883131 5899 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:31.884020 5899 heartbeater.cc:511] Master 127.1.233.190:39593 requested a full tablet report, sending...
I20240502 01:50:31.885319 1958 webserver.cc:460] Webserver started at http://127.1.233.130:44457/ using document root <none> and password file <none>
I20240502 01:50:31.885998 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:31.886233 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:31.886572 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:31.886464 5772 ts_manager.cc:194] Registered new tserver with Master: 745b7571454f42a9bf74e97c1739c7e1 (127.1.233.129:40253)
I20240502 01:50:31.888366 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-1-root/instance:
uuid: "cc1b2619e8b64cac84ef717d22621eec"
format_stamp: "Formatted at 2024-05-02 01:50:31 on dist-test-slave-2090"
I20240502 01:50:31.889142 5772 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58570
I20240502 01:50:31.893788 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:50:31.897701 5911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:31.898466 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240502 01:50:31.898716 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-1-root
uuid: "cc1b2619e8b64cac84ef717d22621eec"
format_stamp: "Formatted at 2024-05-02 01:50:31 on dist-test-slave-2090"
I20240502 01:50:31.898994 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:31.912123 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:31.913393 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:31.914798 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:31.917089 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:31.917282 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:31.917536 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:31.917671 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:31.956779 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.130:33669
I20240502 01:50:31.956873 5973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.130:33669 every 8 connection(s)
I20240502 01:50:31.961827 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:31.969686 5977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:31.970304 5978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:31.972465 5980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:31.973376 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:31.974514 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:31.974858 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:31.974951 5974 heartbeater.cc:347] Connected to a master server at 127.1.233.190:39593
I20240502 01:50:31.975071 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614631975048 us; error 0 us; skew 500 ppm
I20240502 01:50:31.975440 5974 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:31.975775 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:31.976259 5974 heartbeater.cc:511] Master 127.1.233.190:39593 requested a full tablet report, sending...
I20240502 01:50:31.978562 5772 ts_manager.cc:194] Registered new tserver with Master: cc1b2619e8b64cac84ef717d22621eec (127.1.233.130:33669)
I20240502 01:50:31.978711 1958 webserver.cc:460] Webserver started at http://127.1.233.131:33957/ using document root <none> and password file <none>
I20240502 01:50:31.979384 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:31.979624 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:31.979952 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:31.980562 5772 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58582
I20240502 01:50:31.981703 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-2-root/instance:
uuid: "225b8eb08b8741808f8eddb0d3e02cda"
format_stamp: "Formatted at 2024-05-02 01:50:31 on dist-test-slave-2090"
I20240502 01:50:31.986465 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 01:50:31.989857 5985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:31.990809 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:50:31.991118 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-2-root
uuid: "225b8eb08b8741808f8eddb0d3e02cda"
format_stamp: "Formatted at 2024-05-02 01:50:31 on dist-test-slave-2090"
I20240502 01:50:31.991408 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714614515033618-1958-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:32.038879 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:32.040062 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:32.041662 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:32.043907 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:32.044086 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:32.044328 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:32.044456 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:32.082641 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.131:33579
I20240502 01:50:32.082741 6047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.131:33579 every 8 connection(s)
I20240502 01:50:32.096510 6048 heartbeater.cc:347] Connected to a master server at 127.1.233.190:39593
I20240502 01:50:32.096904 6048 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:32.097616 6048 heartbeater.cc:511] Master 127.1.233.190:39593 requested a full tablet report, sending...
I20240502 01:50:32.099411 5772 ts_manager.cc:194] Registered new tserver with Master: 225b8eb08b8741808f8eddb0d3e02cda (127.1.233.131:33579)
I20240502 01:50:32.099982 1958 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014365121s
I20240502 01:50:32.100941 5772 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58598
I20240502 01:50:32.892290 5899 heartbeater.cc:503] Master 127.1.233.190:39593 was elected leader, sending a full tablet report...
I20240502 01:50:32.983302 5974 heartbeater.cc:503] Master 127.1.233.190:39593 was elected leader, sending a full tablet report...
I20240502 01:50:33.103732 6048 heartbeater.cc:503] Master 127.1.233.190:39593 was elected leader, sending a full tablet report...
I20240502 01:50:33.133455 1958 test_util.cc:268] Using random seed: -735939224
I20240502 01:50:33.157053 5772 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58614:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:50:33.159021 5772 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:50:33.203011 6013 tablet_service.cc:1467] Processing CreateTablet for tablet f5a842c52acd49a7afa9eea18a1e8d46 (DEFAULT_TABLE table=test-workload [id=a5c6599686d949bf9a0a70a547eb15d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:50:33.204445 6013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a842c52acd49a7afa9eea18a1e8d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:33.206929 5864 tablet_service.cc:1467] Processing CreateTablet for tablet f5a842c52acd49a7afa9eea18a1e8d46 (DEFAULT_TABLE table=test-workload [id=a5c6599686d949bf9a0a70a547eb15d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:50:33.207405 5939 tablet_service.cc:1467] Processing CreateTablet for tablet f5a842c52acd49a7afa9eea18a1e8d46 (DEFAULT_TABLE table=test-workload [id=a5c6599686d949bf9a0a70a547eb15d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:50:33.207911 5864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a842c52acd49a7afa9eea18a1e8d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:33.208644 5939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5a842c52acd49a7afa9eea18a1e8d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:33.224536 6069 tablet_bootstrap.cc:492] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda: Bootstrap starting.
I20240502 01:50:33.229542 6070 tablet_bootstrap.cc:492] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec: Bootstrap starting.
I20240502 01:50:33.231045 6068 tablet_bootstrap.cc:492] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1: Bootstrap starting.
I20240502 01:50:33.232299 6069 tablet_bootstrap.cc:654] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:33.235559 6070 tablet_bootstrap.cc:654] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:33.237406 6068 tablet_bootstrap.cc:654] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:33.238072 6069 tablet_bootstrap.cc:492] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda: No bootstrap required, opened a new log
I20240502 01:50:33.238642 6069 ts_tablet_manager.cc:1367] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.005s
I20240502 01:50:33.240078 6070 tablet_bootstrap.cc:492] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec: No bootstrap required, opened a new log
I20240502 01:50:33.240727 6070 ts_tablet_manager.cc:1367] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20240502 01:50:33.241564 6069 raft_consensus.cc:348] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } }
I20240502 01:50:33.242446 6069 raft_consensus.cc:374] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:33.242686 6068 tablet_bootstrap.cc:492] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1: No bootstrap required, opened a new log
I20240502 01:50:33.242801 6069 raft_consensus.cc:729] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225b8eb08b8741808f8eddb0d3e02cda, State: Initialized, Role: FOLLOWER
I20240502 01:50:33.243212 6068 ts_tablet_manager.cc:1367] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240502 01:50:33.242883 6070 raft_consensus.cc:348] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } }
I20240502 01:50:33.243765 6070 raft_consensus.cc:374] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:33.244077 6070 raft_consensus.cc:729] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc1b2619e8b64cac84ef717d22621eec, State: Initialized, Role: FOLLOWER
I20240502 01:50:33.243763 6069 consensus_queue.cc:260] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } }
I20240502 01:50:33.245015 6070 consensus_queue.cc:260] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } }
I20240502 01:50:33.246044 6068 raft_consensus.cc:348] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } }
I20240502 01:50:33.246954 6068 raft_consensus.cc:374] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:33.247385 6068 raft_consensus.cc:729] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745b7571454f42a9bf74e97c1739c7e1, State: Initialized, Role: FOLLOWER
I20240502 01:50:33.248082 6070 ts_tablet_manager.cc:1398] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240502 01:50:33.248467 6068 consensus_queue.cc:260] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } }
I20240502 01:50:33.252358 6068 ts_tablet_manager.cc:1398] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240502 01:50:33.255136 6069 ts_tablet_manager.cc:1398] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
W20240502 01:50:33.338047 6049 tablet.cc:2369] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:50:33.353192 6076 raft_consensus.cc:482] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:50:33.353706 6076 raft_consensus.cc:504] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } }
I20240502 01:50:33.356261 6076 leader_election.cc:288] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 225b8eb08b8741808f8eddb0d3e02cda (127.1.233.131:33579), cc1b2619e8b64cac84ef717d22621eec (127.1.233.130:33669)
I20240502 01:50:33.367488 6023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a842c52acd49a7afa9eea18a1e8d46" candidate_uuid: "745b7571454f42a9bf74e97c1739c7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "225b8eb08b8741808f8eddb0d3e02cda" is_pre_election: true
I20240502 01:50:33.367614 5949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a842c52acd49a7afa9eea18a1e8d46" candidate_uuid: "745b7571454f42a9bf74e97c1739c7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1b2619e8b64cac84ef717d22621eec" is_pre_election: true
I20240502 01:50:33.368289 6023 raft_consensus.cc:2454] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745b7571454f42a9bf74e97c1739c7e1 in term 0.
I20240502 01:50:33.368347 5949 raft_consensus.cc:2454] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745b7571454f42a9bf74e97c1739c7e1 in term 0.
I20240502 01:50:33.369578 5837 leader_election.cc:302] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225b8eb08b8741808f8eddb0d3e02cda, 745b7571454f42a9bf74e97c1739c7e1; no voters:
I20240502 01:50:33.370434 6076 raft_consensus.cc:2781] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:50:33.370709 6076 raft_consensus.cc:482] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:50:33.370976 6076 raft_consensus.cc:3037] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:33.376403 6076 raft_consensus.cc:504] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } }
I20240502 01:50:33.377918 6076 leader_election.cc:288] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [CANDIDATE]: Term 1 election: Requested vote from peers 225b8eb08b8741808f8eddb0d3e02cda (127.1.233.131:33579), cc1b2619e8b64cac84ef717d22621eec (127.1.233.130:33669)
I20240502 01:50:33.378887 6023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a842c52acd49a7afa9eea18a1e8d46" candidate_uuid: "745b7571454f42a9bf74e97c1739c7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "225b8eb08b8741808f8eddb0d3e02cda"
I20240502 01:50:33.378887 5949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a842c52acd49a7afa9eea18a1e8d46" candidate_uuid: "745b7571454f42a9bf74e97c1739c7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1b2619e8b64cac84ef717d22621eec"
I20240502 01:50:33.379706 6023 raft_consensus.cc:3037] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:33.379772 5949 raft_consensus.cc:3037] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:33.384913 6023 raft_consensus.cc:2454] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745b7571454f42a9bf74e97c1739c7e1 in term 1.
I20240502 01:50:33.385012 5949 raft_consensus.cc:2454] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745b7571454f42a9bf74e97c1739c7e1 in term 1.
I20240502 01:50:33.385993 5837 leader_election.cc:302] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225b8eb08b8741808f8eddb0d3e02cda, 745b7571454f42a9bf74e97c1739c7e1; no voters:
I20240502 01:50:33.386656 6076 raft_consensus.cc:2781] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:33.387676 6076 raft_consensus.cc:686] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 1 LEADER]: Becoming Leader. State: Replica: 745b7571454f42a9bf74e97c1739c7e1, State: Running, Role: LEADER
I20240502 01:50:33.388330 6076 consensus_queue.cc:237] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } }
I20240502 01:50:33.395256 5772 catalog_manager.cc:5484] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 745b7571454f42a9bf74e97c1739c7e1 (127.1.233.129). New cstate: current_term: 1 leader_uuid: "745b7571454f42a9bf74e97c1739c7e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } health_report { overall_health: HEALTHY } } }
I20240502 01:50:33.442454 1958 tablet_server.cc:178] TabletServer@127.1.233.129:0 shutting down...
I20240502 01:50:33.466590 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:33.467542 1958 tablet_replica.cc:330] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1: stopping tablet replica
I20240502 01:50:33.468098 1958 raft_consensus.cc:2229] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:33.468696 1958 pending_rounds.cc:62] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1: Trying to abort 1 pending ops.
I20240502 01:50:33.468917 1958 pending_rounds.cc:68] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023061538365165568 op_type: NO_OP noop_request { }
I20240502 01:50:33.469260 1958 raft_consensus.cc:2866] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 01:50:33.469506 1958 raft_consensus.cc:2258] T f5a842c52acd49a7afa9eea18a1e8d46 P 745b7571454f42a9bf74e97c1739c7e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:33.484874 1958 tablet_server.cc:194] TabletServer@127.1.233.129:0 shutdown complete.
I20240502 01:50:33.709544 5825 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:33.711002 5825 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:33.711371 5825 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:35.083256 6081 raft_consensus.cc:482] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:50:35.083729 6081 raft_consensus.cc:504] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } }
I20240502 01:50:35.085633 6081 leader_election.cc:288] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cc1b2619e8b64cac84ef717d22621eec (127.1.233.130:33669), 745b7571454f42a9bf74e97c1739c7e1 (127.1.233.129:40253)
W20240502 01:50:35.089463 5986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.233.129:40253: connect: Connection refused (error 111)
W20240502 01:50:35.092898 5986 leader_election.cc:334] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 745b7571454f42a9bf74e97c1739c7e1 (127.1.233.129:40253): Network error: Client connection negotiation failed: client connection to 127.1.233.129:40253: connect: Connection refused (error 111)
I20240502 01:50:35.095796 5949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a842c52acd49a7afa9eea18a1e8d46" candidate_uuid: "225b8eb08b8741808f8eddb0d3e02cda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1b2619e8b64cac84ef717d22621eec" is_pre_election: true
I20240502 01:50:35.096452 5949 raft_consensus.cc:2454] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 225b8eb08b8741808f8eddb0d3e02cda in term 1.
I20240502 01:50:35.097416 5987 leader_election.cc:302] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 225b8eb08b8741808f8eddb0d3e02cda, cc1b2619e8b64cac84ef717d22621eec; no voters: 745b7571454f42a9bf74e97c1739c7e1
I20240502 01:50:35.098033 6081 raft_consensus.cc:2781] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:50:35.098268 6081 raft_consensus.cc:482] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:50:35.098462 6081 raft_consensus.cc:3037] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:50:35.103283 6081 raft_consensus.cc:504] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } }
I20240502 01:50:35.104761 6081 leader_election.cc:288] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [CANDIDATE]: Term 2 election: Requested vote from peers cc1b2619e8b64cac84ef717d22621eec (127.1.233.130:33669), 745b7571454f42a9bf74e97c1739c7e1 (127.1.233.129:40253)
I20240502 01:50:35.105603 5949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5a842c52acd49a7afa9eea18a1e8d46" candidate_uuid: "225b8eb08b8741808f8eddb0d3e02cda" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1b2619e8b64cac84ef717d22621eec"
I20240502 01:50:35.106094 5949 raft_consensus.cc:3037] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:50:35.108153 5986 leader_election.cc:334] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 745b7571454f42a9bf74e97c1739c7e1 (127.1.233.129:40253): Network error: Client connection negotiation failed: client connection to 127.1.233.129:40253: connect: Connection refused (error 111)
I20240502 01:50:35.111068 5949 raft_consensus.cc:2454] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 225b8eb08b8741808f8eddb0d3e02cda in term 2.
I20240502 01:50:35.112004 5987 leader_election.cc:302] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 225b8eb08b8741808f8eddb0d3e02cda, cc1b2619e8b64cac84ef717d22621eec; no voters: 745b7571454f42a9bf74e97c1739c7e1
I20240502 01:50:35.112673 6081 raft_consensus.cc:2781] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:50:35.113694 6081 raft_consensus.cc:686] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 2 LEADER]: Becoming Leader. State: Replica: 225b8eb08b8741808f8eddb0d3e02cda, State: Running, Role: LEADER
I20240502 01:50:35.114288 6081 consensus_queue.cc:237] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } }
I20240502 01:50:35.120864 5772 catalog_manager.cc:5484] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda reported cstate change: term changed from 1 to 2, leader changed from 745b7571454f42a9bf74e97c1739c7e1 (127.1.233.129) to 225b8eb08b8741808f8eddb0d3e02cda (127.1.233.131). New cstate: current_term: 2 leader_uuid: "225b8eb08b8741808f8eddb0d3e02cda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225b8eb08b8741808f8eddb0d3e02cda" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745b7571454f42a9bf74e97c1739c7e1" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 40253 } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:35.517843 5949 raft_consensus.cc:1261] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec [term 2 FOLLOWER]: Refusing update from remote peer 225b8eb08b8741808f8eddb0d3e02cda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 01:50:35.519239 6081 consensus_queue.cc:1035] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc1b2619e8b64cac84ef717d22621eec" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 33669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:50:35.535909 5986 consensus_peers.cc:485] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda -> Peer 745b7571454f42a9bf74e97c1739c7e1 (127.1.233.129:40253): Couldn't send request to peer 745b7571454f42a9bf74e97c1739c7e1. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:40253: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:35.712591 5825 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:35.713992 5825 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:35.714332 5825 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:37.715533 5825 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:37.717053 5825 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:37.717321 5825 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 01:50:38.009650 5986 consensus_peers.cc:485] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda -> Peer 745b7571454f42a9bf74e97c1739c7e1 (127.1.233.129:40253): Couldn't send request to peer 745b7571454f42a9bf74e97c1739c7e1. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:40253: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:50:39.718410 5825 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:39.719683 5825 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:39.720060 5825 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 01:50:40.576988 5986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.233.129:40253: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 01:50:40.580118 5986 consensus_peers.cc:485] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda -> Peer 745b7571454f42a9bf74e97c1739c7e1 (127.1.233.129:40253): Couldn't send request to peer 745b7571454f42a9bf74e97c1739c7e1. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:40253: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:50:41.721191 5825 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:41.722726 5825 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:41.723039 5825 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 01:50:42.843043 5986 consensus_peers.cc:485] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda -> Peer 745b7571454f42a9bf74e97c1739c7e1 (127.1.233.129:40253): Couldn't send request to peer 745b7571454f42a9bf74e97c1739c7e1. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:40253: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:50:43.724277 5825 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 01:50:43.725808 5825 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 01:50:43.726109 5825 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 01:50:44.558638 1958 tablet_server.cc:178] TabletServer@127.1.233.130:0 shutting down...
I20240502 01:50:44.589643 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:44.590539 1958 tablet_replica.cc:330] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec: stopping tablet replica
I20240502 01:50:44.591287 1958 raft_consensus.cc:2229] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:44.592084 1958 raft_consensus.cc:2258] T f5a842c52acd49a7afa9eea18a1e8d46 P cc1b2619e8b64cac84ef717d22621eec [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:44.614512 1958 tablet_server.cc:194] TabletServer@127.1.233.130:0 shutdown complete.
W20240502 01:50:44.622437 5987 consensus_peers.cc:485] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda -> Peer cc1b2619e8b64cac84ef717d22621eec (127.1.233.130:33669): Couldn't send request to peer cc1b2619e8b64cac84ef717d22621eec. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.130:33669: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:44.630712 1958 tablet_server.cc:178] TabletServer@127.1.233.131:0 shutting down...
I20240502 01:50:44.656585 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:44.657332 1958 tablet_replica.cc:330] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda: stopping tablet replica
I20240502 01:50:44.657896 1958 raft_consensus.cc:2229] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:50:44.658890 1958 raft_consensus.cc:2258] T f5a842c52acd49a7afa9eea18a1e8d46 P 225b8eb08b8741808f8eddb0d3e02cda [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:44.681692 1958 tablet_server.cc:194] TabletServer@127.1.233.131:0 shutdown complete.
I20240502 01:50:44.694626 1958 master.cc:537] Master@127.1.233.190:39593 shutting down...
I20240502 01:50:44.716022 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:44.716841 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:44.717206 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P ec136095488f40cb9d8ed68d084d6c9c: stopping tablet replica
I20240502 01:50:44.736390 1958 master.cc:552] Master@127.1.233.190:39593 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (13217 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20240502 01:50:44.775849 1958 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.233.190:39943
I20240502 01:50:44.777077 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:44.783913 6112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:44.784569 6111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:44.785094 6114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:44.785660 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:44.786480 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:44.786690 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:44.786808 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614644786797 us; error 0 us; skew 500 ppm
I20240502 01:50:44.787261 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:44.794332 1958 webserver.cc:460] Webserver started at http://127.1.233.190:42721/ using document root <none> and password file <none>
I20240502 01:50:44.794852 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:44.795022 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:44.795248 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:44.796667 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/master-0-root/instance:
uuid: "5c58035508a546eb98186d5a534d8b6f"
format_stamp: "Formatted at 2024-05-02 01:50:44 on dist-test-slave-2090"
I20240502 01:50:44.801589 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:50:44.805264 6119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:44.806100 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240502 01:50:44.806349 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/master-0-root
uuid: "5c58035508a546eb98186d5a534d8b6f"
format_stamp: "Formatted at 2024-05-02 01:50:44 on dist-test-slave-2090"
I20240502 01:50:44.806612 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:44.820320 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:44.821578 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:44.860800 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.190:39943
I20240502 01:50:44.860908 6170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.190:39943 every 8 connection(s)
I20240502 01:50:44.865003 6171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:44.877184 6171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f: Bootstrap starting.
I20240502 01:50:44.881554 6171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:44.885857 6171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f: No bootstrap required, opened a new log
I20240502 01:50:44.887954 6171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c58035508a546eb98186d5a534d8b6f" member_type: VOTER }
I20240502 01:50:44.888511 6171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:44.888736 6171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c58035508a546eb98186d5a534d8b6f, State: Initialized, Role: FOLLOWER
I20240502 01:50:44.889279 6171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c58035508a546eb98186d5a534d8b6f" member_type: VOTER }
I20240502 01:50:44.889755 6171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:50:44.889971 6171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:50:44.890219 6171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:44.895191 6171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c58035508a546eb98186d5a534d8b6f" member_type: VOTER }
I20240502 01:50:44.895680 6171 leader_election.cc:302] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c58035508a546eb98186d5a534d8b6f; no voters:
I20240502 01:50:44.896971 6171 leader_election.cc:288] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:50:44.897406 6174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:44.898730 6174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [term 1 LEADER]: Becoming Leader. State: Replica: 5c58035508a546eb98186d5a534d8b6f, State: Running, Role: LEADER
I20240502 01:50:44.899453 6174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c58035508a546eb98186d5a534d8b6f" member_type: VOTER }
I20240502 01:50:44.899967 6171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:50:44.902464 6175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c58035508a546eb98186d5a534d8b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c58035508a546eb98186d5a534d8b6f" member_type: VOTER } }
I20240502 01:50:44.903204 6175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [sys.catalog]: This master's current role is: LEADER
I20240502 01:50:44.902491 6176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c58035508a546eb98186d5a534d8b6f. Latest consensus state: current_term: 1 leader_uuid: "5c58035508a546eb98186d5a534d8b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c58035508a546eb98186d5a534d8b6f" member_type: VOTER } }
I20240502 01:50:44.903980 6176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [sys.catalog]: This master's current role is: LEADER
I20240502 01:50:44.908865 6179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:50:44.917270 6179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:50:44.918462 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:50:44.926784 6179 catalog_manager.cc:1347] Generated new cluster ID: c1239622ded64ae2b0c3b1ff405fd9ae
I20240502 01:50:44.927100 6179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:50:44.951550 6179 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:50:44.952986 6179 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:50:44.965458 6179 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f: Generated new TSK 0
I20240502 01:50:44.966151 6179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:50:44.985245 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:44.992682 6192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:44.993950 6193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:44.994648 6195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:44.995348 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:44.996225 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:44.996423 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:44.996570 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614644996554 us; error 0 us; skew 500 ppm
I20240502 01:50:44.997140 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:44.999679 1958 webserver.cc:460] Webserver started at http://127.1.233.129:35505/ using document root <none> and password file <none>
I20240502 01:50:45.000229 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:45.000420 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:45.000671 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:45.001966 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-0-root/instance:
uuid: "c9849b524f5747dd993da6e85b21d59a"
format_stamp: "Formatted at 2024-05-02 01:50:44 on dist-test-slave-2090"
I20240502 01:50:45.007230 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 01:50:45.011327 6200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:45.012234 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:50:45.012547 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "c9849b524f5747dd993da6e85b21d59a"
format_stamp: "Formatted at 2024-05-02 01:50:44 on dist-test-slave-2090"
I20240502 01:50:45.012853 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:45.028268 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:45.029559 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:45.031107 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:45.033697 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:45.033921 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:45.034158 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:45.034329 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:45.078233 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:43297
I20240502 01:50:45.078328 6262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:43297 every 8 connection(s)
I20240502 01:50:45.083468 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:45.092610 6267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:45.093412 6268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:45.097182 6270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:45.097330 6263 heartbeater.cc:347] Connected to a master server at 127.1.233.190:39943
I20240502 01:50:45.097944 6263 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:45.098114 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:45.098836 6263 heartbeater.cc:511] Master 127.1.233.190:39943 requested a full tablet report, sending...
I20240502 01:50:45.099045 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:45.099311 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:45.099467 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614645099450 us; error 0 us; skew 500 ppm
I20240502 01:50:45.100225 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:45.101315 6136 ts_manager.cc:194] Registered new tserver with Master: c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297)
I20240502 01:50:45.103384 1958 webserver.cc:460] Webserver started at http://127.1.233.130:37671/ using document root <none> and password file <none>
I20240502 01:50:45.103403 6136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46238
I20240502 01:50:45.104040 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:45.104306 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:45.104564 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:45.105934 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-1-root/instance:
uuid: "b9458ded85544ed3afba9be26570a2a5"
format_stamp: "Formatted at 2024-05-02 01:50:45 on dist-test-slave-2090"
I20240502 01:50:45.110949 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 01:50:45.114418 6275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:45.115279 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 01:50:45.115576 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-1-root
uuid: "b9458ded85544ed3afba9be26570a2a5"
format_stamp: "Formatted at 2024-05-02 01:50:45 on dist-test-slave-2090"
I20240502 01:50:45.115890 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:45.127278 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:45.128564 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:45.129945 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:45.132207 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:45.132417 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:45.132670 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:45.132810 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:45.171478 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.130:43321
I20240502 01:50:45.171582 6337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.130:43321 every 8 connection(s)
I20240502 01:50:45.176456 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:45.184042 6341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:45.185336 6342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:45.186677 6344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:45.188270 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:45.189028 6338 heartbeater.cc:347] Connected to a master server at 127.1.233.190:39943
I20240502 01:50:45.189325 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 01:50:45.189464 6338 heartbeater.cc:464] Registering TS with master...
W20240502 01:50:45.189620 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:45.189851 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614645189832 us; error 0 us; skew 500 ppm
I20240502 01:50:45.190387 6338 heartbeater.cc:511] Master 127.1.233.190:39943 requested a full tablet report, sending...
I20240502 01:50:45.190680 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:45.192932 6136 ts_manager.cc:194] Registered new tserver with Master: b9458ded85544ed3afba9be26570a2a5 (127.1.233.130:43321)
I20240502 01:50:45.193923 1958 webserver.cc:460] Webserver started at http://127.1.233.131:41399/ using document root <none> and password file <none>
I20240502 01:50:45.194566 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:45.194808 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:45.194913 6136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46250
I20240502 01:50:45.195152 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:45.196794 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-2-root/instance:
uuid: "f96086da232f4f44932015ca951fd88b"
format_stamp: "Formatted at 2024-05-02 01:50:45 on dist-test-slave-2090"
I20240502 01:50:45.201570 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:50:45.204833 6349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:45.205571 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 01:50:45.205848 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-2-root
uuid: "f96086da232f4f44932015ca951fd88b"
format_stamp: "Formatted at 2024-05-02 01:50:45 on dist-test-slave-2090"
I20240502 01:50:45.206107 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:45.240324 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:45.241578 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:45.243177 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:45.245440 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:45.245617 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:45.245858 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:45.245983 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:45.286180 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.131:33345
I20240502 01:50:45.286293 6411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.131:33345 every 8 connection(s)
I20240502 01:50:45.299867 6412 heartbeater.cc:347] Connected to a master server at 127.1.233.190:39943
I20240502 01:50:45.300331 6412 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:45.301156 6412 heartbeater.cc:511] Master 127.1.233.190:39943 requested a full tablet report, sending...
I20240502 01:50:45.303151 6136 ts_manager.cc:194] Registered new tserver with Master: f96086da232f4f44932015ca951fd88b (127.1.233.131:33345)
I20240502 01:50:45.303979 1958 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014769553s
I20240502 01:50:45.305029 6136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46258
I20240502 01:50:46.106775 6263 heartbeater.cc:503] Master 127.1.233.190:39943 was elected leader, sending a full tablet report...
I20240502 01:50:46.198294 6338 heartbeater.cc:503] Master 127.1.233.190:39943 was elected leader, sending a full tablet report...
I20240502 01:50:46.308092 6412 heartbeater.cc:503] Master 127.1.233.190:39943 was elected leader, sending a full tablet report...
I20240502 01:50:46.337899 1958 test_util.cc:268] Using random seed: -722734843
I20240502 01:50:46.364065 6136 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46264:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:50:46.366770 6136 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:50:46.437721 6377 tablet_service.cc:1467] Processing CreateTablet for tablet da800cb86fb24800b59374018d0ce197 (DEFAULT_TABLE table=test-workload [id=ac87fa1fb7ba4fd6ae3c843042a87d60]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:50:46.438391 6375 tablet_service.cc:1467] Processing CreateTablet for tablet b0a4ee1f9ba9484896bb256105b96565 (DEFAULT_TABLE table=test-workload [id=ac87fa1fb7ba4fd6ae3c843042a87d60]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:50:46.438726 6376 tablet_service.cc:1467] Processing CreateTablet for tablet 39ca1938892648e1a4658b541a260c6d (DEFAULT_TABLE table=test-workload [id=ac87fa1fb7ba4fd6ae3c843042a87d60]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:50:46.440410 6375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0a4ee1f9ba9484896bb256105b96565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:46.441244 6302 tablet_service.cc:1467] Processing CreateTablet for tablet 39ca1938892648e1a4658b541a260c6d (DEFAULT_TABLE table=test-workload [id=ac87fa1fb7ba4fd6ae3c843042a87d60]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:50:46.441491 6374 tablet_service.cc:1467] Processing CreateTablet for tablet 5cada54fde9f4c0498067e9a5ce6962c (DEFAULT_TABLE table=test-workload [id=ac87fa1fb7ba4fd6ae3c843042a87d60]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:50:46.442910 6302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ca1938892648e1a4658b541a260c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:46.442971 6376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ca1938892648e1a4658b541a260c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:46.438781 6303 tablet_service.cc:1467] Processing CreateTablet for tablet da800cb86fb24800b59374018d0ce197 (DEFAULT_TABLE table=test-workload [id=ac87fa1fb7ba4fd6ae3c843042a87d60]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:50:46.446012 6303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da800cb86fb24800b59374018d0ce197. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:46.450243 6301 tablet_service.cc:1467] Processing CreateTablet for tablet b0a4ee1f9ba9484896bb256105b96565 (DEFAULT_TABLE table=test-workload [id=ac87fa1fb7ba4fd6ae3c843042a87d60]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:50:46.451857 6301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0a4ee1f9ba9484896bb256105b96565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:46.458232 6300 tablet_service.cc:1467] Processing CreateTablet for tablet 5cada54fde9f4c0498067e9a5ce6962c (DEFAULT_TABLE table=test-workload [id=ac87fa1fb7ba4fd6ae3c843042a87d60]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:50:46.459683 6300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cada54fde9f4c0498067e9a5ce6962c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:46.463652 6377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da800cb86fb24800b59374018d0ce197. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:46.470059 6228 tablet_service.cc:1467] Processing CreateTablet for tablet da800cb86fb24800b59374018d0ce197 (DEFAULT_TABLE table=test-workload [id=ac87fa1fb7ba4fd6ae3c843042a87d60]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:50:46.471714 6228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da800cb86fb24800b59374018d0ce197. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:46.472846 6374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cada54fde9f4c0498067e9a5ce6962c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:46.473577 6227 tablet_service.cc:1467] Processing CreateTablet for tablet 39ca1938892648e1a4658b541a260c6d (DEFAULT_TABLE table=test-workload [id=ac87fa1fb7ba4fd6ae3c843042a87d60]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:50:46.474959 6227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ca1938892648e1a4658b541a260c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:46.477572 6226 tablet_service.cc:1467] Processing CreateTablet for tablet b0a4ee1f9ba9484896bb256105b96565 (DEFAULT_TABLE table=test-workload [id=ac87fa1fb7ba4fd6ae3c843042a87d60]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:50:46.479012 6226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0a4ee1f9ba9484896bb256105b96565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:46.481456 6225 tablet_service.cc:1467] Processing CreateTablet for tablet 5cada54fde9f4c0498067e9a5ce6962c (DEFAULT_TABLE table=test-workload [id=ac87fa1fb7ba4fd6ae3c843042a87d60]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:50:46.482805 6225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cada54fde9f4c0498067e9a5ce6962c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:46.501895 6433 tablet_bootstrap.cc:492] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5: Bootstrap starting.
I20240502 01:50:46.509343 6434 tablet_bootstrap.cc:492] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b: Bootstrap starting.
I20240502 01:50:46.516358 6434 tablet_bootstrap.cc:654] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:46.517467 6433 tablet_bootstrap.cc:654] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:46.531383 6433 tablet_bootstrap.cc:492] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5: No bootstrap required, opened a new log
I20240502 01:50:46.532078 6433 ts_tablet_manager.cc:1367] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.008s
I20240502 01:50:46.535099 6433 raft_consensus.cc:348] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.548427 6434 tablet_bootstrap.cc:492] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b: No bootstrap required, opened a new log
I20240502 01:50:46.549117 6434 ts_tablet_manager.cc:1367] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.014s
I20240502 01:50:46.546581 6433 raft_consensus.cc:374] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:46.552986 6433 raft_consensus.cc:729] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9458ded85544ed3afba9be26570a2a5, State: Initialized, Role: FOLLOWER
I20240502 01:50:46.553125 6434 raft_consensus.cc:348] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.553246 6437 tablet_bootstrap.cc:492] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a: Bootstrap starting.
I20240502 01:50:46.554183 6434 raft_consensus.cc:374] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:46.554100 6433 consensus_queue.cc:260] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.554764 6434 raft_consensus.cc:729] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96086da232f4f44932015ca951fd88b, State: Initialized, Role: FOLLOWER
I20240502 01:50:46.555580 6434 consensus_queue.cc:260] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.557512 6433 ts_tablet_manager.cc:1398] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5: Time spent starting tablet: real 0.025s user 0.006s sys 0.002s
I20240502 01:50:46.560000 6433 tablet_bootstrap.cc:492] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5: Bootstrap starting.
I20240502 01:50:46.561489 6437 tablet_bootstrap.cc:654] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:46.566462 6433 tablet_bootstrap.cc:654] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:46.575095 6434 ts_tablet_manager.cc:1398] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b: Time spent starting tablet: real 0.026s user 0.013s sys 0.007s
I20240502 01:50:46.576407 6434 tablet_bootstrap.cc:492] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b: Bootstrap starting.
I20240502 01:50:46.582748 6434 tablet_bootstrap.cc:654] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:46.586246 6437 tablet_bootstrap.cc:492] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a: No bootstrap required, opened a new log
I20240502 01:50:46.586822 6433 tablet_bootstrap.cc:492] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5: No bootstrap required, opened a new log
I20240502 01:50:46.586851 6437 ts_tablet_manager.cc:1367] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.012s
I20240502 01:50:46.587507 6433 ts_tablet_manager.cc:1367] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.007s
I20240502 01:50:46.589984 6434 tablet_bootstrap.cc:492] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b: No bootstrap required, opened a new log
I20240502 01:50:46.590052 6433 raft_consensus.cc:348] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } }
I20240502 01:50:46.590641 6434 ts_tablet_manager.cc:1367] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.001s
I20240502 01:50:46.590373 6437 raft_consensus.cc:348] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } }
I20240502 01:50:46.590826 6433 raft_consensus.cc:374] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:46.591068 6437 raft_consensus.cc:374] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:46.591174 6433 raft_consensus.cc:729] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9458ded85544ed3afba9be26570a2a5, State: Initialized, Role: FOLLOWER
I20240502 01:50:46.591516 6437 raft_consensus.cc:729] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9849b524f5747dd993da6e85b21d59a, State: Initialized, Role: FOLLOWER
I20240502 01:50:46.592494 6433 consensus_queue.cc:260] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } }
I20240502 01:50:46.592432 6437 consensus_queue.cc:260] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } }
I20240502 01:50:46.593535 6434 raft_consensus.cc:348] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } }
I20240502 01:50:46.594409 6434 raft_consensus.cc:374] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:46.594542 6433 ts_tablet_manager.cc:1398] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:50:46.594808 6434 raft_consensus.cc:729] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96086da232f4f44932015ca951fd88b, State: Initialized, Role: FOLLOWER
I20240502 01:50:46.595875 6433 tablet_bootstrap.cc:492] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5: Bootstrap starting.
I20240502 01:50:46.595790 6434 consensus_queue.cc:260] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } }
I20240502 01:50:46.598829 6434 ts_tablet_manager.cc:1398] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 01:50:46.600152 6434 tablet_bootstrap.cc:492] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b: Bootstrap starting.
I20240502 01:50:46.604614 6433 tablet_bootstrap.cc:654] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:46.607975 6434 tablet_bootstrap.cc:654] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:46.608003 6437 ts_tablet_manager.cc:1398] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a: Time spent starting tablet: real 0.020s user 0.012s sys 0.007s
I20240502 01:50:46.609393 6437 tablet_bootstrap.cc:492] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a: Bootstrap starting.
I20240502 01:50:46.617115 6437 tablet_bootstrap.cc:654] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:46.625485 6434 tablet_bootstrap.cc:492] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b: No bootstrap required, opened a new log
I20240502 01:50:46.625945 6434 ts_tablet_manager.cc:1367] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.008s
I20240502 01:50:46.627962 6434 raft_consensus.cc:348] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.628947 6434 raft_consensus.cc:374] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:46.629376 6434 raft_consensus.cc:729] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96086da232f4f44932015ca951fd88b, State: Initialized, Role: FOLLOWER
I20240502 01:50:46.630434 6434 consensus_queue.cc:260] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.632932 6434 ts_tablet_manager.cc:1398] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 01:50:46.633652 6433 tablet_bootstrap.cc:492] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5: No bootstrap required, opened a new log
I20240502 01:50:46.634037 6434 tablet_bootstrap.cc:492] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b: Bootstrap starting.
I20240502 01:50:46.634229 6433 ts_tablet_manager.cc:1367] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.018s
I20240502 01:50:46.634083 6437 tablet_bootstrap.cc:492] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a: No bootstrap required, opened a new log
I20240502 01:50:46.635138 6437 ts_tablet_manager.cc:1367] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.005s
I20240502 01:50:46.637303 6433 raft_consensus.cc:348] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.638180 6433 raft_consensus.cc:374] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:46.638662 6433 raft_consensus.cc:729] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9458ded85544ed3afba9be26570a2a5, State: Initialized, Role: FOLLOWER
I20240502 01:50:46.638209 6437 raft_consensus.cc:348] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.639381 6437 raft_consensus.cc:374] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:46.639760 6437 raft_consensus.cc:729] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9849b524f5747dd993da6e85b21d59a, State: Initialized, Role: FOLLOWER
I20240502 01:50:46.639518 6433 consensus_queue.cc:260] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.641074 6434 tablet_bootstrap.cc:654] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:46.640878 6437 consensus_queue.cc:260] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.643460 6437 ts_tablet_manager.cc:1398] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 01:50:46.644616 6433 ts_tablet_manager.cc:1398] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20240502 01:50:46.645069 6437 tablet_bootstrap.cc:492] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a: Bootstrap starting.
I20240502 01:50:46.645697 6433 tablet_bootstrap.cc:492] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5: Bootstrap starting.
I20240502 01:50:46.652359 6433 tablet_bootstrap.cc:654] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:46.653863 6437 tablet_bootstrap.cc:654] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:46.660832 6434 tablet_bootstrap.cc:492] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b: No bootstrap required, opened a new log
I20240502 01:50:46.661739 6434 ts_tablet_manager.cc:1367] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.013s
I20240502 01:50:46.665247 6433 tablet_bootstrap.cc:492] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5: No bootstrap required, opened a new log
I20240502 01:50:46.665928 6433 ts_tablet_manager.cc:1367] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.008s
I20240502 01:50:46.665609 6434 raft_consensus.cc:348] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.666941 6434 raft_consensus.cc:374] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:46.667440 6434 raft_consensus.cc:729] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96086da232f4f44932015ca951fd88b, State: Initialized, Role: FOLLOWER
I20240502 01:50:46.668677 6434 consensus_queue.cc:260] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.669219 6433 raft_consensus.cc:348] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.672223 6433 raft_consensus.cc:374] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:46.672637 6433 raft_consensus.cc:729] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9458ded85544ed3afba9be26570a2a5, State: Initialized, Role: FOLLOWER
I20240502 01:50:46.673552 6433 consensus_queue.cc:260] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.676390 6433 ts_tablet_manager.cc:1398] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240502 01:50:46.679083 6434 ts_tablet_manager.cc:1398] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b: Time spent starting tablet: real 0.017s user 0.004s sys 0.004s
I20240502 01:50:46.686628 6437 tablet_bootstrap.cc:492] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a: No bootstrap required, opened a new log
I20240502 01:50:46.687158 6437 ts_tablet_manager.cc:1367] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.024s
I20240502 01:50:46.689340 6437 raft_consensus.cc:348] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.689920 6437 raft_consensus.cc:374] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:46.690135 6437 raft_consensus.cc:729] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9849b524f5747dd993da6e85b21d59a, State: Initialized, Role: FOLLOWER
I20240502 01:50:46.690703 6437 consensus_queue.cc:260] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.692469 6437 ts_tablet_manager.cc:1398] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a: Time spent starting tablet: real 0.005s user 0.004s sys 0.004s
I20240502 01:50:46.693245 6437 tablet_bootstrap.cc:492] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a: Bootstrap starting.
I20240502 01:50:46.697633 6437 tablet_bootstrap.cc:654] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:46.702590 6437 tablet_bootstrap.cc:492] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a: No bootstrap required, opened a new log
I20240502 01:50:46.703013 6437 ts_tablet_manager.cc:1367] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 01:50:46.705125 6437 raft_consensus.cc:348] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.705646 6437 raft_consensus.cc:374] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:46.705855 6437 raft_consensus.cc:729] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9849b524f5747dd993da6e85b21d59a, State: Initialized, Role: FOLLOWER
I20240502 01:50:46.706447 6437 consensus_queue.cc:260] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.708051 6437 ts_tablet_manager.cc:1398] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:50:46.735610 6443 raft_consensus.cc:482] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:50:46.736066 6443 raft_consensus.cc:504] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } }
I20240502 01:50:46.738107 6443 leader_election.cc:288] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f96086da232f4f44932015ca951fd88b (127.1.233.131:33345), b9458ded85544ed3afba9be26570a2a5 (127.1.233.130:43321)
I20240502 01:50:46.739158 6438 raft_consensus.cc:482] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:50:46.739704 6438 raft_consensus.cc:504] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.746152 6438 leader_election.cc:288] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297), f96086da232f4f44932015ca951fd88b (127.1.233.131:33345)
I20240502 01:50:46.757412 6387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da800cb86fb24800b59374018d0ce197" candidate_uuid: "c9849b524f5747dd993da6e85b21d59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96086da232f4f44932015ca951fd88b" is_pre_election: true
I20240502 01:50:46.758744 6387 raft_consensus.cc:2454] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9849b524f5747dd993da6e85b21d59a in term 0.
I20240502 01:50:46.764389 6203 leader_election.cc:302] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9849b524f5747dd993da6e85b21d59a, f96086da232f4f44932015ca951fd88b; no voters:
I20240502 01:50:46.765466 6443 raft_consensus.cc:2781] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:50:46.765844 6443 raft_consensus.cc:482] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:50:46.766163 6443 raft_consensus.cc:3037] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:46.767539 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da800cb86fb24800b59374018d0ce197" candidate_uuid: "c9849b524f5747dd993da6e85b21d59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9458ded85544ed3afba9be26570a2a5" is_pre_election: true
I20240502 01:50:46.768364 6313 raft_consensus.cc:2454] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9849b524f5747dd993da6e85b21d59a in term 0.
I20240502 01:50:46.773310 6443 raft_consensus.cc:504] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } }
I20240502 01:50:46.775089 6387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ca1938892648e1a4658b541a260c6d" candidate_uuid: "b9458ded85544ed3afba9be26570a2a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96086da232f4f44932015ca951fd88b" is_pre_election: true
I20240502 01:50:46.775182 6238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ca1938892648e1a4658b541a260c6d" candidate_uuid: "b9458ded85544ed3afba9be26570a2a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9849b524f5747dd993da6e85b21d59a" is_pre_election: true
I20240502 01:50:46.775941 6387 raft_consensus.cc:2454] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9458ded85544ed3afba9be26570a2a5 in term 0.
I20240502 01:50:46.776021 6238 raft_consensus.cc:2454] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9458ded85544ed3afba9be26570a2a5 in term 0.
I20240502 01:50:46.776089 6443 leader_election.cc:288] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [CANDIDATE]: Term 1 election: Requested vote from peers f96086da232f4f44932015ca951fd88b (127.1.233.131:33345), b9458ded85544ed3afba9be26570a2a5 (127.1.233.130:43321)
I20240502 01:50:46.776697 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da800cb86fb24800b59374018d0ce197" candidate_uuid: "c9849b524f5747dd993da6e85b21d59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9458ded85544ed3afba9be26570a2a5"
I20240502 01:50:46.777587 6278 leader_election.cc:302] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9458ded85544ed3afba9be26570a2a5, f96086da232f4f44932015ca951fd88b; no voters:
I20240502 01:50:46.777711 6386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da800cb86fb24800b59374018d0ce197" candidate_uuid: "c9849b524f5747dd993da6e85b21d59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96086da232f4f44932015ca951fd88b"
I20240502 01:50:46.778151 6313 raft_consensus.cc:3037] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:46.778438 6386 raft_consensus.cc:3037] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:46.778698 6438 raft_consensus.cc:2781] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:50:46.779132 6438 raft_consensus.cc:482] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:50:46.779481 6438 raft_consensus.cc:3037] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:46.785194 6438 raft_consensus.cc:504] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.786178 6313 raft_consensus.cc:2454] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9849b524f5747dd993da6e85b21d59a in term 1.
I20240502 01:50:46.786182 6386 raft_consensus.cc:2454] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9849b524f5747dd993da6e85b21d59a in term 1.
I20240502 01:50:46.787293 6438 leader_election.cc:288] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [CANDIDATE]: Term 1 election: Requested vote from peers c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297), f96086da232f4f44932015ca951fd88b (127.1.233.131:33345)
I20240502 01:50:46.787983 6387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ca1938892648e1a4658b541a260c6d" candidate_uuid: "b9458ded85544ed3afba9be26570a2a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96086da232f4f44932015ca951fd88b"
I20240502 01:50:46.787802 6203 leader_election.cc:302] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9849b524f5747dd993da6e85b21d59a, f96086da232f4f44932015ca951fd88b; no voters:
I20240502 01:50:46.787817 6238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ca1938892648e1a4658b541a260c6d" candidate_uuid: "b9458ded85544ed3afba9be26570a2a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9849b524f5747dd993da6e85b21d59a"
I20240502 01:50:46.788703 6387 raft_consensus.cc:3037] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:46.788841 6238 raft_consensus.cc:3037] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:46.789085 6443 raft_consensus.cc:2781] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:46.790556 6443 raft_consensus.cc:686] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 1 LEADER]: Becoming Leader. State: Replica: c9849b524f5747dd993da6e85b21d59a, State: Running, Role: LEADER
I20240502 01:50:46.792238 6443 consensus_queue.cc:237] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } }
I20240502 01:50:46.795150 6238 raft_consensus.cc:2454] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9458ded85544ed3afba9be26570a2a5 in term 1.
I20240502 01:50:46.795317 6439 raft_consensus.cc:482] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:50:46.795958 6439 raft_consensus.cc:504] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.796751 6279 leader_election.cc:302] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9458ded85544ed3afba9be26570a2a5, c9849b524f5747dd993da6e85b21d59a; no voters:
I20240502 01:50:46.797062 6387 raft_consensus.cc:2454] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9458ded85544ed3afba9be26570a2a5 in term 1.
I20240502 01:50:46.797652 6438 raft_consensus.cc:2781] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:46.800065 6439 leader_election.cc:288] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297), b9458ded85544ed3afba9be26570a2a5 (127.1.233.130:43321)
I20240502 01:50:46.807567 6133 catalog_manager.cc:5484] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a reported cstate change: term changed from 0 to 1, leader changed from <none> to c9849b524f5747dd993da6e85b21d59a (127.1.233.129). New cstate: current_term: 1 leader_uuid: "c9849b524f5747dd993da6e85b21d59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:46.818591 6438 raft_consensus.cc:686] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 1 LEADER]: Becoming Leader. State: Replica: b9458ded85544ed3afba9be26570a2a5, State: Running, Role: LEADER
I20240502 01:50:46.819671 6438 consensus_queue.cc:237] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.830140 6238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cada54fde9f4c0498067e9a5ce6962c" candidate_uuid: "f96086da232f4f44932015ca951fd88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9849b524f5747dd993da6e85b21d59a" is_pre_election: true
I20240502 01:50:46.830919 6238 raft_consensus.cc:2454] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96086da232f4f44932015ca951fd88b in term 0.
I20240502 01:50:46.831179 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cada54fde9f4c0498067e9a5ce6962c" candidate_uuid: "f96086da232f4f44932015ca951fd88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9458ded85544ed3afba9be26570a2a5" is_pre_election: true
I20240502 01:50:46.832090 6313 raft_consensus.cc:2454] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96086da232f4f44932015ca951fd88b in term 0.
I20240502 01:50:46.831254 6133 catalog_manager.cc:5484] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9458ded85544ed3afba9be26570a2a5 (127.1.233.130). New cstate: current_term: 1 leader_uuid: "b9458ded85544ed3afba9be26570a2a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:46.832266 6353 leader_election.cc:302] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9849b524f5747dd993da6e85b21d59a, f96086da232f4f44932015ca951fd88b; no voters:
I20240502 01:50:46.833218 6439 raft_consensus.cc:2781] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:50:46.833508 6439 raft_consensus.cc:482] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:50:46.833762 6439 raft_consensus.cc:3037] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:46.838824 6439 raft_consensus.cc:504] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.840700 6439 leader_election.cc:288] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 1 election: Requested vote from peers c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297), b9458ded85544ed3afba9be26570a2a5 (127.1.233.130:43321)
I20240502 01:50:46.841576 6238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cada54fde9f4c0498067e9a5ce6962c" candidate_uuid: "f96086da232f4f44932015ca951fd88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9849b524f5747dd993da6e85b21d59a"
I20240502 01:50:46.841754 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cada54fde9f4c0498067e9a5ce6962c" candidate_uuid: "f96086da232f4f44932015ca951fd88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9458ded85544ed3afba9be26570a2a5"
I20240502 01:50:46.842123 6238 raft_consensus.cc:3037] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:46.842305 6313 raft_consensus.cc:3037] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:46.847188 6238 raft_consensus.cc:2454] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96086da232f4f44932015ca951fd88b in term 1.
I20240502 01:50:46.847278 6313 raft_consensus.cc:2454] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96086da232f4f44932015ca951fd88b in term 1.
I20240502 01:50:46.848359 6353 leader_election.cc:302] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9849b524f5747dd993da6e85b21d59a, f96086da232f4f44932015ca951fd88b; no voters:
I20240502 01:50:46.849092 6439 raft_consensus.cc:2781] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:46.850091 6439 raft_consensus.cc:686] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 1 LEADER]: Becoming Leader. State: Replica: f96086da232f4f44932015ca951fd88b, State: Running, Role: LEADER
I20240502 01:50:46.850872 6439 consensus_queue.cc:237] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.858144 6133 catalog_manager.cc:5484] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b reported cstate change: term changed from 0 to 1, leader changed from <none> to f96086da232f4f44932015ca951fd88b (127.1.233.131). New cstate: current_term: 1 leader_uuid: "f96086da232f4f44932015ca951fd88b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } health_report { overall_health: HEALTHY } } }
W20240502 01:50:46.914986 6184 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 01:50:46.969156 6439 raft_consensus.cc:482] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:50:46.969691 6439 raft_consensus.cc:504] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.971396 6439 leader_election.cc:288] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297), b9458ded85544ed3afba9be26570a2a5 (127.1.233.130:43321)
I20240502 01:50:46.972517 6238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4ee1f9ba9484896bb256105b96565" candidate_uuid: "f96086da232f4f44932015ca951fd88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9849b524f5747dd993da6e85b21d59a" is_pre_election: true
I20240502 01:50:46.972534 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4ee1f9ba9484896bb256105b96565" candidate_uuid: "f96086da232f4f44932015ca951fd88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9458ded85544ed3afba9be26570a2a5" is_pre_election: true
I20240502 01:50:46.973258 6238 raft_consensus.cc:2454] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96086da232f4f44932015ca951fd88b in term 0.
I20240502 01:50:46.973351 6313 raft_consensus.cc:2454] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96086da232f4f44932015ca951fd88b in term 0.
I20240502 01:50:46.974345 6353 leader_election.cc:302] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9849b524f5747dd993da6e85b21d59a, f96086da232f4f44932015ca951fd88b; no voters:
I20240502 01:50:46.974999 6439 raft_consensus.cc:2781] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:50:46.975368 6439 raft_consensus.cc:482] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:50:46.975647 6439 raft_consensus.cc:3037] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:46.981009 6439 raft_consensus.cc:504] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:46.982802 6439 leader_election.cc:288] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 1 election: Requested vote from peers c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297), b9458ded85544ed3afba9be26570a2a5 (127.1.233.130:43321)
I20240502 01:50:46.983826 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4ee1f9ba9484896bb256105b96565" candidate_uuid: "f96086da232f4f44932015ca951fd88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9458ded85544ed3afba9be26570a2a5"
I20240502 01:50:46.983716 6238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0a4ee1f9ba9484896bb256105b96565" candidate_uuid: "f96086da232f4f44932015ca951fd88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9849b524f5747dd993da6e85b21d59a"
I20240502 01:50:46.984562 6313 raft_consensus.cc:3037] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:46.984570 6238 raft_consensus.cc:3037] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:46.992031 6238 raft_consensus.cc:2454] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96086da232f4f44932015ca951fd88b in term 1.
I20240502 01:50:46.992031 6313 raft_consensus.cc:2454] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96086da232f4f44932015ca951fd88b in term 1.
I20240502 01:50:46.993628 6353 leader_election.cc:302] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9849b524f5747dd993da6e85b21d59a, f96086da232f4f44932015ca951fd88b; no voters:
I20240502 01:50:46.994400 6439 raft_consensus.cc:2781] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:46.994832 6439 raft_consensus.cc:686] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 1 LEADER]: Becoming Leader. State: Replica: f96086da232f4f44932015ca951fd88b, State: Running, Role: LEADER
I20240502 01:50:46.995569 6439 consensus_queue.cc:237] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } }
I20240502 01:50:47.002557 6133 catalog_manager.cc:5484] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b reported cstate change: term changed from 0 to 1, leader changed from <none> to f96086da232f4f44932015ca951fd88b (127.1.233.131). New cstate: current_term: 1 leader_uuid: "f96086da232f4f44932015ca951fd88b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } health_report { overall_health: HEALTHY } } }
I20240502 01:50:47.147990 1958 tablet_server.cc:178] TabletServer@127.1.233.129:0 shutting down...
I20240502 01:50:47.170464 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:47.171437 1958 tablet_replica.cc:330] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a: stopping tablet replica
I20240502 01:50:47.172345 1958 raft_consensus.cc:2229] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:47.172797 1958 raft_consensus.cc:2258] T 39ca1938892648e1a4658b541a260c6d P c9849b524f5747dd993da6e85b21d59a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:47.174958 1958 tablet_replica.cc:330] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a: stopping tablet replica
I20240502 01:50:47.175516 1958 raft_consensus.cc:2229] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:47.175894 1958 raft_consensus.cc:2258] T b0a4ee1f9ba9484896bb256105b96565 P c9849b524f5747dd993da6e85b21d59a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:47.177963 1958 tablet_replica.cc:330] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a: stopping tablet replica
I20240502 01:50:47.178498 1958 raft_consensus.cc:2229] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:47.178879 1958 raft_consensus.cc:2258] T 5cada54fde9f4c0498067e9a5ce6962c P c9849b524f5747dd993da6e85b21d59a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:47.180749 1958 tablet_replica.cc:330] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a: stopping tablet replica
I20240502 01:50:47.181212 1958 raft_consensus.cc:2229] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:47.181859 1958 pending_rounds.cc:62] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a: Trying to abort 1 pending ops.
I20240502 01:50:47.182113 1958 pending_rounds.cc:68] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023061593271033856 op_type: NO_OP noop_request { }
I20240502 01:50:47.182515 1958 raft_consensus.cc:2866] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 01:50:47.182803 1958 raft_consensus.cc:2258] T da800cb86fb24800b59374018d0ce197 P c9849b524f5747dd993da6e85b21d59a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:47.207125 1958 tablet_server.cc:194] TabletServer@127.1.233.129:0 shutdown complete.
I20240502 01:50:47.295120 6439 consensus_queue.cc:1035] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:50:47.306465 6353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 01:50:47.313139 6353 consensus_peers.cc:485] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b -> Peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Couldn't send request to peer c9849b524f5747dd993da6e85b21d59a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:47.333804 6438 consensus_queue.cc:1035] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:50:47.352082 6279 consensus_peers.cc:485] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 -> Peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Couldn't send request to peer c9849b524f5747dd993da6e85b21d59a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:50:47.408529 6353 consensus_peers.cc:485] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b -> Peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Couldn't send request to peer c9849b524f5747dd993da6e85b21d59a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:47.505613 6462 consensus_queue.cc:1035] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:50:47.827514 6471 consensus_queue.cc:579] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [LEADER]: Leader has been unable to successfully communicate with peer c9849b524f5747dd993da6e85b21d59a for more than 1 seconds (1.007s)
W20240502 01:50:47.834961 6132 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 39ca1938892648e1a4658b541a260c6d with cas_config_opid_index -1: no extra replica candidate found for tablet 39ca1938892648e1a4658b541a260c6d (table test-workload [id=ac87fa1fb7ba4fd6ae3c843042a87d60]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20240502 01:50:47.917255 6184 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver c9849b524f5747dd993da6e85b21d59a not available for placement
I20240502 01:50:48.194818 6462 consensus_queue.cc:579] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [LEADER]: Leader has been unable to successfully communicate with peer c9849b524f5747dd993da6e85b21d59a for more than 1 seconds (1.343s)
I20240502 01:50:48.224637 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:48.231792 6478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:48.232441 6479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:48.233462 6481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:48.234169 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:48.235162 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:48.235350 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:48.235479 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614648235469 us; error 0 us; skew 500 ppm
I20240502 01:50:48.235965 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:48.238667 1958 webserver.cc:460] Webserver started at http://127.1.233.132:41197/ using document root <none> and password file <none>
I20240502 01:50:48.239130 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:48.239324 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:48.239578 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:48.240809 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-3-root/instance:
uuid: "4976324b2fd64be59fd9c32fe90f86b4"
format_stamp: "Formatted at 2024-05-02 01:50:48 on dist-test-slave-2090"
I20240502 01:50:48.245455 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:50:48.248804 6486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:48.249552 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:50:48.249832 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-3-root
uuid: "4976324b2fd64be59fd9c32fe90f86b4"
format_stamp: "Formatted at 2024-05-02 01:50:48 on dist-test-slave-2090"
I20240502 01:50:48.250090 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714614515033618-1958-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:48.270502 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:48.271698 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:48.273276 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:48.275605 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:48.275792 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:48.276062 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:48.276235 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 01:50:48.316375 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.132:33777
I20240502 01:50:48.316498 6548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.132:33777 every 8 connection(s)
I20240502 01:50:48.331004 6549 heartbeater.cc:347] Connected to a master server at 127.1.233.190:39943
I20240502 01:50:48.331339 6549 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:48.332072 6549 heartbeater.cc:511] Master 127.1.233.190:39943 requested a full tablet report, sending...
I20240502 01:50:48.333824 6132 ts_manager.cc:194] Registered new tserver with Master: 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132:33777)
I20240502 01:50:48.335166 6132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46268
I20240502 01:50:48.343427 6387 consensus_queue.cc:237] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } }
I20240502 01:50:48.350021 6313 raft_consensus.cc:1261] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:48.351630 6476 consensus_queue.cc:579] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [LEADER]: Leader has been unable to successfully communicate with peer c9849b524f5747dd993da6e85b21d59a for more than 1 seconds (1.355s)
I20240502 01:50:48.353312 6476 consensus_queue.cc:1035] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:50:48.357918 6353 consensus_peers.cc:485] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b -> Peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Couldn't send request to peer c9849b524f5747dd993da6e85b21d59a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:50:48.362730 6353 consensus_peers.cc:485] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b -> Peer 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132:33777): Couldn't send request to peer 4976324b2fd64be59fd9c32fe90f86b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cada54fde9f4c0498067e9a5ce6962c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:48.362742 6462 raft_consensus.cc:2932] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } } }
I20240502 01:50:48.363938 6387 consensus_queue.cc:237] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } }
I20240502 01:50:48.364481 6313 raft_consensus.cc:2932] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } } }
I20240502 01:50:48.370476 6312 raft_consensus.cc:1261] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:50:48.371968 6353 consensus_peers.cc:485] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b -> Peer 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132:33777): Couldn't send request to peer 4976324b2fd64be59fd9c32fe90f86b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0a4ee1f9ba9484896bb256105b96565. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:48.372177 6477 consensus_queue.cc:1035] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:50:48.374532 6122 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5cada54fde9f4c0498067e9a5ce6962c with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
W20240502 01:50:48.375365 6353 consensus_peers.cc:485] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b -> Peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Couldn't send request to peer c9849b524f5747dd993da6e85b21d59a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:48.378975 6133 catalog_manager.cc:5484] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b reported cstate change: config changed from index -1 to 2, NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) added. New cstate: current_term: 1 leader_uuid: "f96086da232f4f44932015ca951fd88b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:48.380826 6462 raft_consensus.cc:2932] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } } }
I20240502 01:50:48.382716 6313 raft_consensus.cc:2932] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } } }
I20240502 01:50:48.390300 6122 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b0a4ee1f9ba9484896bb256105b96565 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:50:48.398839 6133 catalog_manager.cc:5484] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) added. New cstate: current_term: 1 leader_uuid: "f96086da232f4f44932015ca951fd88b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } } }
I20240502 01:50:48.502646 6313 consensus_queue.cc:237] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } }
I20240502 01:50:48.509428 6387 raft_consensus.cc:1261] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Refusing update from remote peer b9458ded85544ed3afba9be26570a2a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:50:48.510862 6471 consensus_queue.cc:1035] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:50:48.514817 6279 consensus_peers.cc:485] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 -> Peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Couldn't send request to peer c9849b524f5747dd993da6e85b21d59a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:50:48.518852 6279 consensus_peers.cc:485] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 -> Peer 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132:33777): Couldn't send request to peer 4976324b2fd64be59fd9c32fe90f86b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39ca1938892648e1a4658b541a260c6d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:48.518481 6562 raft_consensus.cc:2932] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } } }
I20240502 01:50:48.520010 6387 raft_consensus.cc:2932] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } } }
I20240502 01:50:48.525462 6122 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 39ca1938892648e1a4658b541a260c6d with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20240502 01:50:48.530002 6132 catalog_manager.cc:5484] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b reported cstate change: config changed from index -1 to 2, NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) added. New cstate: current_term: 1 leader_uuid: "b9458ded85544ed3afba9be26570a2a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } } }
I20240502 01:50:48.532169 6476 raft_consensus.cc:482] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:50:48.532681 6476 raft_consensus.cc:504] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } }
I20240502 01:50:48.534279 6476 leader_election.cc:288] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297), b9458ded85544ed3afba9be26570a2a5 (127.1.233.130:43321)
I20240502 01:50:48.535439 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da800cb86fb24800b59374018d0ce197" candidate_uuid: "f96086da232f4f44932015ca951fd88b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9458ded85544ed3afba9be26570a2a5" is_pre_election: true
I20240502 01:50:48.536067 6313 raft_consensus.cc:2454] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96086da232f4f44932015ca951fd88b in term 1.
I20240502 01:50:48.537846 6352 leader_election.cc:302] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9458ded85544ed3afba9be26570a2a5, f96086da232f4f44932015ca951fd88b; no voters:
W20240502 01:50:48.538993 6353 leader_election.cc:334] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111)
I20240502 01:50:48.539119 6476 raft_consensus.cc:2781] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:50:48.539595 6476 raft_consensus.cc:482] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:50:48.539906 6476 raft_consensus.cc:3037] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:50:48.547163 6476 raft_consensus.cc:504] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } }
I20240502 01:50:48.548858 6476 leader_election.cc:288] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 2 election: Requested vote from peers c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297), b9458ded85544ed3afba9be26570a2a5 (127.1.233.130:43321)
I20240502 01:50:48.549849 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da800cb86fb24800b59374018d0ce197" candidate_uuid: "f96086da232f4f44932015ca951fd88b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9458ded85544ed3afba9be26570a2a5"
I20240502 01:50:48.550390 6313 raft_consensus.cc:3037] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:50:48.552032 6353 leader_election.cc:334] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111)
I20240502 01:50:48.555548 6313 raft_consensus.cc:2454] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96086da232f4f44932015ca951fd88b in term 2.
I20240502 01:50:48.556718 6352 leader_election.cc:302] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9458ded85544ed3afba9be26570a2a5, f96086da232f4f44932015ca951fd88b; no voters: c9849b524f5747dd993da6e85b21d59a
I20240502 01:50:48.557400 6476 raft_consensus.cc:2781] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:50:48.557806 6476 raft_consensus.cc:686] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 2 LEADER]: Becoming Leader. State: Replica: f96086da232f4f44932015ca951fd88b, State: Running, Role: LEADER
I20240502 01:50:48.558367 6476 consensus_queue.cc:237] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } }
I20240502 01:50:48.565373 6133 catalog_manager.cc:5484] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b reported cstate change: term changed from 1 to 2, leader changed from c9849b524f5747dd993da6e85b21d59a (127.1.233.129) to f96086da232f4f44932015ca951fd88b (127.1.233.131). New cstate: current_term: 2 leader_uuid: "f96086da232f4f44932015ca951fd88b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:48.762342 6567 ts_tablet_manager.cc:902] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4: Initiating tablet copy from peer f96086da232f4f44932015ca951fd88b (127.1.233.131:33345)
I20240502 01:50:48.765789 6567 tablet_copy_client.cc:308] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Beginning tablet copy session from remote peer at address 127.1.233.131:33345
I20240502 01:50:48.775557 6397 tablet_copy_service.cc:140] P f96086da232f4f44932015ca951fd88b: Received BeginTabletCopySession request for tablet 5cada54fde9f4c0498067e9a5ce6962c from peer 4976324b2fd64be59fd9c32fe90f86b4 ({username='slave'} at 127.0.0.1:55668)
I20240502 01:50:48.776033 6397 tablet_copy_service.cc:161] P f96086da232f4f44932015ca951fd88b: Beginning new tablet copy session on tablet 5cada54fde9f4c0498067e9a5ce6962c from peer 4976324b2fd64be59fd9c32fe90f86b4 at {username='slave'} at 127.0.0.1:55668: session id = 4976324b2fd64be59fd9c32fe90f86b4-5cada54fde9f4c0498067e9a5ce6962c
I20240502 01:50:48.781627 6397 tablet_copy_source_session.cc:205] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:48.784122 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cada54fde9f4c0498067e9a5ce6962c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:48.794759 6567 tablet_copy_client.cc:783] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:48.795214 6567 tablet_copy_client.cc:647] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:48.798338 6567 tablet_copy_client.cc:523] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:48.799970 6573 ts_tablet_manager.cc:902] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4: Initiating tablet copy from peer f96086da232f4f44932015ca951fd88b (127.1.233.131:33345)
I20240502 01:50:48.804563 6573 tablet_copy_client.cc:308] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Beginning tablet copy session from remote peer at address 127.1.233.131:33345
I20240502 01:50:48.805963 6397 tablet_copy_service.cc:140] P f96086da232f4f44932015ca951fd88b: Received BeginTabletCopySession request for tablet b0a4ee1f9ba9484896bb256105b96565 from peer 4976324b2fd64be59fd9c32fe90f86b4 ({username='slave'} at 127.0.0.1:55668)
I20240502 01:50:48.806335 6567 tablet_bootstrap.cc:492] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4: Bootstrap starting.
I20240502 01:50:48.806360 6397 tablet_copy_service.cc:161] P f96086da232f4f44932015ca951fd88b: Beginning new tablet copy session on tablet b0a4ee1f9ba9484896bb256105b96565 from peer 4976324b2fd64be59fd9c32fe90f86b4 at {username='slave'} at 127.0.0.1:55668: session id = 4976324b2fd64be59fd9c32fe90f86b4-b0a4ee1f9ba9484896bb256105b96565
I20240502 01:50:48.811673 6397 tablet_copy_source_session.cc:205] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:48.814306 6573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0a4ee1f9ba9484896bb256105b96565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:48.822423 6567 tablet_bootstrap.cc:492] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:48.823143 6567 tablet_bootstrap.cc:492] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4: Bootstrap complete.
I20240502 01:50:48.823681 6567 ts_tablet_manager.cc:1367] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.006s
I20240502 01:50:48.825572 6567 raft_consensus.cc:348] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } }
I20240502 01:50:48.826175 6567 raft_consensus.cc:729] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4976324b2fd64be59fd9c32fe90f86b4, State: Initialized, Role: LEARNER
I20240502 01:50:48.826638 6573 tablet_copy_client.cc:783] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:48.827272 6573 tablet_copy_client.cc:647] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:48.826648 6567 consensus_queue.cc:260] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } }
I20240502 01:50:48.829160 6549 heartbeater.cc:503] Master 127.1.233.190:39943 was elected leader, sending a full tablet report...
I20240502 01:50:48.829583 6567 ts_tablet_manager.cc:1398] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20240502 01:50:48.832053 6397 tablet_copy_service.cc:342] P f96086da232f4f44932015ca951fd88b: Request end of tablet copy session 4976324b2fd64be59fd9c32fe90f86b4-5cada54fde9f4c0498067e9a5ce6962c received from {username='slave'} at 127.0.0.1:55668
I20240502 01:50:48.832216 6573 tablet_copy_client.cc:523] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:48.832489 6397 tablet_copy_service.cc:433] P f96086da232f4f44932015ca951fd88b: ending tablet copy session 4976324b2fd64be59fd9c32fe90f86b4-5cada54fde9f4c0498067e9a5ce6962c on tablet 5cada54fde9f4c0498067e9a5ce6962c with peer 4976324b2fd64be59fd9c32fe90f86b4
I20240502 01:50:48.840546 6573 tablet_bootstrap.cc:492] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4: Bootstrap starting.
I20240502 01:50:48.853957 6573 tablet_bootstrap.cc:492] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:48.854574 6573 tablet_bootstrap.cc:492] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4: Bootstrap complete.
I20240502 01:50:48.855053 6573 ts_tablet_manager.cc:1367] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20240502 01:50:48.856911 6573 raft_consensus.cc:348] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } }
I20240502 01:50:48.857493 6573 raft_consensus.cc:729] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4976324b2fd64be59fd9c32fe90f86b4, State: Initialized, Role: LEARNER
I20240502 01:50:48.857931 6573 consensus_queue.cc:260] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } }
I20240502 01:50:48.859711 6573 ts_tablet_manager.cc:1398] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:50:48.861407 6397 tablet_copy_service.cc:342] P f96086da232f4f44932015ca951fd88b: Request end of tablet copy session 4976324b2fd64be59fd9c32fe90f86b4-b0a4ee1f9ba9484896bb256105b96565 received from {username='slave'} at 127.0.0.1:55668
I20240502 01:50:48.861827 6397 tablet_copy_service.cc:433] P f96086da232f4f44932015ca951fd88b: ending tablet copy session 4976324b2fd64be59fd9c32fe90f86b4-b0a4ee1f9ba9484896bb256105b96565 on tablet b0a4ee1f9ba9484896bb256105b96565 with peer 4976324b2fd64be59fd9c32fe90f86b4
W20240502 01:50:48.919345 6184 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver c9849b524f5747dd993da6e85b21d59a not available for placement
I20240502 01:50:49.072853 6573 ts_tablet_manager.cc:902] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4: Initiating tablet copy from peer b9458ded85544ed3afba9be26570a2a5 (127.1.233.130:43321)
I20240502 01:50:49.076175 6573 tablet_copy_client.cc:308] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Beginning tablet copy session from remote peer at address 127.1.233.130:43321
I20240502 01:50:49.084756 6323 tablet_copy_service.cc:140] P b9458ded85544ed3afba9be26570a2a5: Received BeginTabletCopySession request for tablet 39ca1938892648e1a4658b541a260c6d from peer 4976324b2fd64be59fd9c32fe90f86b4 ({username='slave'} at 127.0.0.1:41298)
I20240502 01:50:49.085251 6323 tablet_copy_service.cc:161] P b9458ded85544ed3afba9be26570a2a5: Beginning new tablet copy session on tablet 39ca1938892648e1a4658b541a260c6d from peer 4976324b2fd64be59fd9c32fe90f86b4 at {username='slave'} at 127.0.0.1:41298: session id = 4976324b2fd64be59fd9c32fe90f86b4-39ca1938892648e1a4658b541a260c6d
I20240502 01:50:49.091255 6323 tablet_copy_source_session.cc:205] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 01:50:49.093886 6353 consensus_peers.cc:485] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b -> Peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Couldn't send request to peer c9849b524f5747dd993da6e85b21d59a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:49.094264 6573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ca1938892648e1a4658b541a260c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:49.095098 6313 raft_consensus.cc:1261] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 2 FOLLOWER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 01:50:49.096508 6476 consensus_queue.cc:1035] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:50:49.110522 6573 tablet_copy_client.cc:783] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:49.111220 6573 tablet_copy_client.cc:647] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:49.114825 6573 tablet_copy_client.cc:523] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:49.121636 6573 tablet_bootstrap.cc:492] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4: Bootstrap starting.
I20240502 01:50:49.134845 6573 tablet_bootstrap.cc:492] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:49.135579 6573 tablet_bootstrap.cc:492] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4: Bootstrap complete.
I20240502 01:50:49.136127 6573 ts_tablet_manager.cc:1367] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.004s
I20240502 01:50:49.137943 6573 raft_consensus.cc:348] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } }
I20240502 01:50:49.138535 6573 raft_consensus.cc:729] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4976324b2fd64be59fd9c32fe90f86b4, State: Initialized, Role: LEARNER
I20240502 01:50:49.138995 6573 consensus_queue.cc:260] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } }
I20240502 01:50:49.140805 6573 ts_tablet_manager.cc:1398] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:50:49.142479 6323 tablet_copy_service.cc:342] P b9458ded85544ed3afba9be26570a2a5: Request end of tablet copy session 4976324b2fd64be59fd9c32fe90f86b4-39ca1938892648e1a4658b541a260c6d received from {username='slave'} at 127.0.0.1:41298
I20240502 01:50:49.142783 6323 tablet_copy_service.cc:433] P b9458ded85544ed3afba9be26570a2a5: ending tablet copy session 4976324b2fd64be59fd9c32fe90f86b4-39ca1938892648e1a4658b541a260c6d on tablet 39ca1938892648e1a4658b541a260c6d with peer 4976324b2fd64be59fd9c32fe90f86b4
I20240502 01:50:49.233209 6524 raft_consensus.cc:1203] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:49.364337 6524 raft_consensus.cc:1203] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:49.381242 6591 consensus_queue.cc:579] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [LEADER]: Leader has been unable to successfully communicate with peer c9849b524f5747dd993da6e85b21d59a for more than 1 seconds (1.016s)
I20240502 01:50:49.596731 6524 raft_consensus.cc:1203] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:50:49.769769 6591 raft_consensus.cc:1050] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b: attempting to promote NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 to VOTER
I20240502 01:50:49.771984 6591 consensus_queue.cc:237] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } }
I20240502 01:50:49.779798 6524 raft_consensus.cc:1261] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 LEARNER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:49.779798 6313 raft_consensus.cc:1261] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:49.781543 6594 consensus_queue.cc:1035] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:50:49.782164 6353 consensus_peers.cc:485] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b -> Peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Couldn't send request to peer c9849b524f5747dd993da6e85b21d59a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:49.783238 6594 consensus_queue.cc:1035] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:49.790236 6599 raft_consensus.cc:2932] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:49.791831 6313 raft_consensus.cc:2932] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:49.799485 6524 raft_consensus.cc:2932] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:49.802443 6132 catalog_manager.cc:5484] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b reported cstate change: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f96086da232f4f44932015ca951fd88b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:50:49.818260 6387 consensus_queue.cc:237] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } }
I20240502 01:50:49.822861 6524 raft_consensus.cc:1261] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:49.822861 6313 raft_consensus.cc:1261] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:49.824424 6594 consensus_queue.cc:1035] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:49.825268 6599 consensus_queue.cc:1035] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:49.831100 6594 raft_consensus.cc:2932] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:49.832757 6313 raft_consensus.cc:2932] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:49.834326 6524 raft_consensus.cc:2932] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:49.840394 6122 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5cada54fde9f4c0498067e9a5ce6962c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:49.845604 6136 catalog_manager.cc:5484] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 reported cstate change: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New cstate: current_term: 1 leader_uuid: "f96086da232f4f44932015ca951fd88b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
W20240502 01:50:49.858999 6123 catalog_manager.cc:4573] TS c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5cada54fde9f4c0498067e9a5ce6962c: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111)
I20240502 01:50:49.877389 6599 raft_consensus.cc:1050] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b: attempting to promote NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 to VOTER
I20240502 01:50:49.879599 6599 consensus_queue.cc:237] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } }
I20240502 01:50:49.885668 6524 raft_consensus.cc:1261] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 LEARNER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:49.885668 6313 raft_consensus.cc:1261] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:49.887621 6476 consensus_queue.cc:1035] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:49.888284 6599 consensus_queue.cc:1035] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:50:49.888842 6353 consensus_peers.cc:485] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b -> Peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Couldn't send request to peer c9849b524f5747dd993da6e85b21d59a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:49.897120 6582 consensus_queue.cc:579] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [LEADER]: Leader has been unable to successfully communicate with peer c9849b524f5747dd993da6e85b21d59a for more than 1 seconds (1.393s)
I20240502 01:50:49.897701 6599 raft_consensus.cc:2932] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:49.900476 6313 raft_consensus.cc:2932] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:49.909830 6524 raft_consensus.cc:2932] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:49.912583 6133 catalog_manager.cc:5484] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b reported cstate change: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f96086da232f4f44932015ca951fd88b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:50:49.922075 6184 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver c9849b524f5747dd993da6e85b21d59a not available for placement
I20240502 01:50:49.924376 6387 consensus_queue.cc:237] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } }
I20240502 01:50:49.929211 6313 raft_consensus.cc:1261] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:49.929256 6524 raft_consensus.cc:1261] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:49.930554 6599 consensus_queue.cc:1035] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:49.931280 6476 consensus_queue.cc:1035] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:49.938350 6594 raft_consensus.cc:2932] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:49.939898 6313 raft_consensus.cc:2932] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:49.941339 6524 raft_consensus.cc:2932] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:49.948261 6122 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0a4ee1f9ba9484896bb256105b96565 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:49.951948 6132 catalog_manager.cc:5484] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4 reported cstate change: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New cstate: current_term: 1 leader_uuid: "f96086da232f4f44932015ca951fd88b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:50.027999 6599 consensus_queue.cc:579] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [LEADER]: Leader has been unable to successfully communicate with peer c9849b524f5747dd993da6e85b21d59a for more than 1 seconds (1.469s)
I20240502 01:50:50.036950 6387 consensus_queue.cc:237] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } }
I20240502 01:50:50.042666 6313 raft_consensus.cc:1261] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 2 FOLLOWER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240502 01:50:50.042960 6353 consensus_peers.cc:485] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b -> Peer 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132:33777): Couldn't send request to peer 4976324b2fd64be59fd9c32fe90f86b4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da800cb86fb24800b59374018d0ce197. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:50.044651 6599 consensus_queue.cc:1035] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:50:50.045913 6353 consensus_peers.cc:485] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b -> Peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Couldn't send request to peer c9849b524f5747dd993da6e85b21d59a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:50.046933 6596 raft_consensus.cc:1050] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5: attempting to promote NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 to VOTER
I20240502 01:50:50.048918 6596 consensus_queue.cc:237] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } }
I20240502 01:50:50.051002 6594 raft_consensus.cc:2932] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } } }
I20240502 01:50:50.052980 6313 raft_consensus.cc:2932] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } } }
I20240502 01:50:50.055528 6524 raft_consensus.cc:1261] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 LEARNER]: Refusing update from remote peer b9458ded85544ed3afba9be26570a2a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:50.056978 6597 consensus_queue.cc:1035] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:50.059728 6387 raft_consensus.cc:1261] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Refusing update from remote peer b9458ded85544ed3afba9be26570a2a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:50:50.062554 6582 consensus_queue.cc:1035] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:50.063378 6122 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet da800cb86fb24800b59374018d0ce197 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 01:50:50.068544 6279 consensus_peers.cc:485] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 -> Peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Couldn't send request to peer c9849b524f5747dd993da6e85b21d59a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:50.070876 6136 catalog_manager.cc:5484] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b reported cstate change: config changed from index -1 to 2, NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) added. New cstate: current_term: 2 leader_uuid: "f96086da232f4f44932015ca951fd88b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:50:50.073982 6597 raft_consensus.cc:2932] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:50.075621 6387 raft_consensus.cc:2932] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:50.089495 6524 raft_consensus.cc:2932] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:50.090420 6136 catalog_manager.cc:5484] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 reported cstate change: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9458ded85544ed3afba9be26570a2a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:50:50.103250 6313 consensus_queue.cc:237] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } }
I20240502 01:50:50.107920 6387 raft_consensus.cc:1261] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Refusing update from remote peer b9458ded85544ed3afba9be26570a2a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:50.107900 6524 raft_consensus.cc:1261] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Refusing update from remote peer b9458ded85544ed3afba9be26570a2a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:50:50.109836 6582 consensus_queue.cc:1035] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:50.110680 6614 consensus_queue.cc:1035] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:50.116670 6597 raft_consensus.cc:2932] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:50.118340 6524 raft_consensus.cc:2932] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:50.123870 6122 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39ca1938892648e1a4658b541a260c6d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:50.124815 6387 raft_consensus.cc:2932] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:50.127974 6132 catalog_manager.cc:5484] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 reported cstate change: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New cstate: current_term: 1 leader_uuid: "b9458ded85544ed3afba9be26570a2a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:50:50.551046 6615 ts_tablet_manager.cc:902] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4: Initiating tablet copy from peer f96086da232f4f44932015ca951fd88b (127.1.233.131:33345)
I20240502 01:50:50.555104 6615 tablet_copy_client.cc:308] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Beginning tablet copy session from remote peer at address 127.1.233.131:33345
I20240502 01:50:50.556437 6397 tablet_copy_service.cc:140] P f96086da232f4f44932015ca951fd88b: Received BeginTabletCopySession request for tablet da800cb86fb24800b59374018d0ce197 from peer 4976324b2fd64be59fd9c32fe90f86b4 ({username='slave'} at 127.0.0.1:55668)
I20240502 01:50:50.556854 6397 tablet_copy_service.cc:161] P f96086da232f4f44932015ca951fd88b: Beginning new tablet copy session on tablet da800cb86fb24800b59374018d0ce197 from peer 4976324b2fd64be59fd9c32fe90f86b4 at {username='slave'} at 127.0.0.1:55668: session id = 4976324b2fd64be59fd9c32fe90f86b4-da800cb86fb24800b59374018d0ce197
I20240502 01:50:50.561695 6397 tablet_copy_source_session.cc:205] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:50:50.564376 6615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da800cb86fb24800b59374018d0ce197. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:50.574860 6615 tablet_copy_client.cc:783] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Starting download of 0 data blocks...
I20240502 01:50:50.575377 6615 tablet_copy_client.cc:647] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:50:50.578532 6615 tablet_copy_client.cc:523] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:50:50.585000 6615 tablet_bootstrap.cc:492] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4: Bootstrap starting.
I20240502 01:50:50.598280 6615 tablet_bootstrap.cc:492] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:50:50.598908 6615 tablet_bootstrap.cc:492] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4: Bootstrap complete.
I20240502 01:50:50.599380 6615 ts_tablet_manager.cc:1367] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20240502 01:50:50.601311 6615 raft_consensus.cc:348] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } }
I20240502 01:50:50.601926 6615 raft_consensus.cc:729] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4976324b2fd64be59fd9c32fe90f86b4, State: Initialized, Role: LEARNER
I20240502 01:50:50.602385 6615 consensus_queue.cc:260] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: true } }
I20240502 01:50:50.604228 6615 ts_tablet_manager.cc:1398] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4: Time spent starting tablet: real 0.005s user 0.005s sys 0.003s
I20240502 01:50:50.605892 6397 tablet_copy_service.cc:342] P f96086da232f4f44932015ca951fd88b: Request end of tablet copy session 4976324b2fd64be59fd9c32fe90f86b4-da800cb86fb24800b59374018d0ce197 received from {username='slave'} at 127.0.0.1:55668
I20240502 01:50:50.606288 6397 tablet_copy_service.cc:433] P f96086da232f4f44932015ca951fd88b: ending tablet copy session 4976324b2fd64be59fd9c32fe90f86b4-da800cb86fb24800b59374018d0ce197 on tablet da800cb86fb24800b59374018d0ce197 with peer 4976324b2fd64be59fd9c32fe90f86b4
W20240502 01:50:50.924532 6184 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver c9849b524f5747dd993da6e85b21d59a not available for placement
W20240502 01:50:50.979070 6123 catalog_manager.cc:4573] TS c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5cada54fde9f4c0498067e9a5ce6962c: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111)
I20240502 01:50:51.098229 6524 raft_consensus.cc:1203] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240502 01:50:51.509254 6622 consensus_queue.cc:579] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [LEADER]: Leader has been unable to successfully communicate with peer c9849b524f5747dd993da6e85b21d59a for more than 1 seconds (1.471s)
I20240502 01:50:51.593138 6599 raft_consensus.cc:1050] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b: attempting to promote NON_VOTER 4976324b2fd64be59fd9c32fe90f86b4 to VOTER
I20240502 01:50:51.595058 6599 consensus_queue.cc:237] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } }
I20240502 01:50:51.601464 6524 raft_consensus.cc:1261] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4 [term 2 LEARNER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:50:51.601476 6313 raft_consensus.cc:1261] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 2 FOLLOWER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:50:51.602905 6626 consensus_queue.cc:1035] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:50:51.603653 6599 consensus_queue.cc:1035] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:50:51.604350 6353 consensus_peers.cc:485] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b -> Peer c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): Couldn't send request to peer c9849b524f5747dd993da6e85b21d59a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:51.611219 6599 raft_consensus.cc:2932] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:51.612908 6313 raft_consensus.cc:2932] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:51.620190 6524 raft_consensus.cc:2932] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:51.622974 6136 catalog_manager.cc:5484] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b reported cstate change: config changed from index 2 to 3, 4976324b2fd64be59fd9c32fe90f86b4 (127.1.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f96086da232f4f44932015ca951fd88b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9849b524f5747dd993da6e85b21d59a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:50:51.638640 6387 consensus_queue.cc:237] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } }
I20240502 01:50:51.643239 6313 raft_consensus.cc:1261] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 2 FOLLOWER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:50:51.643241 6524 raft_consensus.cc:1261] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4 [term 2 FOLLOWER]: Refusing update from remote peer f96086da232f4f44932015ca951fd88b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:50:51.644793 6599 consensus_queue.cc:1035] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:51.645494 6621 consensus_queue.cc:1035] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:50:51.650974 6599 raft_consensus.cc:2932] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:51.652709 6524 raft_consensus.cc:2932] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:51.653748 6313 raft_consensus.cc:2932] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } } }
I20240502 01:50:51.658957 6122 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet da800cb86fb24800b59374018d0ce197 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:50:51.663036 6132 catalog_manager.cc:5484] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b reported cstate change: config changed from index 3 to 4, VOTER c9849b524f5747dd993da6e85b21d59a (127.1.233.129) evicted. New cstate: current_term: 2 leader_uuid: "f96086da232f4f44932015ca951fd88b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f96086da232f4f44932015ca951fd88b" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9458ded85544ed3afba9be26570a2a5" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4976324b2fd64be59fd9c32fe90f86b4" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 33777 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:50:52.031213 6123 catalog_manager.cc:4573] TS c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5cada54fde9f4c0498067e9a5ce6962c: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111)
W20240502 01:50:52.855566 6123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 01:50:53.406899 6123 catalog_manager.cc:4573] TS c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet da800cb86fb24800b59374018d0ce197: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111)
W20240502 01:50:53.552305 6408 debug-util.cc:398] Leaking SignalData structure 0x7b08000d6d60 after lost signal to thread 1961
W20240502 01:50:53.553416 6408 debug-util.cc:398] Leaking SignalData structure 0x7b08001bd6c0 after lost signal to thread 6170
W20240502 01:50:53.554489 6408 debug-util.cc:398] Leaking SignalData structure 0x7b080008f0e0 after lost signal to thread 6337
W20240502 01:50:53.555403 6408 debug-util.cc:398] Leaking SignalData structure 0x7b080013fa40 after lost signal to thread 6411
W20240502 01:50:53.556356 6408 debug-util.cc:398] Leaking SignalData structure 0x7b0800047b40 after lost signal to thread 6548
W20240502 01:50:53.556618 6408 debug-util.cc:398] Leaking SignalData structure 0x7b08001fe000 after lost signal to thread 6606
W20240502 01:50:54.481032 6123 catalog_manager.cc:4573] TS c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet da800cb86fb24800b59374018d0ce197: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111)
I20240502 01:50:55.440654 1958 tablet_server.cc:178] TabletServer@127.1.233.130:0 shutting down...
W20240502 01:50:55.457859 6352 consensus_peers.cc:485] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b -> Peer b9458ded85544ed3afba9be26570a2a5 (127.1.233.130:43321): Couldn't send request to peer b9458ded85544ed3afba9be26570a2a5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:50:55.477458 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:55.478204 1958 tablet_replica.cc:330] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5: stopping tablet replica
I20240502 01:50:55.478813 1958 raft_consensus.cc:2229] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:55.479420 1958 raft_consensus.cc:2258] T da800cb86fb24800b59374018d0ce197 P b9458ded85544ed3afba9be26570a2a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:55.481355 1958 tablet_replica.cc:330] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5: stopping tablet replica
I20240502 01:50:55.481859 1958 raft_consensus.cc:2229] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:55.482481 1958 raft_consensus.cc:2258] T 5cada54fde9f4c0498067e9a5ce6962c P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:55.484375 1958 tablet_replica.cc:330] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5: stopping tablet replica
I20240502 01:50:55.484872 1958 raft_consensus.cc:2229] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:55.485739 1958 raft_consensus.cc:2258] T 39ca1938892648e1a4658b541a260c6d P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:55.487593 1958 tablet_replica.cc:330] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5: stopping tablet replica
I20240502 01:50:55.488039 1958 raft_consensus.cc:2229] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:55.488642 1958 raft_consensus.cc:2258] T b0a4ee1f9ba9484896bb256105b96565 P b9458ded85544ed3afba9be26570a2a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:55.513042 1958 tablet_server.cc:194] TabletServer@127.1.233.130:0 shutdown complete.
I20240502 01:50:55.527740 1958 tablet_server.cc:178] TabletServer@127.1.233.131:0 shutting down...
I20240502 01:50:55.557605 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:55.558475 1958 tablet_replica.cc:330] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b: stopping tablet replica
I20240502 01:50:55.559253 1958 raft_consensus.cc:2229] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:50:55.560436 1958 raft_consensus.cc:2258] T da800cb86fb24800b59374018d0ce197 P f96086da232f4f44932015ca951fd88b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:55.563174 1958 tablet_replica.cc:330] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b: stopping tablet replica
I20240502 01:50:55.563990 1958 raft_consensus.cc:2229] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:55.565091 1958 raft_consensus.cc:2258] T b0a4ee1f9ba9484896bb256105b96565 P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:55.567545 1958 tablet_replica.cc:330] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b: stopping tablet replica
I20240502 01:50:55.568248 1958 raft_consensus.cc:2229] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:55.569304 1958 raft_consensus.cc:2258] T 5cada54fde9f4c0498067e9a5ce6962c P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:55.571736 1958 tablet_replica.cc:330] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b: stopping tablet replica
I20240502 01:50:55.572490 1958 raft_consensus.cc:2229] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:55.573362 1958 raft_consensus.cc:2258] T 39ca1938892648e1a4658b541a260c6d P f96086da232f4f44932015ca951fd88b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:50:55.952678 6184 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver f96086da232f4f44932015ca951fd88b not available for placement
I20240502 01:50:56.229935 1958 tablet_server.cc:194] TabletServer@127.1.233.131:0 shutdown complete.
I20240502 01:50:56.252122 1958 tablet_server.cc:178] TabletServer@127.1.233.132:0 shutting down...
I20240502 01:50:56.275832 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:50:56.276664 1958 tablet_replica.cc:330] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4: stopping tablet replica
I20240502 01:50:56.277433 1958 raft_consensus.cc:2229] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:56.278550 1958 raft_consensus.cc:2258] T da800cb86fb24800b59374018d0ce197 P 4976324b2fd64be59fd9c32fe90f86b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:56.280977 1958 tablet_replica.cc:330] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4: stopping tablet replica
I20240502 01:50:56.281464 1958 raft_consensus.cc:2229] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:56.282418 1958 raft_consensus.cc:2258] T 39ca1938892648e1a4658b541a260c6d P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:56.284741 1958 tablet_replica.cc:330] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4: stopping tablet replica
I20240502 01:50:56.285308 1958 raft_consensus.cc:2229] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:56.286166 1958 raft_consensus.cc:2258] T b0a4ee1f9ba9484896bb256105b96565 P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:56.288339 1958 tablet_replica.cc:330] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4: stopping tablet replica
I20240502 01:50:56.288753 1958 raft_consensus.cc:2229] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:50:56.289547 1958 raft_consensus.cc:2258] T 5cada54fde9f4c0498067e9a5ce6962c P 4976324b2fd64be59fd9c32fe90f86b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:56.313822 1958 tablet_server.cc:194] TabletServer@127.1.233.132:0 shutdown complete.
I20240502 01:50:56.331166 1958 master.cc:537] Master@127.1.233.190:39943 shutting down...
W20240502 01:50:56.583483 6123 catalog_manager.cc:4573] TS c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet da800cb86fb24800b59374018d0ce197: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111)
W20240502 01:50:58.223742 6123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 01:50:58.226718 6123 catalog_manager.cc:4573] TS c9849b524f5747dd993da6e85b21d59a (127.1.233.129:43297): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5cada54fde9f4c0498067e9a5ce6962c: Network error: Client connection negotiation failed: client connection to 127.1.233.129:43297: connect: Connection refused (error 111)
I20240502 01:50:59.230662 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:50:59.231498 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:50:59.231864 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5c58035508a546eb98186d5a534d8b6f: stopping tablet replica
I20240502 01:50:59.253662 1958 master.cc:552] Master@127.1.233.190:39943 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (14516 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20240502 01:50:59.292505 1958 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.233.190:35433
I20240502 01:50:59.293584 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:59.299402 6653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:59.300191 6654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:59.300849 6656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:59.301883 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:59.302769 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:59.302958 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:59.303099 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614659303084 us; error 0 us; skew 500 ppm
I20240502 01:50:59.303706 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:59.306049 1958 webserver.cc:460] Webserver started at http://127.1.233.190:41823/ using document root <none> and password file <none>
I20240502 01:50:59.306475 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:59.306629 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:59.306883 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:59.308132 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/master-0-root/instance:
uuid: "dd278b125980448e86ebc6ba0762b157"
format_stamp: "Formatted at 2024-05-02 01:50:59 on dist-test-slave-2090"
I20240502 01:50:59.312839 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:50:59.316100 6661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:59.317026 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:50:59.317404 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/master-0-root
uuid: "dd278b125980448e86ebc6ba0762b157"
format_stamp: "Formatted at 2024-05-02 01:50:59 on dist-test-slave-2090"
I20240502 01:50:59.317714 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:59.329305 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:59.330438 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:59.367936 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.190:35433
I20240502 01:50:59.368048 6712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.190:35433 every 8 connection(s)
I20240502 01:50:59.371902 6713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:50:59.383973 6713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157: Bootstrap starting.
I20240502 01:50:59.388445 6713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157: Neither blocks nor log segments found. Creating new log.
I20240502 01:50:59.392740 6713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157: No bootstrap required, opened a new log
I20240502 01:50:59.394780 6713 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd278b125980448e86ebc6ba0762b157" member_type: VOTER }
I20240502 01:50:59.395239 6713 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:50:59.395447 6713 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd278b125980448e86ebc6ba0762b157, State: Initialized, Role: FOLLOWER
I20240502 01:50:59.395951 6713 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd278b125980448e86ebc6ba0762b157" member_type: VOTER }
I20240502 01:50:59.396412 6713 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:50:59.396611 6713 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:50:59.396847 6713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:50:59.401479 6713 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd278b125980448e86ebc6ba0762b157" member_type: VOTER }
I20240502 01:50:59.401970 6713 leader_election.cc:302] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd278b125980448e86ebc6ba0762b157; no voters:
I20240502 01:50:59.403122 6713 leader_election.cc:288] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:50:59.403556 6716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:50:59.404863 6716 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [term 1 LEADER]: Becoming Leader. State: Replica: dd278b125980448e86ebc6ba0762b157, State: Running, Role: LEADER
I20240502 01:50:59.405541 6716 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd278b125980448e86ebc6ba0762b157" member_type: VOTER }
I20240502 01:50:59.406033 6713 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:50:59.411746 6718 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd278b125980448e86ebc6ba0762b157. Latest consensus state: current_term: 1 leader_uuid: "dd278b125980448e86ebc6ba0762b157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd278b125980448e86ebc6ba0762b157" member_type: VOTER } }
I20240502 01:50:59.412467 6718 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [sys.catalog]: This master's current role is: LEADER
I20240502 01:50:59.413038 6717 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd278b125980448e86ebc6ba0762b157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd278b125980448e86ebc6ba0762b157" member_type: VOTER } }
I20240502 01:50:59.413718 6717 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [sys.catalog]: This master's current role is: LEADER
I20240502 01:50:59.415127 6725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:50:59.420228 6725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:50:59.421443 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:50:59.429185 6725 catalog_manager.cc:1347] Generated new cluster ID: b7ebb4dad20546c593c2c0024a4149cc
I20240502 01:50:59.429463 6725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:50:59.461046 6725 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:50:59.462450 6725 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:50:59.478483 6725 catalog_manager.cc:5857] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157: Generated new TSK 0
I20240502 01:50:59.479068 6725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:50:59.488323 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:59.495347 6734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:59.496526 6735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:59.497457 6737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:59.498122 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:59.498934 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:59.499123 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:59.499259 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614659499234 us; error 0 us; skew 500 ppm
I20240502 01:50:59.499737 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:59.502137 1958 webserver.cc:460] Webserver started at http://127.1.233.129:43163/ using document root <none> and password file <none>
I20240502 01:50:59.502593 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:59.502748 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:59.502962 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:59.504079 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-0-root/instance:
uuid: "c1318f37ec884127be5a4081d001801f"
format_stamp: "Formatted at 2024-05-02 01:50:59 on dist-test-slave-2090"
I20240502 01:50:59.508991 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:50:59.512264 6742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:59.512984 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:50:59.513263 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "c1318f37ec884127be5a4081d001801f"
format_stamp: "Formatted at 2024-05-02 01:50:59 on dist-test-slave-2090"
I20240502 01:50:59.513564 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:59.540293 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:59.541496 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:59.543063 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:59.545580 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:59.545766 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:59.546017 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:59.546166 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:59.587318 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:36447
I20240502 01:50:59.587433 6804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:36447 every 8 connection(s)
I20240502 01:50:59.592167 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:59.601115 6809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:59.601876 6810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:59.604895 6812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:59.605923 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:59.606781 6805 heartbeater.cc:347] Connected to a master server at 127.1.233.190:35433
I20240502 01:50:59.606945 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:59.607208 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:59.607228 6805 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:59.607573 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614659607549 us; error 0 us; skew 500 ppm
I20240502 01:50:59.608408 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:59.608350 6805 heartbeater.cc:511] Master 127.1.233.190:35433 requested a full tablet report, sending...
I20240502 01:50:59.610941 6678 ts_manager.cc:194] Registered new tserver with Master: c1318f37ec884127be5a4081d001801f (127.1.233.129:36447)
I20240502 01:50:59.611533 1958 webserver.cc:460] Webserver started at http://127.1.233.130:35047/ using document root <none> and password file <none>
I20240502 01:50:59.612032 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:59.612267 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:59.612525 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:59.613104 6678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44006
I20240502 01:50:59.613873 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-1-root/instance:
uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7"
format_stamp: "Formatted at 2024-05-02 01:50:59 on dist-test-slave-2090"
I20240502 01:50:59.619367 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20240502 01:50:59.622838 6817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:59.623697 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:50:59.623994 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-1-root
uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7"
format_stamp: "Formatted at 2024-05-02 01:50:59 on dist-test-slave-2090"
I20240502 01:50:59.624315 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:59.636819 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:59.638074 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:59.639498 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:59.641912 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:59.642150 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:59.642500 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:59.642711 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:59.685050 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.130:43245
I20240502 01:50:59.685164 6879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.130:43245 every 8 connection(s)
I20240502 01:50:59.689817 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:50:59.697365 6883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:59.697819 6884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:50:59.700816 6886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:50:59.701633 1958 server_base.cc:1034] running on GCE node
I20240502 01:50:59.701758 6880 heartbeater.cc:347] Connected to a master server at 127.1.233.190:35433
I20240502 01:50:59.702209 6880 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:59.702811 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:50:59.703097 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:50:59.703222 6880 heartbeater.cc:511] Master 127.1.233.190:35433 requested a full tablet report, sending...
I20240502 01:50:59.703330 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614659703307 us; error 0 us; skew 500 ppm
I20240502 01:50:59.704257 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:50:59.705706 6678 ts_manager.cc:194] Registered new tserver with Master: 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245)
I20240502 01:50:59.707227 1958 webserver.cc:460] Webserver started at http://127.1.233.131:46459/ using document root <none> and password file <none>
I20240502 01:50:59.707767 6678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44018
I20240502 01:50:59.707916 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:50:59.708272 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:50:59.708653 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:50:59.709900 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-2-root/instance:
uuid: "a57d4b3613e24ffc9165c04b1e64eba4"
format_stamp: "Formatted at 2024-05-02 01:50:59 on dist-test-slave-2090"
I20240502 01:50:59.714646 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:50:59.718075 6891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:59.718837 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240502 01:50:59.719132 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-2-root
uuid: "a57d4b3613e24ffc9165c04b1e64eba4"
format_stamp: "Formatted at 2024-05-02 01:50:59 on dist-test-slave-2090"
I20240502 01:50:59.719411 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:50:59.740494 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:50:59.741597 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:50:59.743109 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:50:59.745455 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:50:59.745652 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:59.745870 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:50:59.746017 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:50:59.785511 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.131:46787
I20240502 01:50:59.785619 6953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.131:46787 every 8 connection(s)
I20240502 01:50:59.799028 6954 heartbeater.cc:347] Connected to a master server at 127.1.233.190:35433
I20240502 01:50:59.799397 6954 heartbeater.cc:464] Registering TS with master...
I20240502 01:50:59.800124 6954 heartbeater.cc:511] Master 127.1.233.190:35433 requested a full tablet report, sending...
I20240502 01:50:59.802008 6678 ts_manager.cc:194] Registered new tserver with Master: a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:50:59.802382 1958 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013649737s
I20240502 01:50:59.804009 6678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44028
I20240502 01:51:00.616437 6805 heartbeater.cc:503] Master 127.1.233.190:35433 was elected leader, sending a full tablet report...
I20240502 01:51:00.710613 6880 heartbeater.cc:503] Master 127.1.233.190:35433 was elected leader, sending a full tablet report...
I20240502 01:51:00.807256 6954 heartbeater.cc:503] Master 127.1.233.190:35433 was elected leader, sending a full tablet report...
I20240502 01:51:00.836340 1958 test_util.cc:268] Using random seed: -708236326
I20240502 01:51:00.868793 6678 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44036:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:51:00.871277 6678 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:51:00.931484 6845 tablet_service.cc:1467] Processing CreateTablet for tablet 0335246fb7914702a35bb99ad5bc4705 (DEFAULT_TABLE table=test-workload [id=f3bd73de4f514bf6a8378283b590ae31]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:51:00.932590 6844 tablet_service.cc:1467] Processing CreateTablet for tablet 7c13a16e9e474b07b30df5a73fc6cb50 (DEFAULT_TABLE table=test-workload [id=f3bd73de4f514bf6a8378283b590ae31]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:51:00.933072 6845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0335246fb7914702a35bb99ad5bc4705. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:00.934028 6844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c13a16e9e474b07b30df5a73fc6cb50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:00.935051 6842 tablet_service.cc:1467] Processing CreateTablet for tablet 0dae112f99f04e58ba8843992ad40465 (DEFAULT_TABLE table=test-workload [id=f3bd73de4f514bf6a8378283b590ae31]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:51:00.935671 6919 tablet_service.cc:1467] Processing CreateTablet for tablet 0335246fb7914702a35bb99ad5bc4705 (DEFAULT_TABLE table=test-workload [id=f3bd73de4f514bf6a8378283b590ae31]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:51:00.936311 6842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dae112f99f04e58ba8843992ad40465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:00.936909 6919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0335246fb7914702a35bb99ad5bc4705. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:00.940687 6918 tablet_service.cc:1467] Processing CreateTablet for tablet 7c13a16e9e474b07b30df5a73fc6cb50 (DEFAULT_TABLE table=test-workload [id=f3bd73de4f514bf6a8378283b590ae31]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:51:00.941252 6843 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8b85e1ca954ea7ab30c2361781fc34 (DEFAULT_TABLE table=test-workload [id=f3bd73de4f514bf6a8378283b590ae31]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:51:00.942023 6918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c13a16e9e474b07b30df5a73fc6cb50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:00.942526 6843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8b85e1ca954ea7ab30c2361781fc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:00.946764 6917 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8b85e1ca954ea7ab30c2361781fc34 (DEFAULT_TABLE table=test-workload [id=f3bd73de4f514bf6a8378283b590ae31]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:51:00.948056 6917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8b85e1ca954ea7ab30c2361781fc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:00.952075 6916 tablet_service.cc:1467] Processing CreateTablet for tablet 0dae112f99f04e58ba8843992ad40465 (DEFAULT_TABLE table=test-workload [id=f3bd73de4f514bf6a8378283b590ae31]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:51:00.953492 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dae112f99f04e58ba8843992ad40465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:00.954429 6770 tablet_service.cc:1467] Processing CreateTablet for tablet 0335246fb7914702a35bb99ad5bc4705 (DEFAULT_TABLE table=test-workload [id=f3bd73de4f514bf6a8378283b590ae31]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:51:00.955624 6770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0335246fb7914702a35bb99ad5bc4705. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:00.961405 6769 tablet_service.cc:1467] Processing CreateTablet for tablet 7c13a16e9e474b07b30df5a73fc6cb50 (DEFAULT_TABLE table=test-workload [id=f3bd73de4f514bf6a8378283b590ae31]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:51:00.962826 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c13a16e9e474b07b30df5a73fc6cb50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:00.966408 6768 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8b85e1ca954ea7ab30c2361781fc34 (DEFAULT_TABLE table=test-workload [id=f3bd73de4f514bf6a8378283b590ae31]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:51:00.967783 6768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8b85e1ca954ea7ab30c2361781fc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:00.969273 6767 tablet_service.cc:1467] Processing CreateTablet for tablet 0dae112f99f04e58ba8843992ad40465 (DEFAULT_TABLE table=test-workload [id=f3bd73de4f514bf6a8378283b590ae31]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:51:00.970575 6767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dae112f99f04e58ba8843992ad40465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:00.990980 6974 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap starting.
I20240502 01:51:00.997623 6974 tablet_bootstrap.cc:654] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:01.004037 6975 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap starting.
I20240502 01:51:01.004451 6976 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f: Bootstrap starting.
I20240502 01:51:01.013638 6975 tablet_bootstrap.cc:654] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:01.014328 6976 tablet_bootstrap.cc:654] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:01.019261 6975 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4: No bootstrap required, opened a new log
I20240502 01:51:01.019734 6975 ts_tablet_manager.cc:1367] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.004s
I20240502 01:51:01.020296 6976 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f: No bootstrap required, opened a new log
I20240502 01:51:01.020709 6976 ts_tablet_manager.cc:1367] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.001s
I20240502 01:51:01.021751 6975 raft_consensus.cc:348] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.022274 6975 raft_consensus.cc:374] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:01.022532 6975 raft_consensus.cc:729] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57d4b3613e24ffc9165c04b1e64eba4, State: Initialized, Role: FOLLOWER
I20240502 01:51:01.022545 6976 raft_consensus.cc:348] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.023245 6976 raft_consensus.cc:374] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:01.023474 6976 raft_consensus.cc:729] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1318f37ec884127be5a4081d001801f, State: Initialized, Role: FOLLOWER
I20240502 01:51:01.023376 6975 consensus_queue.cc:260] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.024197 6976 consensus_queue.cc:260] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.026669 6975 ts_tablet_manager.cc:1398] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20240502 01:51:01.026970 6974 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: No bootstrap required, opened a new log
I20240502 01:51:01.027437 6974 ts_tablet_manager.cc:1367] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent bootstrapping tablet: real 0.037s user 0.017s sys 0.006s
I20240502 01:51:01.027698 6975 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap starting.
I20240502 01:51:01.030217 6974 raft_consensus.cc:348] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.031060 6974 raft_consensus.cc:374] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:01.031351 6974 raft_consensus.cc:729] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447a5c3a2b7d4f5d8e6cd800cc9501e7, State: Initialized, Role: FOLLOWER
I20240502 01:51:01.032275 6974 consensus_queue.cc:260] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.034714 6975 tablet_bootstrap.cc:654] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:01.044719 6974 ts_tablet_manager.cc:1398] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent starting tablet: real 0.017s user 0.009s sys 0.004s
I20240502 01:51:01.046018 6974 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap starting.
I20240502 01:51:01.046414 6976 ts_tablet_manager.cc:1398] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f: Time spent starting tablet: real 0.025s user 0.013s sys 0.006s
I20240502 01:51:01.047401 6976 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: Bootstrap starting.
I20240502 01:51:01.051836 6974 tablet_bootstrap.cc:654] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:01.052316 6976 tablet_bootstrap.cc:654] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:01.057075 6975 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4: No bootstrap required, opened a new log
I20240502 01:51:01.057580 6975 ts_tablet_manager.cc:1367] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.003s
I20240502 01:51:01.060130 6975 raft_consensus.cc:348] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.060810 6975 raft_consensus.cc:374] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:01.061085 6975 raft_consensus.cc:729] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57d4b3613e24ffc9165c04b1e64eba4, State: Initialized, Role: FOLLOWER
I20240502 01:51:01.062024 6975 consensus_queue.cc:260] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.066071 6974 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: No bootstrap required, opened a new log
I20240502 01:51:01.066591 6974 ts_tablet_manager.cc:1367] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20240502 01:51:01.067101 6976 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: No bootstrap required, opened a new log
I20240502 01:51:01.067586 6976 ts_tablet_manager.cc:1367] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.011s
I20240502 01:51:01.069123 6975 ts_tablet_manager.cc:1398] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240502 01:51:01.069193 6974 raft_consensus.cc:348] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.069989 6975 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap starting.
I20240502 01:51:01.069985 6974 raft_consensus.cc:374] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:01.070066 6976 raft_consensus.cc:348] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.070722 6974 raft_consensus.cc:729] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447a5c3a2b7d4f5d8e6cd800cc9501e7, State: Initialized, Role: FOLLOWER
I20240502 01:51:01.071082 6976 raft_consensus.cc:374] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:01.071393 6976 raft_consensus.cc:729] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1318f37ec884127be5a4081d001801f, State: Initialized, Role: FOLLOWER
I20240502 01:51:01.071696 6974 consensus_queue.cc:260] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.072202 6976 consensus_queue.cc:260] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.074462 6976 ts_tablet_manager.cc:1398] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 01:51:01.075388 6976 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f: Bootstrap starting.
I20240502 01:51:01.077884 6974 ts_tablet_manager.cc:1398] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20240502 01:51:01.079013 6974 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap starting.
I20240502 01:51:01.081233 6976 tablet_bootstrap.cc:654] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:01.085156 6974 tablet_bootstrap.cc:654] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:01.088591 6982 raft_consensus.cc:482] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:01.089491 6975 tablet_bootstrap.cc:654] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:01.089275 6982 raft_consensus.cc:504] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.093276 6976 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f: No bootstrap required, opened a new log
I20240502 01:51:01.093853 6976 ts_tablet_manager.cc:1367] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20240502 01:51:01.095057 6975 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: No bootstrap required, opened a new log
I20240502 01:51:01.095469 6975 ts_tablet_manager.cc:1367] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.003s
I20240502 01:51:01.096985 6976 raft_consensus.cc:348] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.097513 6975 raft_consensus.cc:348] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.098027 6976 raft_consensus.cc:374] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:01.098167 6975 raft_consensus.cc:374] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:01.098223 6982 leader_election.cc:288] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1318f37ec884127be5a4081d001801f (127.1.233.129:36447), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:01.098536 6976 raft_consensus.cc:729] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1318f37ec884127be5a4081d001801f, State: Initialized, Role: FOLLOWER
I20240502 01:51:01.098948 6975 raft_consensus.cc:729] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57d4b3613e24ffc9165c04b1e64eba4, State: Initialized, Role: FOLLOWER
I20240502 01:51:01.099612 6976 consensus_queue.cc:260] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.100044 6975 consensus_queue.cc:260] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.104710 6975 ts_tablet_manager.cc:1398] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240502 01:51:01.105784 6975 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap starting.
I20240502 01:51:01.106711 6976 ts_tablet_manager.cc:1398] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f: Time spent starting tablet: real 0.013s user 0.009s sys 0.000s
I20240502 01:51:01.106818 6974 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: No bootstrap required, opened a new log
I20240502 01:51:01.107664 6974 ts_tablet_manager.cc:1367] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.008s
I20240502 01:51:01.110347 6980 raft_consensus.cc:482] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:01.112920 6975 tablet_bootstrap.cc:654] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:01.110669 6974 raft_consensus.cc:348] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.113737 6974 raft_consensus.cc:374] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:01.114147 6974 raft_consensus.cc:729] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447a5c3a2b7d4f5d8e6cd800cc9501e7, State: Initialized, Role: FOLLOWER
I20240502 01:51:01.110853 6980 raft_consensus.cc:504] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.122000 6980 leader_election.cc:288] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1318f37ec884127be5a4081d001801f (127.1.233.129:36447), 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245)
I20240502 01:51:01.122520 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c13a16e9e474b07b30df5a73fc6cb50" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" is_pre_election: true
I20240502 01:51:01.123556 6929 raft_consensus.cc:2454] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in term 0.
I20240502 01:51:01.125123 6821 leader_election.cc:302] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7, a57d4b3613e24ffc9165c04b1e64eba4; no voters:
I20240502 01:51:01.112681 6976 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: Bootstrap starting.
I20240502 01:51:01.126865 6780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c13a16e9e474b07b30df5a73fc6cb50" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1318f37ec884127be5a4081d001801f" is_pre_election: true
I20240502 01:51:01.127712 6780 raft_consensus.cc:2454] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in term 0.
I20240502 01:51:01.130160 6982 raft_consensus.cc:2781] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:51:01.132535 6982 raft_consensus.cc:482] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:01.132983 6982 raft_consensus.cc:3037] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:01.134390 6976 tablet_bootstrap.cc:654] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:01.146113 6982 raft_consensus.cc:504] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.149147 6974 consensus_queue.cc:260] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.161042 6982 leader_election.cc:288] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 1 election: Requested vote from peers c1318f37ec884127be5a4081d001801f (127.1.233.129:36447), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:01.162249 6780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c13a16e9e474b07b30df5a73fc6cb50" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1318f37ec884127be5a4081d001801f"
I20240502 01:51:01.162962 6780 raft_consensus.cc:3037] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:01.169128 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c13a16e9e474b07b30df5a73fc6cb50" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4"
I20240502 01:51:01.169842 6929 raft_consensus.cc:3037] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:01.172546 6981 raft_consensus.cc:482] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:01.173115 6981 raft_consensus.cc:504] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.174201 6976 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: No bootstrap required, opened a new log
I20240502 01:51:01.174753 6976 ts_tablet_manager.cc:1367] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: Time spent bootstrapping tablet: real 0.062s user 0.026s sys 0.019s
I20240502 01:51:01.177798 6976 raft_consensus.cc:348] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.178659 6976 raft_consensus.cc:374] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:01.179059 6976 raft_consensus.cc:729] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1318f37ec884127be5a4081d001801f, State: Initialized, Role: FOLLOWER
I20240502 01:51:01.179966 6976 consensus_queue.cc:260] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.181540 6975 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4: No bootstrap required, opened a new log
I20240502 01:51:01.182089 6975 ts_tablet_manager.cc:1367] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent bootstrapping tablet: real 0.077s user 0.018s sys 0.044s
I20240502 01:51:01.183636 6981 leader_election.cc:288] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:01.187572 6780 raft_consensus.cc:2454] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in term 1.
I20240502 01:51:01.184003 6929 raft_consensus.cc:2454] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in term 1.
I20240502 01:51:01.194909 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8b85e1ca954ea7ab30c2361781fc34" candidate_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1318f37ec884127be5a4081d001801f" is_pre_election: true
I20240502 01:51:01.185057 6975 raft_consensus.cc:348] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.195894 6779 raft_consensus.cc:2454] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a57d4b3613e24ffc9165c04b1e64eba4 in term 0.
I20240502 01:51:01.196125 6975 raft_consensus.cc:374] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:01.196640 6975 raft_consensus.cc:729] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57d4b3613e24ffc9165c04b1e64eba4, State: Initialized, Role: FOLLOWER
I20240502 01:51:01.197691 6975 consensus_queue.cc:260] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.201650 6855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8b85e1ca954ea7ab30c2361781fc34" candidate_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" is_pre_election: true
I20240502 01:51:01.204128 6975 ts_tablet_manager.cc:1398] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent starting tablet: real 0.022s user 0.007s sys 0.003s
W20240502 01:51:01.206703 6894 leader_election.cc:341] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:51:01.206969 6976 ts_tablet_manager.cc:1398] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: Time spent starting tablet: real 0.032s user 0.021s sys 0.012s
I20240502 01:51:01.207365 6821 leader_election.cc:302] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7, a57d4b3613e24ffc9165c04b1e64eba4; no voters:
I20240502 01:51:01.207744 6994 raft_consensus.cc:482] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:01.208273 6994 raft_consensus.cc:504] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.208720 6982 raft_consensus.cc:2781] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:51:01.219450 6982 raft_consensus.cc:686] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 LEADER]: Becoming Leader. State: Replica: 447a5c3a2b7d4f5d8e6cd800cc9501e7, State: Running, Role: LEADER
I20240502 01:51:01.220312 6982 consensus_queue.cc:237] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.221988 6994 leader_election.cc:288] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1318f37ec884127be5a4081d001801f (127.1.233.129:36447), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:01.222884 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1318f37ec884127be5a4081d001801f" is_pre_election: true
I20240502 01:51:01.215553 6894 leader_election.cc:302] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a57d4b3613e24ffc9165c04b1e64eba4, c1318f37ec884127be5a4081d001801f; no voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7
I20240502 01:51:01.217825 6855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" is_pre_election: true
I20240502 01:51:01.223610 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" is_pre_election: true
I20240502 01:51:01.223912 6855 raft_consensus.cc:2454] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 0.
I20240502 01:51:01.224314 6929 raft_consensus.cc:2454] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in term 0.
I20240502 01:51:01.224602 6980 raft_consensus.cc:2781] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:51:01.224977 6980 raft_consensus.cc:482] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:01.223654 6779 raft_consensus.cc:2454] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in term 0.
I20240502 01:51:01.225358 6980 raft_consensus.cc:3037] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:01.225304 6745 leader_election.cc:302] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7, c1318f37ec884127be5a4081d001801f; no voters:
I20240502 01:51:01.225325 6821 leader_election.cc:302] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7, a57d4b3613e24ffc9165c04b1e64eba4; no voters:
I20240502 01:51:01.227677 7000 raft_consensus.cc:2781] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:51:01.228091 7000 raft_consensus.cc:482] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:01.228526 7000 raft_consensus.cc:3037] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:01.233793 6980 raft_consensus.cc:504] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.236492 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" is_pre_election: true
I20240502 01:51:01.223945 6974 ts_tablet_manager.cc:1398] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent starting tablet: real 0.116s user 0.012s sys 0.000s
I20240502 01:51:01.237375 6929 raft_consensus.cc:2454] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 0.
I20240502 01:51:01.238221 6974 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap starting.
I20240502 01:51:01.238375 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8b85e1ca954ea7ab30c2361781fc34" candidate_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1318f37ec884127be5a4081d001801f"
I20240502 01:51:01.238945 6779 raft_consensus.cc:3037] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:01.239691 6678 catalog_manager.cc:5484] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130). New cstate: current_term: 1 leader_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } health_report { overall_health: UNKNOWN } } }
I20240502 01:51:01.236305 7000 raft_consensus.cc:504] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.244112 6982 raft_consensus.cc:2781] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:51:01.244604 6974 tablet_bootstrap.cc:654] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:01.244715 7000 leader_election.cc:288] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 1 election: Requested vote from peers 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:01.244566 6982 raft_consensus.cc:482] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:01.245422 6982 raft_consensus.cc:3037] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:01.244480 6855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7"
I20240502 01:51:01.251586 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4"
I20240502 01:51:01.252053 6982 raft_consensus.cc:504] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.252305 6779 raft_consensus.cc:2454] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57d4b3613e24ffc9165c04b1e64eba4 in term 1.
I20240502 01:51:01.252481 6929 raft_consensus.cc:3037] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:01.253492 6855 raft_consensus.cc:2379] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c1318f37ec884127be5a4081d001801f in current term 1: Already voted for candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in this term.
I20240502 01:51:01.255606 6854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8b85e1ca954ea7ab30c2361781fc34" candidate_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7"
I20240502 01:51:01.260721 6982 leader_election.cc:288] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 1 election: Requested vote from peers c1318f37ec884127be5a4081d001801f (127.1.233.129:36447), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:01.260625 6929 raft_consensus.cc:2454] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 1.
I20240502 01:51:01.261148 6928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4"
I20240502 01:51:01.262077 6928 raft_consensus.cc:2379] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in current term 1: Already voted for candidate c1318f37ec884127be5a4081d001801f in this term.
W20240502 01:51:01.258250 6894 leader_election.cc:341] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:51:01.265014 6894 leader_election.cc:302] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a57d4b3613e24ffc9165c04b1e64eba4, c1318f37ec884127be5a4081d001801f; no voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7
I20240502 01:51:01.265137 6746 leader_election.cc:302] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a57d4b3613e24ffc9165c04b1e64eba4, c1318f37ec884127be5a4081d001801f; no voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7
I20240502 01:51:01.253434 6980 leader_election.cc:288] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [CANDIDATE]: Term 1 election: Requested vote from peers c1318f37ec884127be5a4081d001801f (127.1.233.129:36447), 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245)
I20240502 01:51:01.266073 7000 raft_consensus.cc:2781] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:51:01.266813 6980 raft_consensus.cc:2781] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:51:01.266844 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1318f37ec884127be5a4081d001801f"
I20240502 01:51:01.266966 7000 raft_consensus.cc:686] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 1 LEADER]: Becoming Leader. State: Replica: c1318f37ec884127be5a4081d001801f, State: Running, Role: LEADER
I20240502 01:51:01.267588 6980 raft_consensus.cc:686] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 LEADER]: Becoming Leader. State: Replica: a57d4b3613e24ffc9165c04b1e64eba4, State: Running, Role: LEADER
I20240502 01:51:01.268409 6980 consensus_queue.cc:237] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.268432 7000 consensus_queue.cc:237] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.271126 6974 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: No bootstrap required, opened a new log
I20240502 01:51:01.271687 6974 ts_tablet_manager.cc:1367] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.006s
I20240502 01:51:01.275913 6974 raft_consensus.cc:348] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.278156 6974 raft_consensus.cc:374] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:01.278605 6974 raft_consensus.cc:729] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447a5c3a2b7d4f5d8e6cd800cc9501e7, State: Initialized, Role: FOLLOWER
I20240502 01:51:01.279603 6974 consensus_queue.cc:260] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.280581 6678 catalog_manager.cc:5484] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131). New cstate: current_term: 1 leader_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } health_report { overall_health: HEALTHY } } }
I20240502 01:51:01.283077 6974 ts_tablet_manager.cc:1398] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20240502 01:51:01.284603 6820 leader_election.cc:302] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7; no voters: a57d4b3613e24ffc9165c04b1e64eba4, c1318f37ec884127be5a4081d001801f
I20240502 01:51:01.285910 6982 raft_consensus.cc:2726] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:51:01.288278 6676 catalog_manager.cc:5484] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f reported cstate change: term changed from 0 to 1, leader changed from <none> to c1318f37ec884127be5a4081d001801f (127.1.233.129). New cstate: current_term: 1 leader_uuid: "c1318f37ec884127be5a4081d001801f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } health_report { overall_health: UNKNOWN } } }
I20240502 01:51:01.377169 7000 raft_consensus.cc:482] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:01.377692 7000 raft_consensus.cc:504] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.379223 7000 leader_election.cc:288] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:01.380293 6928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dae112f99f04e58ba8843992ad40465" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" is_pre_election: true
I20240502 01:51:01.380302 6855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dae112f99f04e58ba8843992ad40465" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" is_pre_election: true
I20240502 01:51:01.380947 6928 raft_consensus.cc:2454] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 0.
I20240502 01:51:01.381000 6855 raft_consensus.cc:2454] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 0.
I20240502 01:51:01.381872 6746 leader_election.cc:302] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a57d4b3613e24ffc9165c04b1e64eba4, c1318f37ec884127be5a4081d001801f; no voters:
I20240502 01:51:01.382535 7000 raft_consensus.cc:2781] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:51:01.382813 7000 raft_consensus.cc:482] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:01.383064 7000 raft_consensus.cc:3037] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:01.387714 7000 raft_consensus.cc:504] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.389336 7000 leader_election.cc:288] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 1 election: Requested vote from peers 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:01.390117 6855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dae112f99f04e58ba8843992ad40465" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7"
I20240502 01:51:01.390287 6928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dae112f99f04e58ba8843992ad40465" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4"
I20240502 01:51:01.390628 6855 raft_consensus.cc:3037] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:01.390787 6928 raft_consensus.cc:3037] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:01.395741 6928 raft_consensus.cc:2454] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 1.
I20240502 01:51:01.395730 6855 raft_consensus.cc:2454] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 1.
I20240502 01:51:01.396876 6746 leader_election.cc:302] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a57d4b3613e24ffc9165c04b1e64eba4, c1318f37ec884127be5a4081d001801f; no voters:
I20240502 01:51:01.397516 7000 raft_consensus.cc:2781] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:51:01.397867 7000 raft_consensus.cc:686] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 1 LEADER]: Becoming Leader. State: Replica: c1318f37ec884127be5a4081d001801f, State: Running, Role: LEADER
I20240502 01:51:01.398591 7000 consensus_queue.cc:237] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:01.405217 6676 catalog_manager.cc:5484] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f reported cstate change: term changed from 0 to 1, leader changed from <none> to c1318f37ec884127be5a4081d001801f (127.1.233.129). New cstate: current_term: 1 leader_uuid: "c1318f37ec884127be5a4081d001801f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } health_report { overall_health: UNKNOWN } } }
I20240502 01:51:01.414150 1958 tablet_server.cc:178] TabletServer@127.1.233.129:0 shutting down...
I20240502 01:51:01.436307 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:51:01.437150 1958 tablet_replica.cc:330] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: stopping tablet replica
I20240502 01:51:01.437867 1958 raft_consensus.cc:2229] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:51:01.438479 1958 pending_rounds.cc:62] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: Trying to abort 1 pending ops.
I20240502 01:51:01.438730 1958 pending_rounds.cc:68] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023061653096529920 op_type: NO_OP noop_request { }
I20240502 01:51:01.439061 1958 raft_consensus.cc:2866] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 01:51:01.439419 1958 raft_consensus.cc:2258] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:01.441781 1958 tablet_replica.cc:330] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f: stopping tablet replica
I20240502 01:51:01.442317 1958 raft_consensus.cc:2229] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:01.442786 1958 raft_consensus.cc:2258] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:01.444860 1958 tablet_replica.cc:330] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: stopping tablet replica
I20240502 01:51:01.445497 1958 raft_consensus.cc:2229] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:51:01.446111 1958 pending_rounds.cc:62] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: Trying to abort 1 pending ops.
I20240502 01:51:01.446282 1958 pending_rounds.cc:68] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023061652565880832 op_type: NO_OP noop_request { }
I20240502 01:51:01.446556 1958 raft_consensus.cc:2866] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 01:51:01.446803 1958 raft_consensus.cc:2258] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:01.449321 1958 tablet_replica.cc:330] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f: stopping tablet replica
I20240502 01:51:01.449831 1958 raft_consensus.cc:2229] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:01.450196 1958 raft_consensus.cc:2258] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:01.474036 1958 tablet_server.cc:194] TabletServer@127.1.233.129:0 shutdown complete.
I20240502 01:51:01.485606 1958 tablet_server.cc:178] TabletServer@127.1.233.130:0 shutting down...
I20240502 01:51:01.503208 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:51:01.504048 1958 tablet_replica.cc:330] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: stopping tablet replica
I20240502 01:51:01.504648 1958 raft_consensus.cc:2229] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:01.505121 1958 raft_consensus.cc:2258] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:01.507081 1958 tablet_replica.cc:330] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: stopping tablet replica
I20240502 01:51:01.507678 1958 raft_consensus.cc:2229] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:01.508059 1958 raft_consensus.cc:2258] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:01.509866 1958 tablet_replica.cc:330] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: stopping tablet replica
I20240502 01:51:01.510309 1958 raft_consensus.cc:2229] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:01.510627 1958 raft_consensus.cc:2258] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:01.512585 1958 tablet_replica.cc:330] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: stopping tablet replica
I20240502 01:51:01.513044 1958 raft_consensus.cc:2229] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:51:01.513626 1958 pending_rounds.cc:62] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Trying to abort 1 pending ops.
I20240502 01:51:01.513774 1958 pending_rounds.cc:68] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023061652365742080 op_type: NO_OP noop_request { }
I20240502 01:51:01.514047 1958 raft_consensus.cc:2866] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 01:51:01.514271 1958 raft_consensus.cc:2258] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:01.537286 1958 tablet_server.cc:194] TabletServer@127.1.233.130:0 shutdown complete.
I20240502 01:51:01.549048 1958 tablet_server.cc:178] TabletServer@127.1.233.131:0 shutting down...
I20240502 01:51:01.565673 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:51:01.566380 1958 tablet_replica.cc:330] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: stopping tablet replica
I20240502 01:51:01.566902 1958 raft_consensus.cc:2229] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:01.567278 1958 raft_consensus.cc:2258] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:01.569100 1958 tablet_replica.cc:330] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4: stopping tablet replica
I20240502 01:51:01.569617 1958 raft_consensus.cc:2229] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:51:01.570180 1958 pending_rounds.cc:62] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4: Trying to abort 1 pending ops.
I20240502 01:51:01.570334 1958 pending_rounds.cc:68] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023061652564291584 op_type: NO_OP noop_request { }
I20240502 01:51:01.570641 1958 raft_consensus.cc:2866] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 01:51:01.570879 1958 raft_consensus.cc:2258] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:01.573014 1958 tablet_replica.cc:330] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4: stopping tablet replica
I20240502 01:51:01.573523 1958 raft_consensus.cc:2229] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:01.573866 1958 raft_consensus.cc:2258] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:01.575618 1958 tablet_replica.cc:330] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4: stopping tablet replica
I20240502 01:51:01.576025 1958 raft_consensus.cc:2229] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:01.576404 1958 raft_consensus.cc:2258] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:01.597069 1958 tablet_server.cc:194] TabletServer@127.1.233.131:0 shutdown complete.
I20240502 01:51:01.609061 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:51:01.615290 7008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:01.616207 7009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:01.616623 7011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:01.617393 1958 server_base.cc:1034] running on GCE node
I20240502 01:51:01.618386 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:51:01.618695 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:51:01.618952 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614661618938 us; error 0 us; skew 500 ppm
I20240502 01:51:01.619508 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:51:01.621876 1958 webserver.cc:460] Webserver started at http://127.1.233.132:37049/ using document root <none> and password file <none>
I20240502 01:51:01.622411 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:51:01.622704 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:51:01.623049 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:51:01.624336 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-3-root/instance:
uuid: "a12546b72c37471c933460636bb19670"
format_stamp: "Formatted at 2024-05-02 01:51:01 on dist-test-slave-2090"
I20240502 01:51:01.629052 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240502 01:51:01.632277 7016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:01.633173 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:51:01.633567 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-3-root
uuid: "a12546b72c37471c933460636bb19670"
format_stamp: "Formatted at 2024-05-02 01:51:01 on dist-test-slave-2090"
I20240502 01:51:01.633946 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:51:01.645747 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:51:01.646771 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:51:01.648248 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:51:01.650596 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:51:01.650889 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:01.651168 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:51:01.651396 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:01.687845 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.132:37529
I20240502 01:51:01.687963 7078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.132:37529 every 8 connection(s)
I20240502 01:51:01.701710 7079 heartbeater.cc:347] Connected to a master server at 127.1.233.190:35433
I20240502 01:51:01.702217 7079 heartbeater.cc:464] Registering TS with master...
I20240502 01:51:01.703115 7079 heartbeater.cc:511] Master 127.1.233.190:35433 requested a full tablet report, sending...
I20240502 01:51:01.705250 6676 ts_manager.cc:194] Registered new tserver with Master: a12546b72c37471c933460636bb19670 (127.1.233.132:37529)
I20240502 01:51:01.706842 6676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44038
W20240502 01:51:02.440871 6726 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.1.233.130:43245: connect: Connection refused (error 111)
W20240502 01:51:02.451972 6726 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.1.233.130:43245: connect: Connection refused (error 111)
W20240502 01:51:02.452495 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.1.233.130:43245: connect: Connection refused (error 111)
W20240502 01:51:02.456585 6726 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.1.233.131:46787: connect: Connection refused (error 111)
W20240502 01:51:02.456964 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.1.233.131:46787: connect: Connection refused (error 111)
W20240502 01:51:02.460929 6726 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.1.233.129:36447: connect: Connection refused (error 111)
W20240502 01:51:02.461264 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.1.233.129:36447: connect: Connection refused (error 111)
I20240502 01:51:02.709599 7079 heartbeater.cc:503] Master 127.1.233.190:35433 was elected leader, sending a full tablet report...
W20240502 01:51:03.492127 6726 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.1.233.129:36447: connect: Connection refused (error 111)
W20240502 01:51:03.496767 6664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.233.129:36447: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240502 01:51:03.500211 6726 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.1.233.129:36447: connect: Connection refused (error 111)
W20240502 01:51:03.500478 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.1.233.129:36447: connect: Connection refused (error 111)
W20240502 01:51:03.504871 6726 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.1.233.129:36447: connect: Connection refused (error 111)
W20240502 01:51:03.505127 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.1.233.129:36447: connect: Connection refused (error 111)
W20240502 01:51:03.509661 6726 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.1.233.130:43245: connect: Connection refused (error 111)
W20240502 01:51:03.509961 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.1.233.130:43245: connect: Connection refused (error 111)
W20240502 01:51:04.518628 6726 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table f3bd73de4f514bf6a8378283b590ae31: could not find any suitable replica to move from server c1318f37ec884127be5a4081d001801f to server a12546b72c37471c933460636bb19670
W20240502 01:51:05.530663 6726 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.1.233.129:36447: connect: Connection refused (error 111)
W20240502 01:51:05.537523 6726 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.1.233.129:36447: connect: Connection refused (error 111)
W20240502 01:51:05.537797 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.1.233.129:36447: connect: Connection refused (error 111)
W20240502 01:51:05.542057 6726 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.1.233.131:46787: connect: Connection refused (error 111)
W20240502 01:51:05.542310 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.1.233.131:46787: connect: Connection refused (error 111)
W20240502 01:51:05.546640 6726 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.1.233.130:43245: connect: Connection refused (error 111)
W20240502 01:51:05.546883 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.1.233.130:43245: connect: Connection refused (error 111)
W20240502 01:51:06.549608 6726 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 447a5c3a2b7d4f5d8e6cd800cc9501e7 not available for placement
W20240502 01:51:07.552265 6726 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 447a5c3a2b7d4f5d8e6cd800cc9501e7 not available for placement
I20240502 01:51:07.727700 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:51:07.738566 7093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:07.739082 7094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:07.739909 7096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:07.743616 1958 server_base.cc:1034] running on GCE node
I20240502 01:51:07.745383 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:51:07.745635 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:51:07.745785 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614667745773 us; error 0 us; skew 500 ppm
I20240502 01:51:07.746368 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:51:07.749770 1958 webserver.cc:460] Webserver started at http://127.1.233.129:43163/ using document root <none> and password file <none>
I20240502 01:51:07.750312 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:51:07.750476 1958 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240502 01:51:07.754971 1958 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20240502 01:51:07.758759 7101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:07.759630 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:51:07.759946 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "c1318f37ec884127be5a4081d001801f"
format_stamp: "Formatted at 2024-05-02 01:50:59 on dist-test-slave-2090"
I20240502 01:51:07.760435 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:51:07.783404 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:51:07.785082 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:51:07.786847 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:51:07.791034 7108 ts_tablet_manager.cc:522] Loading tablet metadata (0/4 complete)
I20240502 01:51:07.817620 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240502 01:51:07.818017 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.028s user 0.002s sys 0.000s
I20240502 01:51:07.818367 1958 ts_tablet_manager.cc:574] Registering tablets (0/4 complete)
I20240502 01:51:07.824661 7108 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: Bootstrap starting.
I20240502 01:51:07.838249 1958 ts_tablet_manager.cc:590] Registered 4 tablets
I20240502 01:51:07.838524 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.020s user 0.015s sys 0.003s
I20240502 01:51:07.839075 7108 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:51:07.839962 7108 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: Bootstrap complete.
I20240502 01:51:07.840643 7108 ts_tablet_manager.cc:1367] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20240502 01:51:07.843117 7108 raft_consensus.cc:348] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:07.844364 7108 raft_consensus.cc:729] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1318f37ec884127be5a4081d001801f, State: Initialized, Role: FOLLOWER
I20240502 01:51:07.845083 7108 consensus_queue.cc:260] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:07.847733 7108 ts_tablet_manager.cc:1398] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:51:07.848690 7108 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f: Bootstrap starting.
I20240502 01:51:07.862304 7108 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:07.863440 7108 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f: Bootstrap complete.
I20240502 01:51:07.864421 7108 ts_tablet_manager.cc:1367] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20240502 01:51:07.867915 7108 raft_consensus.cc:348] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:07.869027 7108 raft_consensus.cc:729] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1318f37ec884127be5a4081d001801f, State: Initialized, Role: FOLLOWER
I20240502 01:51:07.870021 7108 consensus_queue.cc:260] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:07.873044 7108 ts_tablet_manager.cc:1398] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20240502 01:51:07.874224 7108 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: Bootstrap starting.
I20240502 01:51:07.892760 7108 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:51:07.893703 7108 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: Bootstrap complete.
I20240502 01:51:07.894353 7108 ts_tablet_manager.cc:1367] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.012s
I20240502 01:51:07.896916 7108 raft_consensus.cc:348] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:07.897670 7108 raft_consensus.cc:729] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1318f37ec884127be5a4081d001801f, State: Initialized, Role: FOLLOWER
I20240502 01:51:07.898259 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:36447
I20240502 01:51:07.898375 7168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:36447 every 8 connection(s)
I20240502 01:51:07.898376 7108 consensus_queue.cc:260] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:07.900522 7108 ts_tablet_manager.cc:1398] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240502 01:51:07.901273 7108 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f: Bootstrap starting.
I20240502 01:51:07.905159 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:51:07.910442 7170 tablet.cc:2369] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:51:07.920640 7108 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 01:51:07.921306 7175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:07.922187 7108 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f: Bootstrap complete.
I20240502 01:51:07.923426 7108 ts_tablet_manager.cc:1367] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20240502 01:51:07.927582 7108 raft_consensus.cc:348] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:07.928730 7108 raft_consensus.cc:729] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1318f37ec884127be5a4081d001801f, State: Initialized, Role: FOLLOWER
I20240502 01:51:07.930266 7108 consensus_queue.cc:260] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
W20240502 01:51:07.928779 7173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:07.933985 1958 server_base.cc:1034] running on GCE node
I20240502 01:51:07.934720 7169 heartbeater.cc:347] Connected to a master server at 127.1.233.190:35433
I20240502 01:51:07.934186 7108 ts_tablet_manager.cc:1398] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240502 01:51:07.935204 7169 heartbeater.cc:464] Registering TS with master...
I20240502 01:51:07.936199 7169 heartbeater.cc:511] Master 127.1.233.190:35433 requested a full tablet report, sending...
W20240502 01:51:07.937273 7177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:07.938333 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:51:07.938591 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:51:07.938776 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614667938761 us; error 0 us; skew 500 ppm
I20240502 01:51:07.939532 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:51:07.940855 6676 ts_manager.cc:194] Re-registered known tserver with Master: c1318f37ec884127be5a4081d001801f (127.1.233.129:36447)
I20240502 01:51:07.942975 1958 webserver.cc:460] Webserver started at http://127.1.233.130:35047/ using document root <none> and password file <none>
I20240502 01:51:07.943711 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:51:07.943948 1958 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240502 01:51:07.948657 6676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50320
I20240502 01:51:07.949657 1958 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:51:07.952185 7169 heartbeater.cc:503] Master 127.1.233.190:35433 was elected leader, sending a full tablet report...
I20240502 01:51:07.953809 7182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:07.954836 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:51:07.955214 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-1-root
uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7"
format_stamp: "Formatted at 2024-05-02 01:50:59 on dist-test-slave-2090"
I20240502 01:51:07.955505 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:51:07.975620 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:51:07.976898 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:51:07.978417 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:51:08.007392 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240502 01:51:08.007722 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.027s user 0.001s sys 0.001s
I20240502 01:51:08.013198 7189 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap starting.
I20240502 01:51:08.024891 7189 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:08.024909 1958 ts_tablet_manager.cc:590] Registered 4 tablets
I20240502 01:51:08.025324 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.017s user 0.018s sys 0.000s
I20240502 01:51:08.025719 7189 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap complete.
I20240502 01:51:08.026351 7189 ts_tablet_manager.cc:1367] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20240502 01:51:08.028450 7189 raft_consensus.cc:348] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.029044 7189 raft_consensus.cc:729] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447a5c3a2b7d4f5d8e6cd800cc9501e7, State: Initialized, Role: FOLLOWER
I20240502 01:51:08.029652 7189 consensus_queue.cc:260] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.032481 7189 ts_tablet_manager.cc:1398] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240502 01:51:08.033210 7189 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap starting.
I20240502 01:51:08.048395 7189 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:51:08.049312 7189 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap complete.
I20240502 01:51:08.049948 7189 ts_tablet_manager.cc:1367] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.006s
I20240502 01:51:08.052114 7189 raft_consensus.cc:348] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.052943 7189 raft_consensus.cc:729] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447a5c3a2b7d4f5d8e6cd800cc9501e7, State: Initialized, Role: FOLLOWER
I20240502 01:51:08.053622 7189 consensus_queue.cc:260] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.055361 7189 ts_tablet_manager.cc:1398] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240502 01:51:08.056214 7189 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap starting.
I20240502 01:51:08.068655 7189 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:08.069432 7189 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap complete.
I20240502 01:51:08.070125 7189 ts_tablet_manager.cc:1367] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20240502 01:51:08.072536 7189 raft_consensus.cc:348] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.073195 7189 raft_consensus.cc:729] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447a5c3a2b7d4f5d8e6cd800cc9501e7, State: Initialized, Role: FOLLOWER
I20240502 01:51:08.073889 7189 consensus_queue.cc:260] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.075642 7189 ts_tablet_manager.cc:1398] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:51:08.076483 7189 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap starting.
I20240502 01:51:08.088743 7189 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:08.089280 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.130:43245
I20240502 01:51:08.089540 7189 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Bootstrap complete.
I20240502 01:51:08.089540 7250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.130:43245 every 8 connection(s)
I20240502 01:51:08.090526 7189 ts_tablet_manager.cc:1367] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240502 01:51:08.092823 7189 raft_consensus.cc:348] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.093827 7189 raft_consensus.cc:374] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:08.094238 7189 raft_consensus.cc:729] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447a5c3a2b7d4f5d8e6cd800cc9501e7, State: Initialized, Role: FOLLOWER
I20240502 01:51:08.094406 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:51:08.095247 7189 consensus_queue.cc:260] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.097906 7189 ts_tablet_manager.cc:1398] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
W20240502 01:51:08.106670 7254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:08.108587 7255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:08.110147 7251 heartbeater.cc:347] Connected to a master server at 127.1.233.190:35433
I20240502 01:51:08.110534 7251 heartbeater.cc:464] Registering TS with master...
W20240502 01:51:08.110886 7257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:08.111368 7251 heartbeater.cc:511] Master 127.1.233.190:35433 requested a full tablet report, sending...
I20240502 01:51:08.111639 1958 server_base.cc:1034] running on GCE node
I20240502 01:51:08.112917 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:51:08.113206 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:51:08.113446 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614668113422 us; error 0 us; skew 500 ppm
I20240502 01:51:08.114293 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:51:08.116737 6676 ts_manager.cc:194] Re-registered known tserver with Master: 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245)
I20240502 01:51:08.117617 1958 webserver.cc:460] Webserver started at http://127.1.233.131:46459/ using document root <none> and password file <none>
I20240502 01:51:08.118328 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:51:08.118606 1958 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240502 01:51:08.123596 6676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50330
I20240502 01:51:08.123996 1958 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:51:08.127043 7251 heartbeater.cc:503] Master 127.1.233.190:35433 was elected leader, sending a full tablet report...
I20240502 01:51:08.128429 7262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:08.129392 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:51:08.129724 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-2-root
uuid: "a57d4b3613e24ffc9165c04b1e64eba4"
format_stamp: "Formatted at 2024-05-02 01:50:59 on dist-test-slave-2090"
I20240502 01:51:08.130034 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:51:08.149296 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:51:08.150616 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:51:08.152216 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:51:08.181329 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240502 01:51:08.181699 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.027s user 0.002s sys 0.000s
I20240502 01:51:08.187203 7269 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap starting.
I20240502 01:51:08.198455 7269 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:08.199172 7269 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap complete.
I20240502 01:51:08.199421 1958 ts_tablet_manager.cc:590] Registered 4 tablets
I20240502 01:51:08.199695 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.018s user 0.015s sys 0.000s
I20240502 01:51:08.199824 7269 ts_tablet_manager.cc:1367] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240502 01:51:08.202267 7269 raft_consensus.cc:348] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.202867 7269 raft_consensus.cc:729] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57d4b3613e24ffc9165c04b1e64eba4, State: Initialized, Role: FOLLOWER
I20240502 01:51:08.203572 7269 consensus_queue.cc:260] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.205955 7269 ts_tablet_manager.cc:1398] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240502 01:51:08.206720 7269 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap starting.
I20240502 01:51:08.220892 7269 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:08.222019 7269 tablet_bootstrap.cc:492] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap complete.
I20240502 01:51:08.222970 7269 ts_tablet_manager.cc:1367] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20240502 01:51:08.226377 7269 raft_consensus.cc:348] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.227418 7269 raft_consensus.cc:729] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57d4b3613e24ffc9165c04b1e64eba4, State: Initialized, Role: FOLLOWER
I20240502 01:51:08.228395 7269 consensus_queue.cc:260] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.231178 7269 ts_tablet_manager.cc:1398] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 01:51:08.232389 7269 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap starting.
I20240502 01:51:08.249598 7269 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:08.250661 7269 tablet_bootstrap.cc:492] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap complete.
I20240502 01:51:08.251461 7269 ts_tablet_manager.cc:1367] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.007s
I20240502 01:51:08.254621 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.131:46787
I20240502 01:51:08.254788 7329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.131:46787 every 8 connection(s)
I20240502 01:51:08.254977 7269 raft_consensus.cc:348] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.255825 7269 raft_consensus.cc:729] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57d4b3613e24ffc9165c04b1e64eba4, State: Initialized, Role: FOLLOWER
I20240502 01:51:08.256691 7269 consensus_queue.cc:260] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.259682 7269 ts_tablet_manager.cc:1398] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20240502 01:51:08.261051 7269 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap starting.
I20240502 01:51:08.272346 7330 heartbeater.cc:347] Connected to a master server at 127.1.233.190:35433
I20240502 01:51:08.273089 7330 heartbeater.cc:464] Registering TS with master...
I20240502 01:51:08.274207 7330 heartbeater.cc:511] Master 127.1.233.190:35433 requested a full tablet report, sending...
I20240502 01:51:08.277518 7269 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:51:08.278731 7269 tablet_bootstrap.cc:492] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4: Bootstrap complete.
I20240502 01:51:08.279444 6676 ts_manager.cc:194] Re-registered known tserver with Master: a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:08.279661 7269 ts_tablet_manager.cc:1367] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.000s
I20240502 01:51:08.283149 7269 raft_consensus.cc:348] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.284487 7269 raft_consensus.cc:729] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57d4b3613e24ffc9165c04b1e64eba4, State: Initialized, Role: FOLLOWER
I20240502 01:51:08.285526 7269 consensus_queue.cc:260] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.286577 6676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50346
I20240502 01:51:08.288022 7269 ts_tablet_manager.cc:1398] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:51:08.289790 7330 heartbeater.cc:503] Master 127.1.233.190:35433 was elected leader, sending a full tablet report...
I20240502 01:51:08.537531 7192 raft_consensus.cc:482] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:08.538343 7192 raft_consensus.cc:504] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.540627 7192 leader_election.cc:288] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1318f37ec884127be5a4081d001801f (127.1.233.129:36447), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:08.555905 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8b85e1ca954ea7ab30c2361781fc34" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" is_pre_election: true
I20240502 01:51:08.555917 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8b85e1ca954ea7ab30c2361781fc34" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1318f37ec884127be5a4081d001801f" is_pre_election: true
I20240502 01:51:08.557389 7304 raft_consensus.cc:2379] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in current term 1: Already voted for candidate a57d4b3613e24ffc9165c04b1e64eba4 in this term.
I20240502 01:51:08.557775 7143 raft_consensus.cc:2379] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in current term 1: Already voted for candidate a57d4b3613e24ffc9165c04b1e64eba4 in this term.
I20240502 01:51:08.559365 7185 leader_election.cc:302] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7; no voters: a57d4b3613e24ffc9165c04b1e64eba4, c1318f37ec884127be5a4081d001801f
I20240502 01:51:08.560637 7192 raft_consensus.cc:3037] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:08.568097 7192 raft_consensus.cc:2726] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 01:51:08.576891 6726 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica c1318f37ec884127be5a4081d001801f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } } }
W20240502 01:51:08.581040 6726 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0dae112f99f04e58ba8843992ad40465, TS 447a5c3a2b7d4f5d8e6cd800cc9501e7 -> TS a12546b72c37471c933460636bb19670 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
W20240502 01:51:08.581542 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0dae112f99f04e58ba8843992ad40465, TS 447a5c3a2b7d4f5d8e6cd800cc9501e7 -> TS a12546b72c37471c933460636bb19670 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
W20240502 01:51:08.585278 6726 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0335246fb7914702a35bb99ad5bc4705, TS a57d4b3613e24ffc9165c04b1e64eba4 -> TS a12546b72c37471c933460636bb19670 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
W20240502 01:51:08.585719 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0335246fb7914702a35bb99ad5bc4705, TS a57d4b3613e24ffc9165c04b1e64eba4 -> TS a12546b72c37471c933460636bb19670 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
W20240502 01:51:08.599370 6726 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7c13a16e9e474b07b30df5a73fc6cb50, TS c1318f37ec884127be5a4081d001801f -> TS a12546b72c37471c933460636bb19670 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
W20240502 01:51:08.599815 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7c13a16e9e474b07b30df5a73fc6cb50, TS c1318f37ec884127be5a4081d001801f -> TS a12546b72c37471c933460636bb19670 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.986989 7339 raft_consensus.cc:482] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:08.987540 7339 raft_consensus.cc:504] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:08.989413 7339 leader_election.cc:288] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:08.999197 7224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dae112f99f04e58ba8843992ad40465" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" is_pre_election: true
I20240502 01:51:08.999445 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dae112f99f04e58ba8843992ad40465" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" is_pre_election: true
I20240502 01:51:08.999877 7224 raft_consensus.cc:2454] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 1.
I20240502 01:51:09.000069 7304 raft_consensus.cc:2454] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 1.
I20240502 01:51:09.001073 7104 leader_election.cc:302] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7, c1318f37ec884127be5a4081d001801f; no voters:
I20240502 01:51:09.001787 7339 raft_consensus.cc:2781] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:51:09.002141 7339 raft_consensus.cc:482] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:09.002393 7339 raft_consensus.cc:3037] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:51:09.007846 7339 raft_consensus.cc:504] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.009557 7339 leader_election.cc:288] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 2 election: Requested vote from peers 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:09.010449 7224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dae112f99f04e58ba8843992ad40465" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7"
I20240502 01:51:09.010630 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dae112f99f04e58ba8843992ad40465" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4"
I20240502 01:51:09.010970 7224 raft_consensus.cc:3037] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:51:09.011162 7304 raft_consensus.cc:3037] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:51:09.016396 7224 raft_consensus.cc:2454] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 2.
I20240502 01:51:09.016830 7304 raft_consensus.cc:2454] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 2.
I20240502 01:51:09.017453 7104 leader_election.cc:302] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7, c1318f37ec884127be5a4081d001801f; no voters:
I20240502 01:51:09.018296 7339 raft_consensus.cc:2781] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:51:09.019189 7339 raft_consensus.cc:686] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 2 LEADER]: Becoming Leader. State: Replica: c1318f37ec884127be5a4081d001801f, State: Running, Role: LEADER
I20240502 01:51:09.019950 7339 consensus_queue.cc:237] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.026682 6676 catalog_manager.cc:5484] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "c1318f37ec884127be5a4081d001801f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } health_report { overall_health: UNKNOWN } } }
I20240502 01:51:09.209969 7348 raft_consensus.cc:482] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:09.210453 7348 raft_consensus.cc:504] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.212275 7348 leader_election.cc:288] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1318f37ec884127be5a4081d001801f (127.1.233.129:36447), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:09.213111 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1318f37ec884127be5a4081d001801f" is_pre_election: true
I20240502 01:51:09.213383 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" is_pre_election: true
I20240502 01:51:09.213948 7143 raft_consensus.cc:2396] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 01:51:09.213980 7304 raft_consensus.cc:2454] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in term 1.
I20240502 01:51:09.215222 7186 leader_election.cc:302] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7, a57d4b3613e24ffc9165c04b1e64eba4; no voters: c1318f37ec884127be5a4081d001801f
I20240502 01:51:09.215943 7348 raft_consensus.cc:2781] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:51:09.216243 7348 raft_consensus.cc:482] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:09.216492 7348 raft_consensus.cc:3037] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:51:09.221987 7348 raft_consensus.cc:504] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.223704 7348 leader_election.cc:288] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 2 election: Requested vote from peers c1318f37ec884127be5a4081d001801f (127.1.233.129:36447), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:09.224622 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1318f37ec884127be5a4081d001801f"
I20240502 01:51:09.224800 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4"
I20240502 01:51:09.225144 7143 raft_consensus.cc:3037] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:51:09.225343 7304 raft_consensus.cc:3037] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:51:09.230527 7304 raft_consensus.cc:2454] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in term 2.
I20240502 01:51:09.230818 7143 raft_consensus.cc:2396] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 01:51:09.231655 7186 leader_election.cc:302] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7, a57d4b3613e24ffc9165c04b1e64eba4; no voters:
I20240502 01:51:09.232509 7348 raft_consensus.cc:2781] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:51:09.233398 7348 raft_consensus.cc:686] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 LEADER]: Becoming Leader. State: Replica: 447a5c3a2b7d4f5d8e6cd800cc9501e7, State: Running, Role: LEADER
I20240502 01:51:09.234236 7348 consensus_queue.cc:237] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.241245 6676 catalog_manager.cc:5484] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 reported cstate change: term changed from 1 to 2, leader changed from c1318f37ec884127be5a4081d001801f (127.1.233.129) to 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130). New cstate: current_term: 2 leader_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } health_report { overall_health: UNKNOWN } } }
I20240502 01:51:09.293772 7339 raft_consensus.cc:482] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:09.294363 7339 raft_consensus.cc:504] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.295984 7339 leader_election.cc:288] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:09.296963 7224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c13a16e9e474b07b30df5a73fc6cb50" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" is_pre_election: true
I20240502 01:51:09.297065 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c13a16e9e474b07b30df5a73fc6cb50" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" is_pre_election: true
I20240502 01:51:09.297746 7224 raft_consensus.cc:2396] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c1318f37ec884127be5a4081d001801f for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 01:51:09.297675 7304 raft_consensus.cc:2454] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 1.
I20240502 01:51:09.298826 7105 leader_election.cc:302] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a57d4b3613e24ffc9165c04b1e64eba4, c1318f37ec884127be5a4081d001801f; no voters:
I20240502 01:51:09.299539 7339 raft_consensus.cc:2781] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:51:09.299870 7339 raft_consensus.cc:482] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:09.300233 7339 raft_consensus.cc:3037] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:51:09.305713 7339 raft_consensus.cc:504] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.307468 7339 leader_election.cc:288] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 2 election: Requested vote from peers 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:09.308399 7224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c13a16e9e474b07b30df5a73fc6cb50" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7"
I20240502 01:51:09.308642 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c13a16e9e474b07b30df5a73fc6cb50" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4"
I20240502 01:51:09.309093 7224 raft_consensus.cc:3037] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:51:09.309185 7304 raft_consensus.cc:3037] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:51:09.316455 7304 raft_consensus.cc:2454] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 2.
I20240502 01:51:09.316833 7224 raft_consensus.cc:2396] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c1318f37ec884127be5a4081d001801f for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 01:51:09.317662 7105 leader_election.cc:302] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a57d4b3613e24ffc9165c04b1e64eba4, c1318f37ec884127be5a4081d001801f; no voters:
I20240502 01:51:09.318578 7339 raft_consensus.cc:2781] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:51:09.319022 7339 raft_consensus.cc:686] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 2 LEADER]: Becoming Leader. State: Replica: c1318f37ec884127be5a4081d001801f, State: Running, Role: LEADER
I20240502 01:51:09.319851 7339 consensus_queue.cc:237] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.326292 6676 catalog_manager.cc:5484] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f reported cstate change: term changed from 1 to 2, leader changed from 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130) to c1318f37ec884127be5a4081d001801f (127.1.233.129). New cstate: current_term: 2 leader_uuid: "c1318f37ec884127be5a4081d001801f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } health_report { overall_health: UNKNOWN } } }
I20240502 01:51:09.418838 7304 raft_consensus.cc:1261] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Refusing update from remote peer c1318f37ec884127be5a4081d001801f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:51:09.420212 7339 consensus_queue.cc:1035] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:51:09.434059 7224 raft_consensus.cc:1261] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Refusing update from remote peer c1318f37ec884127be5a4081d001801f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:51:09.435762 7339 consensus_queue.cc:1035] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Connected to new peer: Peer: permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:51:09.439980 7349 raft_consensus.cc:482] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:09.440555 7349 raft_consensus.cc:504] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.442837 7349 leader_election.cc:288] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1318f37ec884127be5a4081d001801f (127.1.233.129:36447), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:09.443620 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c13a16e9e474b07b30df5a73fc6cb50" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c1318f37ec884127be5a4081d001801f" is_pre_election: true
I20240502 01:51:09.445329 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c13a16e9e474b07b30df5a73fc6cb50" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" is_pre_election: true
I20240502 01:51:09.446120 7304 raft_consensus.cc:2454] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in term 2.
I20240502 01:51:09.447266 7186 leader_election.cc:302] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7, a57d4b3613e24ffc9165c04b1e64eba4; no voters: c1318f37ec884127be5a4081d001801f
I20240502 01:51:09.448949 7349 raft_consensus.cc:2781] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 01:51:09.449331 7349 raft_consensus.cc:482] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:09.449700 7349 raft_consensus.cc:3037] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:51:09.456012 7349 raft_consensus.cc:504] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.457800 7349 leader_election.cc:288] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 3 election: Requested vote from peers c1318f37ec884127be5a4081d001801f (127.1.233.129:36447), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:09.458575 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c13a16e9e474b07b30df5a73fc6cb50" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c1318f37ec884127be5a4081d001801f"
I20240502 01:51:09.458818 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c13a16e9e474b07b30df5a73fc6cb50" candidate_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4"
I20240502 01:51:09.459503 7304 raft_consensus.cc:3037] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:51:09.464924 7304 raft_consensus.cc:2454] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 447a5c3a2b7d4f5d8e6cd800cc9501e7 in term 3.
I20240502 01:51:09.465935 7186 leader_election.cc:302] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7, a57d4b3613e24ffc9165c04b1e64eba4; no voters: c1318f37ec884127be5a4081d001801f
I20240502 01:51:09.466667 7349 raft_consensus.cc:2781] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:51:09.467094 7349 raft_consensus.cc:686] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 3 LEADER]: Becoming Leader. State: Replica: 447a5c3a2b7d4f5d8e6cd800cc9501e7, State: Running, Role: LEADER
I20240502 01:51:09.467594 7357 raft_consensus.cc:482] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:09.468016 7357 raft_consensus.cc:504] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.467970 7349 consensus_queue.cc:237] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.470204 7357 leader_election.cc:288] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1318f37ec884127be5a4081d001801f (127.1.233.129:36447), 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245)
I20240502 01:51:09.474365 7339 raft_consensus.cc:482] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:09.474939 7339 raft_consensus.cc:504] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.477108 7339 leader_election.cc:288] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:09.477957 7224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" is_pre_election: true
I20240502 01:51:09.478520 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" is_pre_election: true
I20240502 01:51:09.479240 7304 raft_consensus.cc:2454] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 2.
I20240502 01:51:09.478713 6677 catalog_manager.cc:5484] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 reported cstate change: term changed from 2 to 3, leader changed from c1318f37ec884127be5a4081d001801f (127.1.233.129) to 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130). New cstate: current_term: 3 leader_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } health_report { overall_health: UNKNOWN } } }
I20240502 01:51:09.480269 7105 leader_election.cc:302] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a57d4b3613e24ffc9165c04b1e64eba4, c1318f37ec884127be5a4081d001801f; no voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7
I20240502 01:51:09.481420 7339 raft_consensus.cc:2781] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 01:51:09.481757 7339 raft_consensus.cc:482] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:09.482074 7339 raft_consensus.cc:3037] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:51:09.489722 7339 raft_consensus.cc:504] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.490725 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8b85e1ca954ea7ab30c2361781fc34" candidate_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c1318f37ec884127be5a4081d001801f" is_pre_election: true
I20240502 01:51:09.491472 7143 raft_consensus.cc:2454] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a57d4b3613e24ffc9165c04b1e64eba4 in term 1.
I20240502 01:51:09.491822 7224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8b85e1ca954ea7ab30c2361781fc34" candidate_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" is_pre_election: true
I20240502 01:51:09.492580 7224 raft_consensus.cc:2454] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a57d4b3613e24ffc9165c04b1e64eba4 in term 1.
I20240502 01:51:09.492919 7265 leader_election.cc:302] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a57d4b3613e24ffc9165c04b1e64eba4, c1318f37ec884127be5a4081d001801f; no voters:
I20240502 01:51:09.492967 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a57d4b3613e24ffc9165c04b1e64eba4"
I20240502 01:51:09.493728 7339 leader_election.cc:288] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 3 election: Requested vote from peers 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245), a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787)
I20240502 01:51:09.493139 7223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0335246fb7914702a35bb99ad5bc4705" candidate_uuid: "c1318f37ec884127be5a4081d001801f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7"
I20240502 01:51:09.493894 7357 raft_consensus.cc:2781] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:51:09.493917 7304 raft_consensus.cc:3037] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:51:09.494586 7357 raft_consensus.cc:482] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:09.494997 7357 raft_consensus.cc:3037] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:51:09.500276 7304 raft_consensus.cc:2454] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1318f37ec884127be5a4081d001801f in term 3.
I20240502 01:51:09.501464 7105 leader_election.cc:302] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a57d4b3613e24ffc9165c04b1e64eba4, c1318f37ec884127be5a4081d001801f; no voters: 447a5c3a2b7d4f5d8e6cd800cc9501e7
I20240502 01:51:09.501658 7357 raft_consensus.cc:504] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.502368 7339 raft_consensus.cc:2781] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:51:09.502794 7339 raft_consensus.cc:686] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 3 LEADER]: Becoming Leader. State: Replica: c1318f37ec884127be5a4081d001801f, State: Running, Role: LEADER
I20240502 01:51:09.503686 7357 leader_election.cc:288] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [CANDIDATE]: Term 2 election: Requested vote from peers c1318f37ec884127be5a4081d001801f (127.1.233.129:36447), 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130:43245)
I20240502 01:51:09.503567 7339 consensus_queue.cc:237] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.504729 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8b85e1ca954ea7ab30c2361781fc34" candidate_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c1318f37ec884127be5a4081d001801f"
I20240502 01:51:09.504951 7223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8b85e1ca954ea7ab30c2361781fc34" candidate_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7"
I20240502 01:51:09.505275 7143 raft_consensus.cc:3037] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:51:09.505635 7223 raft_consensus.cc:3037] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:51:09.511771 7143 raft_consensus.cc:2454] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57d4b3613e24ffc9165c04b1e64eba4 in term 2.
I20240502 01:51:09.513088 7265 leader_election.cc:302] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a57d4b3613e24ffc9165c04b1e64eba4, c1318f37ec884127be5a4081d001801f; no voters:
I20240502 01:51:09.513674 7223 raft_consensus.cc:2454] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57d4b3613e24ffc9165c04b1e64eba4 in term 2.
I20240502 01:51:09.514000 7357 raft_consensus.cc:2781] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:51:09.514740 7357 raft_consensus.cc:686] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 LEADER]: Becoming Leader. State: Replica: a57d4b3613e24ffc9165c04b1e64eba4, State: Running, Role: LEADER
I20240502 01:51:09.513976 6677 catalog_manager.cc:5484] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f reported cstate change: term changed from 2 to 3, leader changed from 447a5c3a2b7d4f5d8e6cd800cc9501e7 (127.1.233.130) to c1318f37ec884127be5a4081d001801f (127.1.233.129). New cstate: current_term: 3 leader_uuid: "c1318f37ec884127be5a4081d001801f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } health_report { overall_health: UNKNOWN } } }
I20240502 01:51:09.515580 7357 consensus_queue.cc:237] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.523878 6676 catalog_manager.cc:5484] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } health_report { overall_health: HEALTHY } } }
I20240502 01:51:09.610734 7143 consensus_queue.cc:237] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: true } }
I20240502 01:51:09.616928 7304 raft_consensus.cc:1261] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Refusing update from remote peer c1318f37ec884127be5a4081d001801f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:51:09.617926 7303 raft_consensus.cc:1226] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 3 FOLLOWER]: Rejecting Update request from peer 447a5c3a2b7d4f5d8e6cd800cc9501e7 for earlier term 2. Current term is 3. Ops: []
I20240502 01:51:09.618700 7342 consensus_queue.cc:1035] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:51:09.620330 7349 consensus_queue.cc:1046] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:51:09.621325 7349 raft_consensus.cc:3032] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:51:09.621680 7349 raft_consensus.cc:729] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 447a5c3a2b7d4f5d8e6cd800cc9501e7, State: Running, Role: LEADER
I20240502 01:51:09.622653 7349 consensus_queue.cc:260] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.623806 7223 raft_consensus.cc:1261] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Refusing update from remote peer c1318f37ec884127be5a4081d001801f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:51:09.624333 7349 raft_consensus.cc:3037] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:51:09.625542 7342 consensus_queue.cc:1035] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Connected to new peer: Peer: permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:51:09.628422 7342 raft_consensus.cc:2932] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a12546b72c37471c933460636bb19670 (127.1.233.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: true } } }
I20240502 01:51:09.632620 7304 raft_consensus.cc:2932] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a12546b72c37471c933460636bb19670 (127.1.233.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: true } } }
I20240502 01:51:09.634452 7223 raft_consensus.cc:2932] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a12546b72c37471c933460636bb19670 (127.1.233.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: true } } }
W20240502 01:51:09.636933 7103 consensus_peers.cc:485] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f -> Peer a12546b72c37471c933460636bb19670 (127.1.233.132:37529): Couldn't send request to peer a12546b72c37471c933460636bb19670. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0dae112f99f04e58ba8843992ad40465. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:51:09.647421 6676 catalog_manager.cc:5484] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 reported cstate change: config changed from index -1 to 3, NON_VOTER a12546b72c37471c933460636bb19670 (127.1.233.132) added. New cstate: current_term: 2 leader_uuid: "c1318f37ec884127be5a4081d001801f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: true } } }
W20240502 01:51:09.663519 6726 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240502 01:51:09.671649 6726 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0b8b85e1ca954ea7ab30c2361781fc34, TS 447a5c3a2b7d4f5d8e6cd800cc9501e7 -> TS a12546b72c37471c933460636bb19670 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
W20240502 01:51:09.671932 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0b8b85e1ca954ea7ab30c2361781fc34, TS 447a5c3a2b7d4f5d8e6cd800cc9501e7 -> TS a12546b72c37471c933460636bb19670 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
W20240502 01:51:09.677757 6726 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7c13a16e9e474b07b30df5a73fc6cb50, TS c1318f37ec884127be5a4081d001801f -> TS a12546b72c37471c933460636bb19670 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
W20240502 01:51:09.678004 6726 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7c13a16e9e474b07b30df5a73fc6cb50, TS c1318f37ec884127be5a4081d001801f -> TS a12546b72c37471c933460636bb19670 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.741185 7223 raft_consensus.cc:1226] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 3 LEADER]: Rejecting Update request from peer c1318f37ec884127be5a4081d001801f for earlier term 2. Current term is 3. Ops: []
I20240502 01:51:09.742570 7342 consensus_queue.cc:1046] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:51:09.743288 7342 consensus_queue.cc:1225] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [LEADER]: Peer 447a5c3a2b7d4f5d8e6cd800cc9501e7 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240502 01:51:09.743779 7356 raft_consensus.cc:3032] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:51:09.744063 7356 raft_consensus.cc:729] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c1318f37ec884127be5a4081d001801f, State: Running, Role: LEADER
I20240502 01:51:09.744877 7356 consensus_queue.cc:260] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } }
I20240502 01:51:09.745893 7356 raft_consensus.cc:3037] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:51:09.935907 7304 raft_consensus.cc:1261] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 3 FOLLOWER]: Refusing update from remote peer 447a5c3a2b7d4f5d8e6cd800cc9501e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 01:51:09.937304 7349 consensus_queue.cc:1035] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:51:09.949617 7223 raft_consensus.cc:1261] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Refusing update from remote peer a57d4b3613e24ffc9165c04b1e64eba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:51:09.951620 7357 consensus_queue.cc:1035] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:51:09.961529 7143 raft_consensus.cc:1261] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 3 FOLLOWER]: Refusing update from remote peer 447a5c3a2b7d4f5d8e6cd800cc9501e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 01:51:09.967819 7371 consensus_queue.cc:1035] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:51:09.968854 7371 consensus_queue.cc:1225] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [LEADER]: Peer c1318f37ec884127be5a4081d001801f log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240502 01:51:09.973770 7142 pending_rounds.cc:76] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f: Aborting all ops after (but not including) 0
I20240502 01:51:09.973809 7141 raft_consensus.cc:1261] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Refusing update from remote peer a57d4b3613e24ffc9165c04b1e64eba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:51:09.974349 7142 pending_rounds.cc:97] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240502 01:51:09.974721 7142 raft_consensus.cc:2866] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:51:09.975937 7352 consensus_queue.cc:1035] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:51:10.062788 7304 raft_consensus.cc:1261] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 3 FOLLOWER]: Refusing update from remote peer c1318f37ec884127be5a4081d001801f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 01:51:10.064414 7356 consensus_queue.cc:1035] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:51:10.078275 7223 raft_consensus.cc:1261] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 3 FOLLOWER]: Refusing update from remote peer c1318f37ec884127be5a4081d001801f: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 01:51:10.080206 7339 consensus_queue.cc:1035] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [LEADER]: Connected to new peer: Peer: permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:51:10.081360 7339 consensus_queue.cc:1225] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [LEADER]: Peer 447a5c3a2b7d4f5d8e6cd800cc9501e7 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240502 01:51:10.084885 7223 pending_rounds.cc:76] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Aborting all ops after (but not including) 0
I20240502 01:51:10.085211 7223 pending_rounds.cc:97] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240502 01:51:10.085477 7223 raft_consensus.cc:2866] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:51:10.209236 7378 ts_tablet_manager.cc:902] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670: Initiating tablet copy from peer c1318f37ec884127be5a4081d001801f (127.1.233.129:36447)
I20240502 01:51:10.214462 7378 tablet_copy_client.cc:308] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670: tablet copy: Beginning tablet copy session from remote peer at address 127.1.233.129:36447
I20240502 01:51:10.225486 7153 tablet_copy_service.cc:140] P c1318f37ec884127be5a4081d001801f: Received BeginTabletCopySession request for tablet 0dae112f99f04e58ba8843992ad40465 from peer a12546b72c37471c933460636bb19670 ({username='slave'} at 127.0.0.1:60978)
I20240502 01:51:10.226022 7153 tablet_copy_service.cc:161] P c1318f37ec884127be5a4081d001801f: Beginning new tablet copy session on tablet 0dae112f99f04e58ba8843992ad40465 from peer a12546b72c37471c933460636bb19670 at {username='slave'} at 127.0.0.1:60978: session id = a12546b72c37471c933460636bb19670-0dae112f99f04e58ba8843992ad40465
I20240502 01:51:10.232683 7153 tablet_copy_source_session.cc:205] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:51:10.235482 7378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dae112f99f04e58ba8843992ad40465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:10.249025 7378 tablet_copy_client.cc:783] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670: tablet copy: Starting download of 0 data blocks...
I20240502 01:51:10.249639 7378 tablet_copy_client.cc:647] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670: tablet copy: Starting download of 1 WAL segments...
I20240502 01:51:10.253393 7378 tablet_copy_client.cc:523] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:51:10.261746 7378 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670: Bootstrap starting.
I20240502 01:51:10.280825 7378 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:10.281756 7378 tablet_bootstrap.cc:492] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670: Bootstrap complete.
I20240502 01:51:10.282377 7378 ts_tablet_manager.cc:1367] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.004s
I20240502 01:51:10.285166 7378 raft_consensus.cc:348] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: true } }
I20240502 01:51:10.285841 7378 raft_consensus.cc:729] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a12546b72c37471c933460636bb19670, State: Initialized, Role: LEARNER
I20240502 01:51:10.286505 7378 consensus_queue.cc:260] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: true } }
I20240502 01:51:10.289177 7378 ts_tablet_manager.cc:1398] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20240502 01:51:10.290848 7153 tablet_copy_service.cc:342] P c1318f37ec884127be5a4081d001801f: Request end of tablet copy session a12546b72c37471c933460636bb19670-0dae112f99f04e58ba8843992ad40465 received from {username='slave'} at 127.0.0.1:60978
I20240502 01:51:10.291249 7153 tablet_copy_service.cc:433] P c1318f37ec884127be5a4081d001801f: ending tablet copy session a12546b72c37471c933460636bb19670-0dae112f99f04e58ba8843992ad40465 on tablet 0dae112f99f04e58ba8843992ad40465 with peer a12546b72c37471c933460636bb19670
I20240502 01:51:10.819486 7054 raft_consensus.cc:1203] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240502 01:51:11.327952 7339 raft_consensus.cc:1050] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f: attempting to promote NON_VOTER a12546b72c37471c933460636bb19670 to VOTER
I20240502 01:51:11.330252 7339 consensus_queue.cc:237] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: false } }
I20240502 01:51:11.337119 7054 raft_consensus.cc:1261] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670 [term 2 LEARNER]: Refusing update from remote peer c1318f37ec884127be5a4081d001801f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:51:11.339046 7339 consensus_queue.cc:1035] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:51:11.339560 7223 raft_consensus.cc:1261] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Refusing update from remote peer c1318f37ec884127be5a4081d001801f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:51:11.339560 7304 raft_consensus.cc:1261] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Refusing update from remote peer c1318f37ec884127be5a4081d001801f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:51:11.341449 7394 consensus_queue.cc:1035] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Connected to new peer: Peer: permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:51:11.342478 7339 consensus_queue.cc:1035] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:51:11.358758 7339 raft_consensus.cc:2932] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, a12546b72c37471c933460636bb19670 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: false } } }
I20240502 01:51:11.360634 7054 raft_consensus.cc:2932] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, a12546b72c37471c933460636bb19670 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: false } } }
I20240502 01:51:11.362808 7304 raft_consensus.cc:2932] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, a12546b72c37471c933460636bb19670 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: false } } }
I20240502 01:51:11.373610 7223 raft_consensus.cc:2932] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, a12546b72c37471c933460636bb19670 (127.1.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: false } } }
I20240502 01:51:11.381552 6676 catalog_manager.cc:5484] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f reported cstate change: config changed from index 3 to 4, a12546b72c37471c933460636bb19670 (127.1.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c1318f37ec884127be5a4081d001801f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57d4b3613e24ffc9165c04b1e64eba4" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 46787 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:51:11.410993 7141 consensus_queue.cc:237] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: false } }
I20240502 01:51:11.415513 7223 raft_consensus.cc:1261] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Refusing update from remote peer c1318f37ec884127be5a4081d001801f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:51:11.415776 7054 raft_consensus.cc:1261] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670 [term 2 FOLLOWER]: Refusing update from remote peer c1318f37ec884127be5a4081d001801f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:51:11.417091 7392 consensus_queue.cc:1035] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Connected to new peer: Peer: permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:51:11.418058 7399 consensus_queue.cc:1035] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:51:11.425972 7339 raft_consensus.cc:2932] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: false } } }
I20240502 01:51:11.426761 7223 raft_consensus.cc:2932] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: false } } }
I20240502 01:51:11.435070 6723 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0dae112f99f04e58ba8843992ad40465 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:51:11.426761 7054 raft_consensus.cc:2932] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: false } } }
I20240502 01:51:11.437608 6676 catalog_manager.cc:5484] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f reported cstate change: config changed from index 4 to 5, VOTER a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131) evicted. New cstate: current_term: 2 leader_uuid: "c1318f37ec884127be5a4081d001801f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c1318f37ec884127be5a4081d001801f" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 36447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "447a5c3a2b7d4f5d8e6cd800cc9501e7" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 43245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a12546b72c37471c933460636bb19670" member_type: VOTER last_known_addr { host: "127.1.233.132" port: 37529 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:51:11.465376 7294 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dae112f99f04e58ba8843992ad40465 with delete_type TABLET_DATA_TOMBSTONED (TS a57d4b3613e24ffc9165c04b1e64eba4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44504
I20240502 01:51:11.467258 7406 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:11.468056 7406 raft_consensus.cc:2229] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:11.468644 7406 raft_consensus.cc:2258] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:11.471035 7406 ts_tablet_manager.cc:1875] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:51:11.482229 7406 ts_tablet_manager.cc:1888] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 01:51:11.482575 7406 log.cc:1196] T 0dae112f99f04e58ba8843992ad40465 P a57d4b3613e24ffc9165c04b1e64eba4: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714614515033618-1958-0/minicluster-data/ts-2-root/wals/0dae112f99f04e58ba8843992ad40465
I20240502 01:51:11.483989 6665 catalog_manager.cc:4830] TS a57d4b3613e24ffc9165c04b1e64eba4 (127.1.233.131:46787): tablet 0dae112f99f04e58ba8843992ad40465 (table test-workload [id=f3bd73de4f514bf6a8378283b590ae31]) successfully deleted
I20240502 01:51:11.781085 1958 tablet_server.cc:178] TabletServer@127.1.233.129:36447 shutting down...
I20240502 01:51:11.800801 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:51:11.801457 1958 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:11.802021 1958 raft_consensus.cc:2229] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:51:11.802866 1958 raft_consensus.cc:2258] T 0335246fb7914702a35bb99ad5bc4705 P c1318f37ec884127be5a4081d001801f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:11.804884 1958 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:11.805402 1958 raft_consensus.cc:2229] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:11.805831 1958 raft_consensus.cc:2258] T 7c13a16e9e474b07b30df5a73fc6cb50 P c1318f37ec884127be5a4081d001801f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:11.807693 1958 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:11.808187 1958 raft_consensus.cc:2229] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:11.808598 1958 raft_consensus.cc:2258] T 0b8b85e1ca954ea7ab30c2361781fc34 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:11.810277 1958 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:11.810736 1958 raft_consensus.cc:2229] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:51:11.811518 1958 raft_consensus.cc:2258] T 0dae112f99f04e58ba8843992ad40465 P c1318f37ec884127be5a4081d001801f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:11.836329 1958 tablet_server.cc:194] TabletServer@127.1.233.129:36447 shutdown complete.
I20240502 01:51:11.850273 1958 tablet_server.cc:178] TabletServer@127.1.233.130:43245 shutting down...
I20240502 01:51:11.867877 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:51:11.868624 1958 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:11.869212 1958 raft_consensus.cc:2229] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:11.869623 1958 raft_consensus.cc:2258] T 0335246fb7914702a35bb99ad5bc4705 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:11.871287 1958 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:11.871722 1958 raft_consensus.cc:2229] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 3 LEADER]: Raft consensus shutting down.
I20240502 01:51:11.872542 1958 raft_consensus.cc:2258] T 7c13a16e9e474b07b30df5a73fc6cb50 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:11.874281 1958 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:11.874737 1958 raft_consensus.cc:2229] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:11.875114 1958 raft_consensus.cc:2258] T 0b8b85e1ca954ea7ab30c2361781fc34 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:11.876763 1958 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:11.877223 1958 raft_consensus.cc:2229] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:11.877768 1958 raft_consensus.cc:2258] T 0dae112f99f04e58ba8843992ad40465 P 447a5c3a2b7d4f5d8e6cd800cc9501e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:11.899484 1958 tablet_server.cc:194] TabletServer@127.1.233.130:43245 shutdown complete.
I20240502 01:51:11.911294 1958 tablet_server.cc:178] TabletServer@127.1.233.131:46787 shutting down...
I20240502 01:51:11.930132 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:51:11.930984 1958 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:11.931619 1958 raft_consensus.cc:2229] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:11.932210 1958 raft_consensus.cc:2258] T 0335246fb7914702a35bb99ad5bc4705 P a57d4b3613e24ffc9165c04b1e64eba4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:11.934092 1958 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:11.934583 1958 raft_consensus.cc:2229] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:11.935027 1958 raft_consensus.cc:2258] T 7c13a16e9e474b07b30df5a73fc6cb50 P a57d4b3613e24ffc9165c04b1e64eba4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:11.936925 1958 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:11.937353 1958 raft_consensus.cc:2229] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 LEADER]: Raft consensus shutting down.
I20240502 01:51:11.938145 1958 raft_consensus.cc:2258] T 0b8b85e1ca954ea7ab30c2361781fc34 P a57d4b3613e24ffc9165c04b1e64eba4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:11.961660 1958 tablet_server.cc:194] TabletServer@127.1.233.131:46787 shutdown complete.
I20240502 01:51:11.973966 1958 tablet_server.cc:178] TabletServer@127.1.233.132:0 shutting down...
I20240502 01:51:11.995465 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:51:11.996105 1958 tablet_replica.cc:330] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670: stopping tablet replica
I20240502 01:51:11.996691 1958 raft_consensus.cc:2229] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:11.997239 1958 raft_consensus.cc:2258] T 0dae112f99f04e58ba8843992ad40465 P a12546b72c37471c933460636bb19670 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:12.016556 1958 tablet_server.cc:194] TabletServer@127.1.233.132:0 shutdown complete.
I20240502 01:51:12.027021 1958 master.cc:537] Master@127.1.233.190:35433 shutting down...
I20240502 01:51:12.044306 1958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:51:12.044859 1958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:12.045159 1958 tablet_replica.cc:330] T 00000000000000000000000000000000 P dd278b125980448e86ebc6ba0762b157: stopping tablet replica
I20240502 01:51:12.063704 1958 master.cc:552] Master@127.1.233.190:35433 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (12812 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20240502 01:51:12.105719 1958 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.233.190:39913
I20240502 01:51:12.106910 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:51:12.113323 7407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:12.113956 7408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:12.114385 7410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:12.115092 1958 server_base.cc:1034] running on GCE node
I20240502 01:51:12.115963 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:51:12.116171 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:51:12.116313 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614672116300 us; error 0 us; skew 500 ppm
I20240502 01:51:12.116813 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:51:12.125793 1958 webserver.cc:460] Webserver started at http://127.1.233.190:46233/ using document root <none> and password file <none>
I20240502 01:51:12.126281 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:51:12.126443 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:51:12.126665 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:51:12.127841 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/master-0-root/instance:
uuid: "2720d009733a4f609c1859836042e885"
format_stamp: "Formatted at 2024-05-02 01:51:12 on dist-test-slave-2090"
I20240502 01:51:12.132851 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:51:12.136658 7415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:12.137526 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240502 01:51:12.137845 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/master-0-root
uuid: "2720d009733a4f609c1859836042e885"
format_stamp: "Formatted at 2024-05-02 01:51:12 on dist-test-slave-2090"
I20240502 01:51:12.138135 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:51:12.159399 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:51:12.160773 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:51:12.200079 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.190:39913
I20240502 01:51:12.200232 7466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.190:39913 every 8 connection(s)
I20240502 01:51:12.203928 7467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:12.214802 7467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885: Bootstrap starting.
I20240502 01:51:12.219107 7467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:12.223286 7467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885: No bootstrap required, opened a new log
I20240502 01:51:12.225369 7467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2720d009733a4f609c1859836042e885" member_type: VOTER }
I20240502 01:51:12.225760 7467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:12.226001 7467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2720d009733a4f609c1859836042e885, State: Initialized, Role: FOLLOWER
I20240502 01:51:12.226626 7467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2720d009733a4f609c1859836042e885" member_type: VOTER }
I20240502 01:51:12.227079 7467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:51:12.227293 7467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:51:12.227535 7467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:12.232311 7467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2720d009733a4f609c1859836042e885" member_type: VOTER }
I20240502 01:51:12.232831 7467 leader_election.cc:302] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2720d009733a4f609c1859836042e885; no voters:
I20240502 01:51:12.234097 7467 leader_election.cc:288] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:51:12.234552 7470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:51:12.235909 7470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [term 1 LEADER]: Becoming Leader. State: Replica: 2720d009733a4f609c1859836042e885, State: Running, Role: LEADER
I20240502 01:51:12.236537 7470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2720d009733a4f609c1859836042e885" member_type: VOTER }
I20240502 01:51:12.237109 7467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:51:12.239784 7471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2720d009733a4f609c1859836042e885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2720d009733a4f609c1859836042e885" member_type: VOTER } }
I20240502 01:51:12.239969 7472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2720d009733a4f609c1859836042e885. Latest consensus state: current_term: 1 leader_uuid: "2720d009733a4f609c1859836042e885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2720d009733a4f609c1859836042e885" member_type: VOTER } }
I20240502 01:51:12.240483 7471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [sys.catalog]: This master's current role is: LEADER
I20240502 01:51:12.240862 7472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885 [sys.catalog]: This master's current role is: LEADER
I20240502 01:51:12.243433 7476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:51:12.252249 7476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:51:12.253396 1958 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:51:12.261504 7476 catalog_manager.cc:1347] Generated new cluster ID: 9cb16e3e7c1e43bf95cdb265dc5acac1
I20240502 01:51:12.261822 7476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:51:12.276120 7476 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:51:12.277495 7476 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:51:12.293084 7476 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885: Generated new TSK 0
I20240502 01:51:12.293812 7476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:51:12.320974 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:51:12.328307 7488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:12.328609 7489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:12.330344 7491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:12.331045 1958 server_base.cc:1034] running on GCE node
I20240502 01:51:12.333022 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:51:12.333212 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:51:12.333361 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614672333345 us; error 0 us; skew 500 ppm
I20240502 01:51:12.333855 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:51:12.336405 1958 webserver.cc:460] Webserver started at http://127.1.233.129:37541/ using document root <none> and password file <none>
I20240502 01:51:12.336853 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:51:12.337047 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:51:12.337273 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:51:12.338619 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-0-root/instance:
uuid: "7c6075468bc949f08b209b5aebe0c49a"
format_stamp: "Formatted at 2024-05-02 01:51:12 on dist-test-slave-2090"
I20240502 01:51:12.343622 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20240502 01:51:12.347450 7496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:12.348469 1958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:51:12.348767 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "7c6075468bc949f08b209b5aebe0c49a"
format_stamp: "Formatted at 2024-05-02 01:51:12 on dist-test-slave-2090"
I20240502 01:51:12.349062 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:51:12.376076 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:51:12.377223 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:51:12.378723 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:51:12.381160 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:51:12.381343 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:12.381589 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:51:12.381762 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:12.423086 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:33345
I20240502 01:51:12.423193 7558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:33345 every 8 connection(s)
I20240502 01:51:12.427887 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:51:12.436107 7563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:12.437214 7564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:12.438740 7566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:12.439723 1958 server_base.cc:1034] running on GCE node
I20240502 01:51:12.440768 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 01:51:12.440954 7559 heartbeater.cc:347] Connected to a master server at 127.1.233.190:39913
W20240502 01:51:12.441031 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:51:12.441350 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614672441329 us; error 0 us; skew 500 ppm
I20240502 01:51:12.441437 7559 heartbeater.cc:464] Registering TS with master...
I20240502 01:51:12.442123 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:51:12.442327 7559 heartbeater.cc:511] Master 127.1.233.190:39913 requested a full tablet report, sending...
I20240502 01:51:12.444828 7432 ts_manager.cc:194] Registered new tserver with Master: 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345)
I20240502 01:51:12.445103 1958 webserver.cc:460] Webserver started at http://127.1.233.130:44581/ using document root <none> and password file <none>
I20240502 01:51:12.445681 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:51:12.445866 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:51:12.446168 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:51:12.446991 7432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47848
I20240502 01:51:12.447438 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-1-root/instance:
uuid: "56327090ee9245b8b42724ae0edcb365"
format_stamp: "Formatted at 2024-05-02 01:51:12 on dist-test-slave-2090"
I20240502 01:51:12.452304 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:51:12.455765 7571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:12.456586 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:51:12.456885 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-1-root
uuid: "56327090ee9245b8b42724ae0edcb365"
format_stamp: "Formatted at 2024-05-02 01:51:12 on dist-test-slave-2090"
I20240502 01:51:12.457279 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:51:12.478974 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:51:12.480335 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:51:12.481976 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:51:12.484722 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:51:12.484979 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:12.485225 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:51:12.485375 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:12.524878 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.130:41677
I20240502 01:51:12.524977 7633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.130:41677 every 8 connection(s)
I20240502 01:51:12.529942 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:51:12.538161 7637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:12.539111 7638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:12.542467 7640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:12.542688 7634 heartbeater.cc:347] Connected to a master server at 127.1.233.190:39913
I20240502 01:51:12.543292 7634 heartbeater.cc:464] Registering TS with master...
I20240502 01:51:12.544063 1958 server_base.cc:1034] running on GCE node
I20240502 01:51:12.544176 7634 heartbeater.cc:511] Master 127.1.233.190:39913 requested a full tablet report, sending...
I20240502 01:51:12.545243 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:51:12.545490 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:51:12.545670 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614672545654 us; error 0 us; skew 500 ppm
I20240502 01:51:12.546480 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:51:12.546803 7432 ts_manager.cc:194] Registered new tserver with Master: 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
I20240502 01:51:12.548836 7432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47864
I20240502 01:51:12.549651 1958 webserver.cc:460] Webserver started at http://127.1.233.131:32995/ using document root <none> and password file <none>
I20240502 01:51:12.550328 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:51:12.550521 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:51:12.550799 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:51:12.551995 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-2-root/instance:
uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc"
format_stamp: "Formatted at 2024-05-02 01:51:12 on dist-test-slave-2090"
I20240502 01:51:12.557010 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20240502 01:51:12.560384 7645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:12.561206 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:51:12.561511 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-2-root
uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc"
format_stamp: "Formatted at 2024-05-02 01:51:12 on dist-test-slave-2090"
I20240502 01:51:12.561782 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:51:12.583140 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:51:12.584286 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:51:12.585796 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:51:12.588280 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:51:12.588487 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:12.588744 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:51:12.588891 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:12.629565 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.131:42861
I20240502 01:51:12.629675 7707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.131:42861 every 8 connection(s)
I20240502 01:51:12.643478 7708 heartbeater.cc:347] Connected to a master server at 127.1.233.190:39913
I20240502 01:51:12.643929 7708 heartbeater.cc:464] Registering TS with master...
I20240502 01:51:12.644709 7708 heartbeater.cc:511] Master 127.1.233.190:39913 requested a full tablet report, sending...
I20240502 01:51:12.646857 7432 ts_manager.cc:194] Registered new tserver with Master: d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861)
I20240502 01:51:12.647686 1958 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014899881s
I20240502 01:51:12.648644 7432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47870
I20240502 01:51:13.449548 7559 heartbeater.cc:503] Master 127.1.233.190:39913 was elected leader, sending a full tablet report...
I20240502 01:51:13.552009 7634 heartbeater.cc:503] Master 127.1.233.190:39913 was elected leader, sending a full tablet report...
I20240502 01:51:13.651701 7708 heartbeater.cc:503] Master 127.1.233.190:39913 was elected leader, sending a full tablet report...
I20240502 01:51:13.681828 1958 test_util.cc:268] Using random seed: -695390839
I20240502 01:51:13.705212 7432 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47886:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:51:13.707507 7432 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:51:13.759339 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 7a354a74fa444714abd15b3dc6c9b5ce (DEFAULT_TABLE table=test-workload [id=309a236d6e9541abba4b7f6756db454b]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:51:13.760814 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a354a74fa444714abd15b3dc6c9b5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:13.761917 7671 tablet_service.cc:1467] Processing CreateTablet for tablet d945ad77c1284b33a14e47ddda96f868 (DEFAULT_TABLE table=test-workload [id=309a236d6e9541abba4b7f6756db454b]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:51:13.763252 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d945ad77c1284b33a14e47ddda96f868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:13.766455 7672 tablet_service.cc:1467] Processing CreateTablet for tablet fadba1fb5a5c41cc8904ab3920871362 (DEFAULT_TABLE table=test-workload [id=309a236d6e9541abba4b7f6756db454b]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:51:13.767797 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fadba1fb5a5c41cc8904ab3920871362. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:13.760849 7670 tablet_service.cc:1467] Processing CreateTablet for tablet e503a6bf258643ea8035d4b6f7131f83 (DEFAULT_TABLE table=test-workload [id=309a236d6e9541abba4b7f6756db454b]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:51:13.769676 7670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e503a6bf258643ea8035d4b6f7131f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:13.781242 7599 tablet_service.cc:1467] Processing CreateTablet for tablet 7a354a74fa444714abd15b3dc6c9b5ce (DEFAULT_TABLE table=test-workload [id=309a236d6e9541abba4b7f6756db454b]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:51:13.781606 7598 tablet_service.cc:1467] Processing CreateTablet for tablet fadba1fb5a5c41cc8904ab3920871362 (DEFAULT_TABLE table=test-workload [id=309a236d6e9541abba4b7f6756db454b]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:51:13.782672 7599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a354a74fa444714abd15b3dc6c9b5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:13.784160 7597 tablet_service.cc:1467] Processing CreateTablet for tablet d945ad77c1284b33a14e47ddda96f868 (DEFAULT_TABLE table=test-workload [id=309a236d6e9541abba4b7f6756db454b]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:51:13.785521 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d945ad77c1284b33a14e47ddda96f868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:13.788589 7598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fadba1fb5a5c41cc8904ab3920871362. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:13.793911 7596 tablet_service.cc:1467] Processing CreateTablet for tablet e503a6bf258643ea8035d4b6f7131f83 (DEFAULT_TABLE table=test-workload [id=309a236d6e9541abba4b7f6756db454b]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:51:13.795225 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e503a6bf258643ea8035d4b6f7131f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:13.804081 7728 tablet_bootstrap.cc:492] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc: Bootstrap starting.
I20240502 01:51:13.811334 7728 tablet_bootstrap.cc:654] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:13.832211 7524 tablet_service.cc:1467] Processing CreateTablet for tablet 7a354a74fa444714abd15b3dc6c9b5ce (DEFAULT_TABLE table=test-workload [id=309a236d6e9541abba4b7f6756db454b]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:51:13.833660 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a354a74fa444714abd15b3dc6c9b5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:13.837786 7523 tablet_service.cc:1467] Processing CreateTablet for tablet fadba1fb5a5c41cc8904ab3920871362 (DEFAULT_TABLE table=test-workload [id=309a236d6e9541abba4b7f6756db454b]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:51:13.839474 7523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fadba1fb5a5c41cc8904ab3920871362. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:13.840004 7521 tablet_service.cc:1467] Processing CreateTablet for tablet e503a6bf258643ea8035d4b6f7131f83 (DEFAULT_TABLE table=test-workload [id=309a236d6e9541abba4b7f6756db454b]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:51:13.841349 7521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e503a6bf258643ea8035d4b6f7131f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:13.843147 7728 tablet_bootstrap.cc:492] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc: No bootstrap required, opened a new log
I20240502 01:51:13.843744 7728 ts_tablet_manager.cc:1367] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent bootstrapping tablet: real 0.040s user 0.023s sys 0.012s
I20240502 01:51:13.845201 7522 tablet_service.cc:1467] Processing CreateTablet for tablet d945ad77c1284b33a14e47ddda96f868 (DEFAULT_TABLE table=test-workload [id=309a236d6e9541abba4b7f6756db454b]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:51:13.846513 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d945ad77c1284b33a14e47ddda96f868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:13.847299 7729 tablet_bootstrap.cc:492] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365: Bootstrap starting.
I20240502 01:51:13.846968 7728 raft_consensus.cc:348] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.849679 7728 raft_consensus.cc:374] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:13.850636 7728 raft_consensus.cc:729] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Initialized, Role: FOLLOWER
I20240502 01:51:13.856264 7729 tablet_bootstrap.cc:654] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:13.865912 7731 tablet_bootstrap.cc:492] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:13.865465 7728 consensus_queue.cc:260] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.873145 7728 ts_tablet_manager.cc:1398] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent starting tablet: real 0.029s user 0.007s sys 0.001s
I20240502 01:51:13.879134 7729 tablet_bootstrap.cc:492] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365: No bootstrap required, opened a new log
I20240502 01:51:13.879751 7729 ts_tablet_manager.cc:1367] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.010s
I20240502 01:51:13.880098 7731 tablet_bootstrap.cc:654] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:13.882495 7729 raft_consensus.cc:348] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } }
I20240502 01:51:13.883517 7729 raft_consensus.cc:374] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:13.883860 7729 raft_consensus.cc:729] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56327090ee9245b8b42724ae0edcb365, State: Initialized, Role: FOLLOWER
I20240502 01:51:13.884774 7729 consensus_queue.cc:260] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } }
I20240502 01:51:13.887934 7728 tablet_bootstrap.cc:492] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc: Bootstrap starting.
I20240502 01:51:13.889196 7729 ts_tablet_manager.cc:1398] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240502 01:51:13.889891 7731 tablet_bootstrap.cc:492] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a: No bootstrap required, opened a new log
I20240502 01:51:13.890260 7729 tablet_bootstrap.cc:492] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365: Bootstrap starting.
I20240502 01:51:13.890434 7731 ts_tablet_manager.cc:1367] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.004s
I20240502 01:51:13.893339 7731 raft_consensus.cc:348] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } }
I20240502 01:51:13.894158 7731 raft_consensus.cc:374] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:13.894505 7731 raft_consensus.cc:729] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:13.895254 7731 consensus_queue.cc:260] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } }
I20240502 01:51:13.896559 7729 tablet_bootstrap.cc:654] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:13.901515 7731 ts_tablet_manager.cc:1398] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20240502 01:51:13.901865 7728 tablet_bootstrap.cc:654] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:13.902324 7729 tablet_bootstrap.cc:492] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365: No bootstrap required, opened a new log
I20240502 01:51:13.902556 7731 tablet_bootstrap.cc:492] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:13.902966 7729 ts_tablet_manager.cc:1367] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240502 01:51:13.905788 7729 raft_consensus.cc:348] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.906596 7729 raft_consensus.cc:374] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:13.906960 7729 raft_consensus.cc:729] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56327090ee9245b8b42724ae0edcb365, State: Initialized, Role: FOLLOWER
I20240502 01:51:13.907598 7731 tablet_bootstrap.cc:654] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:13.907696 7729 consensus_queue.cc:260] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.914244 7729 ts_tablet_manager.cc:1398] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240502 01:51:13.915390 7729 tablet_bootstrap.cc:492] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365: Bootstrap starting.
I20240502 01:51:13.921181 7728 tablet_bootstrap.cc:492] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc: No bootstrap required, opened a new log
I20240502 01:51:13.921423 7731 tablet_bootstrap.cc:492] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a: No bootstrap required, opened a new log
I20240502 01:51:13.921880 7728 ts_tablet_manager.cc:1367] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.014s
I20240502 01:51:13.922088 7731 ts_tablet_manager.cc:1367] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.008s
I20240502 01:51:13.922600 7729 tablet_bootstrap.cc:654] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:13.924744 7728 raft_consensus.cc:348] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.925606 7728 raft_consensus.cc:374] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:13.925062 7731 raft_consensus.cc:348] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.925976 7728 raft_consensus.cc:729] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Initialized, Role: FOLLOWER
I20240502 01:51:13.926134 7731 raft_consensus.cc:374] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:13.926604 7731 raft_consensus.cc:729] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:13.926792 7728 consensus_queue.cc:260] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.927559 7731 consensus_queue.cc:260] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.929549 7731 ts_tablet_manager.cc:1398] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:51:13.931059 7728 ts_tablet_manager.cc:1398] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:51:13.931682 7731 tablet_bootstrap.cc:492] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:13.932308 7728 tablet_bootstrap.cc:492] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc: Bootstrap starting.
I20240502 01:51:13.931157 7729 tablet_bootstrap.cc:492] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365: No bootstrap required, opened a new log
I20240502 01:51:13.937835 7731 tablet_bootstrap.cc:654] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:13.937893 7728 tablet_bootstrap.cc:654] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:13.941334 7729 ts_tablet_manager.cc:1367] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.004s
I20240502 01:51:13.943524 7728 tablet_bootstrap.cc:492] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc: No bootstrap required, opened a new log
I20240502 01:51:13.944072 7728 ts_tablet_manager.cc:1367] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240502 01:51:13.944326 7729 raft_consensus.cc:348] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.944962 7729 raft_consensus.cc:374] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:13.945202 7729 raft_consensus.cc:729] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56327090ee9245b8b42724ae0edcb365, State: Initialized, Role: FOLLOWER
I20240502 01:51:13.945438 7731 tablet_bootstrap.cc:492] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a: No bootstrap required, opened a new log
I20240502 01:51:13.946033 7731 ts_tablet_manager.cc:1367] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240502 01:51:13.945844 7729 consensus_queue.cc:260] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.946923 7728 raft_consensus.cc:348] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } }
I20240502 01:51:13.947754 7728 raft_consensus.cc:374] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:13.948022 7729 ts_tablet_manager.cc:1398] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:51:13.948064 7728 raft_consensus.cc:729] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Initialized, Role: FOLLOWER
I20240502 01:51:13.948767 7731 raft_consensus.cc:348] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.949297 7729 tablet_bootstrap.cc:492] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365: Bootstrap starting.
I20240502 01:51:13.949633 7731 raft_consensus.cc:374] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:13.949172 7728 consensus_queue.cc:260] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } }
I20240502 01:51:13.950024 7731 raft_consensus.cc:729] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:13.950816 7731 consensus_queue.cc:260] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.953331 7731 ts_tablet_manager.cc:1398] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:51:13.954433 7732 raft_consensus.cc:482] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:13.954995 7732 raft_consensus.cc:504] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.959033 7729 tablet_bootstrap.cc:654] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:13.965816 7732 leader_election.cc:288] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
I20240502 01:51:13.969141 7728 ts_tablet_manager.cc:1398] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20240502 01:51:13.954443 7731 tablet_bootstrap.cc:492] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:13.973541 7728 tablet_bootstrap.cc:492] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc: Bootstrap starting.
I20240502 01:51:13.975883 7731 tablet_bootstrap.cc:654] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:13.980371 7728 tablet_bootstrap.cc:654] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:13.977965 7729 tablet_bootstrap.cc:492] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365: No bootstrap required, opened a new log
I20240502 01:51:13.981279 7729 ts_tablet_manager.cc:1367] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.004s
I20240502 01:51:13.981591 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e503a6bf258643ea8035d4b6f7131f83" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6075468bc949f08b209b5aebe0c49a" is_pre_election: true
I20240502 01:51:13.982374 7534 raft_consensus.cc:2454] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 0.
I20240502 01:51:13.983934 7648 leader_election.cc:302] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6075468bc949f08b209b5aebe0c49a, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:13.984333 7729 raft_consensus.cc:348] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.984941 7732 raft_consensus.cc:2781] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:51:13.985251 7729 raft_consensus.cc:374] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:13.985317 7732 raft_consensus.cc:482] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:13.985641 7729 raft_consensus.cc:729] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56327090ee9245b8b42724ae0edcb365, State: Initialized, Role: FOLLOWER
I20240502 01:51:13.985751 7732 raft_consensus.cc:3037] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:13.986563 7729 consensus_queue.cc:260] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.988459 7728 tablet_bootstrap.cc:492] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc: No bootstrap required, opened a new log
I20240502 01:51:13.989054 7728 ts_tablet_manager.cc:1367] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20240502 01:51:13.985803 7731 tablet_bootstrap.cc:492] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a: No bootstrap required, opened a new log
I20240502 01:51:13.991034 7729 ts_tablet_manager.cc:1398] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:51:13.991528 7731 ts_tablet_manager.cc:1367] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.000s
I20240502 01:51:13.992956 7728 raft_consensus.cc:348] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.993793 7728 raft_consensus.cc:374] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:13.994119 7728 raft_consensus.cc:729] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Initialized, Role: FOLLOWER
I20240502 01:51:13.994264 7731 raft_consensus.cc:348] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.995085 7731 raft_consensus.cc:374] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:13.994949 7728 consensus_queue.cc:260] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.996191 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e503a6bf258643ea8035d4b6f7131f83" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365" is_pre_election: true
I20240502 01:51:13.997120 7609 raft_consensus.cc:2454] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 0.
I20240502 01:51:13.995431 7731 raft_consensus.cc:729] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:13.998796 7728 ts_tablet_manager.cc:1398] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20240502 01:51:13.998740 7731 consensus_queue.cc:260] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:13.998840 7732 raft_consensus.cc:504] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.001184 7732 leader_election.cc:288] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 election: Requested vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
I20240502 01:51:14.001260 7731 ts_tablet_manager.cc:1398] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.009s user 0.002s sys 0.004s
I20240502 01:51:14.003409 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e503a6bf258643ea8035d4b6f7131f83" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6075468bc949f08b209b5aebe0c49a"
I20240502 01:51:14.004021 7534 raft_consensus.cc:3037] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.003995 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e503a6bf258643ea8035d4b6f7131f83" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365"
I20240502 01:51:14.006366 7609 raft_consensus.cc:3037] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.012346 7534 raft_consensus.cc:2454] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 1.
I20240502 01:51:14.012643 7609 raft_consensus.cc:2454] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 1.
I20240502 01:51:14.014006 7648 leader_election.cc:302] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6075468bc949f08b209b5aebe0c49a, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:14.014955 7732 raft_consensus.cc:2781] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:51:14.015450 7732 raft_consensus.cc:686] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 LEADER]: Becoming Leader. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Running, Role: LEADER
I20240502 01:51:14.016273 7732 consensus_queue.cc:237] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.016973 7736 raft_consensus.cc:482] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:14.017419 7736 raft_consensus.cc:504] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.019500 7736 leader_election.cc:288] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677), d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861)
I20240502 01:51:14.028739 7432 catalog_manager.cc:5484] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131). New cstate: current_term: 1 leader_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } health_report { overall_health: HEALTHY } } }
I20240502 01:51:14.033990 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d945ad77c1284b33a14e47ddda96f868" candidate_uuid: "7c6075468bc949f08b209b5aebe0c49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" is_pre_election: true
I20240502 01:51:14.034026 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d945ad77c1284b33a14e47ddda96f868" candidate_uuid: "7c6075468bc949f08b209b5aebe0c49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365" is_pre_election: true
I20240502 01:51:14.034736 7683 raft_consensus.cc:2454] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6075468bc949f08b209b5aebe0c49a in term 0.
I20240502 01:51:14.034896 7609 raft_consensus.cc:2454] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6075468bc949f08b209b5aebe0c49a in term 0.
I20240502 01:51:14.035874 7497 leader_election.cc:302] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6075468bc949f08b209b5aebe0c49a, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:14.037051 7736 raft_consensus.cc:2781] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:51:14.037402 7736 raft_consensus.cc:482] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:14.037729 7736 raft_consensus.cc:3037] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.042810 7736 raft_consensus.cc:504] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.044325 7736 leader_election.cc:288] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [CANDIDATE]: Term 1 election: Requested vote from peers 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677), d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861)
I20240502 01:51:14.045133 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d945ad77c1284b33a14e47ddda96f868" candidate_uuid: "7c6075468bc949f08b209b5aebe0c49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365"
I20240502 01:51:14.045362 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d945ad77c1284b33a14e47ddda96f868" candidate_uuid: "7c6075468bc949f08b209b5aebe0c49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc"
I20240502 01:51:14.045665 7609 raft_consensus.cc:3037] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.045888 7683 raft_consensus.cc:3037] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.050765 7609 raft_consensus.cc:2454] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6075468bc949f08b209b5aebe0c49a in term 1.
I20240502 01:51:14.051026 7683 raft_consensus.cc:2454] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6075468bc949f08b209b5aebe0c49a in term 1.
I20240502 01:51:14.051846 7498 leader_election.cc:302] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56327090ee9245b8b42724ae0edcb365, 7c6075468bc949f08b209b5aebe0c49a; no voters:
I20240502 01:51:14.052623 7736 raft_consensus.cc:2781] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:51:14.053722 7736 raft_consensus.cc:686] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 1 LEADER]: Becoming Leader. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Running, Role: LEADER
I20240502 01:51:14.054384 7736 consensus_queue.cc:237] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.061265 7432 catalog_manager.cc:5484] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129). New cstate: current_term: 1 leader_uuid: "7c6075468bc949f08b209b5aebe0c49a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } health_report { overall_health: UNKNOWN } } }
I20240502 01:51:14.157616 7732 raft_consensus.cc:482] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:14.158147 7732 raft_consensus.cc:504] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.159768 7732 leader_election.cc:288] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
I20240502 01:51:14.160630 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fadba1fb5a5c41cc8904ab3920871362" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6075468bc949f08b209b5aebe0c49a" is_pre_election: true
I20240502 01:51:14.160712 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fadba1fb5a5c41cc8904ab3920871362" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365" is_pre_election: true
I20240502 01:51:14.161214 7534 raft_consensus.cc:2454] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 0.
I20240502 01:51:14.161306 7609 raft_consensus.cc:2454] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 0.
I20240502 01:51:14.162151 7648 leader_election.cc:302] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6075468bc949f08b209b5aebe0c49a, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:14.162817 7732 raft_consensus.cc:2781] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:51:14.163144 7732 raft_consensus.cc:482] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:14.163398 7732 raft_consensus.cc:3037] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.165297 7735 raft_consensus.cc:482] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:14.165675 7735 raft_consensus.cc:504] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } }
I20240502 01:51:14.167686 7735 leader_election.cc:288] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861), 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345)
I20240502 01:51:14.169210 7732 raft_consensus.cc:504] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.172359 7732 leader_election.cc:288] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 election: Requested vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
I20240502 01:51:14.172636 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fadba1fb5a5c41cc8904ab3920871362" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6075468bc949f08b209b5aebe0c49a"
I20240502 01:51:14.173353 7534 raft_consensus.cc:3037] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.175352 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fadba1fb5a5c41cc8904ab3920871362" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365"
I20240502 01:51:14.175963 7609 raft_consensus.cc:3037] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.181154 7534 raft_consensus.cc:2454] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 1.
I20240502 01:51:14.181959 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a354a74fa444714abd15b3dc6c9b5ce" candidate_uuid: "56327090ee9245b8b42724ae0edcb365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" is_pre_election: true
I20240502 01:51:14.182325 7648 leader_election.cc:302] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6075468bc949f08b209b5aebe0c49a, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:14.182660 7683 raft_consensus.cc:2454] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56327090ee9245b8b42724ae0edcb365 in term 0.
I20240502 01:51:14.183744 7572 leader_election.cc:302] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56327090ee9245b8b42724ae0edcb365, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:14.183310 7732 raft_consensus.cc:2781] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:51:14.184666 7735 raft_consensus.cc:2781] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:51:14.184897 7732 raft_consensus.cc:686] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 LEADER]: Becoming Leader. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Running, Role: LEADER
I20240502 01:51:14.185053 7735 raft_consensus.cc:482] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:14.185415 7735 raft_consensus.cc:3037] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.185628 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a354a74fa444714abd15b3dc6c9b5ce" candidate_uuid: "56327090ee9245b8b42724ae0edcb365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6075468bc949f08b209b5aebe0c49a" is_pre_election: true
I20240502 01:51:14.185751 7732 consensus_queue.cc:237] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.186486 7534 raft_consensus.cc:2454] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56327090ee9245b8b42724ae0edcb365 in term 0.
I20240502 01:51:14.188856 7609 raft_consensus.cc:2454] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 1.
I20240502 01:51:14.192706 7735 raft_consensus.cc:504] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } }
I20240502 01:51:14.195132 7735 leader_election.cc:288] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [CANDIDATE]: Term 1 election: Requested vote from peers d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861), 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345)
I20240502 01:51:14.194651 7432 catalog_manager.cc:5484] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131). New cstate: current_term: 1 leader_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } health_report { overall_health: HEALTHY } } }
I20240502 01:51:14.195385 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a354a74fa444714abd15b3dc6c9b5ce" candidate_uuid: "56327090ee9245b8b42724ae0edcb365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc"
I20240502 01:51:14.195739 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a354a74fa444714abd15b3dc6c9b5ce" candidate_uuid: "56327090ee9245b8b42724ae0edcb365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6075468bc949f08b209b5aebe0c49a"
I20240502 01:51:14.196030 7683 raft_consensus.cc:3037] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.196347 7534 raft_consensus.cc:3037] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.203617 7534 raft_consensus.cc:2454] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56327090ee9245b8b42724ae0edcb365 in term 1.
I20240502 01:51:14.204675 7574 leader_election.cc:302] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56327090ee9245b8b42724ae0edcb365, 7c6075468bc949f08b209b5aebe0c49a; no voters:
I20240502 01:51:14.205350 7735 raft_consensus.cc:2781] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:51:14.206004 7683 raft_consensus.cc:2454] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56327090ee9245b8b42724ae0edcb365 in term 1.
I20240502 01:51:14.206336 7735 raft_consensus.cc:686] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [term 1 LEADER]: Becoming Leader. State: Replica: 56327090ee9245b8b42724ae0edcb365, State: Running, Role: LEADER
I20240502 01:51:14.207212 7735 consensus_queue.cc:237] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } }
I20240502 01:51:14.213560 7432 catalog_manager.cc:5484] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56327090ee9245b8b42724ae0edcb365 (127.1.233.130). New cstate: current_term: 1 leader_uuid: "56327090ee9245b8b42724ae0edcb365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } health_report { overall_health: UNKNOWN } } }
I20240502 01:51:14.235065 1958 test_util.cc:268] Using random seed: -694837606
I20240502 01:51:14.258654 7432 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39018:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 01:51:14.260759 7432 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:51:14.288770 7523 tablet_service.cc:1467] Processing CreateTablet for tablet 638b7dba1ac5413d9bf634c58264e22f (DEFAULT_TABLE table=dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:51:14.289517 7599 tablet_service.cc:1467] Processing CreateTablet for tablet 638b7dba1ac5413d9bf634c58264e22f (DEFAULT_TABLE table=dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:51:14.290115 7523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638b7dba1ac5413d9bf634c58264e22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:14.290800 7599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638b7dba1ac5413d9bf634c58264e22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:14.290143 7672 tablet_service.cc:1467] Processing CreateTablet for tablet 638b7dba1ac5413d9bf634c58264e22f (DEFAULT_TABLE table=dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 01:51:14.292412 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638b7dba1ac5413d9bf634c58264e22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:14.293356 7521 tablet_service.cc:1467] Processing CreateTablet for tablet 878d3ccfdac840878a115793a8a913ae (DEFAULT_TABLE table=dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:51:14.294628 7521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878d3ccfdac840878a115793a8a913ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:14.306608 7731 tablet_bootstrap.cc:492] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:14.308002 7729 tablet_bootstrap.cc:492] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365: Bootstrap starting.
I20240502 01:51:14.311648 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 878d3ccfdac840878a115793a8a913ae (DEFAULT_TABLE table=dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:51:14.313144 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878d3ccfdac840878a115793a8a913ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:14.313728 7728 tablet_bootstrap.cc:492] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc: Bootstrap starting.
I20240502 01:51:14.313416 7524 tablet_service.cc:1467] Processing CreateTablet for tablet 13f3a4975c3b43598dde32d8368d2422 (DEFAULT_TABLE table=dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:51:14.315531 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f3a4975c3b43598dde32d8368d2422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:14.317706 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 13f3a4975c3b43598dde32d8368d2422 (DEFAULT_TABLE table=dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:51:14.319003 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f3a4975c3b43598dde32d8368d2422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:14.319475 7728 tablet_bootstrap.cc:654] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:14.321141 7672 tablet_service.cc:1467] Processing CreateTablet for tablet 13f3a4975c3b43598dde32d8368d2422 (DEFAULT_TABLE table=dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 01:51:14.322721 7599 tablet_service.cc:1467] Processing CreateTablet for tablet 878d3ccfdac840878a115793a8a913ae (DEFAULT_TABLE table=dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 01:51:14.325059 7522 tablet_service.cc:1467] Processing CreateTablet for tablet 352a62a7a2c1495387b5785b34c26cbc (DEFAULT_TABLE table=dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:51:14.327256 7598 tablet_service.cc:1467] Processing CreateTablet for tablet 352a62a7a2c1495387b5785b34c26cbc (DEFAULT_TABLE table=dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:51:14.328224 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352a62a7a2c1495387b5785b34c26cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:14.328764 7729 tablet_bootstrap.cc:654] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:14.312186 7731 tablet_bootstrap.cc:654] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:14.330240 7598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352a62a7a2c1495387b5785b34c26cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:14.332306 7599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878d3ccfdac840878a115793a8a913ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:14.352109 7670 tablet_service.cc:1467] Processing CreateTablet for tablet 352a62a7a2c1495387b5785b34c26cbc (DEFAULT_TABLE table=dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 01:51:14.353590 7728 tablet_bootstrap.cc:492] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc: No bootstrap required, opened a new log
I20240502 01:51:14.354447 7728 ts_tablet_manager.cc:1367] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.001s
I20240502 01:51:14.358096 7728 raft_consensus.cc:348] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.359097 7728 raft_consensus.cc:374] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:14.359567 7728 raft_consensus.cc:729] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Initialized, Role: FOLLOWER
I20240502 01:51:14.360599 7728 consensus_queue.cc:260] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.363154 7728 ts_tablet_manager.cc:1398] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20240502 01:51:14.342190 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f3a4975c3b43598dde32d8368d2422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:14.366737 7670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352a62a7a2c1495387b5785b34c26cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:51:14.378664 7728 tablet_bootstrap.cc:492] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc: Bootstrap starting.
I20240502 01:51:14.382436 7731 tablet_bootstrap.cc:492] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: No bootstrap required, opened a new log
I20240502 01:51:14.382949 7731 ts_tablet_manager.cc:1367] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.077s user 0.016s sys 0.012s
I20240502 01:51:14.384513 7728 tablet_bootstrap.cc:654] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:14.385784 7731 raft_consensus.cc:348] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.386365 7731 raft_consensus.cc:374] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:14.386600 7731 raft_consensus.cc:729] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:14.387336 7731 consensus_queue.cc:260] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.390249 7731 ts_tablet_manager.cc:1398] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240502 01:51:14.391050 7731 tablet_bootstrap.cc:492] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:14.394533 7729 tablet_bootstrap.cc:492] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365: No bootstrap required, opened a new log
I20240502 01:51:14.395080 7729 ts_tablet_manager.cc:1367] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365: Time spent bootstrapping tablet: real 0.087s user 0.012s sys 0.000s
I20240502 01:51:14.398691 7729 raft_consensus.cc:348] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.400102 7729 raft_consensus.cc:374] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:14.400422 7729 raft_consensus.cc:729] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56327090ee9245b8b42724ae0edcb365, State: Initialized, Role: FOLLOWER
I20240502 01:51:14.401191 7729 consensus_queue.cc:260] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.410226 7729 ts_tablet_manager.cc:1398] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20240502 01:51:14.411242 7729 tablet_bootstrap.cc:492] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365: Bootstrap starting.
I20240502 01:51:14.413930 7731 tablet_bootstrap.cc:654] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:14.411553 7736 raft_consensus.cc:482] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:14.415146 7736 raft_consensus.cc:504] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.418045 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638b7dba1ac5413d9bf634c58264e22f" candidate_uuid: "7c6075468bc949f08b209b5aebe0c49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" is_pre_election: true
I20240502 01:51:14.418113 7736 leader_election.cc:288] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677), d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861)
I20240502 01:51:14.411614 7728 tablet_bootstrap.cc:492] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc: No bootstrap required, opened a new log
I20240502 01:51:14.418143 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638b7dba1ac5413d9bf634c58264e22f" candidate_uuid: "7c6075468bc949f08b209b5aebe0c49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365" is_pre_election: true
I20240502 01:51:14.418794 7683 raft_consensus.cc:2454] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6075468bc949f08b209b5aebe0c49a in term 0.
I20240502 01:51:14.419380 7609 raft_consensus.cc:2454] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6075468bc949f08b209b5aebe0c49a in term 0.
I20240502 01:51:14.419726 7728 ts_tablet_manager.cc:1367] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.020s
I20240502 01:51:14.420782 7497 leader_election.cc:302] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6075468bc949f08b209b5aebe0c49a, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:14.421870 7736 raft_consensus.cc:2781] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:51:14.422284 7736 raft_consensus.cc:482] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:14.422596 7736 raft_consensus.cc:3037] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.423641 7729 tablet_bootstrap.cc:654] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:14.423776 7728 raft_consensus.cc:348] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.424773 7728 raft_consensus.cc:374] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:14.425097 7728 raft_consensus.cc:729] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Initialized, Role: FOLLOWER
I20240502 01:51:14.425944 7728 consensus_queue.cc:260] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.430032 7736 raft_consensus.cc:504] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.430878 7728 ts_tablet_manager.cc:1398] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20240502 01:51:14.432219 7728 tablet_bootstrap.cc:492] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc: Bootstrap starting.
I20240502 01:51:14.433079 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638b7dba1ac5413d9bf634c58264e22f" candidate_uuid: "7c6075468bc949f08b209b5aebe0c49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc"
I20240502 01:51:14.433725 7683 raft_consensus.cc:3037] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.438907 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638b7dba1ac5413d9bf634c58264e22f" candidate_uuid: "7c6075468bc949f08b209b5aebe0c49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365"
I20240502 01:51:14.439759 7609 raft_consensus.cc:3037] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.440953 7728 tablet_bootstrap.cc:654] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:14.441718 7683 raft_consensus.cc:2454] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6075468bc949f08b209b5aebe0c49a in term 1.
I20240502 01:51:14.443979 7497 leader_election.cc:302] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6075468bc949f08b209b5aebe0c49a, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:14.442632 7736 leader_election.cc:288] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [CANDIDATE]: Term 1 election: Requested vote from peers 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677), d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861)
I20240502 01:51:14.445174 7736 raft_consensus.cc:2781] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:51:14.446502 7736 raft_consensus.cc:686] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 1 LEADER]: Becoming Leader. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Running, Role: LEADER
I20240502 01:51:14.447360 7736 consensus_queue.cc:237] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.449342 7609 raft_consensus.cc:2454] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6075468bc949f08b209b5aebe0c49a in term 1.
I20240502 01:51:14.453953 7731 tablet_bootstrap.cc:492] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: No bootstrap required, opened a new log
I20240502 01:51:14.454701 7731 ts_tablet_manager.cc:1367] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.031s
I20240502 01:51:14.458395 7728 tablet_bootstrap.cc:492] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc: No bootstrap required, opened a new log
I20240502 01:51:14.459041 7728 ts_tablet_manager.cc:1367] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.003s
I20240502 01:51:14.459591 7729 tablet_bootstrap.cc:492] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365: No bootstrap required, opened a new log
I20240502 01:51:14.460031 7729 ts_tablet_manager.cc:1367] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365: Time spent bootstrapping tablet: real 0.049s user 0.017s sys 0.001s
I20240502 01:51:14.460743 7731 raft_consensus.cc:348] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.461475 7731 raft_consensus.cc:374] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:14.461738 7731 raft_consensus.cc:729] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:14.461776 7728 raft_consensus.cc:348] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.462461 7728 raft_consensus.cc:374] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:14.462805 7728 raft_consensus.cc:729] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Initialized, Role: FOLLOWER
I20240502 01:51:14.462489 7731 consensus_queue.cc:260] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.463502 7728 consensus_queue.cc:260] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.463390 7431 catalog_manager.cc:5484] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129). New cstate: current_term: 1 leader_uuid: "7c6075468bc949f08b209b5aebe0c49a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } health_report { overall_health: UNKNOWN } } }
I20240502 01:51:14.465443 7729 raft_consensus.cc:348] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.466140 7729 raft_consensus.cc:374] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:14.466385 7729 raft_consensus.cc:729] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56327090ee9245b8b42724ae0edcb365, State: Initialized, Role: FOLLOWER
I20240502 01:51:14.465014 7731 ts_tablet_manager.cc:1398] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20240502 01:51:14.467010 7728 ts_tablet_manager.cc:1398] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20240502 01:51:14.467794 7731 tablet_bootstrap.cc:492] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:14.467937 7728 tablet_bootstrap.cc:492] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc: Bootstrap starting.
I20240502 01:51:14.474355 7728 tablet_bootstrap.cc:654] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:14.467109 7729 consensus_queue.cc:260] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.477599 7735 raft_consensus.cc:482] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:14.478042 7735 raft_consensus.cc:504] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.480554 7735 leader_election.cc:288] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861)
I20240502 01:51:14.482373 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352a62a7a2c1495387b5785b34c26cbc" candidate_uuid: "56327090ee9245b8b42724ae0edcb365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6075468bc949f08b209b5aebe0c49a" is_pre_election: true
I20240502 01:51:14.482826 7731 tablet_bootstrap.cc:654] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:14.484040 7683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352a62a7a2c1495387b5785b34c26cbc" candidate_uuid: "56327090ee9245b8b42724ae0edcb365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" is_pre_election: true
W20240502 01:51:14.485467 7574 leader_election.cc:341] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:51:14.486816 7572 leader_election.cc:341] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:51:14.487318 7572 leader_election.cc:302] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56327090ee9245b8b42724ae0edcb365; no voters: 7c6075468bc949f08b209b5aebe0c49a, d3a96ac41ed64f778328e3e2fb6ebdfc
I20240502 01:51:14.487955 7735 raft_consensus.cc:2726] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:51:14.489961 7732 consensus_queue.cc:1035] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:51:14.494196 7729 ts_tablet_manager.cc:1398] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365: Time spent starting tablet: real 0.034s user 0.004s sys 0.000s
I20240502 01:51:14.495435 7729 tablet_bootstrap.cc:492] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365: Bootstrap starting.
I20240502 01:51:14.496452 7731 tablet_bootstrap.cc:492] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: No bootstrap required, opened a new log
I20240502 01:51:14.497119 7731 ts_tablet_manager.cc:1367] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.010s
I20240502 01:51:14.500619 7731 raft_consensus.cc:348] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.501994 7729 tablet_bootstrap.cc:654] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:14.501551 7731 raft_consensus.cc:374] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:14.507416 7731 raft_consensus.cc:729] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:14.508302 7731 consensus_queue.cc:260] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.509346 7728 tablet_bootstrap.cc:492] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc: No bootstrap required, opened a new log
I20240502 01:51:14.509886 7728 ts_tablet_manager.cc:1367] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.023s
I20240502 01:51:14.510315 7731 ts_tablet_manager.cc:1398] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20240502 01:51:14.511075 7731 tablet_bootstrap.cc:492] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:14.512519 7728 raft_consensus.cc:348] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.513221 7728 raft_consensus.cc:374] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:14.513504 7728 raft_consensus.cc:729] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Initialized, Role: FOLLOWER
I20240502 01:51:14.514115 7728 consensus_queue.cc:260] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.516687 7731 tablet_bootstrap.cc:654] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:14.518821 7729 tablet_bootstrap.cc:492] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365: No bootstrap required, opened a new log
I20240502 01:51:14.519446 7729 ts_tablet_manager.cc:1367] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.000s
I20240502 01:51:14.522708 7729 raft_consensus.cc:348] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.523736 7729 raft_consensus.cc:374] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:14.524099 7729 raft_consensus.cc:729] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56327090ee9245b8b42724ae0edcb365, State: Initialized, Role: FOLLOWER
I20240502 01:51:14.525108 7729 consensus_queue.cc:260] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.516100 7728 ts_tablet_manager.cc:1398] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:51:14.529407 7729 ts_tablet_manager.cc:1398] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20240502 01:51:14.530732 7729 tablet_bootstrap.cc:492] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365: Bootstrap starting.
I20240502 01:51:14.535455 7748 raft_consensus.cc:482] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:14.536417 7748 raft_consensus.cc:504] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.538470 7748 leader_election.cc:288] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
I20240502 01:51:14.539732 7729 tablet_bootstrap.cc:654] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365: Neither blocks nor log segments found. Creating new log.
I20240502 01:51:14.540385 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f3a4975c3b43598dde32d8368d2422" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365" is_pre_election: true
W20240502 01:51:14.543165 7647 leader_election.cc:341] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:51:14.547705 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f3a4975c3b43598dde32d8368d2422" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6075468bc949f08b209b5aebe0c49a" is_pre_election: true
I20240502 01:51:14.556879 7752 consensus_queue.cc:1035] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [LEADER]: Connected to new peer: Peer: permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:51:14.562474 7534 raft_consensus.cc:2454] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 0.
I20240502 01:51:14.562556 7748 consensus_queue.cc:1035] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:51:14.569474 7748 raft_consensus.cc:482] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:14.570137 7748 raft_consensus.cc:504] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.574539 7648 leader_election.cc:302] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c6075468bc949f08b209b5aebe0c49a, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters: 56327090ee9245b8b42724ae0edcb365
I20240502 01:51:14.579641 7748 leader_election.cc:288] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
I20240502 01:51:14.580497 7748 raft_consensus.cc:2781] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:51:14.580920 7748 raft_consensus.cc:482] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:14.581284 7748 raft_consensus.cc:3037] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.589957 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352a62a7a2c1495387b5785b34c26cbc" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6075468bc949f08b209b5aebe0c49a" is_pre_election: true
W20240502 01:51:14.595525 7648 leader_election.cc:341] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:51:14.628676 7607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352a62a7a2c1495387b5785b34c26cbc" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365" is_pre_election: true
I20240502 01:51:14.629344 7731 tablet_bootstrap.cc:492] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: No bootstrap required, opened a new log
I20240502 01:51:14.629923 7731 ts_tablet_manager.cc:1367] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.119s user 0.056s sys 0.054s
I20240502 01:51:14.629765 7607 raft_consensus.cc:2454] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 0.
I20240502 01:51:14.630247 7736 consensus_queue.cc:1035] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:51:14.634060 7647 leader_election.cc:302] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56327090ee9245b8b42724ae0edcb365, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters: 7c6075468bc949f08b209b5aebe0c49a
I20240502 01:51:14.635854 7781 raft_consensus.cc:2781] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:51:14.636312 7781 raft_consensus.cc:482] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:14.624663 7732 consensus_queue.cc:1035] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240502 01:51:14.636634 7781 raft_consensus.cc:3037] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.641047 7731 raft_consensus.cc:348] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.641955 7731 raft_consensus.cc:374] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:14.642285 7731 raft_consensus.cc:729] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:14.643096 7731 consensus_queue.cc:260] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.643594 7729 tablet_bootstrap.cc:492] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365: No bootstrap required, opened a new log
I20240502 01:51:14.646837 7731 ts_tablet_manager.cc:1398] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.017s user 0.007s sys 0.001s
I20240502 01:51:14.649392 7735 consensus_queue.cc:1035] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:51:14.655395 7735 consensus_queue.cc:1035] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:51:14.658190 7732 consensus_queue.cc:1035] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:51:14.654717 7729 ts_tablet_manager.cc:1367] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365: Time spent bootstrapping tablet: real 0.124s user 0.038s sys 0.053s
I20240502 01:51:14.663789 7781 raft_consensus.cc:504] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.666842 7532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352a62a7a2c1495387b5785b34c26cbc" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6075468bc949f08b209b5aebe0c49a"
I20240502 01:51:14.666715 7748 raft_consensus.cc:504] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.667515 7532 raft_consensus.cc:3037] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.671521 7531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f3a4975c3b43598dde32d8368d2422" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6075468bc949f08b209b5aebe0c49a"
I20240502 01:51:14.674412 7531 raft_consensus.cc:3037] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.680747 7532 raft_consensus.cc:2454] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 1.
I20240502 01:51:14.682003 7648 leader_election.cc:302] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6075468bc949f08b209b5aebe0c49a, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:14.682253 7608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352a62a7a2c1495387b5785b34c26cbc" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365"
I20240502 01:51:14.683010 7608 raft_consensus.cc:3037] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.687650 7781 leader_election.cc:288] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 election: Requested vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
I20240502 01:51:14.683108 7786 raft_consensus.cc:2781] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:51:14.687940 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f3a4975c3b43598dde32d8368d2422" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365"
I20240502 01:51:14.688457 7786 raft_consensus.cc:686] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 LEADER]: Becoming Leader. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Running, Role: LEADER
I20240502 01:51:14.683986 7729 raft_consensus.cc:348] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.689357 7729 raft_consensus.cc:374] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:51:14.689754 7729 raft_consensus.cc:729] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56327090ee9245b8b42724ae0edcb365, State: Initialized, Role: FOLLOWER
I20240502 01:51:14.689404 7786 consensus_queue.cc:237] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.691183 7748 leader_election.cc:288] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 election: Requested vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
I20240502 01:51:14.690536 7729 consensus_queue.cc:260] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.691967 7609 raft_consensus.cc:3037] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.696697 7608 raft_consensus.cc:2454] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 1.
I20240502 01:51:14.699374 7609 raft_consensus.cc:2454] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 1.
I20240502 01:51:14.702800 7729 ts_tablet_manager.cc:1398] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20240502 01:51:14.705307 7647 leader_election.cc:302] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56327090ee9245b8b42724ae0edcb365, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:14.706367 7781 raft_consensus.cc:2781] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:51:14.706825 7781 raft_consensus.cc:686] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 LEADER]: Becoming Leader. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Running, Role: LEADER
I20240502 01:51:14.707595 7781 consensus_queue.cc:237] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.710788 7531 raft_consensus.cc:2454] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 1.
I20240502 01:51:14.748865 7431 catalog_manager.cc:5484] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131). New cstate: current_term: 1 leader_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } health_report { overall_health: HEALTHY } } }
I20240502 01:51:14.750466 7431 catalog_manager.cc:5484] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131). New cstate: current_term: 1 leader_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } health_report { overall_health: HEALTHY } } }
I20240502 01:51:14.794306 7781 raft_consensus.cc:482] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:51:14.794836 7781 raft_consensus.cc:504] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.796546 7781 leader_election.cc:288] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
I20240502 01:51:14.797607 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878d3ccfdac840878a115793a8a913ae" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6075468bc949f08b209b5aebe0c49a" is_pre_election: true
I20240502 01:51:14.797641 7607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878d3ccfdac840878a115793a8a913ae" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365" is_pre_election: true
I20240502 01:51:14.798281 7533 raft_consensus.cc:2454] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 0.
I20240502 01:51:14.798362 7607 raft_consensus.cc:2454] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 0.
I20240502 01:51:14.799271 7648 leader_election.cc:302] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6075468bc949f08b209b5aebe0c49a, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:14.799968 7781 raft_consensus.cc:2781] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:51:14.800330 7781 raft_consensus.cc:482] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:51:14.800573 7781 raft_consensus.cc:3037] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.805500 7781 raft_consensus.cc:504] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.806963 7781 leader_election.cc:288] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 election: Requested vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
I20240502 01:51:14.807833 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878d3ccfdac840878a115793a8a913ae" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6075468bc949f08b209b5aebe0c49a"
I20240502 01:51:14.808045 7607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878d3ccfdac840878a115793a8a913ae" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365"
I20240502 01:51:14.808560 7533 raft_consensus.cc:3037] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.808607 7607 raft_consensus.cc:3037] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:51:14.815814 7607 raft_consensus.cc:2454] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 1.
I20240502 01:51:14.815814 7533 raft_consensus.cc:2454] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 1.
I20240502 01:51:14.817112 7647 leader_election.cc:302] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56327090ee9245b8b42724ae0edcb365, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:14.817802 7781 raft_consensus.cc:2781] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:51:14.818230 7781 raft_consensus.cc:686] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 LEADER]: Becoming Leader. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Running, Role: LEADER
I20240502 01:51:14.818831 7781 consensus_queue.cc:237] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:14.825847 7429 catalog_manager.cc:5484] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc reported cstate change: term changed from 0 to 1, leader changed from <none> to d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131). New cstate: current_term: 1 leader_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } health_report { overall_health: HEALTHY } } }
I20240502 01:51:14.845786 1958 tablet_server.cc:178] TabletServer@127.1.233.129:0 shutting down...
I20240502 01:51:14.867341 1958 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 01:51:14.868252 1958 tablet_replica.cc:330] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: stopping tablet replica
I20240502 01:51:14.869032 1958 raft_consensus.cc:2229] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:14.869406 1958 raft_consensus.cc:2258] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:14.871733 1958 tablet_replica.cc:330] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: stopping tablet replica
I20240502 01:51:14.872560 1958 raft_consensus.cc:2229] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:14.873090 1958 raft_consensus.cc:2258] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:14.875309 1958 tablet_replica.cc:330] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a: stopping tablet replica
I20240502 01:51:14.875856 1958 raft_consensus.cc:2229] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:14.876282 1958 raft_consensus.cc:2258] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:14.878474 1958 tablet_replica.cc:330] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a: stopping tablet replica
I20240502 01:51:14.879174 1958 raft_consensus.cc:2229] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:14.879602 1958 raft_consensus.cc:2258] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:14.880932 7752 consensus_queue.cc:1035] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:51:14.882164 1958 tablet_replica.cc:330] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: stopping tablet replica
I20240502 01:51:14.882768 1958 raft_consensus.cc:2229] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:14.883273 1958 raft_consensus.cc:2258] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:14.885799 1958 tablet_replica.cc:330] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a: stopping tablet replica
I20240502 01:51:14.886312 1958 raft_consensus.cc:2229] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:51:14.887145 1958 raft_consensus.cc:2258] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:14.890040 1958 tablet_replica.cc:330] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: stopping tablet replica
I20240502 01:51:14.891055 1958 raft_consensus.cc:2229] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:51:14.896862 1958 raft_consensus.cc:2258] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:14.898994 1958 tablet_replica.cc:330] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a: stopping tablet replica
I20240502 01:51:14.899480 1958 raft_consensus.cc:2229] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:14.899886 1958 raft_consensus.cc:2258] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:14.918208 1958 tablet_server.cc:194] TabletServer@127.1.233.129:0 shutdown complete.
W20240502 01:51:15.067431 7648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.233.129:33345: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240502 01:51:15.070675 7648 consensus_peers.cc:485] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc -> Peer 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): Couldn't send request to peer 7c6075468bc949f08b209b5aebe0c49a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:33345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:51:15.129412 7648 consensus_peers.cc:485] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc -> Peer 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): Couldn't send request to peer 7c6075468bc949f08b209b5aebe0c49a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:33345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:51:15.221518 7781 consensus_queue.cc:1035] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:51:15.276541 7648 consensus_peers.cc:485] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc -> Peer 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): Couldn't send request to peer 7c6075468bc949f08b209b5aebe0c49a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:33345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:51:15.284054 7574 consensus_peers.cc:485] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 -> Peer 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): Couldn't send request to peer 7c6075468bc949f08b209b5aebe0c49a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:33345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:51:15.335845 7781 consensus_queue.cc:1035] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:51:15.346668 7648 consensus_peers.cc:485] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc -> Peer 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): Couldn't send request to peer 7c6075468bc949f08b209b5aebe0c49a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:33345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:51:15.388875 7807 consensus_queue.cc:1035] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:51:15.393838 7648 consensus_peers.cc:485] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc -> Peer 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): Couldn't send request to peer 7c6075468bc949f08b209b5aebe0c49a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:33345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:51:15.554320 7807 consensus_queue.cc:579] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Leader has been unable to successfully communicate with peer 7c6075468bc949f08b209b5aebe0c49a for more than 1 seconds (1.003s)
I20240502 01:51:15.809973 7735 consensus_queue.cc:579] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [LEADER]: Leader has been unable to successfully communicate with peer 7c6075468bc949f08b209b5aebe0c49a for more than 1 seconds (1.064s)
I20240502 01:51:15.919091 7807 consensus_queue.cc:579] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Leader has been unable to successfully communicate with peer 7c6075468bc949f08b209b5aebe0c49a for more than 1 seconds (1.171s)
I20240502 01:51:15.957331 7432 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47886:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240502 01:51:15.958228 7432 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47886:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240502 01:51:15.962901 7807 consensus_queue.cc:579] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Leader has been unable to successfully communicate with peer 7c6075468bc949f08b209b5aebe0c49a for more than 1 seconds (1.143s)
I20240502 01:51:15.989845 7432 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885: Sending DeleteTablet for 3 replicas of tablet 638b7dba1ac5413d9bf634c58264e22f
I20240502 01:51:15.991941 7432 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885: Sending DeleteTablet for 3 replicas of tablet 878d3ccfdac840878a115793a8a913ae
I20240502 01:51:15.993641 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 638b7dba1ac5413d9bf634c58264e22f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:51:15 UTC) from {username='slave'} at 127.0.0.1:36848
I20240502 01:51:15.994719 7599 tablet_service.cc:1514] Processing DeleteTablet for tablet 878d3ccfdac840878a115793a8a913ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:51:15 UTC) from {username='slave'} at 127.0.0.1:36848
I20240502 01:51:15.995610 7670 tablet_service.cc:1514] Processing DeleteTablet for tablet 878d3ccfdac840878a115793a8a913ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:51:15 UTC) from {username='slave'} at 127.0.0.1:59044
I20240502 01:51:15.996435 7821 tablet_replica.cc:330] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365: stopping tablet replica
I20240502 01:51:15.997037 7432 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885: Sending DeleteTablet for 3 replicas of tablet 13f3a4975c3b43598dde32d8368d2422
I20240502 01:51:15.997371 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet 878d3ccfdac840878a115793a8a913ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:51:15 UTC) from {username='slave'} at 127.0.0.1:59044
I20240502 01:51:15.997965 7673 tablet_service.cc:1514] Processing DeleteTablet for tablet 638b7dba1ac5413d9bf634c58264e22f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:51:15 UTC) from {username='slave'} at 127.0.0.1:59044
I20240502 01:51:15.998972 7821 raft_consensus.cc:2229] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:15.999931 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f3a4975c3b43598dde32d8368d2422 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:51:15 UTC) from {username='slave'} at 127.0.0.1:36848
I20240502 01:51:16.000383 7821 raft_consensus.cc:2258] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:16.002859 7673 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f3a4975c3b43598dde32d8368d2422 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:51:15 UTC) from {username='slave'} at 127.0.0.1:59044
I20240502 01:51:16.003769 7432 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 2720d009733a4f609c1859836042e885: Sending DeleteTablet for 3 replicas of tablet 352a62a7a2c1495387b5785b34c26cbc
I20240502 01:51:16.006641 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 352a62a7a2c1495387b5785b34c26cbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:51:15 UTC) from {username='slave'} at 127.0.0.1:36848
I20240502 01:51:16.007571 7822 tablet_replica.cc:330] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc: stopping tablet replica
I20240502 01:51:16.008858 7673 tablet_service.cc:1514] Processing DeleteTablet for tablet 352a62a7a2c1495387b5785b34c26cbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:51:15 UTC) from {username='slave'} at 127.0.0.1:59044
I20240502 01:51:16.009060 7822 raft_consensus.cc:2229] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:51:16.009090 7821 ts_tablet_manager.cc:1875] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:51:16.011080 7822 raft_consensus.cc:2258] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:16.013695 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:51:16.018510 7822 ts_tablet_manager.cc:1875] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 01:51:16.019212 7418 catalog_manager.cc:4573] TS 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 352a62a7a2c1495387b5785b34c26cbc: Network error: Client connection negotiation failed: client connection to 127.1.233.129:33345: connect: Connection refused (error 111)
W20240502 01:51:16.034569 7824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:16.035506 7825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:16.038105 7821 ts_tablet_manager.cc:1888] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 01:51:16.038689 7821 log.cc:1196] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-1-root/wals/638b7dba1ac5413d9bf634c58264e22f
I20240502 01:51:16.039602 7821 ts_tablet_manager.cc:1909] T 638b7dba1ac5413d9bf634c58264e22f P 56327090ee9245b8b42724ae0edcb365: Deleting consensus metadata
I20240502 01:51:16.043828 7417 catalog_manager.cc:4830] TS 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677): tablet 638b7dba1ac5413d9bf634c58264e22f (table dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]) successfully deleted
I20240502 01:51:16.044976 7821 tablet_replica.cc:330] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365: stopping tablet replica
I20240502 01:51:16.047190 7821 raft_consensus.cc:2229] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:16.050014 7821 raft_consensus.cc:2258] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:51:16.051668 7827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:16.054157 7822 ts_tablet_manager.cc:1888] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 01:51:16.055524 1958 server_base.cc:1034] running on GCE node
I20240502 01:51:16.055660 7822 log.cc:1196] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-2-root/wals/878d3ccfdac840878a115793a8a913ae
I20240502 01:51:16.057600 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 01:51:16.057690 7822 ts_tablet_manager.cc:1909] T 878d3ccfdac840878a115793a8a913ae P d3a96ac41ed64f778328e3e2fb6ebdfc: Deleting consensus metadata
W20240502 01:51:16.058009 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:51:16.058177 7821 ts_tablet_manager.cc:1875] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:51:16.058290 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614676058272 us; error 0 us; skew 500 ppm
I20240502 01:51:16.059455 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:51:16.062386 7416 catalog_manager.cc:4830] TS d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861): tablet 878d3ccfdac840878a115793a8a913ae (table dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]) successfully deleted
I20240502 01:51:16.063095 7822 tablet_replica.cc:330] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc: stopping tablet replica
W20240502 01:51:16.063369 7416 catalog_manager.cc:4797] TS d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861): delete failed for tablet 878d3ccfdac840878a115793a8a913ae because the tablet was not found. No further retry: Not found: Tablet not found: 878d3ccfdac840878a115793a8a913ae
I20240502 01:51:16.064339 7822 raft_consensus.cc:2229] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:16.065274 7822 raft_consensus.cc:2258] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:16.069070 7822 ts_tablet_manager.cc:1875] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:51:16.070101 1958 webserver.cc:460] Webserver started at http://127.1.233.132:37637/ using document root <none> and password file <none>
I20240502 01:51:16.070652 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:51:16.070875 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:51:16.071202 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:51:16.072721 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-3-root/instance:
uuid: "a3af9b0e088744239d1425da594190f2"
format_stamp: "Formatted at 2024-05-02 01:51:16 on dist-test-slave-2090"
I20240502 01:51:16.073690 7807 consensus_queue.cc:579] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Leader has been unable to successfully communicate with peer 7c6075468bc949f08b209b5aebe0c49a for more than 1 seconds (1.365s)
I20240502 01:51:16.077474 7821 ts_tablet_manager.cc:1888] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 01:51:16.077916 7821 log.cc:1196] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-1-root/wals/878d3ccfdac840878a115793a8a913ae
I20240502 01:51:16.078672 7821 ts_tablet_manager.cc:1909] T 878d3ccfdac840878a115793a8a913ae P 56327090ee9245b8b42724ae0edcb365: Deleting consensus metadata
I20240502 01:51:16.083099 7417 catalog_manager.cc:4830] TS 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677): tablet 878d3ccfdac840878a115793a8a913ae (table dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]) successfully deleted
I20240502 01:51:16.084259 7821 tablet_replica.cc:330] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365: stopping tablet replica
I20240502 01:51:16.085363 7821 raft_consensus.cc:2229] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:16.085909 7822 ts_tablet_manager.cc:1888] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 01:51:16.086194 7821 raft_consensus.cc:2258] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:16.086382 7822 log.cc:1196] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-2-root/wals/638b7dba1ac5413d9bf634c58264e22f
I20240502 01:51:16.087303 7822 ts_tablet_manager.cc:1909] T 638b7dba1ac5413d9bf634c58264e22f P d3a96ac41ed64f778328e3e2fb6ebdfc: Deleting consensus metadata
I20240502 01:51:16.090060 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.004s sys 0.008s
I20240502 01:51:16.091105 7821 ts_tablet_manager.cc:1875] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:51:16.091385 7822 tablet_replica.cc:330] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc: stopping tablet replica
I20240502 01:51:16.092525 7822 raft_consensus.cc:2229] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:51:16.093741 7416 catalog_manager.cc:4830] TS d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861): tablet 638b7dba1ac5413d9bf634c58264e22f (table dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]) successfully deleted
I20240502 01:51:16.093928 7822 raft_consensus.cc:2258] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:16.098291 7822 ts_tablet_manager.cc:1875] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:51:16.109050 7821 ts_tablet_manager.cc:1888] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 01:51:16.109536 7821 log.cc:1196] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-1-root/wals/13f3a4975c3b43598dde32d8368d2422
I20240502 01:51:16.110316 7821 ts_tablet_manager.cc:1909] T 13f3a4975c3b43598dde32d8368d2422 P 56327090ee9245b8b42724ae0edcb365: Deleting consensus metadata
I20240502 01:51:16.112794 7832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:16.113898 1958 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.004s sys 0.016s
I20240502 01:51:16.114205 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-3-root
uuid: "a3af9b0e088744239d1425da594190f2"
format_stamp: "Formatted at 2024-05-02 01:51:16 on dist-test-slave-2090"
I20240502 01:51:16.114512 7417 catalog_manager.cc:4830] TS 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677): tablet 13f3a4975c3b43598dde32d8368d2422 (table dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]) successfully deleted
I20240502 01:51:16.114513 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:51:16.115134 7822 ts_tablet_manager.cc:1888] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 01:51:16.115607 7822 log.cc:1196] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-2-root/wals/13f3a4975c3b43598dde32d8368d2422
I20240502 01:51:16.115797 7821 tablet_replica.cc:330] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365: stopping tablet replica
I20240502 01:51:16.116451 7822 ts_tablet_manager.cc:1909] T 13f3a4975c3b43598dde32d8368d2422 P d3a96ac41ed64f778328e3e2fb6ebdfc: Deleting consensus metadata
I20240502 01:51:16.116837 7821 raft_consensus.cc:2229] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:16.117758 7821 raft_consensus.cc:2258] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:16.120581 7822 tablet_replica.cc:330] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc: stopping tablet replica
I20240502 01:51:16.120718 7416 catalog_manager.cc:4830] TS d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861): tablet 13f3a4975c3b43598dde32d8368d2422 (table dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]) successfully deleted
I20240502 01:51:16.121688 7822 raft_consensus.cc:2229] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:51:16.123400 7821 ts_tablet_manager.cc:1875] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:51:16.123484 7822 raft_consensus.cc:2258] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:16.128890 7822 ts_tablet_manager.cc:1875] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:51:16.132483 7810 raft_consensus.cc:482] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7c6075468bc949f08b209b5aebe0c49a)
I20240502 01:51:16.133170 7810 raft_consensus.cc:504] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.137207 7682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d945ad77c1284b33a14e47ddda96f868" candidate_uuid: "56327090ee9245b8b42724ae0edcb365" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" is_pre_election: true
I20240502 01:51:16.139731 7810 leader_election.cc:288] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861)
W20240502 01:51:16.141619 7574 leader_election.cc:334] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): Network error: Client connection negotiation failed: client connection to 127.1.233.129:33345: connect: Connection refused (error 111)
I20240502 01:51:16.142498 7574 leader_election.cc:302] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56327090ee9245b8b42724ae0edcb365; no voters: 7c6075468bc949f08b209b5aebe0c49a, d3a96ac41ed64f778328e3e2fb6ebdfc
I20240502 01:51:16.143721 7821 ts_tablet_manager.cc:1888] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 01:51:16.143710 7810 raft_consensus.cc:2726] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:51:16.144451 7821 log.cc:1196] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-1-root/wals/352a62a7a2c1495387b5785b34c26cbc
I20240502 01:51:16.145292 7821 ts_tablet_manager.cc:1909] T 352a62a7a2c1495387b5785b34c26cbc P 56327090ee9245b8b42724ae0edcb365: Deleting consensus metadata
I20240502 01:51:16.147920 7822 ts_tablet_manager.cc:1888] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 01:51:16.148418 7822 log.cc:1196] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-2-root/wals/352a62a7a2c1495387b5785b34c26cbc
I20240502 01:51:16.149227 7822 ts_tablet_manager.cc:1909] T 352a62a7a2c1495387b5785b34c26cbc P d3a96ac41ed64f778328e3e2fb6ebdfc: Deleting consensus metadata
I20240502 01:51:16.149227 7417 catalog_manager.cc:4830] TS 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677): tablet 352a62a7a2c1495387b5785b34c26cbc (table dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]) successfully deleted
I20240502 01:51:16.152406 7416 catalog_manager.cc:4830] TS d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861): tablet 352a62a7a2c1495387b5785b34c26cbc (table dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]) successfully deleted
I20240502 01:51:16.160591 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:51:16.162384 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:51:16.163899 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:51:16.166513 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:51:16.166720 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:16.167018 1958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:51:16.167178 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:16.214097 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.132:37253
I20240502 01:51:16.214298 7894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.132:37253 every 8 connection(s)
I20240502 01:51:16.214859 7748 raft_consensus.cc:482] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7c6075468bc949f08b209b5aebe0c49a)
I20240502 01:51:16.215483 7748 raft_consensus.cc:504] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.217550 7748 leader_election.cc:288] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
I20240502 01:51:16.219398 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d945ad77c1284b33a14e47ddda96f868" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365" is_pre_election: true
I20240502 01:51:16.220111 7609 raft_consensus.cc:2454] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 1.
I20240502 01:51:16.221951 7647 leader_election.cc:302] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56327090ee9245b8b42724ae0edcb365, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:16.223125 7748 raft_consensus.cc:2781] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:51:16.223611 7748 raft_consensus.cc:482] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7c6075468bc949f08b209b5aebe0c49a)
I20240502 01:51:16.224045 7748 raft_consensus.cc:3037] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:51:16.226995 7648 leader_election.cc:334] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): Network error: Client connection negotiation failed: client connection to 127.1.233.129:33345: connect: Connection refused (error 111)
I20240502 01:51:16.232784 1958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:51:16.233075 7748 raft_consensus.cc:504] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.236287 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d945ad77c1284b33a14e47ddda96f868" candidate_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56327090ee9245b8b42724ae0edcb365"
I20240502 01:51:16.236919 7609 raft_consensus.cc:3037] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:51:16.244727 7895 heartbeater.cc:347] Connected to a master server at 127.1.233.190:39913
I20240502 01:51:16.245232 7895 heartbeater.cc:464] Registering TS with master...
I20240502 01:51:16.245007 7609 raft_consensus.cc:2454] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3a96ac41ed64f778328e3e2fb6ebdfc in term 2.
I20240502 01:51:16.245158 7748 leader_election.cc:288] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 2 election: Requested vote from peers 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345), 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
I20240502 01:51:16.246348 7895 heartbeater.cc:511] Master 127.1.233.190:39913 requested a full tablet report, sending...
I20240502 01:51:16.248751 7647 leader_election.cc:302] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56327090ee9245b8b42724ae0edcb365, d3a96ac41ed64f778328e3e2fb6ebdfc; no voters:
I20240502 01:51:16.249890 7432 ts_manager.cc:194] Registered new tserver with Master: a3af9b0e088744239d1425da594190f2 (127.1.233.132:37253)
W20240502 01:51:16.250680 7648 leader_election.cc:334] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): Network error: Client connection negotiation failed: client connection to 127.1.233.129:33345: connect: Connection refused (error 111)
I20240502 01:51:16.252528 7748 raft_consensus.cc:2781] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:51:16.253111 7748 raft_consensus.cc:686] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 2 LEADER]: Becoming Leader. State: Replica: d3a96ac41ed64f778328e3e2fb6ebdfc, State: Running, Role: LEADER
I20240502 01:51:16.253291 7432 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39036
I20240502 01:51:16.254220 7748 consensus_queue.cc:237] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
W20240502 01:51:16.258860 7898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:51:16.266945 7899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:16.269271 7682 consensus_queue.cc:237] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } }
W20240502 01:51:16.269950 7480 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 7c6075468bc949f08b209b5aebe0c49a not available for placement
W20240502 01:51:16.275393 7902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:51:16.276515 7432 catalog_manager.cc:5484] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc reported cstate change: term changed from 1 to 2, leader changed from 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129) to d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131). New cstate: current_term: 2 leader_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } health_report { overall_health: HEALTHY } } }
I20240502 01:51:16.280771 7609 raft_consensus.cc:1261] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Refusing update from remote peer d3a96ac41ed64f778328e3e2fb6ebdfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:51:16.282440 1958 server_base.cc:1034] running on GCE node
I20240502 01:51:16.282667 7748 consensus_queue.cc:1035] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:51:16.283596 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:51:16.284013 1958 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:51:16.284250 1958 hybrid_clock.cc:648] HybridClock initialized: now 1714614676284229 us; error 0 us; skew 500 ppm
I20240502 01:51:16.285167 1958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:51:16.291503 1958 webserver.cc:460] Webserver started at http://127.1.233.129:37541/ using document root <none> and password file <none>
I20240502 01:51:16.292040 1958 fs_manager.cc:359] Metadata directory not provided
I20240502 01:51:16.292263 1958 fs_manager.cc:362] Using existing metadata directory in first data directory
W20240502 01:51:16.300930 7648 consensus_peers.cc:485] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc -> Peer 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): Couldn't send request to peer 7c6075468bc949f08b209b5aebe0c49a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:33345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:51:16.303069 1958 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.008s sys 0.002s
I20240502 01:51:16.302356 7748 raft_consensus.cc:2932] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } } }
I20240502 01:51:16.305932 7609 raft_consensus.cc:2932] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } } }
W20240502 01:51:16.308794 7649 consensus_peers.cc:485] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc -> Peer a3af9b0e088744239d1425da594190f2 (127.1.233.132:37253): Couldn't send request to peer a3af9b0e088744239d1425da594190f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fadba1fb5a5c41cc8904ab3920871362. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:51:16.310781 7911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:51:16.311846 1958 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.004s
I20240502 01:51:16.312232 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-0-root
uuid: "7c6075468bc949f08b209b5aebe0c49a"
format_stamp: "Formatted at 2024-05-02 01:51:12 on dist-test-slave-2090"
I20240502 01:51:16.312650 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:51:16.314697 7416 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fadba1fb5a5c41cc8904ab3920871362 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20240502 01:51:16.318351 7432 catalog_manager.cc:5484] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc reported cstate change: config changed from index -1 to 2, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) added. New cstate: current_term: 1 leader_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:51:16.342284 1958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:51:16.343760 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:51:16.345319 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:51:16.348901 7918 ts_tablet_manager.cc:522] Loading tablet metadata (0/8 complete)
I20240502 01:51:16.393038 1958 ts_tablet_manager.cc:559] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20240502 01:51:16.393342 1958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.045s user 0.000s sys 0.002s
I20240502 01:51:16.393637 1958 ts_tablet_manager.cc:574] Registering tablets (0/8 complete)
I20240502 01:51:16.398370 7918 tablet_bootstrap.cc:492] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:16.413920 7918 tablet_bootstrap.cc:492] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:16.414924 7918 tablet_bootstrap.cc:492] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap complete.
I20240502 01:51:16.415637 7918 ts_tablet_manager.cc:1367] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.006s
I20240502 01:51:16.418253 7918 raft_consensus.cc:348] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.418970 7918 raft_consensus.cc:729] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:16.419639 7918 consensus_queue.cc:260] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.422360 7918 ts_tablet_manager.cc:1398] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.006s user 0.004s sys 0.005s
I20240502 01:51:16.423173 7918 tablet_bootstrap.cc:492] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:16.438649 7918 tablet_bootstrap.cc:492] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:16.439656 7918 tablet_bootstrap.cc:492] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap complete.
I20240502 01:51:16.439743 1958 ts_tablet_manager.cc:590] Registered 8 tablets
I20240502 01:51:16.440061 1958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.046s user 0.043s sys 0.001s
I20240502 01:51:16.440450 7918 ts_tablet_manager.cc:1367] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.002s
I20240502 01:51:16.443239 7918 raft_consensus.cc:348] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.444041 7918 raft_consensus.cc:729] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:16.444837 7918 consensus_queue.cc:260] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.447105 7918 ts_tablet_manager.cc:1398] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240502 01:51:16.447973 7918 tablet_bootstrap.cc:492] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:16.453310 7609 consensus_queue.cc:237] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } }
I20240502 01:51:16.465934 7682 raft_consensus.cc:1261] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Refusing update from remote peer 56327090ee9245b8b42724ae0edcb365: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:51:16.466758 7918 tablet_bootstrap.cc:492] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:16.467567 7918 tablet_bootstrap.cc:492] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap complete.
I20240502 01:51:16.467667 7810 consensus_queue.cc:1035] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:51:16.468255 7918 ts_tablet_manager.cc:1367] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240502 01:51:16.470808 7918 raft_consensus.cc:348] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.471493 7918 raft_consensus.cc:729] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:16.472204 7918 consensus_queue.cc:260] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.474373 7918 ts_tablet_manager.cc:1398] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:51:16.475170 7918 tablet_bootstrap.cc:492] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
W20240502 01:51:16.483582 7575 consensus_peers.cc:485] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 -> Peer a3af9b0e088744239d1425da594190f2 (127.1.233.132:37253): Couldn't send request to peer a3af9b0e088744239d1425da594190f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a354a74fa444714abd15b3dc6c9b5ce. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:51:16.484102 7574 consensus_peers.cc:485] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 -> Peer 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): Couldn't send request to peer 7c6075468bc949f08b209b5aebe0c49a. Status: Network error: Client connection negotiation failed: client connection to 127.1.233.129:33345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:51:16.491926 7934 raft_consensus.cc:2932] T 7a354a74fa444714abd15b3dc6c9b5ce P 56327090ee9245b8b42724ae0edcb365 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } } }
I20240502 01:51:16.494787 7918 tablet_bootstrap.cc:492] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:16.493764 7682 raft_consensus.cc:2932] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } } }
I20240502 01:51:16.496187 7918 tablet_bootstrap.cc:492] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap complete.
I20240502 01:51:16.497107 7918 ts_tablet_manager.cc:1367] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.011s
I20240502 01:51:16.500311 7918 raft_consensus.cc:348] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.501209 7918 raft_consensus.cc:729] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:16.501971 7918 consensus_queue.cc:260] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.504761 7918 ts_tablet_manager.cc:1398] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 01:51:16.505860 7918 tablet_bootstrap.cc:492] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:16.507787 7417 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7a354a74fa444714abd15b3dc6c9b5ce with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20240502 01:51:16.510828 7432 catalog_manager.cc:5484] T 7a354a74fa444714abd15b3dc6c9b5ce P d3a96ac41ed64f778328e3e2fb6ebdfc reported cstate change: config changed from index -1 to 2, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) added. New cstate: current_term: 1 leader_uuid: "56327090ee9245b8b42724ae0edcb365" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } } }
I20240502 01:51:16.525766 7918 tablet_bootstrap.cc:492] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:16.526970 7918 tablet_bootstrap.cc:492] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap complete.
I20240502 01:51:16.527765 7918 ts_tablet_manager.cc:1367] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.004s
I20240502 01:51:16.530826 7918 raft_consensus.cc:348] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.531688 7918 raft_consensus.cc:729] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:16.532698 7918 consensus_queue.cc:260] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.535530 7918 ts_tablet_manager.cc:1398] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20240502 01:51:16.536515 7918 tablet_bootstrap.cc:492] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:16.552573 7918 tablet_bootstrap.cc:492] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:16.553325 7918 tablet_bootstrap.cc:492] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap complete.
I20240502 01:51:16.553857 7918 ts_tablet_manager.cc:1367] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20240502 01:51:16.555639 7918 raft_consensus.cc:348] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } }
I20240502 01:51:16.556166 7918 raft_consensus.cc:729] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:16.556757 7918 consensus_queue.cc:260] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } }
I20240502 01:51:16.558411 7918 ts_tablet_manager.cc:1398] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240502 01:51:16.559051 7918 tablet_bootstrap.cc:492] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:16.568982 7918 tablet_bootstrap.cc:492] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:16.569692 7918 tablet_bootstrap.cc:492] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap complete.
I20240502 01:51:16.570281 7918 ts_tablet_manager.cc:1367] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240502 01:51:16.571681 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.233.129:33345
I20240502 01:51:16.571916 7987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.233.129:33345 every 8 connection(s)
I20240502 01:51:16.572450 7918 raft_consensus.cc:348] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.573032 7918 raft_consensus.cc:729] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:16.573549 7918 consensus_queue.cc:260] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.575131 7918 ts_tablet_manager.cc:1398] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:51:16.575882 7918 tablet_bootstrap.cc:492] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap starting.
I20240502 01:51:16.588174 7918 tablet_bootstrap.cc:492] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:51:16.588933 7918 tablet_bootstrap.cc:492] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: Bootstrap complete.
I20240502 01:51:16.589381 7988 heartbeater.cc:347] Connected to a master server at 127.1.233.190:39913
I20240502 01:51:16.589540 7918 ts_tablet_manager.cc:1367] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20240502 01:51:16.589818 7988 heartbeater.cc:464] Registering TS with master...
I20240502 01:51:16.590643 7988 heartbeater.cc:511] Master 127.1.233.190:39913 requested a full tablet report, sending...
I20240502 01:51:16.591621 7918 raft_consensus.cc:348] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.592207 7918 raft_consensus.cc:729] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6075468bc949f08b209b5aebe0c49a, State: Initialized, Role: FOLLOWER
I20240502 01:51:16.592746 7918 consensus_queue.cc:260] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.594364 7918 ts_tablet_manager.cc:1398] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:51:16.595683 7429 ts_manager.cc:194] Re-registered known tserver with Master: 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345)
I20240502 01:51:16.602841 7429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39046
I20240502 01:51:16.605765 7988 heartbeater.cc:503] Master 127.1.233.190:39913 was elected leader, sending a full tablet report...
I20240502 01:51:16.637076 7951 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f3a4975c3b43598dde32d8368d2422 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:51:15 UTC) from {username='slave'} at 127.0.0.1:35144
I20240502 01:51:16.638957 7993 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:16.639670 7993 raft_consensus.cc:2229] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:16.640167 7993 raft_consensus.cc:2258] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:16.642036 7993 ts_tablet_manager.cc:1875] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:51:16.650550 7951 tablet_service.cc:1514] Processing DeleteTablet for tablet 878d3ccfdac840878a115793a8a913ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:51:15 UTC) from {username='slave'} at 127.0.0.1:35144
I20240502 01:51:16.654043 7993 ts_tablet_manager.cc:1888] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 01:51:16.654438 7993 log.cc:1196] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-0-root/wals/13f3a4975c3b43598dde32d8368d2422
I20240502 01:51:16.654994 7993 ts_tablet_manager.cc:1909] T 13f3a4975c3b43598dde32d8368d2422 P 7c6075468bc949f08b209b5aebe0c49a: Deleting consensus metadata
I20240502 01:51:16.657292 7993 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:16.657410 7418 catalog_manager.cc:4830] TS 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): tablet 13f3a4975c3b43598dde32d8368d2422 (table dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]) successfully deleted
I20240502 01:51:16.658154 7993 raft_consensus.cc:2229] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:16.658593 7993 raft_consensus.cc:2258] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:16.660778 7993 ts_tablet_manager.cc:1875] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:51:16.673242 7993 ts_tablet_manager.cc:1888] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 01:51:16.673548 7993 log.cc:1196] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-0-root/wals/878d3ccfdac840878a115793a8a913ae
I20240502 01:51:16.674132 7993 ts_tablet_manager.cc:1909] T 878d3ccfdac840878a115793a8a913ae P 7c6075468bc949f08b209b5aebe0c49a: Deleting consensus metadata
I20240502 01:51:16.676980 7418 catalog_manager.cc:4830] TS 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): tablet 878d3ccfdac840878a115793a8a913ae (table dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]) successfully deleted
I20240502 01:51:16.677423 7951 tablet_service.cc:1514] Processing DeleteTablet for tablet 638b7dba1ac5413d9bf634c58264e22f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:51:15 UTC) from {username='slave'} at 127.0.0.1:35144
I20240502 01:51:16.678092 7993 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:16.678692 7993 raft_consensus.cc:2229] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:16.679055 7993 raft_consensus.cc:2258] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:16.681341 7993 ts_tablet_manager.cc:1875] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:51:16.692178 7951 tablet_service.cc:1514] Processing DeleteTablet for tablet 352a62a7a2c1495387b5785b34c26cbc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:51:15 UTC) from {username='slave'} at 127.0.0.1:35144
I20240502 01:51:16.693146 7993 ts_tablet_manager.cc:1888] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 01:51:16.693497 7993 log.cc:1196] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-0-root/wals/638b7dba1ac5413d9bf634c58264e22f
I20240502 01:51:16.694105 7993 ts_tablet_manager.cc:1909] T 638b7dba1ac5413d9bf634c58264e22f P 7c6075468bc949f08b209b5aebe0c49a: Deleting consensus metadata
I20240502 01:51:16.696290 7993 tablet_replica.cc:330] stopping tablet replica
I20240502 01:51:16.696365 7418 catalog_manager.cc:4830] TS 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): tablet 638b7dba1ac5413d9bf634c58264e22f (table dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]) successfully deleted
I20240502 01:51:16.697150 7993 raft_consensus.cc:2229] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:51:16.697546 7993 raft_consensus.cc:2258] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:51:16.700042 7993 ts_tablet_manager.cc:1875] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:51:16.710700 7961 raft_consensus.cc:2932] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } } }
I20240502 01:51:16.715741 7993 ts_tablet_manager.cc:1888] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 01:51:16.716269 7993 log.cc:1196] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: Deleting WAL directory at /tmp/dist-test-taskoTulTI/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714614515033618-1958-0/minicluster-data/ts-0-root/wals/352a62a7a2c1495387b5785b34c26cbc
I20240502 01:51:16.716961 7993 ts_tablet_manager.cc:1909] T 352a62a7a2c1495387b5785b34c26cbc P 7c6075468bc949f08b209b5aebe0c49a: Deleting consensus metadata
I20240502 01:51:16.720228 7418 catalog_manager.cc:4830] TS 7c6075468bc949f08b209b5aebe0c49a (127.1.233.129:33345): tablet 352a62a7a2c1495387b5785b34c26cbc (table dugtrio [id=a8ae1f8fd72549ecb6294ecbca4ebfdf]) successfully deleted
I20240502 01:51:16.744050 7682 consensus_queue.cc:237] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } }
I20240502 01:51:16.750209 7609 raft_consensus.cc:1261] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Refusing update from remote peer d3a96ac41ed64f778328e3e2fb6ebdfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:51:16.750373 7961 raft_consensus.cc:1261] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Refusing update from remote peer d3a96ac41ed64f778328e3e2fb6ebdfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:51:16.750468 7649 consensus_peers.cc:485] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc -> Peer a3af9b0e088744239d1425da594190f2 (127.1.233.132:37253): Couldn't send request to peer a3af9b0e088744239d1425da594190f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e503a6bf258643ea8035d4b6f7131f83. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:51:16.751864 7748 consensus_queue.cc:1035] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:51:16.752768 7807 consensus_queue.cc:1035] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:51:16.760687 7748 raft_consensus.cc:2932] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } } }
I20240502 01:51:16.762622 7609 raft_consensus.cc:2932] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } } }
I20240502 01:51:16.766088 7960 raft_consensus.cc:2932] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } } }
I20240502 01:51:16.770984 7416 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e503a6bf258643ea8035d4b6f7131f83 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 7)
I20240502 01:51:16.777590 7429 catalog_manager.cc:5484] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365 reported cstate change: config changed from index -1 to 2, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) added. New cstate: current_term: 1 leader_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } } }
I20240502 01:51:16.779484 7999 ts_tablet_manager.cc:902] T fadba1fb5a5c41cc8904ab3920871362 P a3af9b0e088744239d1425da594190f2: Initiating tablet copy from peer d3a96ac41ed64f778328e3e2fb6ebdfc (127.1.233.131:42861)
F20240502 01:51:16.780511 7999 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0dae112f99f04e58ba8843992ad40465.queue_time_us, tablet-download-pool-0dae112f99f04e58ba8843992ad40465.queue_time_us) (tablet-download-pool-0dae112f99f04e58ba8843992ad40465.run_cpu_time_us, tablet-download-pool-0dae112f99f04e58ba8843992ad40465.run_cpu_time_us) (tablet-download-pool-0dae112f99f04e58ba8843992ad40465.run_wall_time_us, tablet-download-pool-0dae112f99f04e58ba8843992ad40465.run_wall_time_us) (tablet-download-pool-299f63f8961a4079ac1101429673be59.queue_time_us, tablet-download-pool-299f63f8961a4079ac1101429673be59.queue_time_us) (tablet-download-pool-299f63f8961a4079ac1101429673be59.run_cpu_time_us, tablet-download-pool-299f63f8961a4079ac1101429673be59.run_cpu_time_us) (tablet-download-pool-299f63f8961a4079ac1101429673be59.run_wall_time_us, tablet-download-pool-299f63f8961a4079ac1101429673be59.run_wall_time_us) (tablet-download-pool-39ca1938892648e1a4658b541a260c6d.queue_time_us, tablet-download-pool-39ca1938892648e1a4658b541a260c6d.queue_time_us) (tablet-download-pool-39ca1938892648e1a4658b541a260c6d.run_cpu_time_us, tablet-download-pool-39ca1938892648e1a4658b541a260c6d.run_cpu_time_us) (tablet-download-pool-39ca1938892648e1a4658b541a260c6d.run_wall_time_us, tablet-download-pool-39ca1938892648e1a4658b541a260c6d.run_wall_time_us) (tablet-download-pool-47805d1b4cc44fcca988db5eba2e05c8.queue_time_us, tablet-download-pool-47805d1b4cc44fcca988db5eba2e05c8.queue_time_us) (tablet-download-pool-47805d1b4cc44fcca988db5eba2e05c8.run_cpu_time_us, tablet-download-pool-47805d1b4cc44fcca988db5eba2e05c8.run_cpu_time_us) (tablet-download-pool-47805d1b4cc44fcca988db5eba2e05c8.run_wall_time_us, tablet-download-pool-47805d1b4cc44fcca988db5eba2e05c8.run_wall_time_us) (tablet-download-pool-57e780227c194beda6e8fde8df155c6a.queue_time_us, tablet-download-pool-57e780227c194beda6e8fde8df155c6a.queue_time_us) (tablet-download-pool-57e780227c194beda6e8fde8df155c6a.run_cpu_time_us, tablet-download-pool-57e780227c194beda6e8fde8df155c6a.run_cpu_time_us) (tablet-download-pool-57e780227c194beda6e8fde8df155c6a.run_wall_time_us, tablet-download-pool-57e780227c194beda6e8fde8df155c6a.run_wall_time_us) (tablet-download-pool-5cada54fde9f4c0498067e9a5ce6962c.queue_time_us, tablet-download-pool-5cada54fde9f4c0498067e9a5ce6962c.queue_time_us) (tablet-download-pool-5cada54fde9f4c0498067e9a5ce6962c.run_cpu_time_us, tablet-download-pool-5cada54fde9f4c0498067e9a5ce6962c.run_cpu_time_us) (tablet-download-pool-5cada54fde9f4c0498067e9a5ce6962c.run_wall_time_us, tablet-download-pool-5cada54fde9f4c0498067e9a5ce6962c.run_wall_time_us) (tablet-download-pool-9119e2d663834f5182f41406f021b349.queue_time_us, tablet-download-pool-9119e2d663834f5182f41406f021b349.queue_time_us) (tablet-download-pool-9119e2d663834f5182f41406f021b349.run_cpu_time_us, tablet-download-pool-9119e2d663834f5182f41406f021b349.run_cpu_time_us) (tablet-download-pool-9119e2d663834f5182f41406f021b349.run_wall_time_us, tablet-download-pool-9119e2d663834f5182f41406f021b349.run_wall_time_us) (tablet-download-pool-9d43ecf839ef46eaa968c5c319b22e3c.queue_time_us, tablet-download-pool-9d43ecf839ef46eaa968c5c319b22e3c.queue_time_us) (tablet-download-pool-9d43ecf839ef46eaa968c5c319b22e3c.run_cpu_time_us, tablet-download-pool-9d43ecf839ef46eaa968c5c319b22e3c.run_cpu_time_us) (tablet-download-pool-9d43ecf839ef46eaa968c5c319b22e3c.run_wall_time_us, tablet-download-pool-9d43ecf839ef46eaa968c5c319b22e3c.run_wall_time_us) (tablet-download-pool-b0a4ee1f9ba9484896bb256105b96565.queue_time_us, tablet-download-pool-b0a4ee1f9ba9484896bb256105b96565.queue_time_us) (tablet-download-pool-b0a4ee1f9ba9484896bb256105b96565.run_cpu_time_us, tablet-download-pool-b0a4ee1f9ba9484896bb256105b96565.run_cpu_time_us) (tablet-download-pool-b0a4ee1f9ba9484896bb256105b96565.run_wall_time_us, tablet-download-pool-b0a4ee1f9ba9484896bb256105b96565.run_wall_time_us) (tablet-download-pool-c54b599ba1ee4033b73975c3077abd6f.queue_time_us, tablet-download-pool-c54b599ba1ee4033b73975c3077abd6f.queue_time_us) (tablet-download-pool-c54b599ba1ee4033b73975c3077abd6f.run_cpu_time_us, tablet-download-pool-c54b599ba1ee4033b73975c3077abd6f.run_cpu_time_us) (tablet-download-pool-c54b599ba1ee4033b73975c3077abd6f.run_wall_time_us, tablet-download-pool-c54b599ba1ee4033b73975c3077abd6f.run_wall_time_us) (tablet-download-pool-d19da3a831744a649c477c48f46832ff.queue_time_us, tablet-download-pool-d19da3a831744a649c477c48f46832ff.queue_time_us) (tablet-download-pool-d19da3a831744a649c477c48f46832ff.run_cpu_time_us, tablet-download-pool-d19da3a831744a649c477c48f46832ff.run_cpu_time_us) (tablet-download-pool-d19da3a831744a649c477c48f46832ff.run_wall_time_us, tablet-download-pool-d19da3a831744a649c477c48f46832ff.run_wall_time_us) (tablet-download-pool-da800cb86fb24800b59374018d0ce197.queue_time_us, tablet-download-pool-da800cb86fb24800b59374018d0ce197.queue_time_us) (tablet-download-pool-da800cb86fb24800b59374018d0ce197.run_cpu_time_us, tablet-download-pool-da800cb86fb24800b59374018d0ce197.run_cpu_time_us) (tablet-download-pool-da800cb86fb24800b59374018d0ce197.run_wall_time_us, tablet-download-pool-da800cb86fb24800b59374018d0ce197.run_wall_time_us) (tablet-download-pool-fadba1fb5a5c41cc8904ab3920871362.queue_time_us, tablet-download-pool-fadba1fb5a5c41cc8904ab3920871362.queue_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714614676 (unix time) try "date -d @1714614676" if you are using GNU date ***
I20240502 01:51:16.793079 7682 consensus_queue.cc:237] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
I20240502 01:51:16.798467 7960 raft_consensus.cc:1261] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Refusing update from remote peer d3a96ac41ed64f778328e3e2fb6ebdfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:51:16.799059 7609 raft_consensus.cc:1261] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Refusing update from remote peer d3a96ac41ed64f778328e3e2fb6ebdfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:51:16.800711 7786 consensus_queue.cc:1035] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:51:16.801514 7807 consensus_queue.cc:1035] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:51:16.807700 7807 raft_consensus.cc:2932] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } }
I20240502 01:51:16.812969 7960 raft_consensus.cc:2932] T e503a6bf258643ea8035d4b6f7131f83 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } }
I20240502 01:51:16.812855 7609 raft_consensus.cc:2932] T e503a6bf258643ea8035d4b6f7131f83 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } }
I20240502 01:51:16.819396 7416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e503a6bf258643ea8035d4b6f7131f83 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
PC: @ 0x0 (unknown)
I20240502 01:51:16.823979 7961 raft_consensus.cc:3037] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Advancing to term 2
*** SIGABRT (@0x3e8000007a6) received by PID 1958 (TID 0x7f4db2cfd700) from PID 1958; stack trace: ***
@ 0x2e41a0 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e6683 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20240502 01:51:16.825090 7432 catalog_manager.cc:5484] T e503a6bf258643ea8035d4b6f7131f83 P d3a96ac41ed64f778328e3e2fb6ebdfc reported cstate change: config changed from index 2 to 3, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) evicted. New cstate: current_term: 1 leader_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } health_report { overall_health: HEALTHY } } }
I20240502 01:51:16.852598 7860 tablet_service.cc:1514] Processing DeleteTablet for tablet e503a6bf258643ea8035d4b6f7131f83 with delete_type TABLET_DATA_TOMBSTONED (TS a3af9b0e088744239d1425da594190f2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:42816
@ 0x7f4dec607980 (unknown) at ??:0
I20240502 01:51:16.856688 7961 raft_consensus.cc:1261] T d945ad77c1284b33a14e47ddda96f868 P 7c6075468bc949f08b209b5aebe0c49a [term 2 FOLLOWER]: Refusing update from remote peer d3a96ac41ed64f778328e3e2fb6ebdfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:51:16.858479 7748 consensus_queue.cc:1035] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:51:16.859683 7419 catalog_manager.cc:4797] TS a3af9b0e088744239d1425da594190f2 (127.1.233.132:37253): delete failed for tablet e503a6bf258643ea8035d4b6f7131f83 because the tablet was not found. No further retry: Not found: Tablet not found: e503a6bf258643ea8035d4b6f7131f83
I20240502 01:51:16.866256 7609 raft_consensus.cc:1261] T d945ad77c1284b33a14e47ddda96f868 P 56327090ee9245b8b42724ae0edcb365 [term 2 FOLLOWER]: Refusing update from remote peer d3a96ac41ed64f778328e3e2fb6ebdfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:51:16.867708 7748 consensus_queue.cc:1035] T d945ad77c1284b33a14e47ddda96f868 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
@ 0x7f4de5b7bfb7 gsignal at ??:0
@ 0x7f4de5b7d921 abort at ??:0
@ 0x2ea196 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f4de8971da4 google::LogMessage::SendToLog() at ??:0
@ 0x7f4de8972950 google::LogMessage::Flush() at ??:0
@ 0x7f4de8977a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4de92ff3c7 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f4de92d594f kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f4de92d30a5 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 01:51:17.048499 7961 raft_consensus.cc:2932] T 7a354a74fa444714abd15b3dc6c9b5ce P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "a3af9b0e088744239d1425da594190f2" member_type: NON_VOTER last_known_addr { host: "127.1.233.132" port: 37253 } attrs { promote: true } } }
@ 0x7f4df10f3422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 01:51:17.072242 8021 ts_tablet_manager.cc:902] T 7a354a74fa444714abd15b3dc6c9b5ce P a3af9b0e088744239d1425da594190f2: Initiating tablet copy from peer 56327090ee9245b8b42724ae0edcb365 (127.1.233.130:41677)
@ 0x7f4df10f9c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f4df118f4ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f4df11c03cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 01:51:17.143193 7682 consensus_queue.cc:237] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } }
@ 0x7f4df11a61ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240502 01:51:17.148417 7609 raft_consensus.cc:1261] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Refusing update from remote peer d3a96ac41ed64f778328e3e2fb6ebdfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:51:17.150220 7961 raft_consensus.cc:1261] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Refusing update from remote peer d3a96ac41ed64f778328e3e2fb6ebdfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:51:17.150180 7807 consensus_queue.cc:1035] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:51:17.151865 7748 consensus_queue.cc:1035] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:51:17.156916 7807 raft_consensus.cc:2932] T fadba1fb5a5c41cc8904ab3920871362 P d3a96ac41ed64f778328e3e2fb6ebdfc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } }
I20240502 01:51:17.160923 7961 raft_consensus.cc:2932] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } }
I20240502 01:51:17.163865 7609 raft_consensus.cc:2932] T fadba1fb5a5c41cc8904ab3920871362 P 56327090ee9245b8b42724ae0edcb365 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } }
@ 0x7f4df11a617a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
I20240502 01:51:17.176690 7431 catalog_manager.cc:5484] T fadba1fb5a5c41cc8904ab3920871362 P 7c6075468bc949f08b209b5aebe0c49a reported cstate change: config changed from index 2 to 3, NON_VOTER a3af9b0e088744239d1425da594190f2 (127.1.233.132) evicted. New cstate: current_term: 1 leader_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7c6075468bc949f08b209b5aebe0c49a" member_type: VOTER last_known_addr { host: "127.1.233.129" port: 33345 } } peers { permanent_uuid: "56327090ee9245b8b42724ae0edcb365" member_type: VOTER last_known_addr { host: "127.1.233.130" port: 41677 } } peers { permanent_uuid: "d3a96ac41ed64f778328e3e2fb6ebdfc" member_type: VOTER last_known_addr { host: "127.1.233.131" port: 42861 } } }
I20240502 01:51:17.180100 7416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fadba1fb5a5c41cc8904ab3920871362 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:51:17.190829 7860 tablet_service.cc:1514] Processing DeleteTablet for tablet fadba1fb5a5c41cc8904ab3920871362 with delete_type TABLET_DATA_TOMBSTONED (TS a3af9b0e088744239d1425da594190f2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:42816
W20240502 01:51:17.192504 7419 catalog_manager.cc:4797] TS a3af9b0e088744239d1425da594190f2 (127.1.233.132:37253): delete failed for tablet fadba1fb5a5c41cc8904ab3920871362 because the tablet was not found. No further retry: Not found: Tablet not found: fadba1fb5a5c41cc8904ab3920871362
@ 0x7f4df11a610a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f4df11a60d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f4df11a52fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x7f4df254bda5 std::__1::__function::__value_func<>::operator()() at ??:0
I20240502 01:51:17.259783 7895 heartbeater.cc:503] Master 127.1.233.190:39913 was elected leader, sending a full tablet report...
@ 0x7f4df254bbda std::__1::function<>::operator()() at ??:0
W20240502 01:51:17.282325 7480 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table 309a236d6e9541abba4b7f6756db454b: could not find any suitable replica to move from server d3a96ac41ed64f778328e3e2fb6ebdfc to server a3af9b0e088744239d1425da594190f2
@ 0x7f4de92d7237 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4de92da692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4de92da64a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7f4de92da5da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f4de92da5a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f4de92d989e std::__1::__function::__func<>::operator()() at ??:0
@ 0x7f4df254bda5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7f4df254bbda std::__1::function<>::operator()() at ??:0
@ 0x7f4de92b9937 kudu::Thread::SuperviseThread() at ??:0