forked from TrueCloudLab/frostfs-node
115 lines
7.4 KiB
Markdown
115 lines
7.4 KiB
Markdown
# Shard data evacuation
|
|
|
|
## Overview
|
|
|
|
Evacuation is the process of transferring data from one shard to another. Evacuation is used in case of problems with the shard in order to save data.
|
|
|
|
To start the evacuation, it is necessary that the shard is in read-only mode (read more [here](./shard-modes.md)).
|
|
|
|
First of all, by the evacuation the data is transferred to other shards of the same node; if it is not possible, then the data is transferred to other nodes.
|
|
|
|
Only one running evacuation process is allowed on the node at a time.
|
|
|
|
It is not necessary to turn maintenance mode on storage node.
|
|
|
|
Once evacuation from shard started, it is impossible to read data from it via public API, except the case when evacuation stopped manually or node restarted.
|
|
|
|
Because it is necessary to prevent removing by policer objects with policy `REP 1 ...` from remote node during evacuation.
|
|
|
|
`frostfs-cli` utility is used to manage evacuation.
|
|
|
|
## Commands
|
|
|
|
`frostfs-cli control shards evacuation start` starts evacuation process for shards specified. To start evacuating all node shards, use the `--all` flag.
|
|
By default, objects and trees are evacuated. To limit the evacuation scope, use `--scope` flag (possible values are `all`, `trees`, `objects`).
|
|
To evacuate objects only from containers with policy `REP 1` use option `--rep-one-only`.
|
|
To adjust resource consumption required for evacuation use options:
|
|
- `--container-worker-count` count of concurrent container evacuation workers
|
|
- `--object-worker-count` count of concurrent object evacuation workers
|
|
|
|
`frostfs-cli control shards evacuation stop` stops running evacuation process.
|
|
|
|
`frostfs-cli control shards evacuation status` prints evacuation process status.
|
|
|
|
`frostfs-cli control shards evacuation reset` resets evacuation process status.
|
|
|
|
See commands `--help` output for detailed description.
|
|
|
|
## Examples
|
|
|
|
### Set shard mode to read only
|
|
```bash
|
|
frostfs-cli control shards set-mode --mode read-only --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json --id 8kEBwtvKLU3Hva3PaaodUi
|
|
Enter password >
|
|
Shard mode update request successfully sent.
|
|
```
|
|
|
|
### Start evacuation and get status
|
|
```bash
|
|
frostfs-cli control shards evacuation start --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json --id 8kEBwtvKLU3Hva3PaaodUi
|
|
Enter password >
|
|
Shard evacuation has been successfully started.
|
|
|
|
frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json
|
|
Enter password >
|
|
Shard IDs: 8kEBwtvKLU3Hva3PaaodUi. Status: running. Evacuated 131 objects out of 618, failed to evacuate: 0, skipped: 0; evacuated 0 trees out of 19, failed to evacuate: 0. Started at: 2023-05-10T10:13:06Z UTC. Duration: 00:00:03. Estimated time left: 2 minutes.
|
|
|
|
frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json
|
|
Enter password >
|
|
Shard IDs: 8kEBwtvKLU3Hva3PaaodUi. Status: running. Evacuated 260 objects out of 618, failed to evacuate: 0, skipped: 0; evacuated 0 trees out of 19, failed to evacuate: 0. Started at: 2023-05-10T10:13:06Z UTC. Duration: 00:01:05. Estimated time left: 1 minutes.
|
|
|
|
frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json
|
|
Enter password >
|
|
Shard IDs: 8kEBwtvKLU3Hva3PaaodUi. Status: completed. Evacuated 618 objects out of 618, failed to evacuate: 0, skipped: 0; evacuated 19 trees out of 19, failed to evacuate: 0. Started at: 2023-05-10T10:13:06Z UTC. Duration: 00:02:13.
|
|
```
|
|
|
|
### Stop running evacuation process
|
|
```bash
|
|
frostfs-cli control shards evacuation start --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json --id 54Y8aot9uc7BSadw2XtYr3
|
|
Enter password >
|
|
Shard evacuation has been successfully started.
|
|
|
|
frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json
|
|
Enter password >
|
|
Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Status: running. Evacuated 131 objects out of 618, failed to evacuate: 0, skipped: 0; evacuated 0 trees out of 19, failed to evacuate: 0. Started at: 2023-05-10T10:15:47Z UTC. Duration: 00:00:03. Estimated time left: 0 minutes.
|
|
|
|
frostfs-cli control shards evacuation stop --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json
|
|
Enter password >
|
|
Evacuation stopped.
|
|
|
|
frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json
|
|
Enter password >
|
|
Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Status: completed. Evacuated 131 objects out of 618, failed to evacuate: 0, skipped: 0; evacuated 0 trees out of 19, failed to evacuate: 0. Error: context canceled. Started at: 2023-05-10T10:15:47Z UTC. Duration: 00:00:07.
|
|
```
|
|
|
|
### Start evacuation and await it completes
|
|
```bash
|
|
frostfs-cli control shards evacuation start --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json --id 54Y8aot9uc7BSadw2XtYr3 --await
|
|
Enter password >
|
|
Shard evacuation has been successfully started.
|
|
Progress will be reported every 5 seconds.
|
|
Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Status: running. Evacuated 131 objects out of 618, failed to evacuate: 0, skipped: 0; evacuated 0 trees out of 19, failed to evacuate: 0. Started at: 2023-05-10T10:18:42Z UTC. Duration: 00:00:04. Estimated time left: 0 minutes.
|
|
Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Status: running. Evacuated 343 objects out of 618, failed to evacuate: 0, skipped: 0; evacuated 0 trees out of 19, failed to evacuate: 0. Started at: 2023-05-10T10:18:42Z UTC. Duration: 00:00:09. Estimated time left: 0 minutes.
|
|
Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Status: running. Evacuated 545 objects out of 618, failed to evacuate: 0, skipped: 0; evacuated 0 trees out of 19, failed to evacuate: 0. Started at: 2023-05-10T10:18:42Z UTC. Duration: 00:00:14. Estimated time left: 0 minutes.
|
|
Shard evacuation has been completed.
|
|
Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Evacuated 618 objects out of 618, failed to evacuate: 0, skipped: 0; evacuated 19 trees out of 19, failed to evacuate: 0. Started at: 2023-05-10T10:18:42Z UTC. Duration: 00:00:14.
|
|
```
|
|
|
|
### Start evacuation and await it completes without progress notifications
|
|
```bash
|
|
frostfs-cli control shards evacuation start --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json --id 54Y8aot9uc7BSadw2XtYr3 --await --no-progress
|
|
Enter password >
|
|
Shard evacuation has been successfully started.
|
|
Shard evacuation has been completed.
|
|
Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Evacuated 618 objects out of 618, failed to evacuate: 0, skipped: 0; evacuated 19 trees out of 19, failed to evacuate: 0. Started at: 2023-05-10T10:20:00Z UTC. Duration: 00:00:14.
|
|
```
|
|
|
|
### Start trees evacuation and await it completes
|
|
```bash
|
|
frostfs-cli control shards evacuation start --id FxR6QujButNCHn7jjdhxGP --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json --await --scope trees
|
|
Enter password >
|
|
Shard evacuation has been successfully started.
|
|
Progress will be reported every 5 seconds.
|
|
Shard evacuation has been completed.
|
|
Shard IDs: FxR6QujButNCHn7jjdhxGP. Evacuated 0 objects out of 0, failed to evacuate: 0, skipped: 0; evacuated 2 trees out of 2, failed to evacuate: 0. Started at: 2024-02-08T08:44:17Z UTC. Duration: 00:00:00.
|
|
```
|