From 363105c8d3616951350f699b291ab78fd67ba419 Mon Sep 17 00:00:00 2001 From: Dmitrii Stepanov Date: Wed, 10 May 2023 14:03:49 +0300 Subject: [PATCH] [#329] docs: Add shard evacuation description Signed-off-by: Dmitrii Stepanov --- docs/evacuation.md | 92 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 docs/evacuation.md diff --git a/docs/evacuation.md b/docs/evacuation.md new file mode 100644 index 000000000..7c8beb790 --- /dev/null +++ b/docs/evacuation.md @@ -0,0 +1,92 @@ +# Shard data evacuation + +## Overview + +Evacuation is the process of transferring data from one shard to another. Evacuation is used in case of problems with the shard in order to save data. + +To start the evacuation, it is necessary that the shard is in read-only mode (read more [here](./shard-modes.md)). + +First of all, by the evacuation the data is transferred to other shards of the same node; if it is not possible, then the data is transferred to other nodes. + +Only one running evacuation process is allowed on the node at a time. + +`frostfs-cli` utility is used to manage evacuation. + +## Commands + +`frostfs-cli control shards evacuation start` starts evacuation process for shards specified. To start evacuating all node shards, use the `--all` flag. + +`frostfs-cli control shards evacuation stop` stops running evacuation process. + +`frostfs-cli control shards evacuation status` prints evacuation process status. + +See commands `--help` output for detailed description. + +## Examples + +### Set shard mode to read only +```bash +frostfs-cli control shards set-mode --mode read-only --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json --id 8kEBwtvKLU3Hva3PaaodUi +Enter password > +Shard mode update request successfully sent. +``` + +### Start evacuation and get status +```bash +frostfs-cli control shards evacuation start --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json --id 8kEBwtvKLU3Hva3PaaodUi +Enter password > +Shard evacuation has been successfully started. + +frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json +Enter password > +Shard IDs: 8kEBwtvKLU3Hva3PaaodUi. Status: running. Evacuated 14 object out of 61, failed to evacuate 0 objects. Started at: 2023-05-10T10:13:06Z UTC. Duration: 00:00:03. + +frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json +Enter password > +Shard IDs: 8kEBwtvKLU3Hva3PaaodUi. Status: running. Evacuated 23 object out of 61, failed to evacuate 0 objects. Started at: 2023-05-10T10:13:06Z UTC. Duration: 00:00:05. + +frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json +Enter password > +Shard IDs: 8kEBwtvKLU3Hva3PaaodUi. Status: completed. Evacuated 61 object out of 61, failed to evacuate 0 objects. Started at: 2023-05-10T10:13:06Z UTC. Duration: 00:00:13. +``` + +### Stop running evacuation process +```bash +frostfs-cli control shards evacuation start --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json --id 54Y8aot9uc7BSadw2XtYr3 +Enter password > +Shard evacuation has been successfully started. + +frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json +Enter password > +Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Status: running. Evacuated 15 object out of 73, failed to evacuate 0 objects. Started at: 2023-05-10T10:15:47Z UTC. Duration: 00:00:03. + +frostfs-cli control shards evacuation stop --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json +Enter password > +Evacuation stopped. + +frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json +Enter password > +Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Status: completed. Evacuated 31 object out of 73, failed to evacuate 0 objects. Error: context canceled. Started at: 2023-05-10T10:15:47Z UTC. Duration: 00:00:07. +``` + +### Start evacuation and await it completes +```bash +frostfs-cli control shards evacuation start --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json --id 54Y8aot9uc7BSadw2XtYr3 --await +Enter password > +Shard evacuation has been successfully started. +Progress will be reported every 5 seconds. +Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Status: running. Evacuated 18 object out of 73, failed to evacuate 0 objects. Started at: 2023-05-10T10:18:42Z UTC. Duration: 00:00:04. +Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Status: running. Evacuated 43 object out of 73, failed to evacuate 0 objects. Started at: 2023-05-10T10:18:42Z UTC. Duration: 00:00:09. +Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Status: running. Evacuated 68 object out of 73, failed to evacuate 0 objects. Started at: 2023-05-10T10:18:42Z UTC. Duration: 00:00:14. +Shard evacuation has been completed. +Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Evacuated 73 object out of 73, failed to evacuate 0 objects. Started at: 2023-05-10T10:18:42Z UTC. Duration: 00:00:14. +``` + +### Start evacuation and await it completes without progress notifications +```bash +frostfs-cli control shards evacuation start --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json --id 54Y8aot9uc7BSadw2XtYr3 --await --no-progress +Enter password > +Shard evacuation has been successfully started. +Shard evacuation has been completed. +Shard IDs: 54Y8aot9uc7BSadw2XtYr3. Evacuated 73 object out of 73, failed to evacuate 0 objects. Started at: 2023-05-10T10:20:00Z UTC. Duration: 00:00:14. +``` \ No newline at end of file