Do not fail evacuation in case of REP 1 objects #1222

Merged

dstepanov-yadro merged 1 commit from dstepanov-yadro/frostfs-node:fix/evacuate_rep1 into master

2024-09-04 19:51:09 +00:00

dstepanov-yadro commented

2024-07-03 07:00:01 +00:00

Member

REP 1 SELECT 1 and REP 2 objects are stored on node.

Before fix:

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation start --all --endpoint 127.0.0.1:8081 --wallet ./dev/storage/wallet01.js
on 
Enter password > 
Shard evacuation has been successfully started.

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation status --endpoint 127.0.0.1:8081 --wallet ./dev/storage/wallet01.json 
Enter password > 
Shard IDs: 6JBPcSH6HkvhuvHdNyp1YJ, BCgPcpHGvoq8eTsRK4W2YB. Status: completed. Evacuated 0 objects out of 6542, failed to evacuate: 0, skipped: 0; evacuated 0 trees out of 0, failed to evacuate: 0. Error: object was not replicated. Started at: 2024-07-03T06:21:50Z UTC. Duration: 00:00:00.

After fix:

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation start --all --endpoint 127.0.0.1:8081 --wallet ./dev/storage/wallet01.json 
Enter password > 
Shard evacuation has been successfully started

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation status --endpoint 127.0.0.1:8081 --wallet ./dev/storage/wallet01.json 
Enter password > 
Shard IDs: 82XzYmnrrYA7DunudqCooL, YW1gXt7Rd1ArX9rfFwWBXG. Status: running. Evacuated 973 objects out of 5703, failed to evacuate: 1016, skipped: 0; evacuated 0 trees out of 0, failed to evacuate: 0. Started at: 2024-07-03T06:35:16Z UTC. Duration: 00:01:35. Estimated time left: 2 minutes.

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation status --endpoint 127.0.0.1:8081 --wallet ./dev/storage/wallet01.json 
Enter password > 
Shard IDs: 82XzYmnrrYA7DunudqCooL, YW1gXt7Rd1ArX9rfFwWBXG. Status: completed. Evacuated 2709 objects out of 5703, failed to evacuate: 2994, skipped: 0; evacuated 0 trees out of 0, failed to evacuate: 0. Started at: 2024-07-03T06:35:16Z UTC. Duration: 00:04:34.

The same for trees

Before:

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation start --scope trees --all --endpoint s01.frostfs.devenv:8081 --wallet ./
../frostfs-dev-env/services/storage/wallet01.json 
Enter password > 
Shard evacuation has been successfully started.

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/
services/storage/wallet01.json 
Enter password > 
Shard IDs: GptVR14xFVXCkiUDBikqr, ADGD3at8Nh17H7YDb9JqsS. Status: completed. Evacuated 0 objects out of 0, failed to evacuate: 0, skipped: 0; evacuated 2 trees out of 111, failed to evacuate: 0. Error: no remote nodes available to replicate tree 'version' of container SWQ3e4EAP791haijHd4MMxrWjCqgPHyMZi7uKwxVNZN. Started at: 2024-07-03T08:24:57Z UTC. Duration: 00:00:00.

After:

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation start --scope trees --all --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json 
Enter password > 
Shard evacuation has been successfully started.

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json 
Enter password > 
Shard IDs: DZ3vavsKXVM1DmDGSEvBKd, 82SsxZq3AS5RuGkPw6Ur9S. Status: completed. Evacuated 0 objects out of 0, failed to evacuate: 0, skipped: 0; evacuated 80 trees out of 97, failed to evacuate: 17. Started at: 2024-07-03T08:46:23Z UTC. Duration: 00:00:22

`REP 1 SELECT 1` and `REP 2` objects are stored on node. Before fix: ``` dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation start --all --endpoint 127.0.0.1:8081 --wallet ./dev/storage/wallet01.js on Enter password > Shard evacuation has been successfully started. dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation status --endpoint 127.0.0.1:8081 --wallet ./dev/storage/wallet01.json Enter password > Shard IDs: 6JBPcSH6HkvhuvHdNyp1YJ, BCgPcpHGvoq8eTsRK4W2YB. Status: completed. Evacuated 0 objects out of 6542, failed to evacuate: 0, skipped: 0; evacuated 0 trees out of 0, failed to evacuate: 0. Error: object was not replicated. Started at: 2024-07-03T06:21:50Z UTC. Duration: 00:00:00. ``` After fix: ``` dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation start --all --endpoint 127.0.0.1:8081 --wallet ./dev/storage/wallet01.json Enter password > Shard evacuation has been successfully started dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation status --endpoint 127.0.0.1:8081 --wallet ./dev/storage/wallet01.json Enter password > Shard IDs: 82XzYmnrrYA7DunudqCooL, YW1gXt7Rd1ArX9rfFwWBXG. Status: running. Evacuated 973 objects out of 5703, failed to evacuate: 1016, skipped: 0; evacuated 0 trees out of 0, failed to evacuate: 0. Started at: 2024-07-03T06:35:16Z UTC. Duration: 00:01:35. Estimated time left: 2 minutes. dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation status --endpoint 127.0.0.1:8081 --wallet ./dev/storage/wallet01.json Enter password > Shard IDs: 82XzYmnrrYA7DunudqCooL, YW1gXt7Rd1ArX9rfFwWBXG. Status: completed. Evacuated 2709 objects out of 5703, failed to evacuate: 2994, skipped: 0; evacuated 0 trees out of 0, failed to evacuate: 0. Started at: 2024-07-03T06:35:16Z UTC. Duration: 00:04:34. ``` The same for trees Before: ``` dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation start --scope trees --all --endpoint s01.frostfs.devenv:8081 --wallet ./ ../frostfs-dev-env/services/storage/wallet01.json Enter password > Shard evacuation has been successfully started. dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/ services/storage/wallet01.json Enter password > Shard IDs: GptVR14xFVXCkiUDBikqr, ADGD3at8Nh17H7YDb9JqsS. Status: completed. Evacuated 0 objects out of 0, failed to evacuate: 0, skipped: 0; evacuated 2 trees out of 111, failed to evacuate: 0. Error: no remote nodes available to replicate tree 'version' of container SWQ3e4EAP791haijHd4MMxrWjCqgPHyMZi7uKwxVNZN. Started at: 2024-07-03T08:24:57Z UTC. Duration: 00:00:00. ``` After: ``` dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation start --scope trees --all --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json Enter password > Shard evacuation has been successfully started. dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli control shards evacuation status --endpoint s01.frostfs.devenv:8081 --wallet ./../frostfs-dev-env/services/storage/wallet01.json Enter password > Shard IDs: DZ3vavsKXVM1DmDGSEvBKd, 82SsxZq3AS5RuGkPw6Ur9S. Status: completed. Evacuated 0 objects out of 0, failed to evacuate: 0, skipped: 0; evacuated 80 trees out of 97, failed to evacuate: 17. Started at: 2024-07-03T08:46:23Z UTC. Duration: 00:00:22 ```

dstepanov-yadro changed title from ~~Do not fail evacuation in case of REP 1 objects~~ to WIP: Do not fail evacuation in case of REP 1 objects

2024-07-03 07:00:34 +00:00

dstepanov-yadro force-pushed fix/evacuate_rep1 from aedce44d5c to a6efa4ebcf

2024-07-03 07:02:03 +00:00

Compare

dstepanov-yadro force-pushed fix/evacuate_rep1 from a6efa4ebcf to 28f0a67604

2024-07-03 08:50:32 +00:00

Compare

dstepanov-yadro changed title from ~~WIP: Do not fail evacuation in case of REP 1 objects~~ to Do not fail evacuation in case of REP 1 objects

2024-07-03 08:52:32 +00:00

dstepanov-yadro requested review from storage-core-committers 2024-07-03 11:04:59 +00:00

dstepanov-yadro requested review from storage-core-developers 2024-07-03 11:05:02 +00:00