Allow to migrate objects between storages #661

New issue

Closed

opened 2023-08-29 13:15:48 +00:00 by fyrchik · 1 comment

fyrchik commented

2023-08-29 13:15:48 +00:00

Owner

pkg/local_object_storage should allow to copy items from one storage to another. Usecase: reconfigure width+depth for blobovnicza.

A separate command (or a frostfs-node/frostfs-adm subcommand) can take 2 storage node configs and use the routine to reconfigure everything according to new parameters.
Another (may be easier and better) option is to accept type and parameters specifically for the blobstor substorage -- this will allow to (1) not touch everything else and (2) makes it more straightforward to copy from /path/a to /path/b. For each type1, type2 pair optimized migration could be available, for now we are only interested in width+depth for blobovniczas.

We may assume that there is enough space to have both old an new versions (makes idempotence easier), later can fix this.
Probably, creating new metabase for this shard is an option (or restarting with resync_metabase: true after the migration).

`pkg/local_object_storage` should allow to copy items from one storage to another. Usecase: reconfigure width+depth for blobovnicza. A separate command (or a `frostfs-node`/`frostfs-adm` subcommand) can take 2 storage node configs and use the routine to reconfigure everything according to new parameters. Another (may be easier and better) option is to accept type and parameters specifically for the blobstor substorage -- this will allow to (1) not touch everything else and (2) makes it more straightforward to copy from /path/a to /path/b. For each `type1, type2` pair optimized migration could be available, for now we are only interested in `width+depth` for blobovniczas. We may assume that there is enough space to have both old an new versions (makes idempotence easier), later can fix this. Probably, creating new metabase for this shard is an option (or restarting with `resync_metabase: true` after the migration).

fyrchik added this to the v0.37.0 milestone 2023-08-29 13:15:48 +00:00

fyrchik added the

frostfs-node

triage

labels 2023-08-29 13:15:48 +00:00

dstepanov-yadro was assigned by fyrchik

2023-08-29 13:15:55 +00:00

dstepanov-yadro referenced this issue

2023-09-18 12:55:33 +00:00

blobstor: Refactor storage selection #695

dstepanov-yadro referenced this issue

2023-09-18 15:30:56 +00:00

WIP: blobovnicza migrator #696

dstepanov-yadro referenced this issue

2023-09-22 10:13:29 +00:00

Blobovnicza tree rebuild (support) #703

fyrchik commented

2023-10-09 13:48:40 +00:00

Author

Owner

Decided to do this migration on startup if config parameters were changed.
In the initial implementation only blobovnicza configuration changes are supported.

Decided to do this migration on startup if config parameters were changed. In the initial implementation only blobovnicza configuration changes are supported.

dstepanov-yadro referenced this issue from a pull request that will close it,

2023-11-16 09:00:40 +00:00

Blobovnicza tree rebuild #812

fyrchik referenced this issue from a commit

2023-12-07 13:15:54 +00:00

[#661] shard: Add blobstor rebuilder

fyrchik referenced this issue from a commit

2023-12-07 13:15:54 +00:00

[#661] blobovniczatree: Use .db extension for db files

fyrchik referenced this issue from a commit

2023-12-07 13:15:54 +00:00

[#661] blobovniczatree: Allow to change depth or width