Blobovnicza tree rebuild (support) #703

Merged
fyrchik merged 18 commits from dstepanov-yadro/frostfs-node:feat/shard_migrator_support_v0.37 into support/v0.37 2024-09-04 19:51:03 +00:00

18 commits

Author SHA1 Message Date
02450a9a16 [#661] blobovniczatree: Do not create DB's on init
Some checks failed
Tests and linters / Tests with -race (pull_request) Failing after 10s
DCO action / DCO (pull_request) Successful in 1m24s
Build / Build Components (1.21) (pull_request) Successful in 3m4s
Build / Build Components (1.20) (pull_request) Successful in 4m3s
Vulncheck / Vulncheck (pull_request) Failing after 3m39s
Tests and linters / Lint (pull_request) Successful in 9m41s
Tests and linters / Staticcheck (pull_request) Successful in 9m43s
Tests and linters / Tests (1.21) (pull_request) Failing after 16m43s
Tests and linters / Tests (1.20) (pull_request) Failing after 16m48s
Blobovniczas will be created on write requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:05 +03:00
9de91542fe [#661] blobovnicza: Compute size with record size
To get more accurate size of blobovnicza use record
size (lenght of key + lenght of data).

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:05 +03:00
e0cb29cb6e [#698] blobovnicza: Store counter values
Blobovnicza initialization take a long time because of bucket
Stat() call. So now blobovnicza stores counters in META bucket.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:05 +03:00
736ec847b7 [#661] metabase: Update storage ID in case of logical errors
If object was GC marked or deleted or expired, it is still required
to update storageID to physically delete object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:05 +03:00
4b20f846af [#661] metrcis: Add rebuild percent metric
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:05 +03:00
04b530197e [#661] blobovniczatree: Pass object size limit from config
If actual small object size value lower than default
object size limit, then unnecessary buckets created.
If actual small object size value greated than default
object size limit, then error happens.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:05 +03:00
f8606db7f2 [#661] blobovniczatree: Do not sort DB's and indicies
Put stores object to next active DB, so there is no need to sort DBs.
In addition, it adds unnecessary DB openings.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:05 +03:00
52c625df0f [#661] blobovniczatree: Make Rebuild concurrent for objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:05 +03:00
888f966eb4 [#661] blobovniczatree: Make Rebuild concurrent
Different DBs can be rebuild concurrently.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:04 +03:00
f325b9557c [#661] metrics: Add blobovniczatree rebuild metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:04 +03:00
4f7f67d136 [#661] blobovniczatree: Make Rebuild failover safe
Now move info stores in blobovnicza, so in case of failover
rebuild completes previous operation first.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:04 +03:00
8afb42aae8 [#698] blobovniczatree: Init blobovniczas concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:04 +03:00
71fac2b9c3 [#661] shard: Fix Delete method
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:04 +03:00
9ef4a885de [#661] blobovniczatree: Add Rebuild implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:04 +03:00
b22b703325 [#661] blobstor: Add Rebuild implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:04 +03:00
784efb6155 [#661] blobovniczatree: Allow to change depth or width
Now it is possible to change depth or with of blobovniczatree.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:04 +03:00
df11b40dfd [#661] blobovniczatree: Use .db extension for db files
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:04 +03:00
4e05ce1c3a [#661] shard: Add blobstor rebuilder
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:36:04 +03:00