Metabase upgrade optimizations #1355

dstepanov-yadro · 2024-09-06T08:24:57Z

dstepanov-yadro commented

2024-09-06 08:24:57 +00:00

Run upgrade concurrently
Log every bucket batch delete as this can take a lot of time
Open destination metabase to compact with NoSync
Do not fail if database has latest version: this allows to run compact only on metabase with latest version

1. Run upgrade concurrently 2. Log every bucket batch delete as this can take a lot of time 3. Open destination metabase to compact with NoSync 4. Do not fail if database has latest version: this allows to run compact only on metabase with latest version

dstepanov-yadro added 2 commits 2024-09-06 08:24:58 +00:00

[#9999 ] adm: Run metabase upgrade concurrently 1e0cce3aa9

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>

[#9999 ] metabase: Upgrade improvements

DCO action / DCO (pull_request) Successful in 47s

Details

Vulncheck / Vulncheck (pull_request) Successful in 1m47s

Details

Pre-commit hooks / Pre-commit (pull_request) Successful in 2m12s

Details

Tests and linters / Run gofumpt (pull_request) Successful in 2m10s

Details

Build / Build Components (pull_request) Successful in 2m31s

Details

Tests and linters / gopls check (pull_request) Successful in 2m38s

Details

Tests and linters / Lint (pull_request) Failing after 3m6s

Details

Tests and linters / Staticcheck (pull_request) Successful in 3m6s

Details

Tests and linters / Tests (pull_request) Successful in 5m22s

Details

Tests and linters / Tests with -race (pull_request) Successful in 5m30s

Details

4ba6a9981b

Do not fail on same latest version to run compact on upgraded metabase.
Use NoSync on compact.
Log every batch on bucket delete stage.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>

dstepanov-yadro force-pushed feat/mb_upgrade_optimizations from 4ba6a9981b to ae282aec0a

2024-09-06 08:26:01 +00:00

Compare

dstepanov-yadro force-pushed feat/mb_upgrade_optimizations from ae282aec0a to 6512f86375

2024-09-06 08:33:06 +00:00

Compare

dstepanov-yadro force-pushed feat/mb_upgrade_optimizations from 6512f86375 to df20a973c8

2024-09-06 12:54:58 +00:00

Compare

dstepanov-yadro requested review from storage-core-committers 2024-09-06 12:55:00 +00:00

dstepanov-yadro requested review from storage-core-developers 2024-09-06 12:55:01 +00:00

dstepanov-yadro force-pushed feat/mb_upgrade_optimizations from df20a973c8 to ced3a8461a

2024-09-06 12:58:04 +00:00

Compare

dstepanov-yadro force-pushed feat/mb_upgrade_optimizations from ced3a8461a to 49702daced

2024-09-06 12:58:16 +00:00

Compare

dstepanov-yadro force-pushed feat/mb_upgrade_optimizations from 49702daced to b685d5861e

2024-09-06 12:59:33 +00:00

Compare

acid-ant commented

2024-09-09 08:29:57 +00:00

Please take a look at CI -> Tests and linters / Lint.

Please take a look at `CI -> Tests and linters / Lint`.

dstepanov-yadro force-pushed feat/mb_upgrade_optimizations from b685d5861e to 4668efc0bf

2024-09-09 08:37:56 +00:00

Compare

acid-ant approved these changes 2024-09-09 08:58:26 +00:00

fyrchik approved these changes 2024-09-09 12:23:49 +00:00

pkg/local_object_storage/metabase/upgrade.go

					
				@ -86,6 +90,7 @@ func compactDB(db *bbolt.DB) error {

					}

					dst, err := bbolt.Open(tmpFileName, f.Mode(), &bbolt.Options{

						Timeout: 100 * time.Millisecond,

						NoSync:  true,

fyrchik commented

2024-09-09 12:22:56 +00:00

We can leave garbage in case of premature exit, do we handle this somehow?

fyrchik commented

2024-09-09 12:24:41 +00:00

It is a problem, because we can have no space and need to describe what to do.

dstepanov-yadro commented

2024-09-09 14:02:01 +00:00

do we handle this somehow?

No.

It is a problem, because we can have no space and need to describe what to do.

I dont want to do some delete by pattern to not to delete some manual created files.

> do we handle this somehow? No. > It is a problem, because we can have no space and need to describe what to do. I dont want to do some delete by pattern to not to delete some manual created files.