metabase: optimize Inhume operation #1385

Merged
dstepanov-yadro merged 1 commit from dstepanov-yadro/frostfs-node:fix/metabase_inhume_hangs into master 2024-10-26 11:30:26 +00:00

During load testing, a high latency (> 5s) was detected when performing an Inhume operation.

Fixed:

  • Update replaced with Batch in metabase: based on the analysis of pprof data, it can be seen that single Update operations in the metabase take a long time
  • optimized isTomb check that checks that inhumed object is not tombstone: now it checks only objects from the same container
  • added check that tombstone and tombstoned object have the same containerID: object of type TOMBSTONE stores objectIDs as members, also see Inhume call from Delete RPC:

    addrs[i].SetContainer(tombstone.Container())

Inhume latency before:
image

Inhume latency after:
image

During load testing, a high latency (> 5s) was detected when performing an `Inhume` operation. Fixed: - `Update` replaced with `Batch` in metabase: based on the analysis of pprof data, it can be seen that single `Update` operations in the metabase take a long time - optimized `isTomb` check that checks that inhumed object is not tombstone: now it checks only objects from the same container - added check that tombstone and tombstoned object have the same containerID: object of type `TOMBSTONE` stores objectIDs as members, also see `Inhume` call from `Delete` RPC: https://git.frostfs.info/TrueCloudLab/frostfs-node/src/commit/c34b8acedde282bbe81efccea772a923ee570a8f/cmd/frostfs-node/object.go#L527 Inhume latency before: ![image](/attachments/af1e72dc-a167-4134-8b6b-b6b353833d62) Inhume latency after: ![image](/attachments/dbd898d2-ab6d-4868-96c2-27bfec382d98)
169 KiB
176 KiB
dstepanov-yadro added 1 commit 2024-09-19 15:41:58 +00:00
[#9999] metabase: Use Batch for delete-related operations
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 1m15s
DCO action / DCO (pull_request) Successful in 1m27s
Vulncheck / Vulncheck (pull_request) Successful in 1m25s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m15s
Build / Build Components (pull_request) Successful in 2m23s
Tests and linters / gopls check (pull_request) Successful in 2m42s
Tests and linters / Staticcheck (pull_request) Successful in 2m46s
Tests and linters / Lint (pull_request) Successful in 3m34s
Tests and linters / Tests (pull_request) Successful in 4m15s
Tests and linters / Tests with -race (pull_request) Successful in 5m34s
c1b4c5c0a0
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
dstepanov-yadro force-pushed fix/metabase_inhume_hangs from c1b4c5c0a0 to a193465421 2024-09-19 16:51:42 +00:00 Compare
dstepanov-yadro requested review from storage-core-committers 2024-09-19 16:54:48 +00:00
dstepanov-yadro requested review from storage-core-developers 2024-09-19 16:54:49 +00:00
dstepanov-yadro changed title from metabase: Use Batch for all concurrent operations to WIP: metabase: Use Batch for all concurrent operations 2024-09-20 09:17:28 +00:00
dstepanov-yadro added 2 commits 2024-09-20 12:11:50 +00:00
Target container ID is taken from tombstone: cmd/frostfs-node/object.go:507
Also object of type `TOMBSTONE` contains objectID, so tombstone and
tombstoned object must have the same containerID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
[#1385] metabase: Optimize isTomb check
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 1m56s
DCO action / DCO (pull_request) Successful in 2m13s
Vulncheck / Vulncheck (pull_request) Successful in 2m22s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m39s
Build / Build Components (pull_request) Successful in 2m57s
Tests and linters / Staticcheck (pull_request) Successful in 3m19s
Tests and linters / gopls check (pull_request) Successful in 3m20s
Tests and linters / Lint (pull_request) Successful in 3m56s
Tests and linters / Tests (pull_request) Successful in 4m49s
Tests and linters / Tests with -race (pull_request) Successful in 5m47s
2e35eff8cb
As tombstone and target must have the same containerID, do not iterate
other containers.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
dstepanov-yadro force-pushed fix/metabase_inhume_hangs from 2e35eff8cb to b15437c947 2024-09-20 12:19:09 +00:00 Compare
dstepanov-yadro changed title from WIP: metabase: Use Batch for all concurrent operations to metabase: optimize Inhume operation 2024-09-20 12:27:24 +00:00
dstepanov-yadro force-pushed fix/metabase_inhume_hangs from b15437c947 to 3b5a9d466f 2024-09-23 08:21:17 +00:00 Compare
aarifullin approved these changes 2024-09-24 12:29:47 +00:00
acid-ant approved these changes 2024-09-24 14:01:56 +00:00
Member

Please rebase on master.

Please rebase on master.
dstepanov-yadro force-pushed fix/metabase_inhume_hangs from 3b5a9d466f to fd18aa363b 2024-09-24 14:09:25 +00:00 Compare
dstepanov-yadro merged commit fd18aa363b into master 2024-09-24 14:15:35 +00:00
dstepanov-yadro deleted branch fix/metabase_inhume_hangs 2024-09-24 14:15:37 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#1385
No description provided.