TrueCloudLab/frostfs-node
18
1
Fork 27
Code Issues 90 Pull requests 8 Projects Releases 1 Activity Actions

metabase: optimize Inhume operation #1385

Open
dstepanov-yadro wants to merge 3 commits from dstepanov-yadro/frostfs-node:fix/metabase_inhume_hangs into master
pull from: dstepanov-yadro/frostfs-node:fix/metabase_inhume_hangs
merge into: TrueCloudLab:master
Conversation 0 Commits 3 Files changed 20 +126 -77
dstepanov-yadro commented 2024-09-19 15:41:57 +00:00
Member
Copy link

During load testing, a high latency (> 5s) was detected when performing an Inhume operation.

Fixed:

  • Update replaced with Batch in metabase: based on the analysis of pprof data, it can be seen that single Update operations in the metabase take a long time
  • optimized isTomb check that checks that inhumed object is not tombstone: now it checks only objects from the same container
  • added check that tombstone and tombstoned object have the same containerID: object of type TOMBSTONE stores objectIDs as members, also see Inhume call from Delete RPC:

    Line 527 in c34b8ac
    addrs[i].SetContainer(tombstone.Container())

Inhume latency before:
image

Inhume latency after:
image

During load testing, a high latency (> 5s) was detected when performing an `Inhume` operation. Fixed: - `Update` replaced with `Batch` in metabase: based on the analysis of pprof data, it can be seen that single `Update` operations in the metabase take a long time - optimized `isTomb` check that checks that inhumed object is not tombstone: now it checks only objects from the same container - added check that tombstone and tombstoned object have the same containerID: object of type `TOMBSTONE` stores objectIDs as members, also see `Inhume` call from `Delete` RPC: https://git.frostfs.info/TrueCloudLab/frostfs-node/src/commit/c34b8acedde282bbe81efccea772a923ee570a8f/cmd/frostfs-node/object.go#L527 Inhume latency before: ![image](/attachments/af1e72dc-a167-4134-8b6b-b6b353833d62) Inhume latency after: ![image](/attachments/dbd898d2-ab6d-4868-96c2-27bfec382d98)
image.png
169 KiB
image.png
176 KiB
dstepanov-yadro added 1 commit 2024-09-19 15:41:58 +00:00
[#9999] metabase: Use Batch for delete-related operations
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 1m15s
Details
DCO action / DCO (pull_request) Successful in 1m27s
Details
Vulncheck / Vulncheck (pull_request) Successful in 1m25s
Details
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m15s
Details
Build / Build Components (pull_request) Successful in 2m23s
Details
Tests and linters / gopls check (pull_request) Successful in 2m42s
Details
Tests and linters / Staticcheck (pull_request) Successful in 2m46s
Details
Tests and linters / Lint (pull_request) Successful in 3m34s
Details
Tests and linters / Tests (pull_request) Successful in 4m15s
Details
Tests and linters / Tests with -race (pull_request) Successful in 5m34s
Details
c1b4c5c0a0 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
dstepanov-yadro force-pushed fix/metabase_inhume_hangs from c1b4c5c0a0 to a193465421 2024-09-19 16:51:42 +00:00 Compare
dstepanov-yadro requested review from storage-core-committers 2024-09-19 16:54:48 +00:00
dstepanov-yadro requested review from storage-core-developers 2024-09-19 16:54:49 +00:00
dstepanov-yadro changed title from metabase: Use `Batch` for all concurrent operations to WIP: metabase: Use `Batch` for all concurrent operations 2024-09-20 09:17:28 +00:00
dstepanov-yadro added 2 commits 2024-09-20 12:11:50 +00:00 
Target container ID is taken from tombstone: cmd/frostfs-node/object.go:507
Also object of type `TOMBSTONE` contains objectID, so tombstone and
tombstoned object must have the same containerID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
[#1385] metabase: Optimize isTomb check
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 1m56s
Details
DCO action / DCO (pull_request) Successful in 2m13s
Details
Vulncheck / Vulncheck (pull_request) Successful in 2m22s
Details
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m39s
Details
Build / Build Components (pull_request) Successful in 2m57s
Details
Tests and linters / Staticcheck (pull_request) Successful in 3m19s
Details
Tests and linters / gopls check (pull_request) Successful in 3m20s
Details
Tests and linters / Lint (pull_request) Successful in 3m56s
Details
Tests and linters / Tests (pull_request) Successful in 4m49s
Details
Tests and linters / Tests with -race (pull_request) Successful in 5m47s
Details
2e35eff8cb 
As tombstone and target must have the same containerID, do not iterate
other containers.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
dstepanov-yadro force-pushed fix/metabase_inhume_hangs from 2e35eff8cb to b15437c947 2024-09-20 12:19:09 +00:00 Compare
dstepanov-yadro changed title from WIP: metabase: Use `Batch` for all concurrent operations to metabase: optimize `Inhume` operation 2024-09-20 12:27:24 +00:00
dstepanov-yadro force-pushed fix/metabase_inhume_hangs from b15437c947 to 3b5a9d466f 2024-09-23 08:21:17 +00:00 Compare
All checks were successful
DCO action / DCO (pull_request) Successful in 1m10s
Required
Details
Tests and linters / Run gofumpt (pull_request) Successful in 1m18s
Required
Details
Vulncheck / Vulncheck (pull_request) Successful in 1m31s
Required
Details
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m9s
Required
Details
Build / Build Components (pull_request) Successful in 2m17s
Required
Details
Tests and linters / gopls check (pull_request) Successful in 2m38s
Required
Details
Tests and linters / Staticcheck (pull_request) Successful in 2m44s
Required
Details
Tests and linters / Lint (pull_request) Successful in 3m8s
Required
Details
Tests and linters / Tests (pull_request) Successful in 4m13s
Required
Details
Tests and linters / Tests with -race (pull_request) Successful in 5m50s
Required
Details
This pull request doesn't have enough approvals yet. 0 of 2 approvals granted.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u fix/metabase_inhume_hangs:dstepanov-yadro-fix/metabase_inhume_hangs
git checkout dstepanov-yadro-fix/metabase_inhume_hangs
Sign in to join this conversation.
Reviewers
No reviewers
TrueCloudLab/storage-core-committers
TrueCloudLab/storage-core-developers
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
badger

Badger DB related issues

  
frostfs-adm

   
frostfs-cli

   
frostfs-ir

   
frostfs-lens

   
frostfs-node

   
good first issue

Good for newcomers

  
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#1385
No description provided.
Delete branch "dstepanov-yadro/frostfs-node:fix/metabase_inhume_hangs"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?