Blobovnizca PUT performance degradation #602
Labels
No labels
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: TrueCloudLab/frostfs-node#602
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Current Behavior
Blobovnizca works well in case of low load (around 10-15RPS). But if there are around 30 RPS, then response time in 0.95 quantile goes up to 10 seconds.
See attachment: after RPS increasing at 17:45 latency raises up to 10 seconds
Expected Behavior
I think it must be some linear degradation or something like this. 10 seconds is too much.
Your Environment
MetalCore
12 HDD + 2 SSD
Payload 8KB
Does it happen with any payload size as well?
Haven't tested all possible values.
Does it have any dependency on blobovniczas depth/width?
30 RPS per blobovnicza tree, NOT per node, right?
Right, per shard.
Haven't tested again.
I think this bug requires some research & development, not only development.
Had a thought that this is related to
opened_cache_size(so dbs are closed an opened all the time), but it must've been 1000 in this test.dstepanov-yadro referenced this issue2023-08-15 13:58:27 +00:00
Degradation occurs if a large number of databases are open.
As a solution, the ability to specify the number of databases on leaf nodes has been added.
The calculation of the database size has also been fixed.
It is recommended not to use databases larger than 40-50 MB.