TrueCloudLab/frostfs-node

Fork 27

[#355] Increase tree svc client cache size to test hypotheses #358

Merged

fyrchik merged 1 commit from ale64bit/frostfs-node:fix/355-increase-tree-service-client-cache-size into master

2023-05-17 15:00:40 +00:00

ale64bit commented

2023-05-17 11:15:43 +00:00

Member

Signed-off-by: Alejandro Lopez a.lopez@yadro.com

It's possible that after a certain number of nodes, the cache starts thrashing due to closing connections on eviction. While this PR doesn't solve the problem per se, it's useful to test the hypotheses.

Ultimately, the solution is probably to move away from using a bounded cache (even a configurable value would be prone to thrashing if used incorrectly) and have a better mechanism to detect when certain addresses go out of scope in order to close the corresponding connections. Also, we shouldn't be using WithBlock and instead we should leverage existing backoff and keepalive options to accomplish the same whenever possible.

Signed-off-by: Alejandro Lopez <a.lopez@yadro.com> It's possible that after a certain number of nodes, the cache starts thrashing due to closing connections on eviction. While this PR doesn't solve the problem per se, it's useful to test the hypotheses. Ultimately, the solution is probably to move away from using a bounded cache (even a configurable value would be prone to thrashing if used incorrectly) and have a better mechanism to detect when certain addresses go out of scope in order to close the corresponding connections. Also, [we shouldn't be using `WithBlock`](https://github.com/grpc/grpc-go/blob/master/Documentation/anti-patterns.md#anti-patterns) and instead we should leverage existing backoff and keepalive options to accomplish the same whenever possible.

ale64bit added 1 commit 2023-05-17 11:15:44 +00:00

[#355 ] Increase tree svc client cache size to test hypotheses

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/push/pre-commit Pipeline was successful

Details

13a7a90101

Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>

ale64bit added the

discussion

label 2023-05-17 11:15:51 +00:00

ale64bit requested review from storage-core-committers 2023-05-17 11:15:59 +00:00

ale64bit requested review from storage-core-developers 2023-05-17 11:16:00 +00:00

acid-ant approved these changes 2023-05-17 11:26:57 +00:00

fyrchik commented

2023-05-17 11:36:21 +00:00

Owner

Can you make this PR for support branch too? support/v0.36

Can you make this PR for support branch too? `support/v0.36`

👍 1

ale64bit referenced this pull request

2023-05-17 11:52:59 +00:00

fix/355-increase-tree-service-client-cache-size #359

ale64bit referenced this pull request

2023-05-17 12:18:54 +00:00

[#355] Increase tree svc client cache size to test hypotheses #360

fyrchik approved these changes 2023-05-17 12:23:21 +00:00

dstepanov-yadro approved these changes 2023-05-17 14:09:31 +00:00

fyrchik merged commit 13a7a90101 into master

2023-05-17 15:00:40 +00:00

fyrchik referenced this pull request

2023-05-18 08:21:35 +00:00

Errors 'too many open files' while load is running #355

fyrchik referenced this pull request

2023-05-18 08:26:47 +00:00

goroutine leaks and performance drop on one node #289

No reviewers

No labels

No milestone

No project

No assignees

4 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#358

No description provided.