Fix/tree svc panic #322

Merged

fyrchik merged 1 commit from carpawell/frostfs-node:fix/panic-in-tree-svc into master

2023-05-05 07:06:16 +00:00

carpawell commented

2023-05-04 16:47:55 +00:00

Contributor

If a connection has not been established earlier, it stores nil in LRU
cache. Cache eviction tries to close every connection (even a nil one) and
panics but not crash the app because we are using pools.
That ugly bug also leads to a deadlock where Unlock is not called via
defer func (and that is the way I found it).

Signed-off-by: Pavel Karpy p.karpy@yadro.com

Could be the reason for #260 but should be rechecked.

If a connection has not been established earlier, it stores `nil` in LRU cache. Cache eviction tries to close every connection (even a `nil` one) and panics but not crash the app because we are using pools. That ugly bug also leads to a deadlock where `Unlock` is not called via `defer` func (and that is the way I found it). Signed-off-by: Pavel Karpy <p.karpy@yadro.com> Could be the reason for #260 but should be rechecked.

carpawell added the

bug

frostfs-node

labels 2023-05-04 16:47:55 +00:00

carpawell self-assigned this 2023-05-04 16:47:55 +00:00

carpawell force-pushed fix/panic-in-tree-svc from 1774705f93 to 479c5a65e1

2023-05-04 16:49:00 +00:00

Compare

carpawell changed title from ~~[#xxx] node: Fix tree svc panic~~ to Fix/tree svc panic

2023-05-04 16:49:07 +00:00

carpawell requested review from storage-core-developers 2023-05-04 16:49:19 +00:00

carpawell requested review from storage-core-committers 2023-05-04 16:49:19 +00:00

carpawell requested review from fyrchik 2023-05-04 16:49:23 +00:00

fyrchik approved these changes 2023-05-04 16:54:34 +00:00

fyrchik commented

2023-05-04 16:55:09 +00:00

Owner

We need the same for 1.2 (support/v0.36)

carpawell referenced this pull request

2023-05-04 17:03:03 +00:00

Fix/tree svc panic [support] #323

carpawell commented

2023-05-04 17:03:25 +00:00

Author

Contributor

@fyrchik, no problemo: #323.

@fyrchik, no problemo: https://git.frostfs.info/TrueCloudLab/frostfs-node/pulls/323.

fyrchik commented

2023-05-04 17:08:04 +00:00

Owner

Could you explain, how exactly it could be the reason for #260?

Could you explain, how exactly it _could_ be the reason for #260?

carpawell commented

2023-05-04 17:51:11 +00:00

Author

Contributor

@fyrchik, that is a theory. Every tree svc operation was blocked when i looked at them cause of that bug. Tree sync is related to handling morph events (and event should be handled correctly without any blocks to be handled again next time). At least it stopped some operations and locked them forever. That was the first i saw in the pprof and absolutely no locks in any morph code.