blobovniczatree: Stuck on `Close` #1004

New Issue

Closed

opened 2024-02-22 13:34:35 +00:00 by acid-ant · 1 comment

acid-ant commented

2024-02-22 13:34:35 +00:00

Collaborator

Faced an issue with resource leak when trying to close blobstor during setting DEGRADED mode for shard.
Looks like internal databases opened without control in background.
Routine with an issue from pprof:

...
#	0x46d438	sync.runtime_notifyListWait+0x138													runtime/sema.go:527
#	0x47a564	sync.(*Cond).Wait+0x84															sync/cond.go:70
...blobstor/blobovniczatree.(*openDBCounter).WaitUntilAllClosed+0x2f	git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/blobovniczatree/manager.go:332
...blobstor/blobovniczatree.(*dbManager).Close+0x55			git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/blobovniczatree/manager.go:265
...blobstor/blobovniczatree.(*Blobovniczas).Close+0x36			git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/blobovniczatree/control.go:123
...blobstor.(*BlobStor).Close+0xbd					git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/control.go:76
...blobstor.(*BlobStor).SetMode+0xae					git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/mode.go:23
...shard.(*Shard).setMode+0x4d6					git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/mode.go:61
...shard.(*Shard).SetMode+0x45						git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/mode.go:25
...engine.(*StorageEngine).SetShardMode+0x1b0				git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine/shards.go:327
...er.(*Server).SetShardMode+0x37a					git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/server/set_shard_mode.go:40
...trolService_SetShardMode_Handler+0x16b				git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/service_grpc.pb.go:463
...							google.golang.org/grpc@v1.61.0/server.go:1385
...						google.golang.org/grpc@v1.61.0/server.go:1796
...							google.golang.org/grpc@v1.61.0/server.go:1029

Faced an issue with resource leak when trying to close `blobstor` during setting `DEGRADED` mode for shard. Looks like internal databases opened without control in background. Routine with an issue from `pprof`: ``` ... # 0x46d438 sync.runtime_notifyListWait+0x138 runtime/sema.go:527 # 0x47a564 sync.(*Cond).Wait+0x84 sync/cond.go:70 ...blobstor/blobovniczatree.(*openDBCounter).WaitUntilAllClosed+0x2f git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/blobovniczatree/manager.go:332 ...blobstor/blobovniczatree.(*dbManager).Close+0x55 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/blobovniczatree/manager.go:265 ...blobstor/blobovniczatree.(*Blobovniczas).Close+0x36 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/blobovniczatree/control.go:123 ...blobstor.(*BlobStor).Close+0xbd git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/control.go:76 ...blobstor.(*BlobStor).SetMode+0xae git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/mode.go:23 ...shard.(*Shard).setMode+0x4d6 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/mode.go:61 ...shard.(*Shard).SetMode+0x45 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/mode.go:25 ...engine.(*StorageEngine).SetShardMode+0x1b0 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine/shards.go:327 ...er.(*Server).SetShardMode+0x37a git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/server/set_shard_mode.go:40 ...trolService_SetShardMode_Handler+0x16b git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/service_grpc.pb.go:463 ... google.golang.org/grpc@v1.61.0/server.go:1385 ... google.golang.org/grpc@v1.61.0/server.go:1796 ... google.golang.org/grpc@v1.61.0/server.go:1029 ```