Create tombstone source when reload #393

Merged

fyrchik merged 2 commits from dstepanov-yadro/frostfs-node:fix/gc_panic into master

2023-05-26 12:14:03 +00:00

dstepanov-yadro commented

2023-05-25 13:23:11 +00:00

Member

There is a panic when node's config reloaded and GC deletes objects with tombstones:

2023/05/25 10:58:24 worker exits from a panic: runtime error: invalid memory address or nil pointer dereference
2023/05/25 10:58:24 worker exits from panic: goroutine 7480 [running]:
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
panic({0x104dd40, 0x1d436e0})
        runtime/panic.go:838 +0x207
git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).collectExpiredTombstones(0xc0006d11d0, {0x1487990, 0xc000626000}, {0x14778a0?, 0x1d095d8?})
        git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:410 +0x7de
git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).listenEvents.func1()
        git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:149 +0x74
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65

2023-05-25T10:58:24.027Z        info    frostfs-node/config.go:983      bootstrapping with online state {"previous": "ONLINE"}

Because of panic workgroup.Done() is never called for GC. So there is no receiver for new epoch event and new epoch notification is blocked after panic.

There is a panic when node's config reloaded and GC deletes objects with tombstones: ``` 2023/05/25 10:58:24 worker exits from a panic: runtime error: invalid memory address or nil pointer dereference 2023/05/25 10:58:24 worker exits from panic: goroutine 7480 [running]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1() github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c panic({0x104dd40, 0x1d436e0}) runtime/panic.go:838 +0x207 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).collectExpiredTombstones(0xc0006d11d0, {0x1487990, 0xc000626000}, {0x14778a0?, 0x1d095d8?}) git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:410 +0x7de git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).listenEvents.func1() git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:149 +0x74 github.com/panjf2000/ants/v2.(*goWorker).run.func1() github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97 created by github.com/panjf2000/ants/v2.(*goWorker).run github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65 2023-05-25T10:58:24.027Z info frostfs-node/config.go:983 bootstrapping with online state {"previous": "ONLINE"} ``` Because of panic workgroup.Done() is never called for GC. So there is no receiver for new epoch event and new epoch notification is blocked after panic.

dstepanov-yadro force-pushed fix/gc_panic from 6513fbebf8 to da86da8ddc

2023-05-25 13:24:07 +00:00

Compare