Panic at Inner Ring during new epoch timer processing #205

New Issue

Closed

opened 2023-04-04 06:31:52 +00:00 by alexvanin · 0 comments

alexvanin commented

2023-04-04 06:31:52 +00:00

Owner

Expected Behavior

No panic at new epoch timer tick.

Current Behavior

2023-04-04T02:00:22.766Z        debug        innerring/innerring.go:213        new block        {"index": 73}
2023/04/04 02:00:22 worker exits from a panic: runtime error: invalid memory address or nil pointer dereference
2023/04/04 02:00:22 worker exits from panic: goroutine 135 [running]:
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
panic({0xd8b6e0, 0x17e31b0})
        runtime/panic.go:838 +0x207
git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring.newEpochTimer.func3()
        git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring/blocktimer.go:100 +0x37
git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/timer.(*BlockTimer).tick(0xc0003634a0, 0x49)
        git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/timer/block.go:181 +0x51
git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/timer.(*BlockTimer).tick(0xc0001244e0, 0x49)
        git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/timer/block.go:191 +0xd8
git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/timer.(*BlockTimer).Tick(0xc0001244e0, 0x1785140?)
        git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/timer/block.go:165 +0x56
git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring.(*Server).tickTimers(...)
        git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring/blocktimer.go:78
git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring.(*Server).registerMorphNewBlockEventHandler.func1(0xc0002e4000)
        git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring/innerring.go:224 +0x31a
git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/event.(*listener).listenLoop.func3()
        git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/event/listener.go:286 +0x48
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65
2023-04-04T02:00:30.769Z        debug        innerring/innerring.go:213        new block        {"index": 74}
2023-04-04T02:00:38.771Z        debug        innerring/innerring.go:213        new block        {"index": 75}
2023-04-04T02:00:46.775Z        debug        innerring/innerring.go:213        new block        {"index": 76}

Steps to Reproduce (for bugs)

It just happened randomly in all Inner Ring nodes in the cluster after some time.

Context

At the same time, neofs-adm couldn't update epoch with morph force-new-epoch command, but maybe it is unrelated.

Your Environment

Cloud deploy.
Node version: 0948a280

## Expected Behavior No panic at new epoch timer tick. ## Current Behavior ``` 2023-04-04T02:00:22.766Z debug innerring/innerring.go:213 new block {"index": 73} 2023/04/04 02:00:22 worker exits from a panic: runtime error: invalid memory address or nil pointer dereference 2023/04/04 02:00:22 worker exits from panic: goroutine 135 [running]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1() github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c panic({0xd8b6e0, 0x17e31b0}) runtime/panic.go:838 +0x207 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring.newEpochTimer.func3() git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring/blocktimer.go:100 +0x37 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/timer.(*BlockTimer).tick(0xc0003634a0, 0x49) git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/timer/block.go:181 +0x51 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/timer.(*BlockTimer).tick(0xc0001244e0, 0x49) git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/timer/block.go:191 +0xd8 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/timer.(*BlockTimer).Tick(0xc0001244e0, 0x1785140?) git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/timer/block.go:165 +0x56 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring.(*Server).tickTimers(...) git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring/blocktimer.go:78 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring.(*Server).registerMorphNewBlockEventHandler.func1(0xc0002e4000) git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring/innerring.go:224 +0x31a git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/event.(*listener).listenLoop.func3() git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/event/listener.go:286 +0x48 github.com/panjf2000/ants/v2.(*goWorker).run.func1() github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97 created by github.com/panjf2000/ants/v2.(*goWorker).run github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65 2023-04-04T02:00:30.769Z debug innerring/innerring.go:213 new block {"index": 74} 2023-04-04T02:00:38.771Z debug innerring/innerring.go:213 new block {"index": 75} 2023-04-04T02:00:46.775Z debug innerring/innerring.go:213 new block {"index": 76} ``` ## Steps to Reproduce (for bugs) It just happened randomly in all Inner Ring nodes in the cluster after some time. ## Context At the same time, neofs-adm couldn't update epoch with `morph force-new-epoch` command, but maybe it is unrelated. ## Your Environment Cloud deploy. Node version: 0948a280