Panic on all nodes "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/flush.go:35 +0x30" #143

New issue

Closed

opened 2023-03-17 06:11:51 +00:00 by anikeev-yadro · 0 comments

anikeev-yadro commented

2023-03-17 06:11:51 +00:00

Member

Panics on all nodes while nightly functional tests was running.

Mar 17 06:31:16 az frostfs-node[7200]: 2023-03-17T03:31:16.761Z        debug        replicator/process.go:23        finish work        {"component": "Object Replicator", "amount of unfinished replicas": 1}
Mar 17 06:31:16 az frostfs-node[7200]: 2023-03-17T03:31:16.761Z        error        policer/check.go:100        could not build placement vector for object        {"component": "Object Policer", "cid": "2HjD3Nop5phUsBrKuJxhCw5V7NnG1xN3FonE7rDTBGrf", "error": "could not get container nodes: not enough nodes to SELECT from: 'X'"}
Mar 17 06:31:16 az frostfs-node[7200]: runtime: goroutine stack exceeds 1000000000-byte limit
Mar 17 06:31:16 az frostfs-node[7200]: runtime: sp=0xc0387e6370 stack=[0xc0387e6000, 0xc0587e6000]
Mar 17 06:31:16 az frostfs-node[7200]: fatal error: stack overflow
Mar 17 06:31:16 az frostfs-node[7200]: runtime stack:
Mar 17 06:31:16 az frostfs-node[7200]: runtime.throw({0x116b3f8?, 0x1bf9580?})
Mar 17 06:31:16 az frostfs-node[7200]:         runtime/panic.go:992 +0x71
Mar 17 06:31:16 az frostfs-node[7200]: runtime.newstack()
Mar 17 06:31:16 az frostfs-node[7200]:         runtime/stack.go:1101 +0x5cc
Mar 17 06:31:16 az frostfs-node[7200]: runtime.morestack()
Mar 17 06:31:16 az frostfs-node[7200]:         runtime/asm_amd64.s:547 +0x8b
Mar 17 06:31:16 az frostfs-node[7200]: goroutine 151520 [running]:
Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0xc0002fe2a0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...})
Mar 17 06:31:16 az frostfs-node[7200]:         git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:594 +0x59 fp=0xc0387e6380 sp=0xc0387e6378 pc=0xe87959
Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...})
Mar 17 06:31:16 az frostfs-node[7200]:         git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:595 +0x45 fp=0xc0387e63d8 sp=0xc0387e6380 pc=0xe87945
Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...})
Mar 17 06:31:16 az frostfs-node[7200]:         git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:595 +0x45 fp=0xc0387e6430 sp=0xc0387e63d8 pc=0xe87945
Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...})
Mar 17 06:31:16 az frostfs-node[7200]:         git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:595 +0x45 fp=0xc0387e6488 sp=0xc0387e6430 pc=0xe87945
Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...})
Mar 17 06:31:16 az frostfs-node[7200]:         git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:595 +0x45 fp=0xc0387e64e0 sp=0xc0387e6488 pc=0xe87945
Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...})
Mar 17 06:31:16 az frostfs-node[7200]:         git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:595 +0x45 fp=0xc0387e6538 sp=0xc0387e64e0 pc=0xe87945
Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...})
Mar 17 06:31:16 az frostfs-node[7200]:         git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:595 +0x45 fp=0xc0387e6590 sp=0xc0387e6538 pc=0xe87945
Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...})
...

Then errors was looping on this piece of log:

Mar 17 06:31:16 az frostfs-node[7200]: goroutine 229 [select, 60 minutes]:
Mar 17 06:31:16 az frostfs-node[7200]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache.(*cache).flushWorker(0xc000177b00, 0x0?)
Mar 17 06:31:16 az frostfs-node[7200]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/flush.go:232 +0xf7
Mar 17 06:31:16 az frostfs-node[7200]: created by git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache.(*cache).runFlushLoop
Mar 17 06:31:16 az frostfs-node[7200]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/flush.go:35 +0x30

Versions

0.0.1-231-g64bde68f

Your Environment

Server setup and configuration:
Vitrual, 4 VMs, 4 SN, 4 http qw, 4 s3 gw

Operating System and version (uname -a):
linux vedi 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux

Panics on all nodes while nightly functional tests was running. ``` Mar 17 06:31:16 az frostfs-node[7200]: 2023-03-17T03:31:16.761Z debug replicator/process.go:23 finish work {"component": "Object Replicator", "amount of unfinished replicas": 1} Mar 17 06:31:16 az frostfs-node[7200]: 2023-03-17T03:31:16.761Z error policer/check.go:100 could not build placement vector for object {"component": "Object Policer", "cid": "2HjD3Nop5phUsBrKuJxhCw5V7NnG1xN3FonE7rDTBGrf", "error": "could not get container nodes: not enough nodes to SELECT from: 'X'"} Mar 17 06:31:16 az frostfs-node[7200]: runtime: goroutine stack exceeds 1000000000-byte limit Mar 17 06:31:16 az frostfs-node[7200]: runtime: sp=0xc0387e6370 stack=[0xc0387e6000, 0xc0587e6000] Mar 17 06:31:16 az frostfs-node[7200]: fatal error: stack overflow Mar 17 06:31:16 az frostfs-node[7200]: runtime stack: Mar 17 06:31:16 az frostfs-node[7200]: runtime.throw({0x116b3f8?, 0x1bf9580?}) Mar 17 06:31:16 az frostfs-node[7200]: runtime/panic.go:992 +0x71 Mar 17 06:31:16 az frostfs-node[7200]: runtime.newstack() Mar 17 06:31:16 az frostfs-node[7200]: runtime/stack.go:1101 +0x5cc Mar 17 06:31:16 az frostfs-node[7200]: runtime.morestack() Mar 17 06:31:16 az frostfs-node[7200]: runtime/asm_amd64.s:547 +0x8b Mar 17 06:31:16 az frostfs-node[7200]: goroutine 151520 [running]: Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0xc0002fe2a0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...}) Mar 17 06:31:16 az frostfs-node[7200]: git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:594 +0x59 fp=0xc0387e6380 sp=0xc0387e6378 pc=0xe87959 Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...}) Mar 17 06:31:16 az frostfs-node[7200]: git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:595 +0x45 fp=0xc0387e63d8 sp=0xc0387e6380 pc=0xe87945 Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...}) Mar 17 06:31:16 az frostfs-node[7200]: git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:595 +0x45 fp=0xc0387e6430 sp=0xc0387e63d8 pc=0xe87945 Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...}) Mar 17 06:31:16 az frostfs-node[7200]: git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:595 +0x45 fp=0xc0387e6488 sp=0xc0387e6430 pc=0xe87945 Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...}) Mar 17 06:31:16 az frostfs-node[7200]: git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:595 +0x45 fp=0xc0387e64e0 sp=0xc0387e6488 pc=0xe87945 Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...}) Mar 17 06:31:16 az frostfs-node[7200]: git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:595 +0x45 fp=0xc0387e6538 sp=0xc0387e64e0 pc=0xe87945 Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...}) Mar 17 06:31:16 az frostfs-node[7200]: git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/object.go:595 +0x45 fp=0xc0387e6590 sp=0xc0387e6538 pc=0xe87945 Mar 17 06:31:16 az frostfs-node[7200]: main.engineWithoutNotifications.IsLocked({0x0?}, {{0x6c, 0xf1, 0xa2, 0x62, 0x8, 0x6, 0x6, 0x2c, 0x57, ...}, ...}) ... ``` Then errors was looping on this piece of log: ``` Mar 17 06:31:16 az frostfs-node[7200]: goroutine 229 [select, 60 minutes]: Mar 17 06:31:16 az frostfs-node[7200]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache.(*cache).flushWorker(0xc000177b00, 0x0?) Mar 17 06:31:16 az frostfs-node[7200]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/flush.go:232 +0xf7 Mar 17 06:31:16 az frostfs-node[7200]: created by git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache.(*cache).runFlushLoop Mar 17 06:31:16 az frostfs-node[7200]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/flush.go:35 +0x30 ``` ## Versions ``` 0.0.1-231-g64bde68f ``` ## Your Environment Server setup and configuration: Vitrual, 4 VMs, 4 SN, 4 http qw, 4 s3 gw Operating System and version (uname -a): linux vedi 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux

Test_log.zip

1.5 MiB