NPE in badger writecache tests #598

New issue

Closed

opened 2023-08-11 07:02:41 +00:00 by fyrchik · 1 comment

fyrchik commented

2023-08-11 07:02:41 +00:00

Owner

https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/789/jobs/3

panic: runtime error: invalid memory address or nil pointer dereference
	panic: Unclosed iterator at time of Txn.Discard.
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xecc879]
goroutine 271 [running]:
github.com/dgraph-io/badger/v4.(*Txn).Discard(0xc0002a6200)
	/root/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.1.0/txn.go:524 +0x1fe
panic({0x10a43e0?, 0x1a146e0?})
	/opt/hostedtoolcache/go/1.21.0/x64/src/runtime/panic.go:920 +0x270
github.com/dgraph-io/badger/v4.(*memTable).IncrRef(...)
	/root/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.1.0/memtable.go:229
github.com/dgraph-io/badger/v4.(*DB).getMemTables(0xc000184900)
	/root/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.1.0/db.go:693 +0x199
github.com/dgraph-io/badger/v4.(*Txn).NewIterator(0xc0002a6200, {0x64, 0x1, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, ...})
	/root/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.1.0/iterator.go:482 +0xe5
git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger.(*cache).flushSmallObjects.func1(0xc000184900?)
	/workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger/flush.go:84 +0xc5
github.com/dgraph-io/badger/v4.(*DB).View(0xc000184900, 0xc000357dd0)
	/root/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.1.0/txn.go:807 +0x129
git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger.(*cache).flushSmallObjects(0xc005f12180)
	/workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger/flush.go:83 +0x1a5
git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger.(*cache).runFlushLoop.func1()
	/workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger/flush.go:55 +0x10e
created by git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger.(*cache).runFlushLoop in goroutine 230
	/workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger/flush.go:46 +0x165
FAIL	git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger	10.355s

https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/789/jobs/3 ``` panic: runtime error: invalid memory address or nil pointer dereference panic: Unclosed iterator at time of Txn.Discard. [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xecc879] goroutine 271 [running]: github.com/dgraph-io/badger/v4.(*Txn).Discard(0xc0002a6200) /root/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.1.0/txn.go:524 +0x1fe panic({0x10a43e0?, 0x1a146e0?}) /opt/hostedtoolcache/go/1.21.0/x64/src/runtime/panic.go:920 +0x270 github.com/dgraph-io/badger/v4.(*memTable).IncrRef(...) /root/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.1.0/memtable.go:229 github.com/dgraph-io/badger/v4.(*DB).getMemTables(0xc000184900) /root/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.1.0/db.go:693 +0x199 github.com/dgraph-io/badger/v4.(*Txn).NewIterator(0xc0002a6200, {0x64, 0x1, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, ...}) /root/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.1.0/iterator.go:482 +0xe5 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger.(*cache).flushSmallObjects.func1(0xc000184900?) /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger/flush.go:84 +0xc5 github.com/dgraph-io/badger/v4.(*DB).View(0xc000184900, 0xc000357dd0) /root/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.1.0/txn.go:807 +0x129 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger.(*cache).flushSmallObjects(0xc005f12180) /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger/flush.go:83 +0x1a5 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger.(*cache).runFlushLoop.func1() /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger/flush.go:55 +0x10e created by git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger.(*cache).runFlushLoop in goroutine 230 /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger/flush.go:46 +0x165 FAIL git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebadger 10.355s ```

fyrchik added the

labels 2023-08-11 07:02:41 +00:00

ale64bit self-assigned this 2023-08-11 09:07:32 +00:00

ale64bit commented

2023-08-11 12:33:59 +00:00

Member

It's hard to repro locally, but every time it happens it seems to be related to calling Close when changing mode while being iterated by the flush workers:
https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/local_object_storage/writecache/writecachebadger/mode.go#L45

It's hard to repro locally, but every time it happens it seems to be related to calling `Close` when changing mode while being iterated by the flush workers: https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/local_object_storage/writecache/writecachebadger/mode.go#L45

fyrchik referenced this issue from a commit

2023-08-14 07:12:30 +00:00

[#598] Hold mode mutex when setting mode

~~ale64bit referenced this issue 2023-08-14 10:23:04 +00:00~~

[#598] Fix read-after-close bug in badger writecache #604

ale64bit referenced this issue from a pull request that will close it,

2023-08-14 10:24:07 +00:00

[#598] Fix read-after-close bug in badger writecache #604

fyrchik referenced this issue from a commit

2023-08-15 07:04:08 +00:00

[#598] Fix use-after-close bug in badger writecache

fyrchik closed this issue