`TestListWithCursor` is flaky #326

New issue

Closed

opened 2023-05-05 07:28:53 +00:00 by fyrchik · 3 comments

fyrchik commented

2023-05-05 07:28:53 +00:00

Owner

Cut output for brevity:

--- FAIL: TestListWithCursor (5.30s)
    --- FAIL: TestListWithCursor/many_shards,_many_objects,_big_batch_size (2.34s)
        list_test.go:117:
                Error Trace:    /repo/frostfs/node/pkg/local_object_storage/engine/list_test.go:117
                Error:          Not equal:
                                expected: []object.AddressWithType{object.AddressWithType{Address:oid.Address{cnr:cid.ID

Cut output for brevity: ``` --- FAIL: TestListWithCursor (5.30s) --- FAIL: TestListWithCursor/many_shards,_many_objects,_big_batch_size (2.34s) list_test.go:117: Error Trace: /repo/frostfs/node/pkg/local_object_storage/engine/list_test.go:117 Error: Not equal: expected: []object.AddressWithType{object.AddressWithType{Address:oid.Address{cnr:cid.ID ```

fyrchik added the

bug

triage

labels 2023-05-05 07:28:53 +00:00

aarifullin was assigned by fyrchik

2023-05-05 07:28:53 +00:00

fyrchik added this to the v0.37.0 milestone 2023-05-05 07:29:13 +00:00

fyrchik modified the milestone from v0.37.0 to v0.38.0

2023-06-19 08:02:55 +00:00

fyrchik commented

2023-06-27 11:13:05 +00:00

Author

Owner

Cannot reproduce.

fyrchik closed this issue

2023-06-27 11:13:05 +00:00

fyrchik commented

2023-08-16 07:40:53 +00:00

Author

Owner

Reproduced
https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/888/jobs/3

==================
WARNING: DATA RACE
Read at 0x00c0008a2d43 by goroutine 596:
  testing.(*common).logDepth()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1011 +0xbb
  testing.(*common).log()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1004 +0x99
  testing.(*common).Logf()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1055 +0x62
  testing.(*T).Logf()
      <autogenerated>:1 +0x69
  go.uber.org/zap/zaptest.testingWriter.Write()
      /root/go/pkg/mod/go.uber.org/zap@v1.24.0/zaptest/logger.go:130 +0x11d
  go.uber.org/zap/zaptest.(*testingWriter).Write()
      <autogenerated>:1 +0x74
  go.uber.org/zap/zapcore.(*ioCore).Write()
      /root/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/core.go:99 +0x192
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      /root/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:255 +0x2af
  go.uber.org/zap.(*Logger).Debug()
      /root/go/pkg/mod/go.uber.org/zap@v1.24.0/logger.go:212 +0x67
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).removeGarbage()
      /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:249 +0x20a
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).removeGarbage-fm()
      <autogenerated>:1 +0x47
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).tickRemover()
      /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:212 +0x18c
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).init.func1()
      /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:147 +0x4f
Previous write at 0x00c0008a2d43 by goroutine 323:
  testing.tRunner.func1()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1582 +0x81a
  runtime.deferreturn()
      /opt/hostedtoolcache/go/1.21.0/x64/src/runtime/panic.go:477 +0x30
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1648 +0x44
Goroutine 596 (running) created at:
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).init()
      /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:147 +0x290
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).Init()
      /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/control.go:161 +0xe59
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine.(*StorageEngine).Init.func1()
      /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine/control.go:88 +0x65
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /root/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x76
Goroutine 323 (finished) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1648 +0x82a
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2054 +0x84
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1595 +0x238
  testing.runTests()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2052 +0x896
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1925 +0xb57
  main.main()
      _testmain.go:93 +0x2bd
==================
--- FAIL: TestListWithCursor (0.00s)
    --- FAIL: TestListWithCursor/many_shards,_many_objects,_small_batch_size (30.51s)
        testing.go:1465: race detected during execution of test
    --- FAIL: TestListWithCursor/many_shards,_many_objects,_big_batch_size (30.51s)
        testing.go:1465: race detected during execution of test
FAIL

Reproduced https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/888/jobs/3 ``` ================== WARNING: DATA RACE Read at 0x00c0008a2d43 by goroutine 596: testing.(*common).logDepth() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1011 +0xbb testing.(*common).log() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1004 +0x99 testing.(*common).Logf() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1055 +0x62 testing.(*T).Logf() <autogenerated>:1 +0x69 go.uber.org/zap/zaptest.testingWriter.Write() /root/go/pkg/mod/go.uber.org/zap@v1.24.0/zaptest/logger.go:130 +0x11d go.uber.org/zap/zaptest.(*testingWriter).Write() <autogenerated>:1 +0x74 go.uber.org/zap/zapcore.(*ioCore).Write() /root/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/core.go:99 +0x192 go.uber.org/zap/zapcore.(*CheckedEntry).Write() /root/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:255 +0x2af go.uber.org/zap.(*Logger).Debug() /root/go/pkg/mod/go.uber.org/zap@v1.24.0/logger.go:212 +0x67 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).removeGarbage() /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:249 +0x20a git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).removeGarbage-fm() <autogenerated>:1 +0x47 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).tickRemover() /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:212 +0x18c git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).init.func1() /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:147 +0x4f Previous write at 0x00c0008a2d43 by goroutine 323: testing.tRunner.func1() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1582 +0x81a runtime.deferreturn() /opt/hostedtoolcache/go/1.21.0/x64/src/runtime/panic.go:477 +0x30 testing.(*T).Run.func1() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1648 +0x44 Goroutine 596 (running) created at: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).init() /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:147 +0x290 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).Init() /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/control.go:161 +0xe59 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine.(*StorageEngine).Init.func1() /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine/control.go:88 +0x65 golang.org/x/sync/errgroup.(*Group).Go.func1() /root/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x76 Goroutine 323 (finished) created at: testing.(*T).Run() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1648 +0x82a testing.runTests.func1() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2054 +0x84 testing.tRunner() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1595 +0x238 testing.runTests() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2052 +0x896 testing.(*M).Run() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1925 +0xb57 main.main() _testmain.go:93 +0x2bd ================== --- FAIL: TestListWithCursor (0.00s) --- FAIL: TestListWithCursor/many_shards,_many_objects,_small_batch_size (30.51s) testing.go:1465: race detected during execution of test --- FAIL: TestListWithCursor/many_shards,_many_objects,_big_batch_size (30.51s) testing.go:1465: race detected during execution of test FAIL ```

fyrchik reopened this issue

2023-08-16 07:40:59 +00:00

fyrchik referenced this issue

2023-08-16 07:41:03 +00:00

Data race in TestListWithCursor() #609

fyrchik commented

2023-08-16 07:41:27 +00:00

Author

Owner

No, my bad, different bug.

fyrchik closed this issue

2023-08-16 07:41:29 +00:00