TrueCloudLab
/
frostfs-node
18
1
Fork 20
Code Issues 88 Pull Requests 8 Actions Releases Activity

TestListWithCursor is flaky #326

New Issue
Closed
opened 2023-05-05 07:28:53 +00:00 by fyrchik · 3 comments
fyrchik commented 2023-05-05 07:28:53 +00:00
Owner

Cut output for brevity:

--- FAIL: TestListWithCursor (5.30s)
    --- FAIL: TestListWithCursor/many_shards,_many_objects,_big_batch_size (2.34s)
        list_test.go:117:
                Error Trace:    /repo/frostfs/node/pkg/local_object_storage/engine/list_test.go:117
                Error:          Not equal:
                                expected: []object.AddressWithType{object.AddressWithType{Address:oid.Address{cnr:cid.ID
Cut output for brevity: ``` --- FAIL: TestListWithCursor (5.30s) --- FAIL: TestListWithCursor/many_shards,_many_objects,_big_batch_size (2.34s) list_test.go:117: Error Trace: /repo/frostfs/node/pkg/local_object_storage/engine/list_test.go:117 Error: Not equal: expected: []object.AddressWithType{object.AddressWithType{Address:oid.Address{cnr:cid.ID ```
fyrchik added the
bug
triage
 labels 2023-05-05 07:28:53 +00:00
aarifullin was assigned by fyrchik 2023-05-05 07:28:53 +00:00
fyrchik added this to the v0.37.0 milestone 2023-05-05 07:29:13 +00:00
fyrchik modified the milestone from v0.37.0 to v0.38.0 2023-06-19 08:02:55 +00:00
fyrchik commented 2023-06-27 11:13:05 +00:00
Poster
Owner

Cannot reproduce.

Cannot reproduce.
fyrchik commented 2023-08-16 07:40:53 +00:00
Poster
Owner

Reproduced
https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/888/jobs/3

==================
WARNING: DATA RACE
Read at 0x00c0008a2d43 by goroutine 596:
  testing.(*common).logDepth()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1011 +0xbb
  testing.(*common).log()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1004 +0x99
  testing.(*common).Logf()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1055 +0x62
  testing.(*T).Logf()
      <autogenerated>:1 +0x69
  go.uber.org/zap/zaptest.testingWriter.Write()
      /root/go/pkg/mod/go.uber.org/zap@v1.24.0/zaptest/logger.go:130 +0x11d
  go.uber.org/zap/zaptest.(*testingWriter).Write()
      <autogenerated>:1 +0x74
  go.uber.org/zap/zapcore.(*ioCore).Write()
      /root/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/core.go:99 +0x192
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      /root/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:255 +0x2af
  go.uber.org/zap.(*Logger).Debug()
      /root/go/pkg/mod/go.uber.org/zap@v1.24.0/logger.go:212 +0x67
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).removeGarbage()
      /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:249 +0x20a
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).removeGarbage-fm()
      <autogenerated>:1 +0x47
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).tickRemover()
      /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:212 +0x18c
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).init.func1()
      /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:147 +0x4f
Previous write at 0x00c0008a2d43 by goroutine 323:
  testing.tRunner.func1()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1582 +0x81a
  runtime.deferreturn()
      /opt/hostedtoolcache/go/1.21.0/x64/src/runtime/panic.go:477 +0x30
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1648 +0x44
Goroutine 596 (running) created at:
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).init()
      /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:147 +0x290
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).Init()
      /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/control.go:161 +0xe59
  git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine.(*StorageEngine).Init.func1()
      /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine/control.go:88 +0x65
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /root/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x76
Goroutine 323 (finished) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1648 +0x82a
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2054 +0x84
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1595 +0x238
  testing.runTests()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2052 +0x896
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1925 +0xb57
  main.main()
      _testmain.go:93 +0x2bd
==================
--- FAIL: TestListWithCursor (0.00s)
    --- FAIL: TestListWithCursor/many_shards,_many_objects,_small_batch_size (30.51s)
        testing.go:1465: race detected during execution of test
    --- FAIL: TestListWithCursor/many_shards,_many_objects,_big_batch_size (30.51s)
        testing.go:1465: race detected during execution of test
FAIL
Reproduced https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/888/jobs/3 ``` ================== WARNING: DATA RACE Read at 0x00c0008a2d43 by goroutine 596: testing.(*common).logDepth() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1011 +0xbb testing.(*common).log() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1004 +0x99 testing.(*common).Logf() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1055 +0x62 testing.(*T).Logf() <autogenerated>:1 +0x69 go.uber.org/zap/zaptest.testingWriter.Write() /root/go/pkg/mod/go.uber.org/zap@v1.24.0/zaptest/logger.go:130 +0x11d go.uber.org/zap/zaptest.(*testingWriter).Write() <autogenerated>:1 +0x74 go.uber.org/zap/zapcore.(*ioCore).Write() /root/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/core.go:99 +0x192 go.uber.org/zap/zapcore.(*CheckedEntry).Write() /root/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:255 +0x2af go.uber.org/zap.(*Logger).Debug() /root/go/pkg/mod/go.uber.org/zap@v1.24.0/logger.go:212 +0x67 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).removeGarbage() /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:249 +0x20a git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).removeGarbage-fm() <autogenerated>:1 +0x47 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).tickRemover() /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:212 +0x18c git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).init.func1() /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:147 +0x4f Previous write at 0x00c0008a2d43 by goroutine 323: testing.tRunner.func1() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1582 +0x81a runtime.deferreturn() /opt/hostedtoolcache/go/1.21.0/x64/src/runtime/panic.go:477 +0x30 testing.(*T).Run.func1() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1648 +0x44 Goroutine 596 (running) created at: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*gc).init() /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/gc.go:147 +0x290 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).Init() /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/control.go:161 +0xe59 git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine.(*StorageEngine).Init.func1() /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine/control.go:88 +0x65 golang.org/x/sync/errgroup.(*Group).Go.func1() /root/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x76 Goroutine 323 (finished) created at: testing.(*T).Run() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1648 +0x82a testing.runTests.func1() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2054 +0x84 testing.tRunner() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1595 +0x238 testing.runTests() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2052 +0x896 testing.(*M).Run() /opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1925 +0xb57 main.main() _testmain.go:93 +0x2bd ================== --- FAIL: TestListWithCursor (0.00s) --- FAIL: TestListWithCursor/many_shards,_many_objects,_small_batch_size (30.51s) testing.go:1465: race detected during execution of test --- FAIL: TestListWithCursor/many_shards,_many_objects,_big_batch_size (30.51s) testing.go:1465: race detected during execution of test FAIL ```
fyrchik reopened this issue 2023-08-16 07:40:59 +00:00
fyrchik commented 2023-08-16 07:41:27 +00:00
Poster
Owner

No, my bad, different bug.

No, my bad, different bug.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
support/v0.38
support/v0.37
support/v0.36
support/v0.34
support/v0.30
support/v0.27
v0.42.0-rc.1
v0.38.8
v0.41.0
v0.38.7
v0.40.0
v0.39.0
v0.38.6
v0.38.5
v0.38.4
v0.38.3
v0.38.2
v0.38.1
v0.38.0
v0.38.0-rc.2
v0.38.0-rc.1
v0.37.0
v0.37.0-rc.1
v0.36.0
v0.34.0
Labels
Clear labels   
P0

Highest   
P1

High   
P2

Medium   
P3

Low   
badger

Badger DB related issues   
frostfs-adm
   
frostfs-cli
   
frostfs-ir
   
frostfs-lens
   
frostfs-node
   
good first issue

Good for newcomers   
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff   
blocked

The issue or PR is blocked by something   
bug

Something is not working   
config

Configuration format update or breaking change   
discussion

Talking about something in order to reach a decision or to exchange ideas   
documentation

Documentation related   
duplicate

This issue or pull request already exists   
enhancement

New feature   
go

Language features adoption   
help wanted

Extra attention is needed   
internal
   
invalid

Something is wrong   
kludge

This is a kludge and we know it   
observability

Related to metrics, tracing and logs   
perfomance

Perfomance related   
question

More information is needed   
refactoring

Refactoring   
wontfix

This won't be fixed
No Label
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No Milestone
v0.38.0
Assignees
Clear assignees
No Assignees
aarifullin
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#326
There is no content yet.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?