TrueCloudLab/frostfs-node

Fork 27

`TestFlush` is flaky #667

New issue

Closed

opened 2023-08-30 13:47:55 +00:00 by fyrchik · 2 comments

fyrchik commented

2023-08-30 13:47:55 +00:00

Owner

https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1410/jobs/3

--- FAIL: TestFlush (107.39s)
    --- FAIL: TestFlush/ignore_errors (97.50s)
        --- FAIL: TestFlush/ignore_errors/fs,_read_error (14.31s)
            flush.go:92: 
                	Error Trace:	/workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachetest/flush.go:92
                	Error:      	Not equal: 
                	            	expected: 0x0
                	            	actual  : 0x1
                	Test:       	TestFlush/ignore_errors/fs,_read_error

https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1410/jobs/3 ``` --- FAIL: TestFlush (107.39s) --- FAIL: TestFlush/ignore_errors (97.50s) --- FAIL: TestFlush/ignore_errors/fs,_read_error (14.31s) flush.go:92: Error Trace: /workspace/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachetest/flush.go:92 Error: Not equal: expected: 0x0 actual : 0x1 Test: TestFlush/ignore_errors/fs,_read_error ```

fyrchik added the

bug

triage

labels 2023-08-30 13:47:55 +00:00

fyrchik added this to the v0.37.0 milestone 2023-08-30 13:47:59 +00:00

ale64bit was assigned by fyrchik

2023-08-30 13:48:05 +00:00

acid-ant was assigned by fyrchik

2023-09-05 06:24:53 +00:00

ale64bit was unassigned by fyrchik

2023-09-05 06:24:53 +00:00

fyrchik commented

2023-09-05 06:25:13 +00:00

Author

Owner

Another example https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1572/jobs/3

acid-ant was unassigned by fyrchik

2023-09-05 11:14:36 +00:00

aarifullin was assigned by fyrchik

2023-09-05 11:14:36 +00:00

fyrchik modified the milestone from v0.37.0 to v0.38.0

2023-09-12 07:49:02 +00:00

aarifullin commented

2023-09-14 09:15:05 +00:00

Member

I tried to reproduce the bug many times and no way:

for ((i=1; i<=100; i++)) do
      go clean -testcache;
      /usr/local/go/bin/go test -timeout 300s -run ^TestFlush/ignore_errors/fs,_read_error$ git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebbolt -race;
done

What do you think about the idea to add logging within WithReportErrorFunc. We can see where is the problem place

func TestFlush(t *testing.T) {
	testLog := test.NewLogger(t, true)
	createCacheFn := func(t *testing.T, smallSize uint64, mb *meta.DB,
              bs writecache.MainStorage, opts ...Option) writecache.Cache {
		return New(
			append([]Option{
				WithLogger(testLog),
				WithPath(filepath.Join(t.TempDir(), "writecache")),
				WithSmallObjectSize(smallSize),
				WithMetabase(mb),
				WithBlobstor(bs),
			}, opts...)...)
	}

	errCountOpt := func() (Option, *atomic.Uint32) {
		cnt := &atomic.Uint32{}
		return WithReportErrorFunc(func(msg string, err error) {
            testLog.Info("Report error: " + msg, zap.Error(err))
			cnt.Add(1)
		}), cnt
	}

I tried to reproduce the bug many times and no way: ```bash for ((i=1; i<=100; i++)) do go clean -testcache; /usr/local/go/bin/go test -timeout 300s -run ^TestFlush/ignore_errors/fs,_read_error$ git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/writecachebbolt -race; done ``` What do you think about the idea to add logging within [WithReportErrorFunc](https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/local_object_storage/writecache/writecachebbolt/flush_test.go#L36). We can see where is the problem place ```golang func TestFlush(t *testing.T) { testLog := test.NewLogger(t, true) createCacheFn := func(t *testing.T, smallSize uint64, mb *meta.DB, bs writecache.MainStorage, opts ...Option) writecache.Cache { return New( append([]Option{ WithLogger(testLog), WithPath(filepath.Join(t.TempDir(), "writecache")), WithSmallObjectSize(smallSize), WithMetabase(mb), WithBlobstor(bs), }, opts...)...) } errCountOpt := func() (Option, *atomic.Uint32) { cnt := &atomic.Uint32{} return WithReportErrorFunc(func(msg string, err error) { testLog.Info("Report error: " + msg, zap.Error(err)) cnt.Add(1) }), cnt } ```

aarifullin referenced this issue

2023-09-14 11:13:17 +00:00

[#667] writecache: Add logs for report error func in tests #689

fyrchik referenced this issue from a commit

2023-09-14 17:02:57 +00:00

[#667] writecache: Add logs for report error func in tests

fyrchik added

frostfs-node

and removed

triage

labels 2023-10-06 08:27:39 +00:00

fyrchik changed title from ~~TestFlush is flaky~~ to TestFlush is flaky