Flaky test TestErrorTimeoutChecking #290

New issue

Closed

opened 2024-01-17 07:22:28 +00:00 by dkirillov · 0 comments

dkirillov commented

2024-01-17 07:22:28 +00:00

Member

Test TestErrorTimeoutChecking fails sometimes

Expected Behavior

Test is stable

Current Behavior

Test fails sometimes:

2024-01-17 01:01:30  FAIL
2024-01-17 01:01:30      git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/handler    coverage: 57.3% of statements
2024-01-17 01:01:30  FAIL    git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/handler    8.366s
2024-01-17 01:01:30  ok      git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer    0.426s    coverage: 40.1% of statements
2024-01-17 01:01:30  ok      git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption    0.008s    coverage: 43.2% of statements
2024-01-17 01:01:30  ok      git.frostfs.info/TrueCloudLab/frostfs-s3-gw/authmate    0.010s    coverage: 8.6% of statements
2024-01-17 01:01:30  ok      git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox    0.019s    coverage: 48.8% of statements
2024-01-17 01:01:30  --- FAIL: TestErrorTimeoutChecking (0.16s)
2024-01-17 01:01:30      --- FAIL: TestErrorTimeoutChecking/deadline_exceeded (0.16s)
2024-01-17 01:01:30  panic: runtime error: invalid memory address or nil pointer dereference [recovered]
2024-01-17 01:01:30      panic: runtime error: invalid memory address or nil pointer dereference
2024-01-17 01:01:30  [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x9ef1a2]
2024-01-17 01:01:30  
2024-01-17 01:01:30  goroutine 70 [running]:
2024-01-17 01:01:30  testing.tRunner.func1.2({0xcbd0e0, 0x16ed0a0})
2024-01-17 01:01:30      /usr/local/go/src/testing/testing.go:1526 +0x24e
2024-01-17 01:01:30  testing.tRunner.func1()
2024-01-17 01:01:30      /usr/local/go/src/testing/testing.go:1529 +0x39f
2024-01-17 01:01:30  panic({0xcbd0e0, 0x16ed0a0})
2024-01-17 01:01:30      /usr/local/go/src/runtime/panic.go:884 +0x213
2024-01-17 01:01:30  git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors.IsTimeoutError({0x0, 0x0})
2024-01-17 01:01:30      /home/ci/jenkins_home/workspace/release_v1.3_Bundle_frostfs_build_NB/frostfs-s3-gw/internal/frostfs/errors/errors.go:34 +0x22
2024-01-17 01:01:30  git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs.TestErrorTimeoutChecking.func2(0xc000483380?)
2024-01-17 01:01:30      /home/ci/jenkins_home/workspace/release_v1.3_Bundle_frostfs_build_NB/frostfs-s3-gw/internal/frostfs/frostfs_test.go:43 +0x7a
2024-01-17 01:01:30  testing.tRunner(0xc0004836c0, 0xed80b8)
2024-01-17 01:01:30      /usr/local/go/src/testing/testing.go:1576 +0x10b
2024-01-17 01:01:30  created by testing.(*T).Run
2024-01-17 01:01:30      /usr/local/go/src/testing/testing.go:1629 +0x3ea
2024-01-17 01:01:30  FAIL    git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs    0.189s
2024-01-17 01:01:30  ok      git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/crdt    0.011s    coverage: 61.5% of statements
2024-01-17 01:01:30  ok      git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/xml    0.022s    coverage: 70.0% of statements
2024-01-17 01:01:30  ok      git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/service/tree    0.189s    coverage: 28.3% of statements
2024-01-17 01:01:30  FAIL
2024-01-17 01:01:30  make[2]: *** [Makefile:72: test] Error 1
2024-01-17 01:01:30  make[2]: Leaving directory '/home/ci/jenkins_home/workspace/release_v1.3_Bundle_frostfs_build_NB/frostfs-s3-gw'
2024-01-17 01:01:30  dh_auto_test: error: make -j1 test returned exit code 2
2024-01-17 01:01:30  make[1]: *** [debian/rules:8: build] Error 25
2024-01-17 01:01:30  make[1]: Leaving directory '/home/ci/jenkins_home/workspace/release_v1.3_Bundle_frostfs_build_NB/frostfs-s3-gw'
2024-01-17 01:01:30  dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
2024-01-17 01:01:30  make: *** [Makefile:165: debpackage] Error 2

Steps to Reproduce (for bugs)

Run several times

Context

Couldn't build binary without passing tests in CI.

Regression

Probably no

Your Environment

Version used: master

Test [TestErrorTimeoutChecking](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/src/commit/c32220762f7c30d8917763088d3f04e3e35c7464/internal/frostfs/frostfs_test.go#L33) fails sometimes ## Expected Behavior Test is stable ## Current Behavior Test fails sometimes: ``` 2024-01-17 01:01:30 FAIL 2024-01-17 01:01:30 git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/handler coverage: 57.3% of statements 2024-01-17 01:01:30 FAIL git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/handler 8.366s 2024-01-17 01:01:30 ok git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer 0.426s coverage: 40.1% of statements 2024-01-17 01:01:30 ok git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption 0.008s coverage: 43.2% of statements 2024-01-17 01:01:30 ok git.frostfs.info/TrueCloudLab/frostfs-s3-gw/authmate 0.010s coverage: 8.6% of statements 2024-01-17 01:01:30 ok git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox 0.019s coverage: 48.8% of statements 2024-01-17 01:01:30 --- FAIL: TestErrorTimeoutChecking (0.16s) 2024-01-17 01:01:30 --- FAIL: TestErrorTimeoutChecking/deadline_exceeded (0.16s) 2024-01-17 01:01:30 panic: runtime error: invalid memory address or nil pointer dereference [recovered] 2024-01-17 01:01:30 panic: runtime error: invalid memory address or nil pointer dereference 2024-01-17 01:01:30 [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x9ef1a2] 2024-01-17 01:01:30 2024-01-17 01:01:30 goroutine 70 [running]: 2024-01-17 01:01:30 testing.tRunner.func1.2({0xcbd0e0, 0x16ed0a0}) 2024-01-17 01:01:30 /usr/local/go/src/testing/testing.go:1526 +0x24e 2024-01-17 01:01:30 testing.tRunner.func1() 2024-01-17 01:01:30 /usr/local/go/src/testing/testing.go:1529 +0x39f 2024-01-17 01:01:30 panic({0xcbd0e0, 0x16ed0a0}) 2024-01-17 01:01:30 /usr/local/go/src/runtime/panic.go:884 +0x213 2024-01-17 01:01:30 git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors.IsTimeoutError({0x0, 0x0}) 2024-01-17 01:01:30 /home/ci/jenkins_home/workspace/release_v1.3_Bundle_frostfs_build_NB/frostfs-s3-gw/internal/frostfs/errors/errors.go:34 +0x22 2024-01-17 01:01:30 git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs.TestErrorTimeoutChecking.func2(0xc000483380?) 2024-01-17 01:01:30 /home/ci/jenkins_home/workspace/release_v1.3_Bundle_frostfs_build_NB/frostfs-s3-gw/internal/frostfs/frostfs_test.go:43 +0x7a 2024-01-17 01:01:30 testing.tRunner(0xc0004836c0, 0xed80b8) 2024-01-17 01:01:30 /usr/local/go/src/testing/testing.go:1576 +0x10b 2024-01-17 01:01:30 created by testing.(*T).Run 2024-01-17 01:01:30 /usr/local/go/src/testing/testing.go:1629 +0x3ea 2024-01-17 01:01:30 FAIL git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs 0.189s 2024-01-17 01:01:30 ok git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/crdt 0.011s coverage: 61.5% of statements 2024-01-17 01:01:30 ok git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/xml 0.022s coverage: 70.0% of statements 2024-01-17 01:01:30 ok git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/service/tree 0.189s coverage: 28.3% of statements 2024-01-17 01:01:30 FAIL 2024-01-17 01:01:30 make[2]: *** [Makefile:72: test] Error 1 2024-01-17 01:01:30 make[2]: Leaving directory '/home/ci/jenkins_home/workspace/release_v1.3_Bundle_frostfs_build_NB/frostfs-s3-gw' 2024-01-17 01:01:30 dh_auto_test: error: make -j1 test returned exit code 2 2024-01-17 01:01:30 make[1]: *** [debian/rules:8: build] Error 25 2024-01-17 01:01:30 make[1]: Leaving directory '/home/ci/jenkins_home/workspace/release_v1.3_Bundle_frostfs_build_NB/frostfs-s3-gw' 2024-01-17 01:01:30 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2 2024-01-17 01:01:30 make: *** [Makefile:165: debpackage] Error 2 ``` ## Steps to Reproduce (for bugs) Run several times ## Context Couldn't build binary without passing tests in CI. ## Regression Probably no ## Your Environment  * Version used: [master](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/commit/c32220762f7c30d8917763088d3f04e3e35c7464)