"database not open" error after node restarted #13

New issue

Closed

opened 2023-01-11 16:01:51 +00:00 by anikeev-yadro · 0 comments

anikeev-yadro commented

2023-01-11 16:01:51 +00:00

(Migrated from github.com)

Steps to Reproduce (for bugs)

1.Restart neofs-storage service
2.Logs filled a lot of errors:

Jan 11 15:08:08 vedi neofs-node[1368627]: 2023-01-11T15:08:08.588Z        info        neofs-node/main.go:76        initializing gRPC service...
Jan 11 15:08:08 vedi neofs-node[1368627]: 2023-01-11T15:08:08.589Z        warn        engine/engine.go:160        could not put object to active blobovnicza        {"shard_id": "FRNsLKdPMxv3yaRtFVyXtJ", "error count": 2, "error": "database not open"}

3.Then shard changed mode to degraded-read-only.

Note: It was happened after some failover tests (load on AZ and restart BUKI) have run.

Logs

journal_vedi.tar.xz.zip
journal_glagoli.tar.xz.zip
journal_buki.tar.xz.zip
journal_az.tar.xz.zip

Versions

NeoFS Storage node
Version: v0.34.0-144-g7c3ab4d1-dirty
GoVersion: go1.18.4

Your Environment

Server setup and configuration:
HW, 4 servers, 4 SN, 4 http qw, 4 s3 gw

Operating System and version (uname -a):
linux vedi 5.10.0-16-amd64 https://github.com/nspcc-dev/neofs-node/issues/1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux

## Steps to Reproduce (for bugs) 1.Restart neofs-storage service 2.Logs filled a lot of errors: ``` Jan 11 15:08:08 vedi neofs-node[1368627]: 2023-01-11T15:08:08.588Z info neofs-node/main.go:76 initializing gRPC service... Jan 11 15:08:08 vedi neofs-node[1368627]: 2023-01-11T15:08:08.589Z warn engine/engine.go:160 could not put object to active blobovnicza {"shard_id": "FRNsLKdPMxv3yaRtFVyXtJ", "error count": 2, "error": "database not open"} ``` 3.Then shard changed mode to degraded-read-only. **Note:** It was happened after some failover tests (load on AZ and restart BUKI) have run. ## Logs [journal_vedi.tar.xz.zip](https://github.com/TrueCloudLab/frostfs-node/files/10393996/journal_vedi.tar.xz.zip) [journal_glagoli.tar.xz.zip](https://github.com/TrueCloudLab/frostfs-node/files/10394001/journal_glagoli.tar.xz.zip) [journal_buki.tar.xz.zip](https://github.com/TrueCloudLab/frostfs-node/files/10394002/journal_buki.tar.xz.zip) [journal_az.tar.xz.zip](https://github.com/TrueCloudLab/frostfs-node/files/10394003/journal_az.tar.xz.zip) ## Versions ``` NeoFS Storage node Version: v0.34.0-144-g7c3ab4d1-dirty GoVersion: go1.18.4 ``` ## Your Environment Server setup and configuration: HW, 4 servers, 4 SN, 4 http qw, 4 s3 gw Operating System and version (uname -a): linux vedi 5.10.0-16-amd64 https://github.com/nspcc-dev/neofs-node/issues/1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux