One node cannot start in 30 min due to insufficient funds after all nodes shutdown then start #472

New issue

Closed

opened 2023-06-27 06:44:39 +00:00 by anikeev-yadro · 1 comment

anikeev-yadro commented

2023-06-27 06:44:39 +00:00

Member

Expected Behavior

All nodes should start successfull after all nodes shutdown then start.

Current Behavior

One node cannot start in 30 min due to insufficient funds after all nodes shutdown then start.

Steps to Reproduce (for bugs)

1.Correctly shutdown all nodes (HW)
2.Start all nodes
3.One node cannot start in 30 min due to insufficient funs:

Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z        info        frostfs-node/main.go:75        morph notifications service has been successfully initialized
Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z        info        frostfs-node/main.go:114        starting NATS service...
Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z        info        frostfs-node/main.go:118        NATS service started successfully
Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z        info        frostfs-node/main.go:114        starting gRPC service...
Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z        info        frostfs-node/main.go:114        starting notary service...
Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z        info        frostfs-node/grpc.go:103        start listening gRPC endpoint        {"endpoint": "192.168.201.10:8080"}
Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z        info        frostfs-node/grpc.go:103        start listening gRPC endpoint        {"endpoint": "192.168.201.110:8080"}
Jun 26 11:35:07 dobro kernel: audit: type=1131 audit(1687779307.535:943): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=frostfs-storage comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jun 26 11:35:07 dobro systemd[1]: frostfs-storage.service: Failed with result 'exit-code'.
Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.523Z        info        frostfs-node/grpc.go:103        start listening gRPC endpoint        {"endpoint": "10.78.71.110:8080"}
Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.523Z        info        frostfs-node/grpc.go:103        start listening gRPC endpoint        {"endpoint": "10.78.70.110:8080"}
Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.523Z        info        frostfs-node/grpc.go:103        start listening gRPC endpoint        {"endpoint": "127.0.0.1:8080"}
Jun 26 11:35:07 dobro frostfs-node[23724]: 2023/06/26 11:35:07 can't make notary deposit: Block or transaction validation failed. (-504) - insufficient funds
Jun 26 11:35:07 dobro systemd[1]: frostfs-storage.service: Consumed 7.548s CPU time.
Jun 26 11:35:12 dobro systemd[1]: frostfs-storage.service: Scheduled restart job, restart counter is at 108.

Regression

Version

FrostFS Storage node
Version: v0.22.1-2173-g1cae03c4-dirty
GoVersion: go1.20.5

Your Environment

HW
16 nodes

## Expected Behavior All nodes should start successfull after all nodes shutdown then start. ## Current Behavior One node cannot start in 30 min due to insufficient funds after all nodes shutdown then start. ## Steps to Reproduce (for bugs) 1.Correctly shutdown all nodes (HW) 2.Start all nodes 3.One node cannot start in 30 min due to insufficient funs: ``` Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z info frostfs-node/main.go:75 morph notifications service has been successfully initialized Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z info frostfs-node/main.go:114 starting NATS service... Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z info frostfs-node/main.go:118 NATS service started successfully Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z info frostfs-node/main.go:114 starting gRPC service... Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z info frostfs-node/main.go:114 starting notary service... Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z info frostfs-node/grpc.go:103 start listening gRPC endpoint {"endpoint": "192.168.201.10:8080"} Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.522Z info frostfs-node/grpc.go:103 start listening gRPC endpoint {"endpoint": "192.168.201.110:8080"} Jun 26 11:35:07 dobro kernel: audit: type=1131 audit(1687779307.535:943): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=frostfs-storage comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' Jun 26 11:35:07 dobro systemd[1]: frostfs-storage.service: Failed with result 'exit-code'. Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.523Z info frostfs-node/grpc.go:103 start listening gRPC endpoint {"endpoint": "10.78.71.110:8080"} Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.523Z info frostfs-node/grpc.go:103 start listening gRPC endpoint {"endpoint": "10.78.70.110:8080"} Jun 26 11:35:07 dobro frostfs-node[23724]: 2023-06-26T11:35:07.523Z info frostfs-node/grpc.go:103 start listening gRPC endpoint {"endpoint": "127.0.0.1:8080"} Jun 26 11:35:07 dobro frostfs-node[23724]: 2023/06/26 11:35:07 can't make notary deposit: Block or transaction validation failed. (-504) - insufficient funds Jun 26 11:35:07 dobro systemd[1]: frostfs-storage.service: Consumed 7.548s CPU time. Jun 26 11:35:12 dobro systemd[1]: frostfs-storage.service: Scheduled restart job, restart counter is at 108. ``` ## Regression No ## Version ``` FrostFS Storage node Version: v0.22.1-2173-g1cae03c4-dirty GoVersion: go1.20.5 ``` ## Your Environment HW 16 nodes