Add logs-linter #526

Merged
fyrchik merged 3 commits from achuprov/frostfs-node:logs-linter into master 2024-09-04 19:51:01 +00:00
11 changed files with 67 additions and 23 deletions

View file

@ -14,10 +14,11 @@ jobs:
go-version: '1.21' go-version: '1.21'
cache: true cache: true
- name: golangci-lint - name: Install linters
uses: https://github.com/golangci/golangci-lint-action@v3 run: make lint-install

Can we use single line syntax for a single command run: make lint-install -- like in other places in this file.

Can we use single line syntax for a single command `run: make lint-install` -- like in other places in this file.
with:
fyrchik marked this conversation as resolved Outdated

See staticcheck example below, we would not like to duplicate install instructions.
Maybe introduce make extralinters-install? Or even make lint-install, it depends.

See `staticcheck` example below, we would not like to duplicate install instructions. Maybe introduce `make extralinters-install`? Or even `make lint-install`, it depends.

done

done
version: latest - name: Run linters
run: make lint
tests: tests:
name: Tests name: Tests

View file

@ -37,6 +37,14 @@ linters-settings:
alias: alias:
pkg: git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object pkg: git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object
alias: objectSDK alias: objectSDK
custom:
noliteral:
path: bin/external_linters.so

Will my local make lint continue to work if I don't have this?
Can we restrict it to actions somehow?

Will my local `make lint` continue to work if I don't have this? Can we restrict it to actions somehow?

I am not sure .bin is the right name, how about just bin (we already store protogen plugin there) or .cache (it is in the git ignore).

I am not sure `.bin` is the right name, how about just `bin` (we already store `protogen` plugin there) or `.cache` (it is in the git ignore).

Do we need to change it each time we update linter version? Can it be avoided?

Do we need to change it each time we update linter version? Can it be avoided?
original-url: git.frostfs.info/TrueCloudLab/linters.git
settings:
dstepanov-yadro marked this conversation as resolved Outdated

Invalid URL

Invalid URL

fixed

fixed

It is better to avoid commented out code in RP, especially without comments.

It is better to avoid commented out code in RP, especially without comments.
target-methods : ["reportFlushError", "reportError"]
disable-packages: ["codes", "err", "res","exec"]
constants-package: "git.frostfs.info/TrueCloudLab/frostfs-node/internal/logs"
linters: linters:
enable: enable:
@ -69,5 +77,6 @@ linters:
- gocognit - gocognit
- contextcheck - contextcheck
- importas - importas
- noliteral
disable-all: true disable-all: true
fast: false fast: false

View file

@ -8,7 +8,7 @@ HUB_IMAGE ?= truecloudlab/frostfs
HUB_TAG ?= "$(shell echo ${VERSION} | sed 's/^v//')" HUB_TAG ?= "$(shell echo ${VERSION} | sed 's/^v//')"
GO_VERSION ?= 1.21 GO_VERSION ?= 1.21
LINT_VERSION ?= 1.52.2 LINT_VERSION ?= 1.54.0
ARCH = amd64 ARCH = amd64
BIN = bin BIN = bin
@ -25,6 +25,10 @@ PKG_VERSION ?= $(shell echo $(VERSION) | sed "s/^v//" | \
sed -E "s/(.*)-(g[a-fA-F0-9]{6,8})(.*)/\1\3~\2/" | \ sed -E "s/(.*)-(g[a-fA-F0-9]{6,8})(.*)/\1\3~\2/" | \
sed "s/-/~/")-${OS_RELEASE} sed "s/-/~/")-${OS_RELEASE}
OUTPUT_LINT_DIR ?= $(shell pwd)/bin
acid-ant marked this conversation as resolved Outdated

Why we need this new line?

Why we need this new line?
LINT_DIR = $(OUTPUT_LINT_DIR)/golangci-lint-$(LINT_VERSION)
TMP_DIR := .cache
.PHONY: help all images dep clean fmts fmt imports test lint docker/lint .PHONY: help all images dep clean fmts fmt imports test lint docker/lint
prepare-release debpackage pre-commit unpre-commit prepare-release debpackage pre-commit unpre-commit
@ -131,9 +135,23 @@ test:
pre-commit-run: pre-commit-run:
@pre-commit run -a --hook-stage manual @pre-commit run -a --hook-stage manual
dstepanov-yadro marked this conversation as resolved Outdated

Invalid URL

Invalid URL

fixed

fixed
fyrchik marked this conversation as resolved Outdated

We will clone a git repo on each make lint.
This is not ideal, we should be able to do make lint even without the network.

We will clone a git repo on _each_ `make lint`. This is not ideal, we should be able to do `make lint` even without the network.

make lint command no longer needs internet connectivity

`make lint` command no longer needs internet connectivity

It is a bad idea to clone something in the repo root -- the comman can fail in the middle and we will have garbage. How about .cache or tmp?

It is a bad idea to clone something in the repo root -- the comman can fail in the middle and we will have garbage. How about `.cache` or `tmp`?

Why again .bin ? You can use $(BIN)

Why again `.bin` ? You can use $(BIN)
# Install linters

Some suggestions for this target:

  1. We can always fail unexpectedly, rm ./tmp/linters is a sane thing to do at the beginning.
  2. git clone can clone in a specific directory, no need to cd
  3. yes | rm can be replaced with -f flag for rm.
  4. Again, rm can remove arbitrary path, no need to cd.
  5. Same for make, see -C (it has a nice property of leaving the directory after the make)
  6. make can be replaced with $(MAKE), see https://www.gnu.org/software/make/manual/html_node/Recursion.html
  7. export CGO_ENABLED=1 is useful for 1 command (it does not affect go install -- I still have static binary on my machine. We can set it the same way as OUT_DIR
  8. tmp as a directory where we store stuff, can we a make variable (and I still believe .cache is better, as nothing is stored there manually, usually)
/r/f/node @c0640191fb• | 2 ❱ make lint-install && ldd ./bin/golangci-lint
...
        not a dynamic executable
Some suggestions for this target: 1. We can always fail unexpectedly, `rm ./tmp/linters` is a sane thing to do at the beginning. 2. `git clone` can clone in a specific directory, no need to `cd` 3. `yes | rm` can be replaced with `-f` flag for `rm`. 4. Again, `rm` can remove arbitrary path, no need to `cd`. 5. Same for make, see `-C` (it has a nice property of leaving the directory after the make) 6. `make` can be replaced with `$(MAKE)`, see https://www.gnu.org/software/make/manual/html_node/Recursion.html 7. `export CGO_ENABLED=1` is useful for 1 command (it does not affect `go install` -- I still have static binary on my machine. We can set it the same way as `OUT_DIR` 8. `tmp` as a directory where we store stuff, can we a make variable (and I still believe .cache is better, as nothing is stored there manually, usually) ``` /r/f/node @c0640191fb• | 2 ❱ make lint-install && ldd ./bin/golangci-lint ... not a dynamic executable ```

For (6) and why it is good: if I use custom make (not in my PATH), with $(MAKE) the same binary will be used)
https://stackoverflow.com/questions/50510278/makefile-why-always-use-make-instead-of-make

For (6) and why it is good: if I use custom make (not in my PATH), with $(MAKE) _the same_ binary will be used) https://stackoverflow.com/questions/50510278/makefile-why-always-use-make-instead-of-make

Also, from https://golangci-lint.run/contributing/new-linters/#how-to-add-a-private-linter-to-golangci-lint

This means that golangci-lint needs to be built for whatever machine you intend to run it on (cloning the golangci-lint repository and running a CGO_ENABLED=1 make build should do the trick for your machine).

Also, from https://golangci-lint.run/contributing/new-linters/#how-to-add-a-private-linter-to-golangci-lint >This means that golangci-lint needs to be built for whatever machine you intend to run it on (cloning the golangci-lint repository and running a CGO_ENABLED=1 make build should do the trick for your machine).
lint-install:
@mkdir -p $(TMP_DIR)
dstepanov-yadro marked this conversation as resolved Outdated

I think it's better to not export, just GOBIN=/path/bin go install ...

I think it's better to not export, just `GOBIN=/path/bin go install ...`
@rm -rf $(TMP_DIR)/linters
@git clone --depth 1 https://git.frostfs.info/TrueCloudLab/linters.git $(TMP_DIR)/linters
fyrchik marked this conversation as resolved Outdated

This is a common target across multiple our repos, I would leave it as is.

This is a common target across multiple our repos, I would leave it as is.

reverted

reverted
@@make -C $(TMP_DIR)/linters lib CGO_ENABLED=1 OUT_DIR=$(OUTPUT_LINT_DIR)
@rm -rf $(TMP_DIR)/linters
@rmdir $(TMP_DIR) 2>/dev/null || true
@CGO_ENABLED=1 GOBIN=$(LINT_DIR) go install github.com/golangci/golangci-lint/cmd/golangci-lint@v$(LINT_VERSION)

CGO_ENABLED=1 is what we miss here.

Cloning into '.cache/linters'...
remote: Enumerating objects: 27, done.
remote: Counting objects: 100% (27/27), done.
remote: Compressing objects: 100% (23/23), done.
remote: Total 27 (delta 3), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (27/27), 7.68 KiB | 7.68 MiB/s, done.
Resolving deltas: 100% (3/3), done.
make[1]: Entering directory '/repo/frostfs/node/.cache/linters'
make[1]: Leaving directory '/repo/frostfs/node/.cache/linters'
ERRO Unable to load custom analyzer noliteral:bin/lint-1.54.0/external_linters.so, plugin: not implemented
ERRO Running error: unknown linters: 'noliteral', run 'golangci-lint help linters' to see the list of supported linters
make: *** [Makefile:150: lint] Error 3
`CGO_ENABLED=1` is what we miss here. ``` Cloning into '.cache/linters'... remote: Enumerating objects: 27, done. remote: Counting objects: 100% (27/27), done. remote: Compressing objects: 100% (23/23), done. remote: Total 27 (delta 3), reused 0 (delta 0), pack-reused 0 Receiving objects: 100% (27/27), 7.68 KiB | 7.68 MiB/s, done. Resolving deltas: 100% (3/3), done. make[1]: Entering directory '/repo/frostfs/node/.cache/linters' make[1]: Leaving directory '/repo/frostfs/node/.cache/linters' ERRO Unable to load custom analyzer noliteral:bin/lint-1.54.0/external_linters.so, plugin: not implemented ERRO Running error: unknown linters: 'noliteral', run 'golangci-lint help linters' to see the list of supported linters make: *** [Makefile:150: lint] Error 3 ```
# Run linters # Run linters
lint: lint:
@golangci-lint --timeout=5m run @if [ ! -d "$(LINT_DIR)" ]; then \
echo "Run make lint-install"; \
exit 1; \
fi
$(LINT_DIR)/golangci-lint run

Previously a timeout was different, whats the reason for change?
Anyway, we have it in the .golangci.yml, why do we need it here?

Previously a timeout was different, whats the reason for change? Anyway, we have it in the `.golangci.yml`, why do we need it here?
# Install staticcheck # Install staticcheck
staticcheck-install: staticcheck-install:

View file

@ -333,7 +333,7 @@ func (c *cfg) netmapInitLocalNodeState(epoch uint64) (*netmapSDK.NodeInfo, bool,
if nmState != candidateState { if nmState != candidateState {
// This happens when the node was switched to maintenance without epoch tick. // This happens when the node was switched to maintenance without epoch tick.
// We expect it to continue staying in maintenance. // We expect it to continue staying in maintenance.
c.log.Info("candidate status is different from the netmap status, the former takes priority", c.log.Info(logs.CandidateStatusPriority,
zap.String("netmap", nmState), zap.String("netmap", nmState),
zap.String("candidate", candidateState)) zap.String("candidate", candidateState))
} }

View file

@ -495,6 +495,22 @@ const (
FrostFSNodeNodeIsUnderMaintenanceSkipInitialBootstrap = "the node is under maintenance, skip initial bootstrap" FrostFSNodeNodeIsUnderMaintenanceSkipInitialBootstrap = "the node is under maintenance, skip initial bootstrap"
EngineCouldNotChangeShardModeToDisabled = "could not change shard mode to disabled" EngineCouldNotChangeShardModeToDisabled = "could not change shard mode to disabled"
NetmapNodeAlreadyInCandidateListOnlineSkipInitialBootstrap = "the node is already in candidate list with online state, skip initial bootstrap" NetmapNodeAlreadyInCandidateListOnlineSkipInitialBootstrap = "the node is already in candidate list with online state, skip initial bootstrap"
RPConnectionLost = "RPC connection lost, attempting reconnect"
RPCNodeSwitchFailure = "can't switch RPC node"
FSTreeCantReadFile = "can't read a file"
FSTreeCantUnmarshalObject = "can't unmarshal an object"
FSTreeCantFushObjectBlobstor = "can't flush an object to blobstor"
FSTreeCantUpdateID = "can't update object storage ID"
FSTreeCantDecodeDBObjectAddress = "can't decode object address from the DB"
PutSingleRedirectFailure = "failed to redirect PutSingle request"
StorageIDRetrievalFailure = "can't get storage ID from metabase"
ObjectRemovalFailureBlobStor = "can't remove object from blobStor"
CandidateStatusPriority = "candidate status is different from the netmap status, the former takes priority"
TombstoneExpirationParseFailure = "tombstone getter: could not parse tombstone expiration epoch"
FrostFSNodeCantUpdateObjectStorageID = "can't update object storage ID"
FrostFSNodeCantFlushObjectToBlobstor = "can't flush an object to blobstor"
FrostFSNodeCantDecodeObjectAddressFromDB = "can't decode object address from the DB" // Error in ../node/cmd/frostfs-node/morph.go
FrostFSNodeCantUnmarshalObjectFromDB = "can't unmarshal an object from the DB" // Error in ../node/cmd/frostfs-node/morph.go
RuntimeSoftMemoryLimitUpdated = "soft runtime memory limit value updated" RuntimeSoftMemoryLimitUpdated = "soft runtime memory limit value updated"
RuntimeSoftMemoryDefinedWithGOMEMLIMIT = "soft runtime memory defined with GOMEMLIMIT environment variable, config value skipped" RuntimeSoftMemoryDefinedWithGOMEMLIMIT = "soft runtime memory defined with GOMEMLIMIT environment variable, config value skipped"
) )

View file

@ -92,7 +92,7 @@ func (s *Shard) deleteFromBlobstorSafe(ctx context.Context, addr oid.Address) {
res, err := s.metaBase.StorageID(ctx, sPrm) res, err := s.metaBase.StorageID(ctx, sPrm)
if err != nil { if err != nil {
s.log.Debug("can't get storage ID from metabase", s.log.Debug(logs.StorageIDRetrievalFailure,
zap.Stringer("object", addr), zap.Stringer("object", addr),
zap.String("error", err.Error())) zap.String("error", err.Error()))
} }
@ -104,7 +104,7 @@ func (s *Shard) deleteFromBlobstorSafe(ctx context.Context, addr oid.Address) {
_, err = s.blobStor.Delete(ctx, delPrm) _, err = s.blobStor.Delete(ctx, delPrm)
if err != nil { if err != nil {
s.log.Debug("can't remove object from blobStor", s.log.Debug(logs.ObjectRemovalFailureBlobStor,
zap.Stringer("object_address", addr), zap.Stringer("object_address", addr),
zap.String("error", err.Error())) zap.String("error", err.Error()))
} }

View file

@ -188,7 +188,7 @@ func (c *cache) flushObject(ctx context.Context, obj *objectSDK.Object, data []b
if err != nil { if err != nil {
if !errors.Is(err, common.ErrNoSpace) && !errors.Is(err, common.ErrReadOnly) && if !errors.Is(err, common.ErrNoSpace) && !errors.Is(err, common.ErrReadOnly) &&
!errors.Is(err, blobstor.ErrNoPlaceFound) { !errors.Is(err, blobstor.ErrNoPlaceFound) {
c.reportFlushError("can't flush an object to blobstor", c.reportFlushError(logs.FrostFSNodeCantFlushObjectToBlobstor,
addr.EncodeToString(), err) addr.EncodeToString(), err)
} }
return err return err
@ -200,7 +200,7 @@ func (c *cache) flushObject(ctx context.Context, obj *objectSDK.Object, data []b
_, err = c.metabase.UpdateStorageID(updPrm) _, err = c.metabase.UpdateStorageID(updPrm)
if err != nil { if err != nil {
c.reportFlushError("can't update object storage ID", c.reportFlushError(logs.FrostFSNodeCantUpdateObjectStorageID,
addr.EncodeToString(), err) addr.EncodeToString(), err)
} }
return err return err
@ -230,7 +230,7 @@ func (c *cache) flush(ctx context.Context, ignoreErrors bool) error {
for it.Rewind(); it.Valid(); it.Next() { for it.Rewind(); it.Valid(); it.Next() {
if got, want := int(it.Item().KeySize()), len(key); got != want { if got, want := int(it.Item().KeySize()), len(key); got != want {
err := fmt.Errorf("invalid db key len: got %d, want %d", got, want) err := fmt.Errorf("invalid db key len: got %d, want %d", got, want)
c.reportFlushError("can't decode object address from the DB", hex.EncodeToString(it.Item().Key()), metaerr.Wrap(err)) c.reportFlushError(logs.FrostFSNodeCantDecodeObjectAddressFromDB, hex.EncodeToString(it.Item().Key()), metaerr.Wrap(err))
if ignoreErrors { if ignoreErrors {
continue continue
} }
@ -240,7 +240,7 @@ func (c *cache) flush(ctx context.Context, ignoreErrors bool) error {
var obj objectSDK.Object var obj objectSDK.Object
if err := obj.Unmarshal(data); err != nil { if err := obj.Unmarshal(data); err != nil {
copy(key[:], it.Item().Key()) copy(key[:], it.Item().Key())
c.reportFlushError("can't unmarshal an object from the DB", key.address().EncodeToString(), metaerr.Wrap(err)) c.reportFlushError(logs.FrostFSNodeCantUnmarshalObjectFromDB, key.address().EncodeToString(), metaerr.Wrap(err))
if ignoreErrors { if ignoreErrors {
return nil return nil
} }

View file

@ -194,7 +194,7 @@ func (c *cache) flushFSTree(ctx context.Context, ignoreErrors bool) error {
data, err := f() data, err := f()
if err != nil { if err != nil {
c.reportFlushError("can't read a file", sAddr, metaerr.Wrap(err)) c.reportFlushError(logs.FSTreeCantReadFile, sAddr, metaerr.Wrap(err))
if ignoreErrors { if ignoreErrors {
return nil return nil
} }
@ -204,7 +204,7 @@ func (c *cache) flushFSTree(ctx context.Context, ignoreErrors bool) error {
var obj objectSDK.Object var obj objectSDK.Object
err = obj.Unmarshal(data) err = obj.Unmarshal(data)
if err != nil { if err != nil {
c.reportFlushError("can't unmarshal an object", sAddr, metaerr.Wrap(err)) c.reportFlushError(logs.FSTreeCantUnmarshalObject, sAddr, metaerr.Wrap(err))
if ignoreErrors { if ignoreErrors {
return nil return nil
} }
@ -268,7 +268,7 @@ func (c *cache) flushObject(ctx context.Context, obj *objectSDK.Object, data []b
if err != nil { if err != nil {
if !errors.Is(err, common.ErrNoSpace) && !errors.Is(err, common.ErrReadOnly) && if !errors.Is(err, common.ErrNoSpace) && !errors.Is(err, common.ErrReadOnly) &&
!errors.Is(err, blobstor.ErrNoPlaceFound) { !errors.Is(err, blobstor.ErrNoPlaceFound) {
c.reportFlushError("can't flush an object to blobstor", c.reportFlushError(logs.FSTreeCantFushObjectBlobstor,
addr.EncodeToString(), err) addr.EncodeToString(), err)
} }
return err return err
@ -280,7 +280,7 @@ func (c *cache) flushObject(ctx context.Context, obj *objectSDK.Object, data []b
_, err = c.metabase.UpdateStorageID(updPrm) _, err = c.metabase.UpdateStorageID(updPrm)
if err != nil { if err != nil {
c.reportFlushError("can't update object storage ID", c.reportFlushError(logs.FSTreeCantUpdateID,
addr.EncodeToString(), err) addr.EncodeToString(), err)
} }
return err return err
@ -315,7 +315,7 @@ func (c *cache) flush(ctx context.Context, ignoreErrors bool) error {
for k, data := cs.Seek(nil); k != nil; k, data = cs.Next() { for k, data := cs.Seek(nil); k != nil; k, data = cs.Next() {
sa := string(k) sa := string(k)
if err := addr.DecodeString(sa); err != nil { if err := addr.DecodeString(sa); err != nil {
c.reportFlushError("can't decode object address from the DB", sa, metaerr.Wrap(err)) c.reportFlushError(logs.FSTreeCantDecodeDBObjectAddress, sa, metaerr.Wrap(err))
if ignoreErrors { if ignoreErrors {
continue continue
} }
@ -324,7 +324,7 @@ func (c *cache) flush(ctx context.Context, ignoreErrors bool) error {
var obj objectSDK.Object var obj objectSDK.Object
if err := obj.Unmarshal(data); err != nil { if err := obj.Unmarshal(data); err != nil {
c.reportFlushError("can't unmarshal an object from the DB", sa, metaerr.Wrap(err)) c.reportFlushError(logs.FSTreeCantDecodeDBObjectAddress, sa, metaerr.Wrap(err))
if ignoreErrors { if ignoreErrors {
continue continue
} }

View file

@ -250,9 +250,9 @@ routeloop:
} }
func (s *subscriber) switchEndpoint(ctx context.Context, finishCh chan<- bool) (bool, <-chan rpcclient.Notification) { func (s *subscriber) switchEndpoint(ctx context.Context, finishCh chan<- bool) (bool, <-chan rpcclient.Notification) {
s.log.Info("RPC connection lost, attempting reconnect") s.log.Info(logs.RPConnectionLost)
if !s.client.SwitchRPC(ctx) { if !s.client.SwitchRPC(ctx) {
s.log.Error("can't switch RPC node") s.log.Error(logs.RPCNodeSwitchFailure)
return false, nil return false, nil
} }

View file

@ -343,7 +343,7 @@ func (s *Service) redirectPutSingleRequest(ctx context.Context,
if err != nil { if err != nil {
objID, _ := obj.ID() objID, _ := obj.ID()
cnrID, _ := obj.ContainerID() cnrID, _ := obj.ContainerID()
s.log.Warn("failed to redirect PutSingle request", s.log.Warn(logs.PutSingleRedirectFailure,
zap.Error(err), zap.Error(err),
zap.Stringer("address", addr), zap.Stringer("address", addr),
zap.Stringer("object_id", objID), zap.Stringer("object_id", objID),

View file

@ -78,7 +78,7 @@ func (g *ExpirationChecker) handleTS(addr string, ts *objectSDK.Object, reqEpoch
epoch, err := strconv.ParseUint(atr.Value(), 10, 64) epoch, err := strconv.ParseUint(atr.Value(), 10, 64)
if err != nil { if err != nil {
g.log.Warn( g.log.Warn(
"tombstone getter: could not parse tombstone expiration epoch", logs.TombstoneExpirationParseFailure,
zap.Error(err), zap.Error(err),
) )