TrueCloudLab/frostfs-node
21
1
Fork 27
Code Issues 104 Pull requests 4 Projects Releases 2 Activity Actions
4686 commits 9 branches 96 tags 51 MiB
Commit graph

105 commits

Author SHA1 Message Date
Dmitrii Stepanov
d45d086acd [#1129] policer: Add EC chunk replication 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-16 16:28:48 +03:00
Ekaterina Lebedeva
e07869a8cf [#1100] Remove unused fields
All checks were successful
DCO action / DCO (pull_request) Successful in 2m47s
Details
Build / Build Components (1.21) (pull_request) Successful in 3m21s
Details
Vulncheck / Vulncheck (pull_request) Successful in 3m33s
Details
Build / Build Components (1.22) (pull_request) Successful in 5m25s
Details
Tests and linters / gopls check (pull_request) Successful in 5m13s
Details
Tests and linters / Staticcheck (pull_request) Successful in 6m13s
Details
Tests and linters / Lint (pull_request) Successful in 7m2s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 10m2s
Details
Tests and linters / Tests with -race (pull_request) Successful in 9m55s
Details
Tests and linters / Tests (1.22) (pull_request) Successful in 10m11s
Details 
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2024-05-06 10:14:36 +03:00
Evgenii Stratonikov
3dc81cb4fc Reapply "[#972] Use min/max builtins" 
This reverts commit dad56d2e98.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-04-10 12:09:34 +00:00
Dmitrii Stepanov
f5b67c6735 [#1064] policer: Disable EC processing 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-04-09 07:08:53 +00:00
aarifullin
4738508ce2 [#1063] go.mod: Update SDK version 
* Update frostfs-sdk and frostfs-api-go versions.
* Refactor depreacted method ReplicaNumberByIndex.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-04-03 10:33:51 +00:00
Evgenii Stratonikov
dad56d2e98 Revert "[#972] Use min/max builtins" 
This reverts commit 89784b2e0a.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-02-19 15:36:01 +00:00
Evgenii Stratonikov
89784b2e0a [#972] Use min/max builtins 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-02-19 13:13:09 +00:00
Evgenii Stratonikov
edbe06e07e [#956] policer/test: Reuse testPool helper 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-02-08 15:49:43 +00:00
Evgenii Stratonikov
cbfeb72466 [#956] policer: Remove WithMaxCapacity option 
We already provide the pool and this argument is used only for
preallocation. No functional changes.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-02-08 15:49:43 +00:00
Evgenii Stratonikov
c3fa902780 [#969] policer: Restrict the number of remembered errors
All checks were successful
DCO action / DCO (pull_request) Successful in 4m23s
Details
Vulncheck / Vulncheck (pull_request) Successful in 4m57s
Details
Build / Build Components (1.21) (pull_request) Successful in 6m3s
Details
Build / Build Components (1.20) (pull_request) Successful in 9m2s
Details
Tests and linters / Staticcheck (pull_request) Successful in 10m4s
Details
Tests and linters / Lint (pull_request) Successful in 10m51s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 10m44s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 10m56s
Details
Tests and linters / Tests with -race (pull_request) Successful in 10m57s
Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-02-08 10:10:41 +03:00
Evgenii Stratonikov
6010dfdf3d [#969] policer: Make error skip thread-safe 
Introduces in afd2ba9a66.
Refs #914

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-02-08 10:10:41 +03:00
Ekaterina Lebedeva
afd2ba9a66 [#110] Add check for repeated error log in policer
All checks were successful
DCO action / DCO (pull_request) Successful in 2m55s
Details
Vulncheck / Vulncheck (pull_request) Successful in 3m22s
Details
Tests and linters / Staticcheck (pull_request) Successful in 4m10s
Details
Build / Build Components (1.20) (pull_request) Successful in 4m58s
Details
Build / Build Components (1.21) (pull_request) Successful in 4m54s
Details
Tests and linters / Lint (pull_request) Successful in 5m47s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m45s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m58s
Details
Tests and linters / Tests with -race (pull_request) Successful in 7m42s
Details 
processObject() returns 3 types of errors: container not found errors,
could not get container error and placement vector building error. Every
error will occur for all objects in container simultaneously, so we can
log each error once and safely ignore the rest.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2024-02-06 00:56:41 +03:00
Anton Nikiforov
6534252c22 [#799] policer: Refactor method processNodes 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-02-01 17:49:22 +00:00
Anton Nikiforov
c441296592 [#930] policer: Release task pool when context cancelled 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-01-31 08:30:34 +00:00
Dmitrii Stepanov
681b2c5fd4 [#825] policer: Do not drop required linking objects 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-12-12 11:04:03 +00:00
Dmitrii Stepanov
0f75e48138 [#796] policer: Fix tombstone objects replication 
Tombstone objects must be replicated to all container nodes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-11-09 13:39:33 +03:00
Dmitrii Stepanov
79088baa06 [#772] node: Apply gofumpt 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-10-31 17:03:03 +03:00
Evgenii Stratonikov
aca11d7474 [#735] policer: Allow to provide metrics from the outside
Some checks failed
DCO action / DCO (pull_request) Successful in 3m9s
Details
Vulncheck / Vulncheck (pull_request) Successful in 3m28s
Details
Build / Build Components (1.21) (pull_request) Successful in 4m17s
Details
Build / Build Components (1.20) (pull_request) Successful in 4m35s
Details
Tests and linters / Lint (pull_request) Successful in 7m19s
Details
Tests and linters / Tests with -race (pull_request) Failing after 9m47s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m0s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 11m19s
Details
Tests and linters / Staticcheck (pull_request) Successful in 11m4s
Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-10-11 15:14:13 +03:00
Ekaterina Lebedeva
5e229dc248 [#701] metrics: add metric to evaluate policer performance 
Add processed objects counter in policerMetrics,
add policer field to NodeMetrics

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2023-10-09 19:02:08 +00:00
Anton Nikiforov
88d50e4c77 [#656] policer: Add "bad" testcase 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2023-09-06 08:04:59 +00:00
Airat Arifullin
4ea0df77d0 [#574] policer: Check if the container was really removed 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-08-28 14:21:38 +00:00
Airat Arifullin
554ff2c06b [#574] core: Extend Source interface with DeletionInfo method 
* Introduce common method EverExisted
* Define DeletionInfo for struct that must implement Source
* Refactor tree srv

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2023-08-28 14:21:38 +00:00
Alejandro Lopez
5b7e4a51b7 [#481] Update frostfs-sdk-go and error pointer receivers 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-08-09 10:26:53 +00:00
Dmitrii Stepanov
f0355a453e [#463] policer: Remove capacity rebalance logic
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Current implementation has some quirks. For example,
using only half of object.put.pool_size_remote threads
tells replicator that is node is 50% loaded,
but in reality we could be putting lot's of big objects.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-07-18 10:52:12 +00:00
Alejandro Lopez
3b66f98f27 [#519] Use Address.Equals in policer tests
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-07-17 10:45:07 +00:00
Evgenii Stratonikov
140d970a95 [#498] policer: Fix objectSDK import
Some checks failed
ci/woodpecker/pr/pre-commit Pipeline failed
Details
Build / Build Components (1.19) (pull_request) Successful in 4m23s
Details
Tests and linters / Tests (1.19) (pull_request) Successful in 3m27s
Details
Tests and linters / Lint (pull_request) Successful in 4m11s
Details
Tests and linters / Tests with -race (pull_request) Successful in 5m36s
Details
Build / Build Components (1.20) (pull_request) Successful in 10m18s
Details
Tests and linters / Staticcheck (pull_request) Successful in 5m41s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 11m33s
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-07 18:52:40 +03:00
Evgenii Stratonikov
2310a5c7ba [#498] policer: Allow to set sleep duration between iterations
Some checks failed
ci/woodpecker/push/pre-commit Pipeline failed
Details 
Speed up tests on CI.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-07 14:52:33 +00:00
Evgenii Stratonikov
e858479a74 [#498] policer: Explicitly Rewind() iterator after finish 
Previously, we can continue to return `EndOfListing` infinitely.
Reflect iterator reuse via Rewind() method.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-07-07 14:52:33 +00:00
Alexander Chuprov
033eaf77e1 [#496] node: Fix linter importas
All checks were successful
Build / Build Components (1.20) (pull_request) Successful in 3m52s
Details
Build / Build Components (1.19) (pull_request) Successful in 4m1s
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
Tests and linters / Tests with -race (pull_request) Successful in 5m36s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m55s
Details
Tests and linters / Lint (pull_request) Successful in 14m40s
Details
Tests and linters / Tests (1.19) (pull_request) Successful in 15m29s
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Standardize the alias of the
import frostfs-sdk-go/object as objectSDK.

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2023-07-06 15:36:41 +03:00
Alejandro Lopez
26acf5689e [#92] Ensure policer objects cannot be worked on concurrently
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-07-03 07:05:31 +00:00
Alejandro Lopez
3223402c90 [#92] Embed policer's objectsInWork mutex 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-07-03 07:05:31 +00:00
Alejandro Lopez
f9730f090d [#92] Refactor policer and add some unit tests 
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
 2023-07-03 07:05:31 +00:00
Dmitrii Stepanov
059e9e88a2 [#373] metabase: Add metrics 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-06-21 15:13:26 +03:00
Evgenii Stratonikov
0e31c12e63 [#240] logs: Move log messages to constants 
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-04-14 05:06:09 +00:00
Dmitrii Stepanov
d6486d172e [#210] policier: Refactor nodes processing
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Resolve funlen linter for processNodes method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:55:52 +03:00
Dmitrii Stepanov
080be5cfcd [#210] policier: Refactor object placement 
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:55:52 +03:00
Dmitrii Stepanov
23575e1ac0 [#210] policier: Resolve contextcheck linter 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-04-05 14:55:52 +03:00
Dmitrii Stepanov
44b86bac5a [#148] linter: Add contextcheck linter
Some checks failed
ci/woodpecker/push/full-pre-commit Pipeline failed
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:54:41 +03:00
Dmitrii Stepanov
97c36ed3ec [#148] linter: Add funlen linter 
Long functions are hard to understand and source of errors

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:54:41 +03:00
Dmitrii Stepanov
cc8ff015b4 [#148] linter: Add containedctx linter 
Context has to be passed as an argument: https://pkg.go.dev/context

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2023-03-21 09:52:39 +03:00
Evgenii Stratonikov
6226c3ba86 [#129] policer: Use safer defaults 
If `processNodes` exits earlier for some reason, `needLocalCopy` could
be false.
See https://github.com/nspcc-dev/neofs-node/issues/2267

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2023-03-10 10:59:15 +00:00
Alex Vanin
20de74a505 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 16:38:26 +03:00
Stanislav Bogatyrev
cb016d53a6 [#1] Fix comments and error messages 
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
 2023-02-06 17:41:14 +03:00
Evgenii Stratonikov
8f61cc1dcc [#5] policer: Use generic LRU client 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-31 23:04:06 +03:00
Evgenii Stratonikov
b207dc424f [#2158] policer: Reduce default cache size 
We use cache to avoid policing the same object multiple times in a short
time span (< 30 seconds). If we have 200_000 objects in a blobstor, it is a bit useless
-- if it takes 1 second to process an object and we have `replicator.pool_size: 20`
in config, the next iteration will happen in 10_000 second which is much
larger than 30 second. However we still consume a lot of memory, so it
makes sense to use saner default.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a Move to frostfs-node 
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
 2022-12-28 15:04:29 +03:00
Evgenii Stratonikov
660c38d07e [#2062] services/policer: Use a proper key for object cache 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
1779664644 [#2058] services/policer: Fix panic in shardPolicyWorker 
```
2022/11/15 08:40:56 worker exits from a panic: runtime error: index out of range [0] with length 0
2022/11/15 08:40:56 worker exits from panic: goroutine 1188 [running]:
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
panic({0x1042b60, 0xc0015ae018})
	runtime/panic.go:1038 +0x215
github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).shardPolicyWorker.func1()
	github.com/nspcc-dev/neofs-node/pkg/services/policer/process.go:65 +0x366
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x68
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-11-19 11:01:04 +03:00
Pavel Karpy
634792077e [#1502] node: Store lock object on every container node 
Includes extending listing methods in the Storage Engine with object types.
It allows tuning replication/policer algorithms: container nodes do
not remove `LOCK` objects as redundant and try to fulfill `LOCK` placement
on the ohter container nodes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
846ff515e6 [#1812] policer: Do not remove copies if there are maintenance nodes 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-26 19:13:17 +03:00