Ekaterina Lebedeva
e07869a8cf
[ #1100 ] Remove unused fields
...
DCO action / DCO (pull_request) Successful in 2m47s
Details
Build / Build Components (1.21) (pull_request) Successful in 3m21s
Details
Vulncheck / Vulncheck (pull_request) Successful in 3m33s
Details
Build / Build Components (1.22) (pull_request) Successful in 5m25s
Details
Tests and linters / gopls check (pull_request) Successful in 5m13s
Details
Tests and linters / Staticcheck (pull_request) Successful in 6m13s
Details
Tests and linters / Lint (pull_request) Successful in 7m2s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 10m2s
Details
Tests and linters / Tests with -race (pull_request) Successful in 9m55s
Details
Tests and linters / Tests (1.22) (pull_request) Successful in 10m11s
Details
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-05-06 10:14:36 +03:00
Evgenii Stratonikov
3dc81cb4fc
Reapply "[ #972 ] Use min/max builtins"
...
This reverts commit dad56d2e98
.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-10 12:09:34 +00:00
Dmitrii Stepanov
f5b67c6735
[ #1064 ] policer: Disable EC processing
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 07:08:53 +00:00
Airat Arifullin
4738508ce2
[ #1063 ] go.mod: Update SDK version
...
* Update frostfs-sdk and frostfs-api-go versions.
* Refactor depreacted method ReplicaNumberByIndex.
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-03 10:33:51 +00:00
Evgenii Stratonikov
dad56d2e98
Revert "[ #972 ] Use min/max builtins"
...
This reverts commit 89784b2e0a
.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 15:36:01 +00:00
Evgenii Stratonikov
89784b2e0a
[ #972 ] Use min/max builtins
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 13:13:09 +00:00
Evgenii Stratonikov
edbe06e07e
[ #956 ] policer/test: Reuse testPool helper
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 15:49:43 +00:00
Evgenii Stratonikov
cbfeb72466
[ #956 ] policer: Remove WithMaxCapacity option
...
We already provide the pool and this argument is used only for
preallocation. No functional changes.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 15:49:43 +00:00
Evgenii Stratonikov
c3fa902780
[ #969 ] policer: Restrict the number of remembered errors
...
DCO action / DCO (pull_request) Successful in 4m23s
Details
Vulncheck / Vulncheck (pull_request) Successful in 4m57s
Details
Build / Build Components (1.21) (pull_request) Successful in 6m3s
Details
Build / Build Components (1.20) (pull_request) Successful in 9m2s
Details
Tests and linters / Staticcheck (pull_request) Successful in 10m4s
Details
Tests and linters / Lint (pull_request) Successful in 10m51s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 10m44s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 10m56s
Details
Tests and linters / Tests with -race (pull_request) Successful in 10m57s
Details
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 10:10:41 +03:00
Evgenii Stratonikov
6010dfdf3d
[ #969 ] policer: Make error skip thread-safe
...
Introduces in afd2ba9a66
.
Refs #914
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 10:10:41 +03:00
Ekaterina Lebedeva
afd2ba9a66
[ #110 ] Add check for repeated error log in policer
...
DCO action / DCO (pull_request) Successful in 2m55s
Details
Vulncheck / Vulncheck (pull_request) Successful in 3m22s
Details
Tests and linters / Staticcheck (pull_request) Successful in 4m10s
Details
Build / Build Components (1.20) (pull_request) Successful in 4m58s
Details
Build / Build Components (1.21) (pull_request) Successful in 4m54s
Details
Tests and linters / Lint (pull_request) Successful in 5m47s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 6m45s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 6m58s
Details
Tests and linters / Tests with -race (pull_request) Successful in 7m42s
Details
processObject() returns 3 types of errors: container not found errors,
could not get container error and placement vector building error. Every
error will occur for all objects in container simultaneously, so we can
log each error once and safely ignore the rest.
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-02-06 00:56:41 +03:00
Anton Nikiforov
6534252c22
[ #799 ] policer: Refactor method `processNodes`
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-01 17:49:22 +00:00
Anton Nikiforov
c441296592
[ #930 ] policer: Release task pool when context cancelled
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-31 08:30:34 +00:00
Dmitrii Stepanov
681b2c5fd4
[ #825 ] policer: Do not drop required linking objects
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 11:04:03 +00:00
Dmitrii Stepanov
0f75e48138
[ #796 ] policer: Fix tombstone objects replication
...
Tombstone objects must be replicated to all container nodes.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 13:39:33 +03:00
Dmitrii Stepanov
79088baa06
[ #772 ] node: Apply gofumpt
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:03 +03:00
Evgenii Stratonikov
aca11d7474
[ #735 ] policer: Allow to provide metrics from the outside
...
DCO action / DCO (pull_request) Successful in 3m9s
Details
Vulncheck / Vulncheck (pull_request) Successful in 3m28s
Details
Build / Build Components (1.21) (pull_request) Successful in 4m17s
Details
Build / Build Components (1.20) (pull_request) Successful in 4m35s
Details
Tests and linters / Lint (pull_request) Successful in 7m19s
Details
Tests and linters / Tests with -race (pull_request) Failing after 9m47s
Details
Tests and linters / Tests (1.21) (pull_request) Successful in 8m0s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 11m19s
Details
Tests and linters / Staticcheck (pull_request) Successful in 11m4s
Details
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-11 15:14:13 +03:00
Ekaterina Lebedeva
5e229dc248
[ #701 ] metrics: add metric to evaluate policer performance
...
Add processed objects counter in policerMetrics,
add policer field to NodeMetrics
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-10-09 19:02:08 +00:00
Anton Nikiforov
88d50e4c77
[ #656 ] policer: Add "bad" testcase
...
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-09-06 08:04:59 +00:00
Airat Arifullin
4ea0df77d0
[ #574 ] policer: Check if the container was really removed
...
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
Airat Arifullin
554ff2c06b
[ #574 ] core: Extend Source interface with DeletionInfo method
...
* Introduce common method EverExisted
* Define DeletionInfo for struct that must implement Source
* Refactor tree srv
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
Alejandro Lopez
5b7e4a51b7
[ #481 ] Update frostfs-sdk-go and error pointer receivers
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-09 10:26:53 +00:00
Dmitrii Stepanov
f0355a453e
[ #463 ] policer: Remove capacity rebalance logic
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Current implementation has some quirks. For example,
using only half of object.put.pool_size_remote threads
tells replicator that is node is 50% loaded,
but in reality we could be putting lot's of big objects.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
Alejandro Lopez
3b66f98f27
[ #519 ] Use Address.Equals in policer tests
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-17 10:45:07 +00:00
Evgenii Stratonikov
140d970a95
[ #498 ] policer: Fix objectSDK import
...
ci/woodpecker/pr/pre-commit Pipeline failed
Details
Build / Build Components (1.19) (pull_request) Successful in 4m23s
Details
Tests and linters / Tests (1.19) (pull_request) Successful in 3m27s
Details
Tests and linters / Lint (pull_request) Successful in 4m11s
Details
Tests and linters / Tests with -race (pull_request) Successful in 5m36s
Details
Build / Build Components (1.20) (pull_request) Successful in 10m18s
Details
Tests and linters / Staticcheck (pull_request) Successful in 5m41s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 11m33s
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-07 18:52:40 +03:00
Evgenii Stratonikov
2310a5c7ba
[ #498 ] policer: Allow to set sleep duration between iterations
...
ci/woodpecker/push/pre-commit Pipeline failed
Details
Speed up tests on CI.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-07 14:52:33 +00:00
Evgenii Stratonikov
e858479a74
[ #498 ] policer: Explicitly Rewind() iterator after finish
...
Previously, we can continue to return `EndOfListing` infinitely.
Reflect iterator reuse via Rewind() method.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-07 14:52:33 +00:00
Alexander Chuprov
033eaf77e1
[ #496 ] node: Fix linter importas
...
Build / Build Components (1.20) (pull_request) Successful in 3m52s
Details
Build / Build Components (1.19) (pull_request) Successful in 4m1s
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
Tests and linters / Tests with -race (pull_request) Successful in 5m36s
Details
Tests and linters / Tests (1.20) (pull_request) Successful in 5m55s
Details
Tests and linters / Lint (pull_request) Successful in 14m40s
Details
Tests and linters / Tests (1.19) (pull_request) Successful in 15m29s
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Standardize the alias of the
import frostfs-sdk-go/object as objectSDK.
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-07-06 15:36:41 +03:00
Alejandro Lopez
26acf5689e
[ #92 ] Ensure policer objects cannot be worked on concurrently
...
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
Alejandro Lopez
3223402c90
[ #92 ] Embed policer's objectsInWork mutex
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
Alejandro Lopez
f9730f090d
[ #92 ] Refactor policer and add some unit tests
...
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
Dmitrii Stepanov
059e9e88a2
[ #373 ] metabase: Add metrics
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
Evgenii Stratonikov
0e31c12e63
[ #240 ] logs: Move log messages to constants
...
Drop duplicate entities.
Format entities.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 05:06:09 +00:00
Dmitrii Stepanov
d6486d172e
[ #210 ] policier: Refactor nodes processing
...
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Resolve funlen linter for processNodes method.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
Dmitrii Stepanov
080be5cfcd
[ #210 ] policier: Refactor object placement
...
Resolve containedctx and contextcheck linters.
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
Dmitrii Stepanov
23575e1ac0
[ #210 ] policier: Resolve contextcheck linter
...
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
Dmitrii Stepanov
44b86bac5a
[ #148 ] linter: Add contextcheck linter
...
ci/woodpecker/push/full-pre-commit Pipeline failed
Details
ci/woodpecker/push/pre-commit Pipeline was successful
Details
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
Dmitrii Stepanov
97c36ed3ec
[ #148 ] linter: Add funlen linter
...
Long functions are hard to understand and source of errors
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
Dmitrii Stepanov
cc8ff015b4
[ #148 ] linter: Add containedctx linter
...
Context has to be passed as an argument: https://pkg.go.dev/context
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:52:39 +03:00
Evgenii Stratonikov
6226c3ba86
[ #129 ] policer: Use safer defaults
...
If `processNodes` exits earlier for some reason, `needLocalCopy` could
be false.
See https://github.com/nspcc-dev/neofs-node/issues/2267
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-10 10:59:15 +00:00
Alexey Vanin
20de74a505
Rename package name
...
Due to source code relocation from GitHub.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
Stanislav Bogatyrev
cb016d53a6
[ #1 ] Fix comments and error messages
...
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
Evgenii Stratonikov
8f61cc1dcc
[ #5 ] policer: Use generic LRU client
...
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
Evgenii Stratonikov
b207dc424f
[ #2158 ] policer: Reduce default cache size
...
We use cache to avoid policing the same object multiple times in a short
time span (< 30 seconds). If we have 200_000 objects in a blobstor, it is a bit useless
-- if it takes 1 second to process an object and we have `replicator.pool_size: 20`
in config, the next iteration will happen in 10_000 second which is much
larger than 30 second. However we still consume a lot of memory, so it
makes sense to use saner default.
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a
Move to frostfs-node
...
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-28 15:04:29 +03:00
Evgenii Stratonikov
660c38d07e
[ #2062 ] services/policer: Use a proper key for object cache
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
1779664644
[ #2058 ] services/policer: Fix panic in `shardPolicyWorker`
...
```
2022/11/15 08:40:56 worker exits from a panic: runtime error: index out of range [0] with length 0
2022/11/15 08:40:56 worker exits from panic: goroutine 1188 [running]:
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
panic({0x1042b60, 0xc0015ae018})
runtime/panic.go:1038 +0x215
github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).shardPolicyWorker.func1()
github.com/nspcc-dev/neofs-node/pkg/services/policer/process.go:65 +0x366
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x68
```
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
634792077e
[ #1502 ] node: Store lock object on every container node
...
Includes extending listing methods in the Storage Engine with object types.
It allows tuning replication/policer algorithms: container nodes do
not remove `LOCK` objects as redundant and try to fulfill `LOCK` placement
on the ohter container nodes.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
846ff515e6
[ #1812 ] policer: Do not remove copies if there are maintenance nodes
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 19:13:17 +03:00
Evgenii Stratonikov
0d65888005
[ #1910 ] .golangci.yml: Add `predeclared` linker
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00