Commit graph

3401 commits

Author SHA1 Message Date
d19ade23c8 [#959] node: Set mode to shard's components when open it
Avoid opening database for `metabase` and `cache` in `Degraded` mode.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-09 14:04:01 +00:00
db67c21d55 [#947] engine: Evacuate trees to remote nodes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:33:15 +03:00
728150d1d2 [#947] engine: Evacuate trees to local shards
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:33:15 +03:00
15d853ea22 [#947] controlSvc: Return tree evacuation stat
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:20:39 +03:00
b3f3505ada [#947] cli: Allow to specify evacuation scope
It may be required to evacuate only objects or only tree or all, so
now it spossible to specify.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:20:38 +03:00
a6eb66bf9c [#947] evacuate: Refactor evacuate parameters
Drop methods to make it easier to extend.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:20:38 +03:00
8e2a0611f4 [#947] tree: Add method to list all trees
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:20:38 +03:00
edbe06e07e [#956] policer/test: Reuse testPool helper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 15:49:43 +00:00
cbfeb72466 [#956] policer: Remove WithMaxCapacity option
We already provide the pool and this argument is used only for
preallocation. No functional changes.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 15:49:43 +00:00
cfc5ce7853 [#964] metabase: Drop GC marks if object not found
GC inhumes expired locks and tombstones on all the shards.
So it could be GC mark without object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-08 07:54:39 +00:00
c3fa902780 [#969] policer: Restrict the number of remembered errors
All checks were successful
DCO action / DCO (pull_request) Successful in 4m23s
Vulncheck / Vulncheck (pull_request) Successful in 4m57s
Build / Build Components (1.21) (pull_request) Successful in 6m3s
Build / Build Components (1.20) (pull_request) Successful in 9m2s
Tests and linters / Staticcheck (pull_request) Successful in 10m4s
Tests and linters / Lint (pull_request) Successful in 10m51s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m44s
Tests and linters / Tests (1.20) (pull_request) Successful in 10m56s
Tests and linters / Tests with -race (pull_request) Successful in 10m57s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 10:10:41 +03:00
6010dfdf3d [#969] policer: Make error skip thread-safe
Introduces in afd2ba9a66.
Refs #914

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 10:10:41 +03:00
a6c9a337cd [#965] morph: Get rid of container.List invocations
ContainersOf() is better in almost every aspect, besides creating a
session when the containers number is between 1024 and 2048 (prefetch
script does limited unwrapping). Making List() private helps to ensure
it is no longer used and can be safely removed in future.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-07 08:56:27 +00:00
9ba48c582d [#917] engine: Allow to detach shards
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-06 14:49:47 +03:00
4358d3c423 [#917] controlSvc: Add DetachShards handler
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-06 14:47:52 +03:00
afd2ba9a66 [#110] Add check for repeated error log in policer
All checks were successful
DCO action / DCO (pull_request) Successful in 2m55s
Vulncheck / Vulncheck (pull_request) Successful in 3m22s
Tests and linters / Staticcheck (pull_request) Successful in 4m10s
Build / Build Components (1.20) (pull_request) Successful in 4m58s
Build / Build Components (1.21) (pull_request) Successful in 4m54s
Tests and linters / Lint (pull_request) Successful in 5m47s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m45s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m58s
Tests and linters / Tests with -race (pull_request) Successful in 7m42s
processObject() returns 3 types of errors: container not found errors,
could not get container error and placement vector building error. Every
error will occur for all objects in container simultaneously, so we can
log each error once and safely ignore the rest.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-02-06 00:56:41 +03:00
602ee11123 [#934] containersvc: Marhal public key in short format for APE
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-02 17:51:38 +00:00
c1a5b831b6 [#955] chainbase: Fix rule chain unmarshalling
All checks were successful
DCO action / DCO (pull_request) Successful in 4m41s
Vulncheck / Vulncheck (pull_request) Successful in 4m46s
Tests and linters / Staticcheck (pull_request) Successful in 6m30s
Tests and linters / Lint (pull_request) Successful in 7m36s
Build / Build Components (1.21) (pull_request) Successful in 8m57s
Build / Build Components (1.20) (pull_request) Successful in 9m8s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m32s
Tests and linters / Tests (1.20) (pull_request) Successful in 10m42s
Tests and linters / Tests with -race (pull_request) Successful in 11m23s
* Use correct way DecodeBytes instead unmarshalling by json.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-02 20:28:04 +03:00
9916598dfb [#922] control: Extend api with ListOverrideDefinedTargets
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-02 12:09:51 +00:00
95e15f499f [#922] Update files generated by protoc
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-02 12:09:51 +00:00
2cb04379a4 [#922] go.mod: Update APE
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-02 12:09:51 +00:00
a5446bc17d [#952] object: Pass namespace within context in ACL service
All checks were successful
DCO action / DCO (pull_request) Successful in 6m23s
Vulncheck / Vulncheck (pull_request) Successful in 7m3s
Build / Build Components (1.21) (pull_request) Successful in 8m21s
Build / Build Components (1.20) (pull_request) Successful in 8m31s
Tests and linters / Staticcheck (pull_request) Successful in 11m1s
Tests and linters / Lint (pull_request) Successful in 11m26s
Tests and linters / Tests (1.20) (pull_request) Successful in 12m51s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m14s
Tests and linters / Tests with -race (pull_request) Successful in 13m31s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-02 14:48:11 +03:00
d0eadf7ea2 [#799] engine: Skip put when object removed from shard
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-01 17:49:22 +00:00
6534252c22 [#799] policer: Refactor method processNodes
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-01 17:49:22 +00:00
5be2af881a [#934] container: Make container APE middleware read namespaces
* Those methods that can access already existing containers and thus
  can get container properties should read namespace from Zone
  property. If Zone is not set, take a namespace for root.
* Otherwise, define namespaces by owner ID via frostfs-id contract.
* Improve unit-tests, consider more cases.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-01 17:38:24 +00:00
4352bd0e8e [#934] ape: Transform empty namespace within chainbase
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-01 17:38:24 +00:00
483a67b170 [#937] ape: Validate chain resource name
All checks were successful
DCO action / DCO (pull_request) Successful in 2m13s
Build / Build Components (1.20) (pull_request) Successful in 3m38s
Vulncheck / Vulncheck (pull_request) Successful in 3m13s
Tests and linters / Staticcheck (pull_request) Successful in 6m15s
Build / Build Components (1.21) (pull_request) Successful in 7m11s
Tests and linters / Lint (pull_request) Successful in 9m35s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m56s
Tests and linters / Tests (1.20) (pull_request) Successful in 14m14s
Tests and linters / Tests with -race (pull_request) Successful in 6m2s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-31 11:34:35 +03:00
e3573de6db [#930] gc: Stop internal activity by context
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-31 08:30:34 +00:00
c441296592 [#930] policer: Release task pool when context cancelled
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-31 08:30:34 +00:00
675eec91f3 [#938] shard: Update only changed counters
All checks were successful
DCO action / DCO (pull_request) Successful in 5m40s
Vulncheck / Vulncheck (pull_request) Successful in 5m33s
Tests and linters / Staticcheck (pull_request) Successful in 7m12s
Build / Build Components (1.21) (pull_request) Successful in 7m25s
Build / Build Components (1.20) (pull_request) Successful in 7m29s
Tests and linters / Lint (pull_request) Successful in 8m26s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m48s
Tests and linters / Tests with -race (pull_request) Successful in 9m56s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m10s
If metric value hasn't changed, but we update metric, then
non existed metric will apear with zero value.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-30 12:37:48 +03:00
c681354afd [#938] engine: Fix container count removal
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-30 12:37:48 +03:00
df055fead5 [#931] morph: Provide batch size for container listing explicitly
Besides VM stack item limit we also have restrictions on the size of
JSON for a stackitem, which is 128k. This limit is much harder to
calculate, because JSON representation includes type and the encoding is
different for different items. Thus is makes no sense to invent our own
default, so use the one provided by neo-go. But for container listing we
know exactly what we process, so use big enough value, which is tested.

Introduced in be8607a1f6.

Refs #902
Refs https://github.com/nspcc-dev/neo-go/blob/v0.105.0/pkg/vm/stackitem/json.go#L353

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-01-29 14:00:11 +00:00
6e2cc32768 [#681] objsvc: Validate session token owner for local sessions
Previously, the check was in place only when session token was missing.
Format validator checks are applied only to fully-prepared object, so
this lead to the following situation:
1. Object is put locally with malformed token, because there are no
   checks.
2. Object cannot be replicated, because the token is malformed.

This is now fixed and token check is done before any payload receival.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-01-26 08:52:29 +00:00
b6fc3321c5 [#876] Fix linters
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-25 20:26:13 +03:00
1fe7736d92 [#925] morph: Introduce switch rpc guarded rpc actor
* Introduce switch rpc guarded rpc actor in morph client
  to prevent using invalidated rpc actor when RPC switch
  happens.
* Initialize NewContractStorage with SwitchRPCGuardedActor.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-25 13:24:13 +00:00
5ed330e436 [#927] metabase: Delete GC marks
All checks were successful
DCO action / DCO (pull_request) Successful in 6m34s
Vulncheck / Vulncheck (pull_request) Successful in 7m19s
Build / Build Components (1.20) (pull_request) Successful in 7m48s
Build / Build Components (1.21) (pull_request) Successful in 9m39s
Tests and linters / Staticcheck (pull_request) Successful in 10m36s
Tests and linters / Lint (pull_request) Successful in 11m26s
Tests and linters / Tests (1.20) (pull_request) Successful in 12m57s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m8s
Tests and linters / Tests with -race (pull_request) Successful in 13m7s
`key` is changed inside `db.get`, so encode address again after get.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-24 18:51:16 +03:00
931a5e9aaf [#918] engine: Move shard to degraded mode if metabase open failed
All checks were successful
DCO action / DCO (pull_request) Successful in 1m55s
Build / Build Components (1.20) (pull_request) Successful in 2m14s
Vulncheck / Vulncheck (pull_request) Successful in 2m15s
Build / Build Components (1.21) (pull_request) Successful in 4m6s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m58s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m24s
Tests and linters / Staticcheck (pull_request) Successful in 6m11s
Tests and linters / Lint (pull_request) Successful in 6m37s
Tests and linters / Tests with -race (pull_request) Successful in 6m35s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-23 11:16:40 +03:00
f2f3294fc3 [#919] ape: Improve error messages in ape service
* Wrap all APE middleware errors in apeErr that
  makes errors more explicit with status AccessDenied.
* Use denyingRuleErr for denying status from chain router.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-23 08:11:24 +00:00
f526f49995 [#874] engine: Check object existance concurrently
All checks were successful
DCO action / DCO (pull_request) Successful in 1m36s
Vulncheck / Vulncheck (pull_request) Successful in 2m33s
Build / Build Components (1.21) (pull_request) Successful in 3m9s
Build / Build Components (1.20) (pull_request) Successful in 4m49s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m5s
Tests and linters / Staticcheck (pull_request) Successful in 6m6s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m39s
Tests and linters / Lint (pull_request) Successful in 7m0s
Tests and linters / Tests with -race (pull_request) Successful in 6m46s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-23 09:28:29 +03:00
f5160b27fc [#920] tests: Fix data races
All checks were successful
DCO action / DCO (pull_request) Successful in 2m2s
Build / Build Components (1.21) (pull_request) Successful in 3m35s
Vulncheck / Vulncheck (pull_request) Successful in 3m3s
Tests and linters / Staticcheck (pull_request) Successful in 5m23s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m33s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m41s
Tests and linters / Tests with -race (pull_request) Successful in 7m47s
Tests and linters / Lint (pull_request) Successful in 8m12s
Build / Build Components (1.20) (pull_request) Successful in 1m32s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-19 14:06:05 +03:00
be33070550 [#910] control: Take empty namespace for 'root'
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-19 06:22:10 +00:00
63d3ed1ad8 [#904] tests: Close test engine after test
All checks were successful
DCO action / DCO (pull_request) Successful in 2m7s
Vulncheck / Vulncheck (pull_request) Successful in 2m37s
Build / Build Components (1.20) (pull_request) Successful in 4m0s
Build / Build Components (1.21) (pull_request) Successful in 3m56s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m25s
Tests and linters / Staticcheck (pull_request) Successful in 6m22s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m43s
Tests and linters / Tests with -race (pull_request) Successful in 6m47s
Tests and linters / Lint (pull_request) Successful in 7m5s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 19:04:39 +03:00
57171907e3 [#904] metabase: Return if object was actuall inserted
This requires to count metrics properly.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 19:04:39 +03:00
c1a80235db [#904] metabase: Log Inhume operation
It will be very useful for troubleshooting.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 18:42:52 +03:00
96b020626f [#915] ape: Fix method name in getStreamBasicChecker
All checks were successful
DCO action / DCO (pull_request) Successful in 2m2s
Build / Build Components (1.21) (pull_request) Successful in 2m17s
Build / Build Components (1.20) (pull_request) Successful in 3m2s
Vulncheck / Vulncheck (pull_request) Successful in 2m39s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m54s
Tests and linters / Staticcheck (pull_request) Successful in 5m49s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m11s
Tests and linters / Lint (pull_request) Successful in 6m44s
Tests and linters / Tests with -race (pull_request) Successful in 6m32s
* Replace incorrect MethodGetContainer by MethodGetObject constant.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-16 23:52:37 +03:00
c8baf76fae [#872] object: Introduce APE middlewar for object service
All checks were successful
DCO action / DCO (pull_request) Successful in 2m4s
Vulncheck / Vulncheck (pull_request) Successful in 3m12s
Build / Build Components (1.21) (pull_request) Successful in 4m1s
Build / Build Components (1.20) (pull_request) Successful in 4m13s
Tests and linters / Staticcheck (pull_request) Successful in 4m3s
Tests and linters / Lint (pull_request) Successful in 8m7s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m14s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m18s
Tests and linters / Tests with -race (pull_request) Successful in 8m24s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-12 18:41:35 +03:00
52ffa9f164 [#891] getSvc: Refactor Get service V2 creation
All checks were successful
DCO action / DCO (pull_request) Successful in 2m46s
Vulncheck / Vulncheck (pull_request) Successful in 2m53s
Build / Build Components (1.20) (pull_request) Successful in 4m31s
Build / Build Components (1.21) (pull_request) Successful in 4m26s
Tests and linters / Lint (pull_request) Successful in 6m9s
Tests and linters / Staticcheck (pull_request) Successful in 6m7s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m27s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m4s
Tests and linters / Tests with -race (pull_request) Successful in 9m48s
Use arguments for mandatory fields.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-12 13:35:38 +03:00
394f086fe2 [#891] getSvc: Fix get range hash implementation
Get range can perform GET request, so this request must be done
from container node to not to get access denied error.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-12 13:35:38 +03:00
be8607a1f6 [#902] morph: Avoid creating session in TestInvokeIterator
When the number of items to iterate over is less than 2048, there is no
need to create a session and consume resources.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-01-12 10:25:56 +00:00
a2ab373a0a [#895] metabase: Do not delete GC mark for virtual objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
7166e77c2b [#895] test: Add logger to test shard
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
47dcfa20f3 [#895] test: Use t.Cleanup only for external resources
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
836818fb75 [#895] test: Use zaptest logger
Previous implementation mixes logs from different tests, now logs
are separeted by test.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
f1b2b8bffa [#895] test: Fix NewLogger arguments list
`debug` is always true.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
a8e52ef7aa [#898] control: Fix codes for returning APE errors
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-11 12:31:31 +00:00
5d982976fd [#898] ape: Fix bug with creating type bucket in chainbase
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-11 12:31:31 +00:00
c19396d203 [#885] control: Make chain id bytes in grpc
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-01-11 07:24:22 +00:00
5c0a736a25 [#899] containerSvc: Fix invalid session token type
Some checks failed
DCO action / DCO (pull_request) Successful in 1m23s
Vulncheck / Vulncheck (pull_request) Successful in 3m29s
Tests and linters / Tests (1.21) (pull_request) Failing after 3m58s
Build / Build Components (1.21) (pull_request) Successful in 3m46s
Build / Build Components (1.20) (pull_request) Successful in 3m52s
Tests and linters / Lint (pull_request) Successful in 4m48s
Tests and linters / Staticcheck (pull_request) Successful in 5m5s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m4s
Tests and linters / Tests with -race (pull_request) Successful in 8m36s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-10 18:37:54 +03:00
79bebe4a68 [#884] cli: Fix error message for undefined endpoint
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-01-10 10:02:16 +00:00
4b8b4da681 [#864] engine: Drop container count metric if container removed
All checks were successful
DCO action / DCO (pull_request) Successful in 1m30s
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Build / Build Components (1.20) (pull_request) Successful in 3m53s
Tests and linters / Lint (pull_request) Successful in 4m31s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m1s
Tests and linters / Staticcheck (pull_request) Successful in 4m51s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m13s
Tests and linters / Tests with -race (pull_request) Successful in 8m34s
Vulncheck / Vulncheck (pull_request) Successful in 58s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-10 10:45:32 +03:00
d75e7e9a21 [#864] engine: Drop container size metric if container deleted
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-10 10:44:54 +03:00
dfd62ca6b1 [#864] metabase: Refactor delete/inhume
Available -> Logic, Raw -> Phy for delete/inhume results.
Use single counter instead of vectors.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-09 09:59:42 +03:00
225fe2d4d5 [#894] blobovniczatree: Speedup rebuild test
Down from 3s to 300ms.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-12-29 16:28:54 +00:00
530249e3bd [#893] go.mod: Update neo-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-12-29 15:03:47 +00:00
581887148a [#569] cli: Add control shards writecache seal command
All checks were successful
DCO action / DCO (pull_request) Successful in 2m46s
Vulncheck / Vulncheck (pull_request) Successful in 3m3s
Build / Build Components (1.21) (pull_request) Successful in 3m53s
Build / Build Components (1.20) (pull_request) Successful in 4m0s
Tests and linters / Staticcheck (pull_request) Successful in 5m31s
Tests and linters / Lint (pull_request) Successful in 6m17s
Tests and linters / Tests (1.20) (pull_request) Successful in 12m47s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m14s
Tests and linters / Tests with -race (pull_request) Successful in 13m12s
It does the same as `control shards flush-writecache --seal`, but
has better name.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 16:05:37 +03:00
7a9db5bcdd [#569] writecache: Do not wait modeMtx if mode changes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 16:05:37 +03:00
32c282ca10 [#569] writecache: Refactor flush
Make single RUnlock call instead of two.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 16:05:37 +03:00
0cb0fc1735 [#569] writecache: Allow to seal writecache after flush
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 16:05:37 +03:00
b118734909 [#890] getsvc: Log node PK
Some checks failed
DCO action / DCO (pull_request) Successful in 13m55s
Vulncheck / Vulncheck (pull_request) Successful in 16m51s
Build / Build Components (1.20) (pull_request) Successful in 17m52s
Build / Build Components (1.21) (pull_request) Successful in 18m9s
Tests and linters / Tests (1.20) (pull_request) Failing after 24m34s
Tests and linters / Tests (1.21) (pull_request) Failing after 25m56s
Tests and linters / Staticcheck (pull_request) Successful in 26m46s
Tests and linters / Lint (pull_request) Successful in 28m6s
Tests and linters / Tests with -race (pull_request) Failing after 33m35s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-29 14:24:13 +03:00
764f70634d [#881] containerSvc: Add APE validation
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 2m40s
DCO action / DCO (pull_request) Successful in 2m27s
Build / Build Components (1.21) (pull_request) Successful in 3m22s
Tests and linters / Lint (pull_request) Successful in 4m52s
Tests and linters / Staticcheck (pull_request) Successful in 4m46s
Build / Build Components (1.20) (pull_request) Successful in 4m54s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m59s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m38s
Tests and linters / Tests with -race (pull_request) Successful in 13m10s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-27 11:05:34 +03:00
8180a0664f [#887] node: Drop badger writecache implementation
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 6m1s
Build / Build Components (1.21) (pull_request) Successful in 7m37s
Build / Build Components (1.20) (pull_request) Successful in 7m52s
Tests and linters / Staticcheck (pull_request) Successful in 8m56s
Tests and linters / Lint (pull_request) Successful in 9m26s
Tests and linters / Tests (1.21) (pull_request) Successful in 15m5s
Tests and linters / Tests with -race (pull_request) Successful in 15m7s
DCO action / DCO (pull_request) Successful in 1m1s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m1s
Badger implementation isn't tested and works not well,
but requires human resources to maintain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-22 13:00:54 +03:00
eab981bf1a [#876] Fix linter error
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-12-21 18:21:37 +03:00
d9cbb16bd3 [#866] Use TTL for blobovnicza tree cache
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-12-19 16:36:28 +00:00
7d7cf05575 [#851] ape: Initialize and use policy contract interface
* Replace inmemory policy contract mock by initialized
  policy contract interface.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-18 11:06:35 +00:00
bdd43f6211 [#869] object: Pass just CID to chain router
* Do not convert CID from request to native-schema resource
  format - this step is unneccessary for APE.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-14 11:01:20 +00:00
2d4c0a0f4a [#552] Add systemd notifications to ir service
All checks were successful
DCO action / DCO (pull_request) Successful in 3m19s
Vulncheck / Vulncheck (pull_request) Successful in 3m25s
Build / Build Components (1.21) (pull_request) Successful in 4m27s
Build / Build Components (1.20) (pull_request) Successful in 4m41s
Tests and linters / Staticcheck (pull_request) Successful in 5m5s
Tests and linters / Lint (pull_request) Successful in 5m57s
Tests and linters / Tests (1.20) (pull_request) Successful in 13m25s
Tests and linters / Tests (1.21) (pull_request) Successful in 14m13s
Tests and linters / Tests with -race (pull_request) Successful in 16m15s
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 17:51:41 +03:00
eca7ac9f0d [#552] Add sdnotify package
To avoid using third-party dependencies.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 17:49:26 +03:00
05f8f49289 [#552] gofumpt changes
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-12-13 15:02:25 +03:00
7eb46404a1 [#863] blobovnicza: Fix counters
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 2m27s
DCO action / DCO (pull_request) Successful in 2m39s
Build / Build Components (1.21) (pull_request) Successful in 3m38s
Build / Build Components (1.20) (pull_request) Successful in 3m45s
Tests and linters / Staticcheck (pull_request) Successful in 4m15s
Tests and linters / Lint (pull_request) Successful in 5m47s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m49s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m16s
Tests and linters / Tests with -race (pull_request) Successful in 12m31s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-13 13:34:29 +03:00
11add38e87 [#857] golangci: Add protogetter linter
All checks were successful
DCO action / DCO (pull_request) Successful in 1m38s
Vulncheck / Vulncheck (pull_request) Successful in 3m2s
Build / Build Components (1.21) (pull_request) Successful in 3m51s
Build / Build Components (1.20) (pull_request) Successful in 4m3s
Tests and linters / Staticcheck (pull_request) Successful in 5m17s
Tests and linters / Lint (pull_request) Successful in 6m11s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m7s
Tests and linters / Tests (1.21) (pull_request) Successful in 11m32s
Tests and linters / Tests with -race (pull_request) Successful in 12m22s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 16:27:02 +03:00
94ffe8bb45 [#857] golangci: Add testifylint linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 16:27:02 +03:00
5d7833c89b [#857] golangci: Add perfsprint linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 16:27:02 +03:00
3b7c0362a8 [#861] shard: Fix Delete object
All checks were successful
DCO action / DCO (pull_request) Successful in 12m22s
Build / Build Components (1.20) (pull_request) Successful in 12m44s
Build / Build Components (1.21) (pull_request) Successful in 13m2s
Vulncheck / Vulncheck (pull_request) Successful in 13m3s
Tests and linters / Staticcheck (pull_request) Successful in 15m32s
Tests and linters / Lint (pull_request) Successful in 16m26s
Tests and linters / Tests (1.21) (pull_request) Successful in 26m11s
Tests and linters / Tests with -race (pull_request) Successful in 28m29s
Tests and linters / Tests (1.20) (pull_request) Successful in 3m28s
It is possible that object doesn't exist in metabase.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 14:25:40 +03:00
681b2c5fd4 [#825] policer: Do not drop required linking objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 11:04:03 +00:00
a3ef7b58b4 [#755] innerring: Check container owner namespace
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m12s
DCO action / DCO (pull_request) Successful in 2m43s
Build / Build Components (1.21) (pull_request) Successful in 4m8s
Build / Build Components (1.20) (pull_request) Successful in 4m18s
Tests and linters / Staticcheck (pull_request) Successful in 4m19s
Tests and linters / Lint (pull_request) Successful in 5m30s
Tests and linters / Tests (1.20) (pull_request) Successful in 12m2s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m24s
Tests and linters / Tests with -race (pull_request) Successful in 13m4s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 12:36:34 +03:00
1cd2bfe51a [#755] morph: Drop FrostFSID contract usage
Unused.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-12 12:36:34 +03:00
70e0c1e082 [#841] ir: Execute netmap.addPeerIR only for state online
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-12-11 13:14:41 +00:00
0f45e3d344 [#804] ape: Implement boltdb storage for local overrides
All checks were successful
DCO action / DCO (pull_request) Successful in 2m10s
Vulncheck / Vulncheck (pull_request) Successful in 3m26s
Build / Build Components (1.20) (pull_request) Successful in 5m41s
Build / Build Components (1.21) (pull_request) Successful in 5m44s
Tests and linters / Staticcheck (pull_request) Successful in 7m10s
Tests and linters / Lint (pull_request) Successful in 8m14s
Tests and linters / Tests (1.21) (pull_request) Successful in 14m24s
Tests and linters / Tests (1.20) (pull_request) Successful in 14m41s
Tests and linters / Tests with -race (pull_request) Successful in 14m38s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-07 19:08:41 +03:00
e361e017f3 [#842] control: Pass target instead resource name
* Update policy-engine package version in go.mod, go.sum.
* Refactor CheckIfRequestPermitted: pass container target
  instead container ID.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-07 14:21:55 +00:00
39060382a1 [#842] control: Recieve target in gRPC methods for APE managing
* Introduce Target type and pass it to all gRPC methods
  for APE chain managing instead CID.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-12-07 14:21:55 +00:00
db49ad16cc [#826] blobovniczatree: Do not create DB's on init
All checks were successful
DCO action / DCO (pull_request) Successful in 4m10s
Build / Build Components (1.21) (pull_request) Successful in 7m30s
Vulncheck / Vulncheck (pull_request) Successful in 7m5s
Tests and linters / Staticcheck (pull_request) Successful in 10m6s
Tests and linters / Lint (pull_request) Successful in 10m26s
Build / Build Components (1.20) (pull_request) Successful in 12m39s
Tests and linters / Tests (1.20) (pull_request) Successful in 16m55s
Tests and linters / Tests (1.21) (pull_request) Successful in 17m13s
Tests and linters / Tests with -race (pull_request) Successful in 17m14s
Blobovniczas will be created on write requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
ad0697adc4 [#661] blobovnicza: Compute size with record size
To get more accurate size of blobovnicza use record
size (lenght of key + lenght of data).

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
e54dc3dc7c [#698] blobovnicza: Store counter values
Blobovnicza initialization take a long time because of bucket
Stat() call. So now blobovnicza stores counters in META bucket.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
5e8c08da3e [#661] blobstore: Add address to error logs
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
8911656b1a [#661] metrcis: Add rebuild percent metric
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
2407e5f5ff [#661] blobovniczatree: Do not sort DB's and indicies
Put stores object to next active DB, so there is no need to sort DBs.
In addition, it adds unnecessary DB openings.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
c6a739e746 [#661] blobovniczatree: Make Rebuild concurrent for objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
f1c7905263 [#661] blobovniczatree: Make Rebuild concurrent
Different DBs can be rebuild concurrently.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
d4d905ecc6 [#661] metrics: Add blobovniczatree rebuild metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
b2769ca3de [#661] blobovniczatree: Make Rebuild failover safe
Now move info stores in blobovnicza, so in case of failover
rebuild completes previous operation first.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
da4fee2d0b [#698] blobovniczatree: Init blobovniczas concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:33 +03:00
422226da18 [#661] blobovniczatree: Add Rebuild implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
a531eaf8bc [#661] blobstor: Add Rebuild implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
c1667a11d2 [#661] blobovniczatree: Allow to change depth or width
Now it is possible to change depth or with of blobovniczatree.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
484eb59893 [#661] blobovniczatree: Use .db extension for db files
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
44552a849b [#661] shard: Add blobstor rebuilder
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-07 15:37:32 +03:00
a478050639 [#838] metabase: Resolve funlen linter
All checks were successful
DCO action / DCO (pull_request) Successful in 1m59s
Vulncheck / Vulncheck (pull_request) Successful in 3m0s
Build / Build Components (1.21) (pull_request) Successful in 3m40s
Tests and linters / Lint (pull_request) Successful in 4m57s
Tests and linters / Staticcheck (pull_request) Successful in 4m38s
Build / Build Components (1.20) (pull_request) Successful in 5m17s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m1s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m39s
Tests and linters / Tests with -race (pull_request) Successful in 7m46s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
d30ab5f29e [#838] metabase: Count user objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
f314da4af3 [#838] metabase: Add user object type counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
29550fe600 [#838] shard: Refactor updateMetrics method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-12-06 15:44:21 +03:00
b2c63e57ba [#651] engine/test: Speedup StorageEngine_Inhume
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
445ebcc0e7 [#651] shard/test: Speedup Shard_Delete
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
2302e5d342 [#651] shard/test: Refactor Shard_Delete
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-11-30 13:19:43 +00:00
26e4f7005c [#741] treesvc: Refactor tree sync
Fix linter issues.
Add error logging.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-30 12:45:02 +00:00
b21be1abdd [#741] treesvc: Do not update sync height if some node is unavailable
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-30 12:45:02 +00:00
b215817e14 [#741] treesvc: Remove unused height variables
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-30 12:45:02 +00:00
c516c7c5f4 [#821] node: Pass user.ID by value
All checks were successful
DCO action / DCO (pull_request) Successful in 3m45s
Build / Build Components (1.21) (pull_request) Successful in 5m18s
Build / Build Components (1.20) (pull_request) Successful in 5m28s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m30s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m42s
Tests and linters / Lint (pull_request) Successful in 8m25s
Vulncheck / Vulncheck (pull_request) Successful in 9m22s
Tests and linters / Staticcheck (pull_request) Successful in 10m57s
Tests and linters / Tests with -race (pull_request) Successful in 16m53s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-23 10:21:07 +03:00
07390ad4e3 [#715] node: Unify config parameter names
All checks were successful
DCO action / DCO (pull_request) Successful in 2m54s
Vulncheck / Vulncheck (pull_request) Successful in 3m11s
Tests and linters / Staticcheck (pull_request) Successful in 3m55s
Build / Build Components (1.21) (pull_request) Successful in 3m51s
Build / Build Components (1.20) (pull_request) Successful in 4m6s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m10s
Tests and linters / Lint (pull_request) Successful in 5m24s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m56s
Tests and linters / Tests with -race (pull_request) Successful in 7m46s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-22 17:13:50 +03:00
8d18fa159e [#667] writecache: Fix flush test
All checks were successful
DCO action / DCO (pull_request) Successful in 1m28s
Vulncheck / Vulncheck (pull_request) Successful in 3m55s
Tests and linters / Staticcheck (pull_request) Successful in 4m15s
Build / Build Components (1.21) (pull_request) Successful in 4m40s
Build / Build Components (1.20) (pull_request) Successful in 4m54s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m32s
Tests and linters / Lint (pull_request) Successful in 5m59s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m59s
Tests and linters / Tests with -race (pull_request) Successful in 6m57s
Allow to disable background flusher for testing purposes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-17 17:45:43 +03:00
02454df14a [#809] client: Refactor PrmInit, PrmDial usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-17 13:37:03 +00:00
76ff26039c [#96] node: Drop neo-go's slices package
All checks were successful
DCO action / DCO (pull_request) Successful in 2m13s
Build / Build Components (1.20) (pull_request) Successful in 5m48s
Build / Build Components (1.21) (pull_request) Successful in 5m47s
Vulncheck / Vulncheck (pull_request) Successful in 5m47s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m33s
Tests and linters / Staticcheck (pull_request) Successful in 7m37s
Tests and linters / Lint (pull_request) Successful in 8m8s
Tests and linters / Tests (1.20) (pull_request) Successful in 10m38s
Tests and linters / Tests with -race (pull_request) Successful in 11m22s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-17 13:24:04 +03:00
47286ebf32 [#805] pilorama: Fix TreeDrop
* If treeID is empty then deleting buckets for cursor may get
  invalidated. So, buckets should be gathered before deleting.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-17 10:21:35 +00:00
29fe8c41f3 [#655] storage: Drop ErrorHandler
All checks were successful
DCO action / DCO (pull_request) Successful in 1m42s
Build / Build Components (1.21) (pull_request) Successful in 3m23s
Vulncheck / Vulncheck (pull_request) Successful in 3m25s
Build / Build Components (1.20) (pull_request) Successful in 4m16s
Tests and linters / Staticcheck (pull_request) Successful in 4m24s
Tests and linters / Lint (pull_request) Successful in 5m2s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m36s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m42s
Tests and linters / Tests with -race (pull_request) Successful in 7m18s
The only one usage was for logging.
Now logging performed by storage anyway.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-16 17:27:38 +03:00
137e987a4e [#655] storage: Drop LazyHandler
LazyHandler is implemented and used incorrectly.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-16 17:27:38 +03:00
4d5be5ccb5 [#811] ape: Update policy-engine module version and rebase
All checks were successful
DCO action / DCO (pull_request) Successful in 4m23s
Vulncheck / Vulncheck (pull_request) Successful in 5m31s
Build / Build Components (1.21) (pull_request) Successful in 7m33s
Build / Build Components (1.20) (pull_request) Successful in 7m40s
Tests and linters / Staticcheck (pull_request) Successful in 8m22s
Tests and linters / Lint (pull_request) Successful in 9m23s
Tests and linters / Tests with -race (pull_request) Successful in 11m20s
Tests and linters / Tests (1.21) (pull_request) Successful in 11m32s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m41s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-16 11:31:37 +03:00
a952a406a2 [#506] container: Use uint64 for epoch type
It is `uint64` in netmap source interfaces and other code.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-14 19:00:11 +00:00
f04806ccd3 [#506] container: Use user.ID in DeletionInfo response
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-11-14 19:00:11 +00:00
8088063195 [#787] netmap: Refactor NewEpoch method
All checks were successful
DCO action / DCO (pull_request) Successful in 3m58s
Vulncheck / Vulncheck (pull_request) Successful in 4m51s
Build / Build Components (1.21) (pull_request) Successful in 5m47s
Build / Build Components (1.20) (pull_request) Successful in 5m57s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m25s
Tests and linters / Staticcheck (pull_request) Successful in 6m31s
Tests and linters / Lint (pull_request) Successful in 7m15s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m16s
Tests and linters / Tests with -race (pull_request) Successful in 8m30s
Split for user and control methods.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:22:31 +03:00
c8a62ffedd [#787] morph: Calculate VUB and nonce when hash is nil
All checks were successful
DCO action / DCO (pull_request) Successful in 6m41s
Vulncheck / Vulncheck (pull_request) Successful in 7m54s
Build / Build Components (1.21) (pull_request) Successful in 9m29s
Build / Build Components (1.20) (pull_request) Successful in 9m46s
Tests and linters / Staticcheck (pull_request) Successful in 12m42s
Tests and linters / Tests (1.20) (pull_request) Successful in 13m51s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m50s
Tests and linters / Tests with -race (pull_request) Successful in 14m26s
Tests and linters / Lint (pull_request) Successful in 15m21s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
2393d13e4d [#787] morph: Return VUB for IR service calls
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
518f3baf41 [#787] morph: Return VUB from Invoke method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
bdfa523487 [#787] proto: Add VUB field for IR service
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-13 17:13:03 +03:00
0f75e48138 [#796] policer: Fix tombstone objects replication
Tombstone objects must be replicated to all container nodes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 13:39:33 +03:00
7cdae4f660 [#792] proto: Regenerate with fixed version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 10:09:13 +00:00
9133b4389e [#788] objectsvc: Fix formatting (gofumpt)
All checks were successful
DCO action / DCO (pull_request) Successful in 3m19s
Vulncheck / Vulncheck (pull_request) Successful in 3m40s
Build / Build Components (1.21) (pull_request) Successful in 4m17s
Build / Build Components (1.20) (pull_request) Successful in 4m32s
Tests and linters / Staticcheck (pull_request) Successful in 4m46s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m9s
Tests and linters / Lint (pull_request) Successful in 5m28s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m24s
Tests and linters / Tests with -race (pull_request) Successful in 7m38s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 10:27:32 +03:00
1b22801eed [#788] engine: Fix flaky tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-09 10:25:46 +03:00
3534d6d05b [#794] objectsvc: Return accidentally removed acl checks for Head
All checks were successful
DCO action / DCO (pull_request) Successful in 1m42s
Vulncheck / Vulncheck (pull_request) Successful in 3m23s
Build / Build Components (1.21) (pull_request) Successful in 4m22s
Build / Build Components (1.20) (pull_request) Successful in 5m44s
Tests and linters / Staticcheck (pull_request) Successful in 6m3s
Tests and linters / Lint (pull_request) Successful in 6m35s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m32s
Tests and linters / Tests with -race (pull_request) Successful in 8m47s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m54s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 17:13:58 +03:00
66848d3288 [#770] cli: Add methods to work with APE rules via control svc
* Add methods to frostfs-cli
* Implement rpc in control service

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
8e11ef46b8 [#770] object: Introduce ape chain checker for object svc
* Introduce Request type converted from RequestInfo type
  to implement policy-engine's Request interface
* Implement basic ape checker to check if a request is
  permitted to be performed
* Make put handlers use APE checker instead EACL

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
5ec73fe8a0 [#770] node: Introduce ape chain source
* Provide methods to access rule chains with access
  policy engine (APE) chain source
* Initialize apeChainSource within object service
  initialization
* Share apeChainSource with control service
* Implement dummy apeChainSource instance based on
  in-memory implementation

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
3a2c319b87 [#770] control: Generate gRPC methods to manipulate APE chains
* Define new types and gRPC methods to manipulate APE chains
  in control service.
* Stub gRPC handlers for the generated methods.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-08 13:34:03 +00:00
70ab1ebd54 [#763] metrics: Add container_objects_total metric
All checks were successful
DCO action / DCO (pull_request) Successful in 3m54s
Build / Build Components (1.20) (pull_request) Successful in 4m58s
Build / Build Components (1.21) (pull_request) Successful in 5m16s
Vulncheck / Vulncheck (pull_request) Successful in 9m54s
Tests and linters / Lint (pull_request) Successful in 10m57s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m40s
Tests and linters / Staticcheck (pull_request) Successful in 12m34s
Tests and linters / Tests with -race (pull_request) Successful in 12m48s
Tests and linters / Tests (1.20) (pull_request) Successful in 13m19s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-08 12:30:57 +03:00
9c98fa6152 [#763] metabase: Add container objects counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-08 12:30:57 +03:00
226e84d782 [#684] node: Add skipped objects count to evacuation result
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-07 12:17:11 +00:00
523fb3ca51 [#684] proto: Add skipped count to evacuation status response
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-07 12:17:11 +00:00
74c91eeef5 [#777] client: Refactor PrmContainerList, PrmObjectSearch usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-11-06 06:50:11 +00:00
20d6132f31 [#531] signSvc: Add SetMarshaledData method call
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m25s
DCO action / DCO (pull_request) Successful in 1m36s
Build / Build Components (1.21) (pull_request) Successful in 2m40s
Build / Build Components (1.20) (pull_request) Successful in 3m2s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m41s
Tests and linters / Staticcheck (pull_request) Successful in 5m32s
Tests and linters / Lint (pull_request) Successful in 5m59s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m3s
Tests and linters / Tests with -race (pull_request) Successful in 6m6s
To reduce memory allocations add `SetMarshaledData` method call
to return already marshalled data in next `StableMarshal` calls.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-02 17:34:33 +03:00
a26483fc30 [#749] morph: Fix panic when closing morph client
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-11-01 10:48:10 +00:00
c80b46fad3 [#754] blobstor: Estimate compressability
All checks were successful
DCO action / DCO (pull_request) Successful in 1m59s
Vulncheck / Vulncheck (pull_request) Successful in 3m31s
Build / Build Components (1.20) (pull_request) Successful in 4m37s
Build / Build Components (1.21) (pull_request) Successful in 4m33s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m54s
Tests and linters / Staticcheck (pull_request) Successful in 4m49s
Tests and linters / Tests with -race (pull_request) Successful in 5m9s
Tests and linters / Lint (pull_request) Successful in 6m4s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m9s
Now it is possible to enable compressability estimation.
If data is likely uncompressable, it should reduce CPU time and memory.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-11-01 11:24:32 +03:00
05b508f79a [#772] proto: Fix file ending
All checks were successful
DCO action / DCO (pull_request) Successful in 3m47s
Vulncheck / Vulncheck (pull_request) Successful in 4m47s
Build / Build Components (1.21) (pull_request) Successful in 6m5s
Build / Build Components (1.20) (pull_request) Successful in 6m13s
Tests and linters / Tests (1.20) (pull_request) Successful in 13m9s
Tests and linters / Staticcheck (pull_request) Successful in 13m17s
Tests and linters / Lint (pull_request) Successful in 13m49s
Tests and linters / Tests (1.21) (pull_request) Successful in 14m12s
Tests and linters / Tests with -race (pull_request) Successful in 4m30s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:04 +03:00
79088baa06 [#772] node: Apply gofumpt
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-31 17:03:03 +03:00
00aa6d9749 [#633] shard/test: Fix TestCounters()
Introduced in 362f24953a, forgotten to be changed because test
generator didn't provide payload size.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 12:53:28 +00:00
b8f79f4227 [#633] shard/test: Fix race conditions in TestCounters()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 12:53:28 +00:00
261d281154 [#762] go.mod: Update sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-31 11:22:29 +00:00
869518be0a [#728] writecache: Fix Badger writecache race.
All checks were successful
Build / Build Components (1.21) (pull_request) Successful in 1m47s
DCO action / DCO (pull_request) Successful in 2m1s
Vulncheck / Vulncheck (pull_request) Successful in 3m5s
Build / Build Components (1.20) (pull_request) Successful in 3m58s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m43s
Tests and linters / Staticcheck (pull_request) Successful in 5m49s
Tests and linters / Lint (pull_request) Successful in 6m20s
Tests and linters / Tests with -race (pull_request) Successful in 6m8s
Tests and linters / Tests (1.21) (pull_request) Successful in 3m6s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-30 18:36:41 +03:00
d4b6ebe7e7 [#725] writecache: Fix metric values
All checks were successful
DCO action / DCO (pull_request) Successful in 1m33s
Build / Build Components (1.21) (pull_request) Successful in 3m11s
Build / Build Components (1.20) (pull_request) Successful in 3m22s
Vulncheck / Vulncheck (pull_request) Successful in 3m34s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m0s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m1s
Tests and linters / Staticcheck (pull_request) Successful in 6m4s
Tests and linters / Tests with -race (pull_request) Successful in 3m58s
Tests and linters / Lint (pull_request) Successful in 11m43s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-27 12:22:29 +03:00
121f5c4dd8 [#757] ir: Do not exclude node in maintenance mode from netmap
All checks were successful
Build / Build Components (1.20) (pull_request) Successful in 6m20s
DCO action / DCO (pull_request) Successful in 6m45s
Vulncheck / Vulncheck (pull_request) Successful in 6m20s
Build / Build Components (1.21) (pull_request) Successful in 8m34s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m21s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m17s
Tests and linters / Staticcheck (pull_request) Successful in 9m15s
Tests and linters / Lint (pull_request) Successful in 9m42s
Tests and linters / Tests with -race (pull_request) Successful in 11m34s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-26 10:50:32 +03:00
9f7c2d8810 [#752] innerring: Simplify keyPosition()
All checks were successful
DCO action / DCO (pull_request) Successful in 3m0s
Vulncheck / Vulncheck (pull_request) Successful in 3m10s
Build / Build Components (1.20) (pull_request) Successful in 4m3s
Build / Build Components (1.21) (pull_request) Successful in 4m8s
Tests and linters / Staticcheck (pull_request) Successful in 5m13s
Tests and linters / Lint (pull_request) Successful in 5m44s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m0s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m7s
Tests and linters / Tests with -race (pull_request) Successful in 7m50s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 16:06:44 +03:00
cddc58ace2 [#752] innerring: Optimize keyPosition()
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
              │      old       │                 new                 │
              │     sec/op     │   sec/op     vs base                │
KeyPosition-8   2771.50n ± 10%   40.32n ± 4%  -98.55% (p=0.000 n=10)

              │     old      │                  new                  │
              │     B/op     │     B/op      vs base                 │
KeyPosition-8   1.531Ki ± 0%   0.000Ki ± 0%  -100.00% (p=0.000 n=10)

              │    old     │                new                 │
              │ allocs/op  │ allocs/op  vs base                 │
KeyPosition-8   21.00 ± 0%   0.00 ± 0%  -100.00% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 16:06:44 +03:00
0a9830564f [#752] morph: Adopt neo-go RPC statuses
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 16:06:44 +03:00
6950312967 [#752] morph: Drop loop copy kludges
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 16:06:44 +03:00
fe1acf9e9a [#750] morph: Remove deprecated channel use
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-25 07:57:05 +00:00
559ad58ab1 [#642] writecache: Remove usage of close channel in bbolt
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:57:50 +00:00
c0b86f2d93 [#642] writecache: Remove usage of close channel in badger
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:57:50 +00:00
b0cf100427 [#49] node: React on SIGHUP only when node in READY state
Add more info in logs when node is going to shut down,
but initialization process still in progress.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-24 15:55:29 +00:00
58b6224dd8 [#747] client: Refactor PrmObjectPutInit usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-10-20 11:55:40 +00:00
12b7cf2533 [#747] client: Refactor PrmObjectPutSingle usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-10-20 11:55:40 +00:00
dc4d27201b [#733] morph: Fix delete container signature check
Some checks failed
DCO action / DCO (pull_request) Successful in 3m26s
Vulncheck / Vulncheck (pull_request) Successful in 4m22s
Build / Build Components (1.21) (pull_request) Successful in 5m30s
Build / Build Components (1.20) (pull_request) Successful in 6m4s
Tests and linters / Staticcheck (pull_request) Successful in 6m52s
Tests and linters / Tests with -race (pull_request) Failing after 7m16s
Tests and linters / Lint (pull_request) Successful in 9m46s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m44s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m41s
Committed invalid condition, it was just for debug.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-19 18:07:37 +03:00
189dbb01be [#733] frostfs-cli: Add control ir remove-container
All checks were successful
DCO action / DCO (pull_request) Successful in 3m51s
Vulncheck / Vulncheck (pull_request) Successful in 3m49s
Build / Build Components (1.21) (pull_request) Successful in 4m34s
Build / Build Components (1.20) (pull_request) Successful in 6m21s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m1s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m12s
Tests and linters / Staticcheck (pull_request) Successful in 6m40s
Tests and linters / Tests with -race (pull_request) Successful in 9m21s
Tests and linters / Lint (pull_request) Successful in 9m39s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-19 16:22:18 +03:00
f2437f7ae9 [#734] shard: Fix Delete method
All checks were successful
DCO action / DCO (pull_request) Successful in 1m36s
Build / Build Components (1.20) (pull_request) Successful in 3m46s
Vulncheck / Vulncheck (pull_request) Successful in 3m16s
Tests and linters / Staticcheck (pull_request) Successful in 4m42s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m27s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m35s
Tests and linters / Tests with -race (pull_request) Successful in 6m33s
Build / Build Components (1.21) (pull_request) Successful in 13m7s
Tests and linters / Lint (pull_request) Successful in 19m14s
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
f26233b47a [#734] metabase: Include UpdateStorageID in metrics and traces
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
7e0c5a55de [#734] writecache: Fix flush
Now UpdateStorageID doesn't return error in case of logical error.
If object is in graveyard or GC market, it is still required to
update storage ID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:17 +03:00
3a997d1207 [#680] metrics: Initialize log metrics together with services
All checks were successful
DCO action / DCO (pull_request) Successful in 3m31s
Vulncheck / Vulncheck (pull_request) Successful in 3m17s
Build / Build Components (1.21) (pull_request) Successful in 3m56s
Tests and linters / Staticcheck (pull_request) Successful in 5m13s
Tests and linters / Lint (pull_request) Successful in 5m50s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m31s
Build / Build Components (1.20) (pull_request) Successful in 7m44s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m32s
Tests and linters / Tests with -race (pull_request) Successful in 7m33s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
994f48f8bb [#680] metrics: Export log and morph with script
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
aca11d7474 [#735] policer: Allow to provide metrics from the outside
Some checks failed
DCO action / DCO (pull_request) Successful in 3m9s
Vulncheck / Vulncheck (pull_request) Successful in 3m28s
Build / Build Components (1.21) (pull_request) Successful in 4m17s
Build / Build Components (1.20) (pull_request) Successful in 4m35s
Tests and linters / Lint (pull_request) Successful in 7m19s
Tests and linters / Tests with -race (pull_request) Failing after 9m47s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m0s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m19s
Tests and linters / Staticcheck (pull_request) Successful in 11m4s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-11 15:14:13 +03:00
5e229dc248 [#701] metrics: add metric to evaluate policer performance
Add processed objects counter in policerMetrics,
add policer field to NodeMetrics

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-10-09 19:02:08 +00:00
4caa934eea [#729] containersvc: Remove load announcement
IR code was removed in 8879c6ea.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-09 19:01:13 +00:00
d07afd803c [#726] writecache: Fix small object flush for Badger
All checks were successful
DCO action / DCO (pull_request) Successful in 2m25s
Vulncheck / Vulncheck (pull_request) Successful in 2m41s
Build / Build Components (1.21) (pull_request) Successful in 3m2s
Build / Build Components (1.20) (pull_request) Successful in 3m17s
Tests and linters / Staticcheck (pull_request) Successful in 4m26s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m9s
Tests and linters / Tests (1.21) (pull_request) Successful in 18m15s
Tests and linters / Lint (pull_request) Successful in 20m21s
Tests and linters / Tests with -race (pull_request) Successful in 18m2s
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-06 11:32:50 +03:00
997ac7cd8d [#726] writecache: Fix small object flush for BBolt
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-06 11:32:44 +03:00
c1e4130020 [#146] node: Add trace_id to logs
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 3m7s
DCO action / DCO (pull_request) Successful in 3m36s
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Build / Build Components (1.20) (pull_request) Successful in 3m37s
Tests and linters / Staticcheck (pull_request) Successful in 4m39s
Tests and linters / Lint (pull_request) Successful in 5m2s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m27s
Tests and linters / Tests with -race (pull_request) Successful in 6m29s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m19s
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-09-27 11:05:27 +03:00
3889e829e6 [#667] writecache: Add logs for report error func in tests
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-09-14 17:02:54 +00:00
c6af4a3ec8 [#679] engine: Do not increase error counter on meta mismatch
All checks were successful
DCO action / DCO (pull_request) Successful in 1m43s
Vulncheck / Vulncheck (pull_request) Successful in 2m0s
Tests and linters / Staticcheck (pull_request) Successful in 2m54s
Build / Build Components (1.20) (pull_request) Successful in 2m49s
Build / Build Components (1.21) (pull_request) Successful in 2m44s
Tests and linters / Tests (1.21) (pull_request) Successful in 3m43s
Tests and linters / Lint (pull_request) Successful in 3m58s
Tests and linters / Tests (1.20) (pull_request) Successful in 23m28s
Tests and linters / Tests with -race (pull_request) Successful in 29m0s
It was introduced in 69e1e6ca to help node determine faulty shards.
However, the situation is possible in a real-life scenario:
1. Object O is evacuated from shard A to B.
2. Shard A is unmounted because of lower-level errors.
3. We now have object in meta on A and in blobstor on B. Technically we
   have it in meta on shard B too, but we still got the error if B goes
   to a degraded mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-14 10:39:18 +03:00
8a81af5a3b [#653] Add context parameter to Open functions
All checks were successful
DCO action / DCO (pull_request) Successful in 1m38s
Build / Build Components (1.20) (pull_request) Successful in 4m22s
Build / Build Components (1.21) (pull_request) Successful in 4m25s
Vulncheck / Vulncheck (pull_request) Successful in 4m56s
Tests and linters / Lint (pull_request) Successful in 6m1s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m43s
Tests and linters / Staticcheck (pull_request) Successful in 8m1s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m14s
Tests and linters / Tests with -race (pull_request) Successful in 8m32s
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-07 18:03:29 +03:00
a716db99db [#668] shard/test: Do not alter rootPath option
Supposedly, this was added to allow creating 2 different shards without
subtest. Now we use t.TempDir() everywhere, so this should not be a
problem.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
36759f8434 [#668] shard/test: Properly check event processing
See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
39879fa868 [#668] shard/test: Add dontRelease options
Most of the time we would like to close shard with minor exceptions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
c661ba1312 [#668] shard/test: Use sane defaults in the test constructor
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
268adb79cb [#668] shard/test: Simplify shard construction
newCustomShard() has many parameters but only the first is obligatory.
`enableWriteCache` is left as-is, because it directly affects the
functionality.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
429f941cda [#668] shard/test: Release shard in t.Cleanup()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
382eb8a485 [#668] shard/test: Disable GC where it is not needed
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
42696016de [#668] shard: Close stopChannel in GC
It is done once, but now we could read it from multiple places.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
bdecfbc1be [#668] shard/test: Move tests to the main package
Semantic patch (also, duplicate definitions are removed):
```
@@
var e identifier
@@
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"

-shard.e
+e
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
aa23c6a83a [#668] shard/test: Remove subtest from TestCounters
Otherwise, individual tests cannot be run.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
da8f384324 [#668] shard/test: Fix typo in existence
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
aeeb8193d2 [#676] node: Fix header source creation when checking eacl
Some checks failed
DCO action / DCO (pull_request) Successful in 2m55s
Build / Build Components (1.20) (pull_request) Successful in 4m53s
Vulncheck / Vulncheck (pull_request) Successful in 4m36s
Tests and linters / Staticcheck (pull_request) Successful in 6m35s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m7s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m47s
Tests and linters / Tests with -race (pull_request) Failing after 10m7s
Build / Build Components (1.21) (pull_request) Successful in 11m3s
Tests and linters / Lint (pull_request) Successful in 17m34s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-09-06 17:06:54 +03:00
88d50e4c77 [#656] policer: Add "bad" testcase
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-09-06 08:04:59 +00:00
a54b4472de [#674] network: Close connections on address updates
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-04 15:34:24 +03:00
7456c8556a [#536] blobovnicza: Add blobovniczatree DB cache
All checks were successful
DCO action / DCO (pull_request) Successful in 1m33s
Vulncheck / Vulncheck (pull_request) Successful in 2m43s
Build / Build Components (1.20) (pull_request) Successful in 3m40s
Tests and linters / Staticcheck (pull_request) Successful in 3m59s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m44s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m3s
Tests and linters / Tests with -race (pull_request) Successful in 6m28s
Build / Build Components (1.21) (pull_request) Successful in 12m34s
Tests and linters / Lint (pull_request) Successful in 12m51s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:53:11 +03:00
c672f59ab8 [#536] blobovnicza: Drop cache
Each blobovnicza instance is opened
while is in use.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:51:26 +03:00
b9b86d2ec8 [#666] shard/test: Fix data race in metrics tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-31 08:39:42 +00:00
4dff9555f1 [#568] writecache: Improve flushing scheme for badger
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-30 17:22:28 +00:00
806cc13d9f [#658] client: Refactor PrmObjectGet/Head/Range usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-30 17:13:23 +00:00
1daef2ceeb [#660] writecache: Fix remaining addr2key uses
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-30 17:12:33 +00:00
fe5aa06a75 [#665] node: Bind length of copies number to number of replicas
Allow to use one digit in copies number array for backward compatibility.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-30 17:11:55 +00:00
55b82e744b [#529] objectcore: Use common sender classifier
All checks were successful
DCO action / DCO (pull_request) Successful in 2m19s
Vulncheck / Vulncheck (pull_request) Successful in 3m5s
Build / Build Components (1.21) (pull_request) Successful in 4m8s
Build / Build Components (1.20) (pull_request) Successful in 4m24s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m57s
Tests and linters / Staticcheck (pull_request) Successful in 4m43s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m2s
Tests and linters / Lint (pull_request) Successful in 5m21s
Tests and linters / Tests with -race (pull_request) Successful in 6m17s
Use common sender classifier for ACL service and format validator.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:33:06 +03:00
ae81d6660a [#529] objectcore: Fix object content validation
There are old objects where the owner of the object
may not match the one who issued the token.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:33:06 +03:00
ab2614ec2d [#528] objectcore: Validate token issuer
Add token issuer against object owner validation.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:10:10 +03:00
4ea0df77d0 [#574] policer: Check if the container was really removed
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
554ff2c06b [#574] core: Extend Source interface with DeletionInfo method
* Introduce common method EverExisted
* Define DeletionInfo for struct that must implement Source
* Refactor tree srv

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
9072772a09 [#649] shard/test: Increase GC remover interval
This was set in #348 to speed up tests.
It seems 100ms doesn't increase overall test time,
but it reduces the amount of logs by 100x factor.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
c4db8e7690 [#637] shard/test: Fix data race
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
f8ba60aa0c [#648] objsvc/delete: Handle errors in Go style
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 09:45:35 +00:00
d2084ece41 [#648] objsvc/delete: Remove redundant logs
We never propagate delete requests to the container node, because
tombstone broadcast is done via PUT. No need to pollute logs.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 09:45:35 +00:00
40b556fc19 [#647] objsvc/search: Improve testing coverage
All checks were successful
DCO action / DCO (pull_request) Successful in 1m32s
Build / Build Components (1.20) (pull_request) Successful in 3m54s
Tests and linters / Staticcheck (pull_request) Successful in 3m44s
Tests and linters / Tests (1.21) (pull_request) Successful in 4m18s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m35s
Tests and linters / Lint (pull_request) Successful in 4m42s
Vulncheck / Vulncheck (pull_request) Successful in 5m11s
Tests and linters / Tests with -race (pull_request) Successful in 6m23s
Build / Build Components (1.21) (pull_request) Successful in 8m38s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
4db2cbc927 [#647] objsvc/search: Wrap in uniqueIDWriter during parameter setting
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
966ad22abf [#647] objsvc/search: Simplify error handling
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
56f841b022 [#647] objsvc/search: Remove TraverserGenerator wrapper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
ba58144de1 [#647] objsvc/search: Remove netmap.Source wrapper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
c9e3c9956e [#643] objsvc/put: Unify extraBroadcastEnabled usage
All checks were successful
DCO action / DCO (pull_request) Successful in 1m26s
Vulncheck / Vulncheck (pull_request) Successful in 2m44s
Build / Build Components (1.20) (pull_request) Successful in 3m26s
Tests and linters / Staticcheck (pull_request) Successful in 3m51s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m4s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m46s
Tests and linters / Tests with -race (pull_request) Successful in 6m23s
Build / Build Components (1.21) (pull_request) Successful in 13m1s
Tests and linters / Lint (pull_request) Successful in 19m46s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
facd3b2c4b [#643] objsvc/put: Unify placement iterators
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
3fcf56f2fb [#643] objsvc/put: Copy config to distributedTarget
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
96e690883f [#638] Unify test loggers
In some places we have debug=false, in others debug=true.
Let's be consistent.

Semantic patch:
```
@@
@@
-test.NewLogger(..., false)
+test.NewLogger(..., true)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
322c1dc273 [#638] Use test.NewLogger() in tests
Semantic patch (restricted to **/*_test.go):
```
@@
@@
+import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger/test"
-import "go.uber.org/zap"
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger"

-&logger.Logger{Logger: zap.L()}
+test.NewLogger(t, false)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
02b03d9c4f [#638] logger: Remove sampling from test loggers
Losing logs is always a bad idea, especially when we debug tests.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
82cc453be9 [#xx] shard: Fix data race in metrics tests
Protect test metric store fields with a mutex. Probably, not every field
should be protected, but better safe than sorry.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 10:26:12 +00:00
345a1a69a2 [#635] Use internal key type when deleting from badger wc
All checks were successful
DCO action / DCO (pull_request) Successful in 2m48s
Build / Build Components (1.21) (pull_request) Successful in 4m59s
Build / Build Components (1.20) (pull_request) Successful in 5m5s
Vulncheck / Vulncheck (pull_request) Successful in 4m55s
Tests and linters / Staticcheck (pull_request) Successful in 6m51s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m21s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m53s
Tests and linters / Lint (pull_request) Successful in 8m35s
Tests and linters / Tests with -race (pull_request) Successful in 9m57s
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-22 10:53:19 +03:00
23be3eb627 [#574] tree: Check if container is really removed
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 2m37s
Build / Build Components (1.21) (pull_request) Successful in 3m36s
Tests and linters / Staticcheck (pull_request) Successful in 3m36s
DCO action / DCO (pull_request) Successful in 4m16s
Tests and linters / Lint (pull_request) Successful in 5m34s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m36s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m46s
Tests and linters / Tests with -race (pull_request) Successful in 6m21s
Build / Build Components (1.20) (pull_request) Successful in 8m1s
* Use DeletionInfo method from morph client to check if
  the container has been really removed from neo-go

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-21 12:50:20 +03:00
42fb6fb372 [#574] morph: Add DeletionInfo method for morph client
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-21 12:49:06 +03:00
354a92ea2c [#602] blobovnicza: Add leaf width implementation
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m14s
Build / Build Components (1.21) (pull_request) Successful in 3m36s
DCO action / DCO (pull_request) Successful in 4m4s
Tests and linters / Staticcheck (pull_request) Successful in 4m5s
Tests and linters / Tests (1.20) (pull_request) Successful in 4m49s
Tests and linters / Lint (pull_request) Successful in 5m18s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m44s
Tests and linters / Tests with -race (pull_request) Successful in 5m47s
Build / Build Components (1.20) (pull_request) Successful in 10m45s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-21 10:27:32 +03:00
4d9a6c07fb [#618] core: Replace fmt.Sprintf with strconv.FormatUint
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-21 07:14:50 +00:00
f2811f8585 [#602] metrics: Add blobovnicza items counter
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m29s
Build / Build Components (1.21) (pull_request) Successful in 3m23s
DCO action / DCO (pull_request) Successful in 3m50s
Tests and linters / Tests (1.21) (pull_request) Successful in 4m27s
Tests and linters / Lint (pull_request) Successful in 4m48s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m6s
Tests and linters / Staticcheck (pull_request) Successful in 5m8s
Tests and linters / Tests with -race (pull_request) Successful in 5m38s
Build / Build Components (1.20) (pull_request) Successful in 7m46s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-18 13:01:27 +03:00
c4e1d8eb07 [#602] node: Fix blobovnicza typos
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-18 11:14:10 +03:00
10e63537b2 [#602] metrics: Rename blobovnicza size metric
`Size` is not size, but open db size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-17 19:17:35 +00:00
809e97626b [#602] blobovnicza: Fix size counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-17 19:17:35 +00:00
2e49d7ea7e [#602] blobovnicza: Init before using
Fix blobovnicza size: after restart size metric resets.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-17 19:17:35 +00:00
f7042c5a6f [#609] Replace zaptest.NewLogger() with zap.L()
Semantic patch:
```
@@
@@
-import "go.uber.org/zap/zaptest"
+import "go.uber.org/zap"

-zaptest.NewLogger(t)
+zap.L()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:51:10 +00:00
127c676786 [#607] *: Use keys.PublicKeys.Copy() where possible
Semantic patch:
```
@@
var dst identifier
var src identifier
var keys identifier
@@
 import keys "github.com/nspcc-dev/neo-go/pkg/crypto/keys"

-dst := make(keys.PublicKeys, len(src))
-copy(dst, src)
+dst := src.Copy()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
e604a3d749 [#607] *: Use zap.Stringer() where possible
Semantic patch:
```
@@
var f expression
var t expression
var a expression
@@
 f(
    ...,
-    zap.String(t, a.String()),
+    zap.Stringer(t, a),
    ...,
)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
a8de37c8a2 [#607] *: Remove redundant if on error returns
Semantic patch:
```
@@
@@
-if err != nil { return err }
-return nil
+return err
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
5a51b78946 [#620] object: Send status response for server-side streams
Previously status responses were wrapped in the gRPC error and thus
couldn't be correctly handled on client.

Introduced in c2617baf63, thanks @ale64bit for having found.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 12:52:38 +00:00
6407bb5bd1 [#619] node: Fix object put when copies numbers contains only zeros
All checks were successful
DCO action / DCO (pull_request) Successful in 3m59s
Build / Build Components (1.21) (pull_request) Successful in 4m53s
Vulncheck / Vulncheck (pull_request) Successful in 4m43s
Build / Build Components (1.20) (pull_request) Successful in 5m57s
Tests and linters / Staticcheck (pull_request) Successful in 8m18s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m33s
Tests and linters / Tests with -race (pull_request) Successful in 8m41s
Tests and linters / Lint (pull_request) Successful in 8m58s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m53s
In this case object should placement according to replicas.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-17 11:31:34 +03:00
5335e7089e [#615] pilorama: Speedup TestForest_ApplyRandom()
Some of our pilorama tests fail on CI.
The reasons are not obvious, but one possible improvement
is using `WithNoSync` option for these. It should have much effect,
because we are writing on the tmpfs, but doesn't hurt anyway.

If I replace `t.TempDir()` with a local directory, test execution time
goes down from 5s (sync) to 0.4s (nosync), which is the same time as
with `t.TempDir()`. Maybe we have some strange CI configuration.

```
panic: test timed out after 10m0s
running tests:
	TestForest_ApplyRandom (8m22s)
	TestForest_ApplyRandom/bbolt (8m21s)
...
goroutine 170 [syscall]:
syscall.Syscall(0xc000100000?, 0xc00047b758?, 0x6aff9a?, 0xc00041c1b0?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/syscall_linux.go:69 +0x27
syscall.Fdatasync(0x9e35c0?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/zsyscall_linux_amd64.go:418 +0x2a
go.etcd.io/bbolt.fdatasync(0xc000189000?)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-16 19:09:08 +00:00
2efe9cc1be [#585] writecache: Fix DB counter
Some checks failed
DCO action / DCO (pull_request) Successful in 2m14s
Vulncheck / Vulncheck (pull_request) Successful in 2m44s
Build / Build Components (1.21) (pull_request) Successful in 2m54s
Build / Build Components (1.20) (pull_request) Successful in 3m5s
Tests and linters / Staticcheck (pull_request) Successful in 3m43s
Tests and linters / Lint (pull_request) Successful in 5m12s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m44s
Tests and linters / Tests (1.20) (pull_request) Successful in 21m5s
Tests and linters / Tests with -race (pull_request) Failing after 26m26s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
58c8722c81 [#585] fstree: Add optional file counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
baad49990c [#585] fstree: Return logical error if object deleted
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
0c52186572 [#585] fstree: Remove unused method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
eec97d177e [#585] writecache: Count items periodically
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
d15199c5d8 [#596] engine: Consider context errors as logical
Some checks failed
Vulncheck / Vulncheck (pull_request) Successful in 2m59s
DCO action / DCO (pull_request) Successful in 2m54s
Build / Build Components (1.20) (pull_request) Successful in 4m2s
Build / Build Components (1.21) (pull_request) Successful in 4m51s
Tests and linters / Staticcheck (pull_request) Successful in 14m8s
Tests and linters / Tests (1.20) (pull_request) Failing after 14m56s
Tests and linters / Lint (pull_request) Successful in 15m27s
Tests and linters / Tests (1.21) (pull_request) Failing after 15m36s
Tests and linters / Tests with -race (pull_request) Failing after 16m18s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-16 10:39:41 +03:00
88b6755c5e [#598] Fix use-after-close bug in badger writecache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-15 07:04:06 +00:00
ae8be495c8 [#xx] Avoid manual management of files in tests
Some checks failed
Vulncheck / Vulncheck (pull_request) Successful in 2m36s
Build / Build Components (1.21) (pull_request) Successful in 3m33s
DCO action / DCO (pull_request) Successful in 3m33s
Tests and linters / Staticcheck (pull_request) Successful in 4m15s
Tests and linters / Lint (pull_request) Successful in 5m31s
Build / Build Components (1.20) (pull_request) Successful in 7m2s
Tests and linters / Tests with -race (pull_request) Failing after 8m51s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m55s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m6s
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-14 14:01:39 +03:00
376f03a445 [#598] Hold mode mutex when setting mode
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-14 07:12:27 +00:00
21800e9fcc [#162] core: Move literals to constants
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-11 15:48:42 +03:00
b44a8dd46c [#597] *: Fix linter warnings
Some checks failed
Vulncheck / Vulncheck (pull_request) Successful in 6m26s
Tests and linters / Staticcheck (pull_request) Successful in 7m20s
Build / Build Components (1.21) (pull_request) Successful in 8m31s
Build / Build Components (1.20) (pull_request) Successful in 8m43s
Tests and linters / Tests (1.21) (pull_request) Failing after 15m48s
Tests and linters / Tests (1.20) (pull_request) Failing after 16m2s
Tests and linters / Tests with -race (pull_request) Failing after 16m28s
Tests and linters / Lint (pull_request) Successful in 19m8s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-11 09:55:51 +03:00
e084c47bd6 [#248] innerring: Remove audit from tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
779da6ec35 [#248] morph: Remove audit contract name
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
55ce4dc075 [#248] morph: Remove obsolete network parameters
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
4ad0ebb32f [#565] Add metrics for current GRPC endpoint status
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-10 16:37:49 +00:00
c3e23a1448 [#578] gendoc: Allow to override flags
The command is used in multiple places across the whole FrostFS
ecosystem. While we want to have uniform interfaces everywhere,
sometimes we can't: already defined global flags can be harder to change
because of our obligations to the users. Cobra framework doesn't allow
conflicting flags (we can have global ones), so allow to override them.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
6bcba27757 [#578] gendoc: Remove flag shorthands
Make it harder to encounter conflicts in already existing commands.
Because the command is executed once, I don't think the usability is
worse.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
dca31e8888 [#578] gendoc: Add date to man pages
```
HISTORY
       8-Aug-2023 Auto generated by spf13/cobra
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
b02a1a34c1 [#578] gendoc: Allow to customize man pages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
082370583f [#578] gendoc: Replace NSPCC with FrostFS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
8f994163ee [#586] Fix writecache benchmarks and refactor hacky NeedsCompression
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-10 08:05:18 +00:00
023b90342c [#584] Disable compression in badger writecache
Some checks failed
Build / Build Components (1.20) (pull_request) Failing after 3s
Build / Build Components (1.21) (pull_request) Failing after 2s
Tests and linters / Lint (pull_request) Failing after 4s
Tests and linters / Tests (1.20) (pull_request) Failing after 3s
Tests and linters / Tests (1.21) (pull_request) Failing after 3s
Tests and linters / Tests with -race (pull_request) Failing after 3s
Tests and linters / Staticcheck (pull_request) Failing after 3s
Vulncheck / Vulncheck (pull_request) Failing after 3s
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-10 10:14:06 +03:00
d641cba2fc [#587] Do not use math/rand.Read
Some checks failed
Build / Build Components (1.20) (pull_request) Failing after 3s
Build / Build Components (1.21) (pull_request) Failing after 3s
Tests and linters / Lint (pull_request) Failing after 3s
Tests and linters / Tests (1.20) (pull_request) Failing after 3s
Tests and linters / Tests (1.21) (pull_request) Failing after 3s
Tests and linters / Tests with -race (pull_request) Failing after 5s
Tests and linters / Staticcheck (pull_request) Failing after 8s
Vulncheck / Vulncheck (pull_request) Failing after 2s
Fix staticcheck warnings after go1.20 update.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-09 16:02:44 +03:00
5b7e4a51b7 [#481] Update frostfs-sdk-go and error pointer receivers
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-09 10:26:53 +00:00
ae322e9f73 [#576] Set SyncWrites for badger writecache by default
Some checks failed
Build / Build Components (1.19) (pull_request) Failing after 3s
Build / Build Components (1.20) (pull_request) Failing after 2s
Tests and linters / Tests (1.19) (pull_request) Failing after 3s
Tests and linters / Tests with -race (pull_request) Failing after 3s
Tests and linters / Lint (pull_request) Failing after 12s
Tests and linters / Staticcheck (pull_request) Failing after 3s
Vulncheck / Vulncheck (pull_request) Successful in 5m46s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m37s
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-09 11:28:26 +03:00
8d589314b5 [#560] node: Fix Put in multi REP with intersecting sets of nodes
Once the node was processed it skipped, at the step of forming
result in case when all nodes skipped, because processed for
previous REP, service mark the whole request as incomplete.

Example of policies which are unblocked:
- REP 1 REP 1 CBF 1
- REP 4 IN X REP 4 IN Y
  CBF 4
  SELECT 2 FROM FX AS X SELECT 2 FROM FY AS Y
  FILTER Country EQ Russia OR Country EQ Sweden OR Country EQ Finland AS FY
  FILTER Price GE 0 AS FX

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-08 10:22:53 +00:00
7da4306e38 [#575] writecache: Fix log level for badger writecache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-08 08:16:36 +00:00
0e697266c3 [#563] writecache: Fix metrics and bolt delete
Estimate cache size after delete objects to update metric.
Update counters on small object deletion.
Do not count bbolt DB file as FSTree object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-07 12:53:28 +00:00
1a0cb0f34a [#421] Try using badger for the write-cache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-07 08:16:57 +00:00
65c72f3e0b [#559] Remove manual path handling in fstree tests
Some checks failed
Build / Build Components (1.19) (pull_request) Successful in 2m8s
Build / Build Components (1.20) (pull_request) Successful in 1m54s
Tests and linters / Lint (pull_request) Successful in 2m48s
Tests and linters / Tests (1.19) (pull_request) Successful in 1m53s
Tests and linters / Tests (1.20) (pull_request) Successful in 2m4s
Tests and linters / Tests with -race (pull_request) Successful in 5m21s
Tests and linters / Staticcheck (pull_request) Successful in 2m16s
Vulncheck / Vulncheck (pull_request) Failing after 16m3s
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-03 10:03:41 +03:00
1e8b4b8a17 [#557] services: Regenerate stable marshalers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-02 13:37:40 +00:00
b3695411d9 [#553] eacl: Fix bug with casting to ObjectAccessDenied error
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-02 07:22:48 +00:00
35dc64bd7b [#547] metabase: Fix datarace in tests
Quite an old one bf9e938a3b.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
05ac9e3637 [#547] objectsvc: Work with traversal struct from a single thread
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
7b0fdf0202 [#533] services: Assume API supports status codes
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
ec8b4fdc48 [#541] writecache/test: Close writecache on exit
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-29 10:57:25 +00:00
ad5f527bd3 [#541] writecache/test: Remove initWC()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-29 10:57:25 +00:00
ea32913430 [#543] putsvc: Fix PutSingle implementation
Add Lock and Delete handlers to local PutSingle.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-28 12:09:41 +00:00
99bb488ebd [#539] getsvc: Write payload direct to out stream
Some checks failed
Build / Build Components (1.20) (pull_request) Failing after 1s
Build / Build Components (1.19) (pull_request) Failing after 3s
Tests and linters / Lint (pull_request) Failing after 2s
Tests and linters / Tests (1.19) (pull_request) Failing after 1s
Tests and linters / Tests (1.20) (pull_request) Failing after 2s
Tests and linters / Tests with -race (pull_request) Failing after 1s
Vulncheck / Vulncheck (pull_request) Failing after 2s
Tests and linters / Staticcheck (pull_request) Failing after 2s
To reduce memory allocations.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-27 17:02:08 +03:00
286242cad0 [#539] getsvc: Use buffer to assemble object
To reduce memory consumption.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-27 17:02:08 +03:00
5ff82ff04f [#6] services/object: Simplify local/remote targets
Some checks failed
ci/woodpecker/pr/pre-commit Pipeline was successful
Tests and linters / Tests (1.19) (pull_request) Successful in 14m18s
Vulncheck / Vulncheck (pull_request) Failing after 2m23s
Tests and linters / Staticcheck (pull_request) Successful in 4m8s
Build / Build Components (1.19) (pull_request) Successful in 3m26s
Build / Build Components (1.20) (pull_request) Successful in 2m56s
Tests and linters / Tests with -race (pull_request) Successful in 21m39s
Tests and linters / Tests (1.20) (pull_request) Successful in 22m36s
Tests and linters / Lint (pull_request) Successful in 1m39s
ci/woodpecker/push/pre-commit Pipeline was successful
We do not use the return result from Close() and we always execute both
methods in succession. It makes sense to unite them.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:39:12 +03:00
448b48287c [#6] services/util: Do not panic in sign function
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:39:12 +03:00
c2617baf63 [#6] services/util: Remove remaining stream wrappers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:39:09 +03:00
372160d048 [#6] services/util: Remove SignService.HandleUnaryRequest
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:36:08 +03:00
fef172c5b0 [#6] services/util: Simplify response.Service
It has only 1 parameter, which is obligatory.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:29:57 +03:00
eed594431f [#335] treesvc: Add GetSubTree ordering unit test
All checks were successful
Build / Build Components (1.20) (pull_request) Successful in 3m47s
Build / Build Components (1.19) (pull_request) Successful in 4m2s
Tests and linters / Tests (1.20) (pull_request) Successful in 3m8s
ci/woodpecker/pr/pre-commit Pipeline was successful
Tests and linters / Staticcheck (pull_request) Successful in 3m48s
Tests and linters / Tests with -race (pull_request) Successful in 7m59s
Tests and linters / Tests (1.19) (pull_request) Successful in 15m36s
Tests and linters / Lint (pull_request) Successful in 18m12s
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
af82c2865e [#335] treesvc: Fix inmemory unit tests and nil meta items
Bolt forest saves empty slice of items. Now inmemory forest
does it the same way.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
b4e72a2dfd [#335] treesvc: Sort nodes by Filename in GetSubTree
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
f0355a453e [#463] policer: Remove capacity rebalance logic
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Current implementation has some quirks. For example,
using only half of object.put.pool_size_remote threads
tells replicator that is node is 50% loaded,
but in reality we could be putting lot's of big objects.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
8b78db74bc [#463] replicator: Add tracing span
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
8966dd8e35 [#463] putsvc: Use PutSingle RPC for remote target
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
3b66f98f27 [#519] Use Address.Equals in policer tests
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-17 10:45:07 +00:00
3e8de14e7d [#382] evacuate: Fix unit tests
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-17 10:24:05 +00:00
397131b0ea [#520] objectcore: Refactor format validator
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Remove redundant FIXME.
Move error to consts.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-17 07:07:21 +00:00
11027945d8 [#479] writecache: Fix writecache fstree flush premature ctx cancel
All checks were successful
Build / Build Components (1.20) (pull_request) Successful in 3m49s
Build / Build Components (1.19) (pull_request) Successful in 4m2s
ci/woodpecker/pr/pre-commit Pipeline was successful
Tests and linters / Tests (1.20) (pull_request) Successful in 6m13s
Tests and linters / Lint (pull_request) Successful in 11m25s
Tests and linters / Tests with -race (pull_request) Successful in 9m4s
Tests and linters / Staticcheck (pull_request) Successful in 6m0s
Tests and linters / Tests (1.19) (pull_request) Successful in 25m11s
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-14 10:25:52 +03:00
8a9fc2c372 [#510] treesvc: Rename tableFromBearer to useBearer
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
With impersonation, the old name is no longer descriptive.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-12 10:23:21 +00:00
b8bcfac531 [#510] treesvc: Fix panic in bearer token processing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-12 10:23:21 +00:00
24eb988897 [#294] deletesvc: Drop cast
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
c83e7c875f [#294] searchsvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
e8091101c7 [#294] searchsvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
ec9b738465 [#294] putsvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
800a685e84 [#294] putsvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
1420b8b9ea [#294] deletesvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
a476d8285a [#294] deletesvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
70a1081988 [#294] aclsvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
18d8898b00 [#294] aclsvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
61541eaec2 [#294] aclsvc: Refactor checker constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
7da284f3e8 [#509] go.mod: Update sdk-go
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Tests and linters / Lint (pull_request) Successful in 2m15s
Tests and linters / Tests (1.19) (pull_request) Successful in 5m7s
Tests and linters / Tests with -race (pull_request) Successful in 4m50s
Tests and linters / Staticcheck (pull_request) Successful in 4m10s
Build / Build Components (1.19) (pull_request) Successful in 3m47s
Build / Build Components (1.20) (pull_request) Successful in 3m30s
Tests and linters / Tests (1.20) (pull_request) Successful in 20m39s
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-11 17:38:00 +03:00
80481c015c [#509] putsvc: Omit AccessIdentifiers from iteratePlacement()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-11 17:38:00 +03:00
a65e26878b [#486] put service: Fix error typo
Some checks failed
ci/woodpecker/pr/pre-commit Pipeline failed
Build / Build Components (1.19) (pull_request) Successful in 4m12s
Tests and linters / Tests (1.19) (pull_request) Successful in 2m52s
Tests and linters / Tests (1.20) (pull_request) Successful in 3m49s
Tests and linters / Staticcheck (pull_request) Successful in 3m13s
Tests and linters / Lint (pull_request) Successful in 10m50s
Build / Build Components (1.20) (pull_request) Successful in 11m28s
Tests and linters / Tests with -race (pull_request) Successful in 7m54s
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
fcbf90d31b [#486] node: Add PutSingle implemetation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
7b76527759 [#486] node: Add PutSingle wrappers
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
140d970a95 [#498] policer: Fix objectSDK import
Some checks failed
ci/woodpecker/pr/pre-commit Pipeline failed
Build / Build Components (1.19) (pull_request) Successful in 4m23s
Tests and linters / Tests (1.19) (pull_request) Successful in 3m27s
Tests and linters / Lint (pull_request) Successful in 4m11s
Tests and linters / Tests with -race (pull_request) Successful in 5m36s
Build / Build Components (1.20) (pull_request) Successful in 10m18s
Tests and linters / Staticcheck (pull_request) Successful in 5m41s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m33s
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-07 18:52:40 +03:00
2310a5c7ba [#498] policer: Allow to set sleep duration between iterations
Some checks failed
ci/woodpecker/push/pre-commit Pipeline failed
Speed up tests on CI.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-07 14:52:33 +00:00
e858479a74 [#498] policer: Explicitly Rewind() iterator after finish
Previously, we can continue to return `EndOfListing` infinitely.
Reflect iterator reuse via Rewind() method.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-07 14:52:33 +00:00
033eaf77e1 [#496] node: Fix linter importas
All checks were successful
Build / Build Components (1.20) (pull_request) Successful in 3m52s
Build / Build Components (1.19) (pull_request) Successful in 4m1s
ci/woodpecker/pr/pre-commit Pipeline was successful
Tests and linters / Tests with -race (pull_request) Successful in 5m36s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m55s
Tests and linters / Lint (pull_request) Successful in 14m40s
Tests and linters / Tests (1.19) (pull_request) Successful in 15m29s
ci/woodpecker/push/pre-commit Pipeline was successful
Standardize the alias of the
import frostfs-sdk-go/object as objectSDK.

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-07-06 15:36:41 +03:00
6eefe9747e treesvc: Do not provide credentials unless TLS is used
Some checks failed
ci/woodpecker/push/pre-commit Pipeline failed
4f413fe86e was perfect, except it did the opposite of what we needed.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-06 08:04:28 +00:00
f354b8a270 [#473] Add tests for rem values of traverser
All checks were successful
Build / Build Components (1.20) (pull_request) Successful in 3m52s
Build / Build Components (1.19) (pull_request) Successful in 4m8s
ci/woodpecker/pr/pre-commit Pipeline was successful
Tests and linters / Lint (pull_request) Successful in 9m3s
Tests and linters / Tests with -race (pull_request) Successful in 5m27s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m18s
Tests and linters / Tests (1.19) (pull_request) Successful in 13m54s
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-07-05 20:13:23 +03:00
c6df6c84ae [#473] Do not modify traverse plan when [0] copies number vector is provided
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-07-05 19:54:43 +03:00
90e9a85acc [#483] Update dependencies
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-03 15:44:59 +03:00
8d16d95376 [#484] morph: Add expired tx logging
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-03 08:03:25 +00:00
26acf5689e [#92] Ensure policer objects cannot be worked on concurrently
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
3223402c90 [#92] Embed policer's objectsInWork mutex
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
f9730f090d [#92] Refactor policer and add some unit tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
f437ab8f15 [#197] object: Make Delete method return correct status
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-06-30 12:58:56 +00:00
d01c064674 [#455] Fix non-informative message when parse node attributes
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-28 15:02:29 +00:00
d0ab552a90 [#478] *: Fix funlen linter warnings
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 15:01:49 +00:00
8a4e250dae [#468] *: replace outdated TODO crypto-related links
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
4f413fe86e [#1] treesvc: Properly check for secure transport
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
cab51c8cbe [#1] metabase: Rename blindlyProcess()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
f4c71cea65 [#1] *: Replace outdated FIXME/TODO links
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
0c866f62c5 [#473] placement: Fix backwards compatibility for copies_number
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-27 11:13:16 +03:00
43d263c3d5 [#428] linter: Fix unkeyed assignment
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Thanks to gopls.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 13:29:39 +00:00
cac4ed93d6 [#428] engine: Add low_mem config parameter
Concurrent initialization in case of the metabase resync leads to
high memory consumption and potential OOM.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 13:29:39 +00:00
4bf345225c [#447] pilorama: Use named constant for the key size
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
b4ce0b0412 [#447] pilorama: Do not undo log for create ops
```
goos: linux
goarch: amd64
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
         │     old      │                 new                  │
         │    sec/op    │    sec/op     vs base                │
Create-8   36.48µ ± 11%   30.34µ ± 14%  -16.84% (p=0.000 n=10)

         │     old      │                 new                  │
         │     B/op     │     B/op      vs base                │
Create-8   43.01Ki ± 4%   37.78Ki ± 5%  -12.15% (p=0.000 n=10)

         │    old     │                new                 │
         │ allocs/op  │ allocs/op   vs base                │
Create-8   166.0 ± 3%   146.0 ± 3%  -12.05% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
dd3874eff1 [#447] pilorama: Add benchmark for create ops
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
71889234b7 [#449] tree: Allow reading requests signed by keys from allow list
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-23 14:44:19 +03:00
40eae22109 [#460] services/util: Remove CreateRequestStreamer
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:08:04 +03:00
a64dc9ad70 [#460] services/util: Remove HandleServerStreamRequest
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:08:01 +03:00
167a67f0b8 [#460] services/util: Remove HandleUnaryRequest
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:07:56 +03:00
785d81a68a [#460] services/object: Reduce distibutedTarget memory footprint
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:07:50 +03:00
4d48377cec [#459] blobovniczatree: Fix get error
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 16:44:53 +03:00
03aa210145 [#373] metrics: Move labels to consts
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
To unify label naming all lable keys and other consts are moved to
one file.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
b5d9f4a285 [#373] metrics: Add pilorama metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
e89fa110c7 [#373] metrics: Add metabase metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
af608da952 [#373] metrics: Add blobovnizca metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
56f320dd85 [#373] metrics: Add blobstor metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
16a142cd0c [#373] metrics: Add FSTree metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
d8ecc69d00 [#373] local storage: Pass parent ID
This is required to add shard ID as metric label.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
d5aaec1107 [#373] pilorama: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
059e9e88a2 [#373] metabase: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
f54cc0b607 [#373] blobstor: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
8318d90ad0 [#373] blobovniczatree: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
3ae3c8dfdb [#373] fstree: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
a8526d45e9 [#373] blobovnizca: Add missed/fix tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
028d4a8058 [#373] blobovnicza: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
01a0c97760 [#453] engine: Set Disabled mode to deleted shard
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-20 12:04:07 +03:00
50caa388b0 [#303] ir: Use pub key when validate container deletion
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-20 08:02:48 +00:00
69df0d21c2 [#446] engine: Move to read-only on blobstor errors
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:53:32 +03:00
fe01781811 [#446] los: Wrap SSD errors in a separate type
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:53:31 +03:00
20b84f183a [#446] engine: Simplify logs for shard mode change
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:51:29 +03:00
69b788a90b [#424] metrics: Rename mode to mode_info
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
To follow best practice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 18:39:49 +03:00
26b305f82b [#424] morph: Fix cache metrics
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Use separate morph cache metrics for node and IR

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 18:26:19 +03:00
4449006862 [#424] metrics: Use mode value as metric value for shard
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 18:26:19 +03:00
847732605c [#424] metrics: Rename counter to total
Suffix total should be used for a unit-less accumulating count.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
c348ae35b0 [#424] metrics: Drop embedded metrics
It was not obvious where metrics are used.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
1b364d8cf4 [#424] metrics: Refactor engine metrics
Use histogram vector to measure request duration.
Fix naming like in Prometheus best practice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
c8023a9c8d [#424] morphcache: Use labels for method duration
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
85deb12f4d [#424] writecache: Drop metrics when close
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:52:40 +03:00
07f155ac77 [#424] metrics: Use labels for writecache methods and operations
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:52:40 +03:00
71bbeddb64 [#424] metrics: Drop unused arg
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:51:40 +03:00
fb8fee0c8e [#442] Use strconv.FormatBool instead of untyped conversion
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-14 11:08:36 +00:00
344d6b2ae1 [#xx] Fix invalid log metric namespace identifier
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-13 15:36:34 +03:00
4887f489a1 [#17] Add morph client metrics
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-13 14:06:04 +03:00
90e9247b69 [#371] ir: Add morph cache metrics
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-06-13 10:10:13 +00:00
957a43a124 [#266] services/tree: Add sync check
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Do not accept requests until initial sync is finished.
`Apply` is deliberately left out -- we don't want to miss anything new.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
e69a1e8482 [#266] services/tree: Return operation log up to some height
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
2541d319de [#266] pilorama: Allow to get current tree height
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
0c40d98f7a [#375] Add log metrics
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-13 09:52:45 +00:00
0400153b7d [#424] object: Do not store large slices in pool
Dynamically growing an unbounded buffers can cause a large
amount of memory to be pinned and never be freed.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 07:19:55 +00:00
41ab4d070e [#423] *: Use hrw.StringHash() where possible
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-13 07:18:25 +00:00
9bc1a25c07 [#434] metrics: Fix writecache duration
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 07:18:01 +00:00
5b75432ca2 [#431] metrics: Add missed label
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-07 13:51:36 +00:00
263c6fdc50 [#372] node: Add metrics for the error counter in the engine
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-07 13:04:47 +00:00
63473d0806 [#417] go.mod: Update neo-go
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Update to master, because after the API update notary signer is being
incorrectly formed.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 11:43:48 +00:00
55c28fd5f4 [#412] cache: Pass DialOptions to create new conn
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
dcdfb6ed41 [#412] node: Use observability interceptors
Use metrics and tracing interceptors.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
c09144ecf1 [#412] node: Replace metrics package
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
74578052f9 [#412] node: Replace tracing package
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
f7c4c07453 [#xx] Create innerring metrics before metrics usage
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-01 10:07:48 +03:00
4476a1dbaf [#413] morph/client: Fix govet warnings
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-31 15:38:17 +03:00
dbf41391b5 [#401] engine: Extend evacuation logs
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Add operation-tag to logger.
Log evacuation results.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 13:24:30 +03:00
3220c4df9f [#376] metrics: Add GC metrics
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 10:22:12 +00:00
faca861451 [#411] Remove unnecessary pointers for sync objects
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-31 10:19:14 +00:00
ebcc8afbee [#374] Add inner-ring event metrics
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-31 10:11:48 +00:00
8dcd06c587 [#394] node: Use Context in Blobovniczas.Iterate()
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-31 10:09:18 +00:00
365a7ca0f4 [#366] node: Stop GC once termination signal received
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-29 09:35:08 +03:00
802168c0c6 [#364] node: Stop flushing big object when termination signal received
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-26 16:46:58 +03:00
271a56c2ab [#395] metrics: Drop redundant metrics
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
HistogramVec already has labeled counter.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 13:39:37 +00:00
bc34fee6a7 [#370] Add tree service metrics
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-26 13:39:12 +00:00
f2e5dead7e [#398] pilorama: Disallow applying same operations
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
1. In redo() we save the old state.
2. If we do redo() for the same operation twice, the old state will be
   overritten with the new one.
3. This in turn affects undo() and subsequent isAncestor() check.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-26 12:14:29 +00:00
20a489bdb5 [#393] gc: Use defer to mark handler done
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 12:14:02 +00:00
9119199f6e [#397] cli: Fix evacuation method names
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 11:28:30 +03:00
2613351008 [#387] gc: Cancel GC is change mode requested
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-25 09:38:16 +03:00
2ce43935f9 [#312] metrics: Add writecache metrcis
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
d212d908b5 [#312] wc: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
4503a61997 [#312] wc: Delete unused Iterate method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
53a1b15693 [#338] ir: Drop notaryless code from governance
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
0ab589dd52 [#338] ir: Always enable notary on sidechain
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
81718afb39 [#338] ir: Drop named named put fee
Named put fee value used only when notary disabled.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
656fd7f376 [#338] ir: Drop notaryless code from netmap
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
fb708b3a2d [#338] ir: Drop container notaryless code
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
4b768fd115 [#381] *: Move to sync/atomic
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-23 08:18:01 +03:00
ff570847a4 [#381] go.mod: Update bbolt
Adopt new `ForEachBucket` function where possible.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-22 11:49:14 +03:00
35c9b6b26d [#314] writecache: remove objects right after they are flushed
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-19 09:37:05 +00:00
Pavel Karpy
bf79d06f03 [#314] writecache: Do not lose small objects on disk errors
Do return error if an object could not been stored on WC's disk.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-19 09:37:05 +00:00
Pavel Karpy
9e56592be3 [#314] writecache: Simplify background workers naming
Also, drop not used arg.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-19 09:37:05 +00:00
f7c0b50d70 [#329] cli: Add async evacuate commands
Add start, stop evacuate and evacuate status commands.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
e4889e06ba [#329] node: Make evacuate async
Now it's possible to run evacuate shard in async.
Also only one evacuate process can be in progress.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
100b1b5128 [#329] node: Add async evacuate proto methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
13a7a90101 [#355] Increase tree svc client cache size to test hypotheses
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-17 14:11:46 +03:00
6f47c75e43 [#125] ir: Set extra wallets on SIGHUP
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-16 12:47:43 +00:00
869fcbf591 [#332] gc: Fix expired complex object deletion
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-16 12:44:57 +00:00
ab07bad33d [#332] gc: Add complex object unit test
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-16 12:44:57 +00:00
a6ee7a3087 [#324] Add replicator metrics
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-12 11:00:17 +00:00
6055b18362 [#342] morph: Remove unused toStackParameter()
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 13:24:49 +03:00
02c02974b3 [#309] locode: Parallelize DB generation
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
For v0.4.0 release:
Before:
```
Executed in  571.64 secs    fish           external
   usr time  283.07 secs  744.00 micros  283.07 secs
   sys time    8.41 secs  179.00 micros    8.41 secs
```

After:
```
Executed in   54.23 secs    fish           external
   usr time  418.65 secs    1.01 millis  418.65 secs
   sys time    0.61 secs    0.25 millis    0.60 secs
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:59:13 +00:00
c62025c836 [#321] metabase/test: execute tests in parallel
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
945454f60c [#321] engine/test: Execute tests in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
4578d00619 [#321] shard/test: Execute tests in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
d35e4c389f [#321] shard/test: Parallelize TestWriteCacheObjectLoss
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
969bfb603f [#321] shard/test: Parallelize TestShard_List
```
go test -count=1 -run TestShard_List -race .
Before: 2.492s
After:  0.109s
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
47b0ec33c3 [#337] netmap: Remove unused events
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Done in TrueCloudLab/frostfs-contract#16.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
b480df4985 [#337] container: Remove unused events
Done in TrueCloudLab/frostfs-contract#16.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
Anna Shaleva
bcdb0f330d [#337] morph: Completely remove fallbackTime from client cfg
It's unused and not needed, default fallback lifetime is set by Notary
actor.

Signed-off-by: Anna Shaleva <anna@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
Anna Shaleva
ddcc156ecc [#337] morph: Use Notary Actor for notary requests
Signed-off-by: Anna Shaleva <anna@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
35fdf6f315 [#337] morph: Move subscription logic to subscriber
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
Roman Khimov
a5f118a987 [#337] subscriber: Drop unused UnsubscribeForNotification
It's not really needed, closing the connection works fine when exiting and
normally the app doesn't need to unsubscribe at all.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
800eb5e983 [#125] ir: Reconfigure pprof and metrics on SIGHUP
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-11 12:38:34 +03:00
a181c9e434 [#332] gc: Add additional logging
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 17:34:40 +03:00
90799497d3 [#114] Add remove-node IR control command
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-10 14:31:44 +00:00
ea10abb42a [#296] morph: Add parser unit tests
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 09:40:26 +03:00
1309622b20 [#296] morph: Add listener unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 09:40:26 +03:00
b2ffd7df53 [#291] object: Use PayloadSizeLimiter from SDK
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-05 19:07:06 +03:00
35ea207df6 [#291] object: Split validating target in two
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-05 19:07:06 +03:00
Pavel Karpy
ee58b390bb [#221] node: Allow using vector copies_number
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Also, take into account that value in general (it was not used before at
all).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-05 16:07:13 +00:00
973af12854 [#327] tests: replace os.MkdirTemp with t.TempDir
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-05 11:24:51 +00:00
cedd07bbc8 [#304] morph: Iterate endpoints when create ws client in constructor
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-05 10:07:30 +03:00
Pavel Karpy
479c5a65e1 [#322] node: Fix tree svc panic
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
If a connection has not been established earlier, it stores `nil` in LRU
cache. Cache eviction tries to close every connection (even a `nil` one) and
panics but not crash the app because we are using pools.
That ugly bug also leads to a deadlock where `Unlock` is not called via
`defer` func (and that is the way I found it).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-04 19:48:51 +03:00
2f6757c828 [#311] ir: Fix data race in unit tests
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-04 16:59:15 +03:00
a1b4ba9980 [#209] compression: Do not store uncompressible data
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 16:28:29 +03:00
529d0bc710 [#302] tree: Drop unused ctx
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 16:54:06 +03:00
eca5c210dd [#299] evacuate: Add context cancel checks
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 15:55:30 +03:00
b939e4e5c5 [#299] ir: Drop unused structs
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 14:05:35 +03:00
235fe84ea3 [#298] innerring: Fix broken tests
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-02 11:52:29 +03:00
d00b1c0d29 [#280] ir: Add netmap processor unit tests
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
fb5dcc15d2 [#280] ir: Add governance processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
31b4da225a [#280] ir: Add frostfs processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
5010b35466 [#280] ir: Add container processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
686f01bce5 [#280] ir: Add balance processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
e89fa7f69f [#280] ir: Add alphabet processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
53693071de [#280] ir: Add block timer unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
e70f808dc3 [#280] ir: Add parser unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
2dbe382b5f [#280] ir: Add state unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
500611f3c8 [#280] ir: Add indexer tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
3b64dffda2 [#280] ir: Add fee config tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
22d47376a6 [#277] getsvc: Refactor errors
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Move errors to separate files.
Use zap.Error for error logging.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
45438e7b06 [#277] getsvc: Rename and reorder code
Rename execCtx to request.
Move code to appropriate files.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
1440450606 [#277] getsvc: Drop cyclic struct dependency
Drop cyclic dependency between execCtx and Service.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
591c4e7d50 [#277] getsvc: Move headOnly to request params
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
265d2326a0 [#277] getsvc: Extract remote storage
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
30e1b62b67 [#277] getsvc: Fix service deps
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
8fc082b688 [#277] getsvc: Do not return status error
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
f368ccbdf0 [#248] morph: Remove audit client
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
a9c4ba62c3 [#248] metabase: Remove storage group bucket
Backwards compatible change, so no version increase.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
586f5986bc [#248] core: Remove storage group
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
f1ea8fec93 [#248] object_manager: Remove storage group
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
7f49f07255 [#248] services: Remove audit service
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
8879c6ea4a [#248] innerring: Remove audit and settlement code
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
f73ac6e02d [#290] control: Use generics for response wrappers
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 10:57:24 +03:00
ff25521204 [#270] Add IR epoch tick control call
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-04-28 07:57:00 +00:00
58f1ba4b51 [#288] pilorama: Add missing operation in log
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
daa26f6e9b [#288] pilorama/test: Check operation order for TreeGetByPath()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
291f9e809a [#288] pilorama: Remove getMeta() wrapper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
0045f1bcd4 [#288] pilorama: Use more descriptive names for memory tree
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
f856ad7480 [#288] pilorama: Remove childMap from memory forest
Memory forest is here to check the correctness of boltdb optimized
implementation. Let's keep it simple.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
ada081dfd5 [#19] node: Make policier read shards concurrently
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
* Introduce ListWithMultiCursor that simultaneously reads objects
  from different shards

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-27 11:11:47 +03:00
1f4061c0e2 [#285] blobonicza: Optimize upperPowerOfTwo()
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/push/pre-commit Pipeline was successful
The real reason is this:
```
pkg/local_object_storage/blobovnicza/sizes.go:36:69
    revive empty-block: this block is empty, you can remove it
```

Didn't want to make this function longer or to add `nolint`, thus this
change. To justify:
```
UpperBound/size=1-8          0.4924n ± 1%   0.2472n ± 2%  -49.80% (p=0.000 n=10)
UpperBound/size=1023-8       0.4936n ± 3%   0.2442n ± 1%  -50.52% (p=0.000 n=10)
UpperBound/size=66560-8      0.8201n ± 2%   0.2436n ± 1%  -70.29% (p=0.000 n=10)
UpperBound/size=41943040-8   6.6900n ± 5%   0.2432n ± 0%  -96.36% (p=0.000 n=10)
geomean                       1.075n        0.2446n       -77.24%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:11 +03:00
dfe4ada838 [#285] lint: Resolve revive/if-return
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:08 +03:00
f07e2d4812 [#285] lint: Fix revive/unused-parameter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:04 +03:00
9d01029733 [#166] node: Parallelize background tree service sync by batching
All checks were successful
ci/woodpecker/push/pre-commit Pipeline was successful
* Merge operations

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-26 10:17:56 +00:00
299b24b974 [#166] node: Parallelize background tree service sync by batching
* Concurrently dispatch TreeApply operations for batching in forest

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-26 10:17:56 +00:00
Denis Kirillov
dce5924a89 [#229] services/tree: Use bearer owner as signer
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-26 10:23:33 +03:00
89530534a1 [#229] service/tree: Disable container owner check in tree service
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-04-26 10:23:33 +03:00
c04f6c5e59 [#229] acl: Allow Impersonate
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-04-26 10:23:33 +03:00
6fef2726b8 [#164] metrics: Allow to export metrics description
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00
4ade5339da [#164] metrics: Rename metrics.go -> node.go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00