Commit graph

3438 commits

Author SHA1 Message Date
327d364f34 [#1262] sdnotify: Get rid of go:linkname for nanotime
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-05 12:37:16 +00:00
dc3dcabadc [#1291] morph: Reconnect to the highest priority endpoint
All checks were successful
DCO action / DCO (pull_request) Successful in 3m13s
Vulncheck / Vulncheck (pull_request) Successful in 5m3s
Tests and linters / Run gofumpt (pull_request) Successful in 4m49s
Tests and linters / Staticcheck (pull_request) Successful in 5m20s
Tests and linters / Tests (1.22) (pull_request) Successful in 5m25s
Pre-commit hooks / Pre-commit (pull_request) Successful in 5m40s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m39s
Tests and linters / Tests with -race (pull_request) Successful in 5m35s
Tests and linters / Lint (pull_request) Successful in 5m46s
Build / Build Components (1.22) (pull_request) Successful in 6m8s
Build / Build Components (1.21) (pull_request) Successful in 6m11s
Tests and linters / gopls check (pull_request) Successful in 6m21s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-02 17:54:22 +03:00
8021bacc43 [#1288] putSvc: Respect TTL for EC put
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-02 13:01:42 +00:00
7e04083c27 [#1278] containerSvc: Validate FrostFSID subject exitence on Put
All checks were successful
DCO action / DCO (pull_request) Successful in 1m52s
Vulncheck / Vulncheck (pull_request) Successful in 2m18s
Build / Build Components (1.21) (pull_request) Successful in 2m52s
Tests and linters / Lint (pull_request) Successful in 2m56s
Build / Build Components (1.22) (pull_request) Successful in 2m32s
Tests and linters / Staticcheck (pull_request) Successful in 3m2s
Tests and linters / gopls check (pull_request) Successful in 3m25s
Tests and linters / Tests (1.22) (pull_request) Successful in 5m16s
Pre-commit hooks / Pre-commit (pull_request) Successful in 5m3s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m5s
Tests and linters / Tests with -race (pull_request) Successful in 8m53s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-29 16:01:37 +03:00
a12c39667d [#1278] ir: Do not allow to create container without FrostFSID record
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-29 16:01:37 +03:00
8377372a40 [#1276] go.mod: Update api-go
All checks were successful
Tests and linters / Staticcheck (pull_request) Successful in 2m54s
Build / Build Components (1.22) (pull_request) Successful in 3m28s
Build / Build Components (1.21) (pull_request) Successful in 3m16s
Tests and linters / Lint (pull_request) Successful in 4m14s
Tests and linters / Tests with -race (pull_request) Successful in 8m2s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m41s
Tests and linters / Tests (1.22) (pull_request) Successful in 8m54s
Tests and linters / gopls check (pull_request) Successful in 2m53s
DCO action / DCO (pull_request) Successful in 46s
Vulncheck / Vulncheck (pull_request) Successful in 1m4s
Pre-commit hooks / Pre-commit (pull_request) Successful in 1m47s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-26 16:44:19 +03:00
7fd7961dfa [#1271] getSvc: Fix local EC chunk get
All checks were successful
Build / Build Components (1.21) (pull_request) Successful in 4m22s
Tests and linters / Staticcheck (pull_request) Successful in 4m56s
Tests and linters / gopls check (pull_request) Successful in 5m59s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m20s
DCO action / DCO (pull_request) Successful in 35s
Build / Build Components (1.22) (pull_request) Successful in 1m15s
Vulncheck / Vulncheck (pull_request) Successful in 59s
Tests and linters / Lint (pull_request) Successful in 2m24s
Pre-commit hooks / Pre-commit (pull_request) Successful in 3m1s
Tests and linters / Tests (1.22) (pull_request) Successful in 6m59s
Tests and linters / Tests with -race (pull_request) Successful in 7m32s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-24 08:52:16 +03:00
8398a8b4b3 [#1271] getSvc: Fix head --raw assemble for EC
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-24 08:52:08 +03:00
9c2c76ca32
[#1259] Move pkg/util/locode to frostfs-locode-db
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 4m47s
DCO action / DCO (pull_request) Successful in 4m40s
Build / Build Components (1.21) (pull_request) Successful in 7m27s
Tests and linters / Staticcheck (pull_request) Successful in 8m22s
Tests and linters / gopls check (pull_request) Successful in 10m56s
Build / Build Components (1.22) (pull_request) Successful in 11m48s
Tests and linters / Tests (1.22) (pull_request) Successful in 16m11s
Tests and linters / Lint (pull_request) Successful in 16m27s
Pre-commit hooks / Pre-commit (pull_request) Successful in 19m12s
Tests and linters / Tests (1.21) (pull_request) Successful in 20m31s
Tests and linters / Tests with -race (pull_request) Successful in 8m56s
Removed pkg/util/locode package, added
git.frostfs.info/TrueCloudLab/frostfs-locode-db/pkg/locode dependency.

Signed-off-by: George Bartolomey <george@bh4.ru>
2024-07-23 14:59:14 +03:00
9ee1bd3669
[#1265] *: Run gofumpt
All checks were successful
DCO action / DCO (pull_request) Successful in 7m50s
Vulncheck / Vulncheck (pull_request) Successful in 11m12s
Build / Build Components (1.22) (pull_request) Successful in 15m55s
Build / Build Components (1.21) (pull_request) Successful in 16m2s
Pre-commit hooks / Pre-commit (pull_request) Successful in 20m47s
Tests and linters / Staticcheck (pull_request) Successful in 4m8s
Tests and linters / gopls check (pull_request) Successful in 4m16s
Tests and linters / Lint (pull_request) Successful in 5m35s
Tests and linters / Tests with -race (pull_request) Successful in 7m58s
Tests and linters / Tests (1.22) (pull_request) Successful in 8m12s
Tests and linters / Tests (1.21) (pull_request) Successful in 2m59s
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-07-23 10:36:32 +03:00
18182e578e [#1261] shard: Fix delete objects from FSTree
All checks were successful
DCO action / DCO (pull_request) Successful in 6m56s
Vulncheck / Vulncheck (pull_request) Successful in 7m38s
Build / Build Components (1.22) (pull_request) Successful in 9m57s
Build / Build Components (1.21) (pull_request) Successful in 10m38s
Tests and linters / gopls check (pull_request) Successful in 10m43s
Tests and linters / Staticcheck (pull_request) Successful in 11m1s
Tests and linters / Lint (pull_request) Successful in 13m35s
Pre-commit hooks / Pre-commit (pull_request) Successful in 14m53s
Tests and linters / Tests (1.21) (pull_request) Successful in 15m55s
Tests and linters / Tests with -race (pull_request) Successful in 15m49s
Tests and linters / Tests (1.22) (pull_request) Successful in 16m10s
Replace nil storageID with empty like by shard.Get.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-22 14:38:09 +03:00
3119f2fbc3 [#1257] metabase: Delete EC gc marks and split info
All checks were successful
DCO action / DCO (pull_request) Successful in 2m12s
Vulncheck / Vulncheck (pull_request) Successful in 4m9s
Pre-commit hooks / Pre-commit (pull_request) Successful in 5m37s
Tests and linters / gopls check (pull_request) Successful in 6m7s
Build / Build Components (1.21) (pull_request) Successful in 6m2s
Build / Build Components (1.22) (pull_request) Successful in 5m55s
Tests and linters / Staticcheck (pull_request) Successful in 6m55s
Tests and linters / Lint (pull_request) Successful in 9m2s
Tests and linters / Tests (1.21) (pull_request) Successful in 11m56s
Tests and linters / Tests with -race (pull_request) Successful in 11m47s
Tests and linters / Tests (1.22) (pull_request) Successful in 11m55s
EC parent and split gc marks should be deleted after last EC chunk delete.
Also should delete split info for EC parent and split parent.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-19 17:14:14 +03:00
eadcea8df0 [#1249] object: Remove all APE pre-checks in handlers
* Methods `Head`, `Get`, `GetRangeHash` should no longer use APE pre-checks
  as that leads only to incorrect rule chain processing for requests:
  1. Immediate return with `NoRuleFound` may be unexpected as some `Allow`
     rule is actually defined but can't be matched yet as it gets no object
     attributes;
  2. Immdediate return with `Allow` may be incorrect as some `Deny` rule
     is actually defined but can't bet matched yet as it gets no object
     attirbutes;
  3. Pre-check breaks compatibility for converted EACL-tables.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-18 13:52:43 +00:00
5e5ee545b6 [#1254] policer: Fix svacer warning
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-18 11:30:39 +00:00
27caa8a72f [#1256] metabase: Put split parent object ID for EC chunks
All checks were successful
DCO action / DCO (pull_request) Successful in 6m13s
Vulncheck / Vulncheck (pull_request) Successful in 6m44s
Tests and linters / gopls check (pull_request) Successful in 9m52s
Build / Build Components (1.22) (pull_request) Successful in 10m1s
Build / Build Components (1.21) (pull_request) Successful in 10m33s
Tests and linters / Staticcheck (pull_request) Successful in 11m5s
Pre-commit hooks / Pre-commit (pull_request) Successful in 14m47s
Tests and linters / Lint (pull_request) Successful in 16m21s
Tests and linters / Tests (1.22) (pull_request) Successful in 16m22s
Tests and linters / Tests with -race (pull_request) Successful in 16m32s
Tests and linters / Tests (1.21) (pull_request) Successful in 16m47s
It is required to save split parent ID too, not only split ID.
Otherwise inhume operation works incorrect: shard with last part may be skipped
and parent object will be available.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-17 18:08:07 +03:00
e83d39e33f [#1253] deleteSvc: Use copy of common parameters
All checks were successful
DCO action / DCO (pull_request) Successful in 3m29s
Vulncheck / Vulncheck (pull_request) Successful in 3m31s
Build / Build Components (1.21) (pull_request) Successful in 4m56s
Build / Build Components (1.22) (pull_request) Successful in 4m57s
Tests and linters / Staticcheck (pull_request) Successful in 5m14s
Tests and linters / gopls check (pull_request) Successful in 5m11s
Tests and linters / Lint (pull_request) Successful in 6m23s
Tests and linters / Tests (1.22) (pull_request) Successful in 12m42s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m54s
Tests and linters / Tests with -race (pull_request) Successful in 16m31s
Pre-commit hooks / Pre-commit (pull_request) Successful in 21m57s
getSvc may change the values of some fields, so Head will affect Delete
or Put. In this case, the change is necessary so that the session token
is stored in the tombstone object (EC assemble calls `ForgetTokens`).

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-17 14:24:27 +03:00
fc383ea6ae [#1253] getSvc: Fix EC objects get
Now EC objects assembling is performed concurrently.
Also fixed issue with an error in case of getting
EC object via non-container node.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-17 14:24:27 +03:00
00a88b9936 [#1251] *: Run gofumpt
All checks were successful
DCO action / DCO (pull_request) Successful in 3m50s
Vulncheck / Vulncheck (pull_request) Successful in 3m37s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m45s
Build / Build Components (1.21) (pull_request) Successful in 5m7s
Build / Build Components (1.22) (pull_request) Successful in 5m1s
Tests and linters / Staticcheck (pull_request) Successful in 5m13s
Tests and linters / gopls check (pull_request) Successful in 5m30s
Tests and linters / Lint (pull_request) Successful in 6m31s
Tests and linters / Tests (1.21) (pull_request) Successful in 11m39s
Tests and linters / Tests with -race (pull_request) Successful in 11m37s
Tests and linters / Tests (1.22) (pull_request) Successful in 13m11s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-17 11:25:07 +03:00
3940bc17c1 [#1251] pilorama: Allow traversing multiple branches in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-17 11:25:07 +03:00
e5767c9002 [#1250] *: Reformat proto filets with clang-format
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 15:13:08 +00:00
286df198c9 [#1248] placement: Decouple ContainerNodes() cache from the placement builder.
Also, write tests.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 12:21:30 +00:00
3a48b282b6 [#1248] placement: Use epoch to track netmap versions
Previously we used pointer, this could have worked,
because most of the time, the netmap is cached.
This didn't work, however, because `lastNm` field was always nil.

Rework the mechanism completely:
1. Use epoch to track netmap versions, as it it simpler and
   is unrelated to the TTL of an underlying cache.
2. Fix a bug where the epoch could change while mutex was unlocked.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 12:21:30 +00:00
21431f22c0 [#1248] placement: Use cid.ID as key in the cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 12:21:30 +00:00
d5dc14c639 [#1243] object: Make APE checker set x-headers to request properties
* Update go.mod, go.sum;
* Add x-headers to request properties;
* Add a unit-test.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-16 07:28:42 +00:00
39866a957c [#1196] morph/client: Fix ArrayFromStackItem() description
All checks were successful
DCO action / DCO (pull_request) Successful in 2m38s
Pre-commit hooks / Pre-commit (pull_request) Successful in 3m26s
Vulncheck / Vulncheck (pull_request) Successful in 3m44s
Build / Build Components (1.21) (pull_request) Successful in 6m6s
Build / Build Components (1.22) (pull_request) Successful in 6m1s
Tests and linters / gopls check (pull_request) Successful in 7m47s
Tests and linters / Staticcheck (pull_request) Successful in 8m23s
Tests and linters / Lint (pull_request) Successful in 8m53s
Tests and linters / Tests (1.21) (pull_request) Successful in 11m5s
Tests and linters / Tests with -race (pull_request) Successful in 11m10s
Tests and linters / Tests (1.22) (pull_request) Successful in 11m36s
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-07-15 16:24:58 +03:00
d02a7c2d38 [#1196] morph/client: Remove duplicate utility functions
* We used several utility functions to parse frostfsid client
  subject and extended subject. However, following the changes
  in TrueCloudLab/frostfs-contract#97, these utility functions
  have become public. So there is no more need to have them here.

* There was a mismatch of slice parameter required length between
  frostfs-node's and frostfs-contract's utility functions,
  `checkStackItem()` solves this problem.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-07-15 16:24:48 +03:00
3bf6e6dde6 [#1238] engine: Add non-blocking send in the shard's notification channel
All checks were successful
DCO action / DCO (pull_request) Successful in 17m26s
Build / Build Components (1.21) (pull_request) Successful in 19m19s
Vulncheck / Vulncheck (pull_request) Successful in 19m46s
Build / Build Components (1.22) (pull_request) Successful in 23m58s
Pre-commit hooks / Pre-commit (pull_request) Successful in 32m59s
Tests and linters / Staticcheck (pull_request) Successful in 5m9s
Tests and linters / Lint (pull_request) Successful in 5m47s
Tests and linters / gopls check (pull_request) Successful in 5m22s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m37s
Tests and linters / Tests with -race (pull_request) Successful in 8m38s
Tests and linters / Tests (1.22) (pull_request) Successful in 8m56s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-07-10 11:37:11 +03:00
b027a7f91e [#1234] pilorama: Fix GetByPath() on duplicate directories
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-10 06:11:38 +00:00
6ace2f597e [#1234] pilorama: Add test for duplicate directory behaviour
When AddByPath() is called concurrently on 2 different nodes,
internal path components may be created twice. This violates some
of our assumptions in GetByPath() and, indirectly, in S3 handling of
GetSubTree() results.

Add a test for the correct behaviour, fixes will follow.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-10 06:11:38 +00:00
84ecd61dfd [#1233] putSvc: Try to put EC chunk to any node
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-09 07:54:29 +00:00
d90aab5454 [#1229] util: Fix session token expiration check
* Make session token expired at `current_epoch + 1` but
  not at `current_epoch` when it's still valid.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-08 08:15:56 +00:00
0c2b6f3dac [#1216] ape: Make services use bearer chains fed router
* Refactor object and tree service - they should instantiate
  chain router cheking the bearer token. If there are no bearer
  token rules, then defaul chain router is used.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-05 18:26:48 +00:00
47bcd346d3 [#1216] ape: Introduce BearerChainFeedRouter
* Unlike default chain router, `BearerChainFedRouter` performs checks for
  overrides defined in the bearer token;
* Add unit-test for the introduced router.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-05 18:26:48 +00:00
8eb591d668 [#1231] policer: Add EC node-off unit test
Some checks failed
Build / Build Components (1.21) (pull_request) Successful in 10m29s
Build / Build Components (1.22) (pull_request) Successful in 10m24s
DCO action / DCO (pull_request) Successful in 10m29s
Vulncheck / Vulncheck (pull_request) Successful in 12m18s
Tests and linters / gopls check (pull_request) Successful in 21m40s
Tests and linters / Tests (1.21) (pull_request) Failing after 22m2s
Pre-commit hooks / Pre-commit (pull_request) Successful in 23m40s
Tests and linters / Staticcheck (pull_request) Successful in 24m10s
Tests and linters / Tests (1.22) (pull_request) Successful in 24m19s
Tests and linters / Lint (pull_request) Successful in 25m30s
Tests and linters / Tests with -race (pull_request) Successful in 25m15s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-05 12:11:36 +03:00
cfd5e3d403 [#1227] morph/event: Release ants.Pool on listener stopping
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-04 10:55:05 +00:00
62cbb72a5e [#1226] blobovniczatree: Delete fix db extensions in Init()
Some checks failed
DCO action / DCO (pull_request) Successful in 14m27s
Vulncheck / Vulncheck (pull_request) Successful in 16m45s
Build / Build Components (1.21) (pull_request) Successful in 18m53s
Build / Build Components (1.22) (pull_request) Successful in 20m30s
Tests and linters / gopls check (pull_request) Successful in 25m12s
Tests and linters / Tests (1.21) (pull_request) Failing after 27m52s
Tests and linters / Tests with -race (pull_request) Failing after 27m46s
Tests and linters / Tests (1.22) (pull_request) Failing after 28m13s
Tests and linters / Lint (pull_request) Successful in 30m25s
Tests and linters / Staticcheck (pull_request) Successful in 30m23s
Pre-commit hooks / Pre-commit (pull_request) Successful in 31m27s
Since several releases have been released, this code is not relevant.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 12:22:06 +03:00
78b1d9b18d [#1226] blobovniczatree: Drop leaf width limitation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 12:22:06 +03:00
40c9ddb6ba [#1226] blobovniczatree: Drop init in advance option
To make blobovniczatree unlimited.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 12:22:06 +03:00
3a797e4682 [#1222] engine: Fix tree evacuation
All checks were successful
DCO action / DCO (pull_request) Successful in 13m4s
Vulncheck / Vulncheck (pull_request) Successful in 15m32s
Build / Build Components (1.22) (pull_request) Successful in 16m30s
Build / Build Components (1.21) (pull_request) Successful in 16m37s
Tests and linters / gopls check (pull_request) Successful in 20m33s
Tests and linters / Staticcheck (pull_request) Successful in 21m17s
Tests and linters / Lint (pull_request) Successful in 21m37s
Tests and linters / Tests with -race (pull_request) Successful in 26m40s
Tests and linters / Tests (1.21) (pull_request) Successful in 26m54s
Tests and linters / Tests (1.22) (pull_request) Successful in 27m59s
Pre-commit hooks / Pre-commit (pull_request) Successful in 39m39s
Do not fail if it is unable to evacuate tree to other node.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 10:38:10 +03:00
2bac82cd6f [#1222] engine: Fix object evacuation
Do not fail evacuation if it unable to evacuate object to other node.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 10:38:10 +03:00
bbe95dac8b [#1225] engine: Log the error when check object existence
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-07-04 07:22:54 +00:00
80d7459560 Revert "[#1196] morph/client: Remove duplicate utility functions"
Some checks failed
Tests and linters / Tests (1.21) (pull_request) Failing after 27s
DCO action / DCO (pull_request) Successful in 3m38s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m16s
Vulncheck / Vulncheck (pull_request) Failing after 3m43s
Build / Build Components (1.22) (pull_request) Successful in 4m45s
Build / Build Components (1.21) (pull_request) Successful in 4m54s
Tests and linters / Staticcheck (pull_request) Successful in 5m10s
Tests and linters / gopls check (pull_request) Successful in 5m15s
Tests and linters / Lint (pull_request) Successful in 5m53s
Tests and linters / Tests with -race (pull_request) Successful in 9m12s
Tests and linters / Tests (1.22) (pull_request) Successful in 9m43s
This reverts commit 259007540f.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-03 15:44:35 +03:00
4bd4667791 [#1218] tree: Fix bearer token validation
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-03 07:22:11 +00:00
f3a861806e [#1218] object: Fix bearer token validation
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-03 07:22:11 +00:00
a378ff9cf6 [#1218] object: Pass container owner for backward get method check
* `getStreamBasicChecker` must define `containerOwner` for backward checks,
  otherwise bearer token cannot be validated for the token issuer.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-03 07:22:11 +00:00
91bed3b0ba [#1219] Remove Container.SetEACL method
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-02 13:05:40 +00:00
259007540f [#1196] morph/client: Remove duplicate utility functions
All checks were successful
Build / Build Components (1.21) (pull_request) Successful in 4m1s
DCO action / DCO (pull_request) Successful in 4m3s
Vulncheck / Vulncheck (pull_request) Successful in 3m51s
Pre-commit hooks / Pre-commit (pull_request) Successful in 5m15s
Tests and linters / Lint (pull_request) Successful in 6m59s
Build / Build Components (1.22) (pull_request) Successful in 8m54s
Tests and linters / gopls check (pull_request) Successful in 10m46s
Tests and linters / Staticcheck (pull_request) Successful in 11m30s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m37s
Tests and linters / Tests (1.22) (pull_request) Successful in 13m43s
Tests and linters / Tests with -race (pull_request) Successful in 13m44s
We used several utility functions to parse frostfsid client
subject and extended subject. However, following the changes
in TrueCloudLab/frostfs-contract#97, these utility functions
have become public. So there is no more need to have them here.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-07-02 15:19:59 +03:00
56eeb630b6 [#1217] Fix grammar mistakes and misspelling
All checks were successful
DCO action / DCO (pull_request) Successful in 13m23s
Tests and linters / gopls check (pull_request) Successful in 16m29s
Tests and linters / Lint (pull_request) Successful in 17m2s
Build / Build Components (1.21) (pull_request) Successful in 16m26s
Build / Build Components (1.22) (pull_request) Successful in 16m21s
Vulncheck / Vulncheck (pull_request) Successful in 16m40s
Tests and linters / Staticcheck (pull_request) Successful in 19m9s
Pre-commit hooks / Pre-commit (pull_request) Successful in 21m37s
Tests and linters / Tests with -race (pull_request) Successful in 22m33s
Tests and linters / Tests (1.22) (pull_request) Successful in 3m46s
Tests and linters / Tests (1.21) (pull_request) Successful in 3m51s
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-07-01 19:14:25 +03:00
7a8ac4907a [#1213] engine: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-01 06:49:35 +00:00
dc2867682f [#1213] deleteSvc: Do not allow to delete EC chunks
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-01 06:49:35 +00:00
dc6778f385 [#1213] fmt: Drop unused interfaces
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-01 06:49:35 +00:00
7085723c6b [#1074] pilorama: Allow empty filenames in SortedByFilename()
All checks were successful
DCO action / DCO (pull_request) Successful in 5m38s
Vulncheck / Vulncheck (pull_request) Successful in 5m57s
Tests and linters / gopls check (pull_request) Successful in 6m57s
Build / Build Components (1.22) (pull_request) Successful in 6m48s
Build / Build Components (1.21) (pull_request) Successful in 6m52s
Tests and linters / Lint (pull_request) Successful in 8m26s
Tests and linters / Staticcheck (pull_request) Successful in 8m48s
Tests and linters / Tests (1.22) (pull_request) Successful in 11m39s
Tests and linters / Tests (1.21) (pull_request) Successful in 11m47s
Tests and linters / Tests with -race (pull_request) Successful in 13m49s
Pre-commit hooks / Pre-commit (pull_request) Successful in 17m24s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-28 17:46:24 +03:00
4c7ff159ec [#1201] writecache: Cancel background flush without lock
All checks were successful
DCO action / DCO (pull_request) Successful in 3m14s
Vulncheck / Vulncheck (pull_request) Successful in 5m14s
Tests and linters / gopls check (pull_request) Successful in 6m26s
Tests and linters / Lint (pull_request) Successful in 7m29s
Tests and linters / Staticcheck (pull_request) Successful in 7m22s
Build / Build Components (1.22) (pull_request) Successful in 7m55s
Build / Build Components (1.21) (pull_request) Successful in 8m1s
Pre-commit hooks / Pre-commit (pull_request) Successful in 12m8s
Tests and linters / Tests with -race (pull_request) Successful in 12m41s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m3s
Tests and linters / Tests (1.22) (pull_request) Successful in 13m32s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-28 09:02:36 +03:00
4f7d76c9ef [#1206] audit: Drop not required events
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-27 10:54:31 +00:00
4951babd5f [#1208] blobstor: Fix delete without storage id
All checks were successful
DCO action / DCO (pull_request) Successful in 7m14s
Vulncheck / Vulncheck (pull_request) Successful in 15m56s
Build / Build Components (1.22) (pull_request) Successful in 17m0s
Build / Build Components (1.21) (pull_request) Successful in 17m5s
Tests and linters / gopls check (pull_request) Successful in 18m17s
Tests and linters / Lint (pull_request) Successful in 21m1s
Tests and linters / Staticcheck (pull_request) Successful in 20m41s
Pre-commit hooks / Pre-commit (pull_request) Successful in 22m18s
Tests and linters / Tests (1.21) (pull_request) Successful in 3m0s
Tests and linters / Tests (1.22) (pull_request) Successful in 2m55s
Tests and linters / Tests with -race (pull_request) Successful in 3m42s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-27 11:59:08 +03:00
81ea91de52 [#451] metrics: Move to internal
`metrics` don't look like something others want to import.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-25 08:52:37 +00:00
11a38a0a84 [#1190] tree: GroupIDs must also be target of APE checks
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-25 08:49:20 +00:00
0b87388c18 [#1190] object: GroupIDs must also be target of APE checks
* Also add new test case for ape middleware in container service.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-25 08:49:20 +00:00
621dbf58ab [#1190] container: GroupIDs must also be target of APE checks
* Also add new test case for ape middleware in container service.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-25 08:49:20 +00:00
a1f7615b7e [#1190] ape: Introduce Groups util function to retrieve actor's groupIDs
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-25 08:49:20 +00:00
a83eeddb1d [#60] control: Add GetNetmapStatus method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-20 16:28:42 +03:00
9ac74efc41 [#1173] shard: Use mode from config on reload
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-06-20 11:29:10 +00:00
40b68bcb6c [#1109] object: Validate attribute EXPIRATION_EPOCH on put
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-06-19 17:19:27 +00:00
fd28461def [#1184] ir: Add grpc middleware for control service
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 3m14s
DCO action / DCO (pull_request) Successful in 3m46s
Build / Build Components (1.21) (pull_request) Successful in 4m20s
Build / Build Components (1.22) (pull_request) Successful in 4m21s
Tests and linters / gopls check (pull_request) Successful in 4m45s
Tests and linters / Lint (pull_request) Successful in 5m16s
Tests and linters / Staticcheck (pull_request) Successful in 6m24s
Pre-commit hooks / Pre-commit (pull_request) Successful in 8m54s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m57s
Tests and linters / Tests with -race (pull_request) Successful in 10m52s
Tests and linters / Tests (1.22) (pull_request) Successful in 11m20s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-19 16:05:53 +03:00
ecd1ed7a5e [#1184] node: Add audit middleware for grpc services
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-19 16:05:53 +03:00
75eedf71f3 [#1187] pilorama/test: Remove debug print
Some checks failed
DCO action / DCO (pull_request) Successful in 19m36s
Vulncheck / Vulncheck (pull_request) Successful in 30m13s
Tests and linters / gopls check (pull_request) Successful in 33m40s
Build / Build Components (1.22) (pull_request) Successful in 34m44s
Tests and linters / Lint (pull_request) Successful in 37m6s
Tests and linters / Tests (1.22) (pull_request) Failing after 40m59s
Tests and linters / Tests (1.21) (pull_request) Failing after 41m56s
Tests and linters / Tests with -race (pull_request) Failing after 45m4s
Build / Build Components (1.21) (pull_request) Successful in 44m45s
Tests and linters / Staticcheck (pull_request) Successful in 15m42s
Pre-commit hooks / Pre-commit (pull_request) Successful in 1h0m46s
Introduced in e12fcc041d.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-18 15:09:44 +03:00
5b100699d7 [#566] policer: Move isClientErrMaintenance to frostfs-sdk-go
All checks were successful
DCO action / DCO (pull_request) Successful in 1m5s
Vulncheck / Vulncheck (pull_request) Successful in 4m8s
Pre-commit hooks / Pre-commit (pull_request) Successful in 5m5s
Build / Build Components (1.21) (pull_request) Successful in 4m47s
Build / Build Components (1.22) (pull_request) Successful in 4m44s
Tests and linters / Lint (pull_request) Successful in 6m13s
Tests and linters / Staticcheck (pull_request) Successful in 6m3s
Tests and linters / gopls check (pull_request) Successful in 6m16s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m19s
Tests and linters / Tests with -race (pull_request) Successful in 9m15s
Tests and linters / Tests (1.22) (pull_request) Successful in 9m22s
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-06-18 10:20:45 +03:00
76cf7a051b [#1178] shard: Check metabase existence before read shard ID
All checks were successful
DCO action / DCO (pull_request) Successful in 6m15s
Build / Build Components (1.22) (pull_request) Successful in 11m1s
Build / Build Components (1.21) (pull_request) Successful in 11m6s
Tests and linters / Staticcheck (pull_request) Successful in 16m15s
Tests and linters / gopls check (pull_request) Successful in 16m11s
Tests and linters / Lint (pull_request) Successful in 17m34s
Vulncheck / Vulncheck (pull_request) Successful in 7m47s
Pre-commit hooks / Pre-commit (pull_request) Successful in 21m25s
Tests and linters / Tests (1.21) (pull_request) Successful in 21m22s
Tests and linters / Tests with -race (pull_request) Successful in 21m51s
Tests and linters / Tests (1.22) (pull_request) Successful in 2m46s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-17 09:59:15 +03:00
96fe271bab [#1170] innerring: Support morph mTLS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-14 09:28:32 +03:00
42ecc2f2b9 [#1170] morph: Support mTLS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-14 09:26:27 +03:00
68ac490729 [#1174] shard: Update metric mode_info on Init
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-06-13 08:32:59 +00:00
6a39c3d15e [#1086] engine: Do not use metabase if shard looks bad
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-13 07:35:22 +00:00
9d73f9c2c6 Reapply "[#446] engine: Move to read-only on blobstor errors"
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-13 07:35:22 +00:00
b9fcaad21f [#1168] shard: Set Disabled as default mode for components
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-06-11 15:13:38 +00:00
6cf512e574 [#1166] blobovniczatree: Handle blobovnicza's NoSpaceLeft error
All checks were successful
DCO action / DCO (pull_request) Successful in 10m57s
Vulncheck / Vulncheck (pull_request) Successful in 11m50s
Build / Build Components (1.21) (pull_request) Successful in 14m43s
Build / Build Components (1.22) (pull_request) Successful in 16m38s
Tests and linters / gopls check (pull_request) Successful in 18m17s
Tests and linters / Staticcheck (pull_request) Successful in 23m58s
Tests and linters / Lint (pull_request) Successful in 25m50s
Pre-commit hooks / Pre-commit (pull_request) Successful in 27m13s
Tests and linters / Tests (1.21) (pull_request) Successful in 3m5s
Tests and linters / Tests (1.22) (pull_request) Successful in 4m3s
Tests and linters / Tests with -race (pull_request) Successful in 4m23s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-07 17:15:43 +03:00
e7d479f4c2 [#1166] blobovnicza: Return NoSpaceLeft error instead of syscall.ENOSPC
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-07 17:15:43 +03:00
239323eeef [#1157] tree: Make tree service use Bearer token's APE overrides
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-07 12:11:11 +00:00
04a3f891fd [#1157] object: Make APE checker use Bearer-token's APE overrides
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-07 12:11:11 +00:00
4edff5eea6 [#1157] ape: Introduce single-run chain router
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-07 12:11:11 +00:00
a90310335d [#1156] ape: Return not found when removing local overrides
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-06-07 12:10:57 +00:00
a849236b68 [#1161] node: Remove notification functionality
It is unused and will be reworked in future.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-07 12:10:51 +00:00
3f1961157e [#1163] metabase: Handle multiple splitInfos for EC
All checks were successful
DCO action / DCO (pull_request) Successful in 3m5s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m41s
Vulncheck / Vulncheck (pull_request) Successful in 4m10s
Build / Build Components (1.21) (pull_request) Successful in 5m49s
Build / Build Components (1.22) (pull_request) Successful in 5m46s
Tests and linters / Staticcheck (pull_request) Successful in 6m37s
Tests and linters / Lint (pull_request) Successful in 7m12s
Tests and linters / gopls check (pull_request) Successful in 7m16s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m11s
Tests and linters / Tests (1.22) (pull_request) Successful in 10m20s
Tests and linters / Tests with -race (pull_request) Successful in 10m21s
For REP updating split info is handled explicitly by a high-level PUT logic.
For EC it is trickier, because the address of an object we put is only
distantly related to a split info.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-06 16:26:29 +03:00
2e074d3846 [#1163] metabase: Properly save EC parent split ID
Search by SplitID should return all parts of a complex object.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-05 12:40:16 +03:00
806236da78 [#1121] node: Change mode of shard components
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-06-05 05:55:24 +00:00
6f2187a420 [#1121] node: Refactor mods of shard
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-06-05 05:55:24 +00:00
cc2449beaf [#1158] metabase: Fix EC storage schema
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 3m5s
Build / Build Components (1.21) (pull_request) Successful in 4m0s
Build / Build Components (1.22) (pull_request) Successful in 3m56s
Tests and linters / Staticcheck (pull_request) Successful in 4m54s
Tests and linters / gopls check (pull_request) Successful in 4m55s
Tests and linters / Lint (pull_request) Successful in 5m54s
Pre-commit hooks / Pre-commit (pull_request) Successful in 6m29s
DCO action / DCO (pull_request) Successful in 3m34s
Tests and linters / Tests (1.22) (pull_request) Successful in 7m28s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m50s
Tests and linters / Tests with -race (pull_request) Successful in 9m25s
Do not store EC info twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-04 17:24:40 +03:00
5aacb8fc86 [#1144] metabase: Save parent attributes for ec-chunks
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 4m15s
DCO action / DCO (pull_request) Successful in 4m4s
Build / Build Components (1.22) (pull_request) Successful in 5m17s
Build / Build Components (1.21) (pull_request) Successful in 5m22s
Tests and linters / Lint (pull_request) Successful in 7m6s
Tests and linters / Staticcheck (pull_request) Successful in 7m25s
Tests and linters / gopls check (pull_request) Successful in 7m19s
Pre-commit hooks / Pre-commit (pull_request) Successful in 9m39s
Tests and linters / Tests (1.22) (pull_request) Successful in 10m36s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m50s
Tests and linters / Tests with -race (pull_request) Successful in 10m51s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 19:55:32 +03:00
f8e33f8e3a [#1144] metabase: Proprely choose root OID for EC-splitted objects
* If EC-parent is a part of Split itself, then save to root bucket
  its parent;
* If EC-parent is not a part of Split itself, then save to root bucket
  OID of this EC-parent.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 19:53:32 +03:00
f0edebea18 [#1144] metabase: Support ec parent filter for Search
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 19:53:32 +03:00
0b367007fc [#1152] go.mod: Update api-go and sdk versions
All checks were successful
DCO action / DCO (pull_request) Successful in 5m23s
Build / Build Components (1.22) (pull_request) Successful in 7m33s
Build / Build Components (1.21) (pull_request) Successful in 7m43s
Tests and linters / Lint (pull_request) Successful in 8m51s
Tests and linters / Tests (1.22) (pull_request) Successful in 14m33s
Pre-commit hooks / Pre-commit (pull_request) Successful in 15m48s
Vulncheck / Vulncheck (pull_request) Successful in 15m12s
Tests and linters / Tests (1.21) (pull_request) Successful in 16m38s
Tests and linters / gopls check (pull_request) Successful in 19m24s
Tests and linters / Staticcheck (pull_request) Successful in 21m29s
Tests and linters / Tests with -race (pull_request) Successful in 23m16s
* Resolve conflicts for apemanager since api-go
  contains ape and apemanager packages and SDK only
  ape package.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 15:39:09 +03:00
92e19feb57 [#1147] node: Use public fields for shard.ExistsPrm
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
c1af13b47e [#1147] node: Fix issue from gopls
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
6130650bb6 [#1147] node: Implement Lock\Delete requests for EC object
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
a82c8cc5b8 [#1147] gc: Execute callback for expired tombstones when they exists
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
d355274cd0 [#1147] object: Use methods on pointer for searchsvc.execCtx
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
3555c73225 [#1147] object: Use methods on pointer for deletesvc.execCtx
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
e43e7bec3a [#1147] log: Remove redundant address field from log
Filled when logger created for `request` object from package `getsvc`.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
50923ed81c [#1147] Fix gofumpt issue
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
4a34d0d40e [#1149] go.mod: Bump neo-go up to v0.106.0
Required to work with neo-go v0.106.0 node
with default hardfork configuration. Without
neo-go client version bump, it throws error.

  failed to get network magic: unexpected hardfork: Cockatrice

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-05-30 08:11:58 +00:00
3627b44e92 [#1142] tree: Fill APE-request with source IP property
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-27 10:17:17 +00:00
482c5129ac [#1142] object: Fill APE-request with source IP property
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-27 10:17:17 +00:00
43625e7536 [#1142] container: Fill APE-request property with source IP
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-27 10:17:17 +00:00
542d3adcb2 [#1105] apemanager: Implement apemanager service
* Introduce grpc server for apemanager service and
  its implementation in `pkg/services/apemanager`.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-27 09:34:21 +00:00
51ade979e8 [#1105] ape: Introduce contract storage with proxy contract verification
* `ProxyVerificationContractStorage` uses Proxy contract as a cosigner.
* `ProxyVerificationContractStorage` recreates a contract storage for each handler
  invocation because of an issue: rpc-actor from morph client may be expired. This
  way won't create a bottlenecks because it is expected that this contract storage
  implementation will be used not so often.
* Make morph client return `RPCActor` (that is websocket client in fact).
* Make `SwitchRPCGuardedActor` return `RPCActor` as it will be used for
  `ProxyVerificationContractStorage`.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-27 09:34:21 +00:00
40b04c00ef [#1141] metabase: Fix IsUserObject method
All checks were successful
DCO action / DCO (pull_request) Successful in 6m57s
Build / Build Components (1.22) (pull_request) Successful in 10m4s
Build / Build Components (1.21) (pull_request) Successful in 10m12s
Vulncheck / Vulncheck (pull_request) Successful in 10m4s
Tests and linters / gopls check (pull_request) Successful in 13m13s
Tests and linters / Staticcheck (pull_request) Successful in 13m45s
Tests and linters / Lint (pull_request) Successful in 14m21s
Tests and linters / Tests (1.21) (pull_request) Successful in 16m3s
Pre-commit hooks / Pre-commit (pull_request) Successful in 16m26s
Tests and linters / Tests (1.22) (pull_request) Successful in 16m27s
Tests and linters / Tests with -race (pull_request) Successful in 16m29s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-20 13:24:15 +03:00
89a80e9a0f [#1141] metabase: Fix putUniqueIndexItem
* `GetECHeader` is not correct way to determine if an object's got
  EC-header: `ECHeader` must be used for that.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-20 13:24:08 +03:00
8fd678e269 [#1141] go.mod: Update frostfs-sdk-go and frostfs-api-go/v2
* Also fix unit-test.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-20 13:22:48 +03:00
436c9f5558 [#1129] policer: Restore EC object
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m43s
DCO action / DCO (pull_request) Successful in 2m53s
Build / Build Components (1.21) (pull_request) Successful in 4m6s
Build / Build Components (1.22) (pull_request) Successful in 4m31s
Tests and linters / gopls check (pull_request) Successful in 4m57s
Tests and linters / Staticcheck (pull_request) Successful in 6m14s
Tests and linters / Lint (pull_request) Successful in 6m42s
Pre-commit hooks / Pre-commit (pull_request) Successful in 9m19s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m15s
Tests and linters / Tests (1.22) (pull_request) Successful in 10m36s
Tests and linters / Tests with -race (pull_request) Successful in 10m36s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-17 14:36:18 +03:00
44f2e8f27f [#1129] putSvc: Allow to put single unprepared object to EC container
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:49 +03:00
0e42126ddc [#1129] object: Fix check owner for EC part
Do not validate EC part owner if request from container node.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:49 +03:00
1cd8562db8 [#1129] policer: Refactor shortage
Drop override inside method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:49 +03:00
4ab6c404f7 [#1129] policer: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:49 +03:00
cbe9757490 [#1129] policer: Pull required EC chunks
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:49 +03:00
d45d086acd [#1129] policer: Add EC chunk replication
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:48 +03:00
b078fe5ba1 [#1092] control: Move SignMessage to separate package
All checks were successful
DCO action / DCO (pull_request) Successful in 6m11s
Build / Build Components (1.22) (pull_request) Successful in 7m50s
Build / Build Components (1.21) (pull_request) Successful in 8m3s
Tests and linters / Lint (pull_request) Successful in 9m45s
Tests and linters / gopls check (pull_request) Successful in 12m40s
Vulncheck / Vulncheck (pull_request) Successful in 12m35s
Tests and linters / Staticcheck (pull_request) Successful in 15m34s
Pre-commit hooks / Pre-commit (pull_request) Successful in 19m38s
Tests and linters / Tests with -race (pull_request) Successful in 21m40s
Tests and linters / Tests (1.22) (pull_request) Successful in 3m21s
Tests and linters / Tests (1.21) (pull_request) Successful in 3m36s
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-05-16 12:14:01 +03:00
f3e09cb09b [#1135] sdnotify: Send MONOTONIC_USEC on reload
Fixes #1135

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-15 12:52:52 +00:00
5c582e96fd [#1136] metabase: Fix creation of ECInfoError
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-15 11:04:27 +00:00
b3eaa8a9bc [#1083] objsvc/v2: Check response status in RANGE_HASH forwarder
All checks were successful
DCO action / DCO (pull_request) Successful in 1m45s
Vulncheck / Vulncheck (pull_request) Successful in 4m11s
Pre-commit hooks / Pre-commit (pull_request) Successful in 5m44s
Build / Build Components (1.22) (pull_request) Successful in 5m25s
Build / Build Components (1.21) (pull_request) Successful in 5m30s
Tests and linters / Staticcheck (pull_request) Successful in 5m50s
Tests and linters / gopls check (pull_request) Successful in 5m51s
Tests and linters / Lint (pull_request) Successful in 7m44s
Tests and linters / Tests (1.22) (pull_request) Successful in 11m36s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m13s
Tests and linters / Tests with -race (pull_request) Successful in 12m25s
Fixes #1083

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-15 12:04:21 +03:00
0924b62a95 [#1083] objsvc/v2: Unify response verification after forwarding
1. Use the same routine for HEAD/GET_RANGE methods.
2. Make error message similar.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-15 12:04:06 +03:00
300654b045 [#1083] objsvc/v2: Properly check response status after forwarding
Previously we had cryptic error:
```
debug   get/remote.go:38        remote call failed      {"component": "Object.Get service", "request": "HEAD", "address": "9sTxoVrhJ7WBtXQfK2NJ7zDV5yCF7BPLKK1XTxYPdGsP/BbHV4KZZ8y2BPqAT5kyjdHRLkfbtY2xf5uYoMVqxACn1", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "unexpected header type <nil>"}
```
Now we have and expected error:
```
debug   get/remote.go:38        remote call failed      {"component": "Object.Get service", "request": "HEAD", "address": "D2rqaMG4D2VHdv3HKky8UYSYmwQFH2v9oXXqtyRZPTMy/BbHV4KZZ8y2BPqAT5kyjdHRLkfbtY2xf5uYoMVqxACn1", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-15 12:04:06 +03:00
6e71ae3bda [#1130] fstree: Remove useless Stat() call
All checks were successful
DCO action / DCO (pull_request) Successful in 6m28s
Vulncheck / Vulncheck (pull_request) Successful in 5m52s
Build / Build Components (1.21) (pull_request) Successful in 11m14s
Build / Build Components (1.22) (pull_request) Successful in 11m20s
Tests and linters / gopls check (pull_request) Successful in 11m12s
Tests and linters / Staticcheck (pull_request) Successful in 12m5s
Tests and linters / Lint (pull_request) Successful in 16m52s
Pre-commit hooks / Pre-commit (pull_request) Successful in 17m37s
Tests and linters / Tests (1.21) (pull_request) Successful in 18m43s
Tests and linters / Tests with -race (pull_request) Successful in 18m46s
Tests and linters / Tests (1.22) (pull_request) Successful in 4m22s
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                                           │     old     │                new                 │
                                           │   sec/op    │   sec/op     vs base               │
SubstorageReadPerf/fstree_nosync-seq100-8    2.689µ ± 2%   2.428µ ± 4%  -9.72% (p=0.000 n=10)
SubstorageReadPerf/fstree_nosync-rand100-8   2.727µ ± 1%   2.497µ ± 2%  -8.42% (p=0.000 n=10)
geomean                                      2.708µ        2.462µ       -9.07%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-14 16:05:45 +03:00
bf9bdde8ea [#1128] util/test: Remove unused package
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-14 12:48:13 +00:00
952d13cd2b [#1124] cli: Improve APE rule parsing
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m25s
DCO action / DCO (pull_request) Successful in 1m59s
Build / Build Components (1.21) (pull_request) Successful in 2m27s
Build / Build Components (1.22) (pull_request) Successful in 4m25s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m57s
Tests and linters / Staticcheck (pull_request) Successful in 5m38s
Tests and linters / gopls check (pull_request) Successful in 5m57s
Tests and linters / Lint (pull_request) Successful in 6m26s
Tests and linters / Tests (1.22) (pull_request) Successful in 9m5s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m11s
Tests and linters / Tests with -race (pull_request) Successful in 9m4s
* Make APE rule parser to read condition's kind in unambiguous using lexemes
`ResourceCondition`, `RequestCondition` instead confusing `Object.Request`, `Object.Resource`.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-14 12:23:26 +03:00
20baf6e112 [#1108] ape: Update policy-engine version for listing by iteration
* Update go.mod with a new version of policy-engine pacakge.
* Adapt SwitchRPCGuardedActor to ContractStorage interface.
* Fix `frostfs-adm` util.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-14 12:17:56 +03:00
0144117cc9 [#1125] objectSvc: Add EC header APE check
All checks were successful
Build / Build Components (1.21) (pull_request) Successful in 6m27s
DCO action / DCO (pull_request) Successful in 6m38s
Build / Build Components (1.22) (pull_request) Successful in 8m54s
Vulncheck / Vulncheck (pull_request) Successful in 8m37s
Tests and linters / gopls check (pull_request) Successful in 10m32s
Tests and linters / Staticcheck (pull_request) Successful in 11m3s
Tests and linters / Lint (pull_request) Successful in 11m27s
Pre-commit hooks / Pre-commit (pull_request) Successful in 14m16s
Tests and linters / Tests (1.21) (pull_request) Successful in 14m26s
Tests and linters / Tests (1.22) (pull_request) Successful in 15m14s
Tests and linters / Tests with -race (pull_request) Successful in 15m45s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-08 16:25:55 +03:00
ada1b9f737 [#1120] objectSvc: Fix EC put placement
Use parent object ID to compute placement.
Fix too many copies saving.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-08 15:23:57 +03:00
fe2c1c926f [#1112] node: Fix race warning for GetObjectAndWritePayload
All checks were successful
DCO action / DCO (pull_request) Successful in 2m10s
Vulncheck / Vulncheck (pull_request) Successful in 2m2s
Build / Build Components (1.22) (pull_request) Successful in 3m44s
Build / Build Components (1.21) (pull_request) Successful in 3m52s
Pre-commit hooks / Pre-commit (pull_request) Successful in 5m40s
Tests and linters / Staticcheck (pull_request) Successful in 6m40s
Tests and linters / Lint (pull_request) Successful in 7m11s
Tests and linters / gopls check (pull_request) Successful in 9m16s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m58s
Tests and linters / Tests (1.22) (pull_request) Successful in 11m2s
Tests and linters / Tests with -race (pull_request) Successful in 11m35s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-07 14:47:21 +03:00
3e782527b8 [#1112] node: Add test for Range request for EC object
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-07 14:47:21 +03:00
21a490da8f [#1112] Fix issue from gofumpt
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-07 14:47:21 +03:00
93c0ccad4f [#1077] objectsvc: Fix possible panic in GetRange()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-07 14:47:21 +03:00
00b2b77b26 [#1112] node: Implement Range\RangeHash requests for EC object
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-07 14:47:21 +03:00
b60a51b862 [#1117] ape: Introduce FormFrostfsIDRequestProperties method
* `FormFrostfsIDRequestProperties` gets user claim tags and group id and sets them
  as ape request properties.
* Make tree, container and object service use the method.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-07 10:01:21 +00:00
6c76c9b457 [#1117] core: Introduce SubjectProvider interface for FrostfsID
* Make tree, object and container services use SubjectProvider interface.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-07 10:01:21 +00:00
45f4e6939d [#1117] morph: Make frostfsid client provide GetSubjectExtended method
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-07 10:01:21 +00:00
e07869a8cf [#1100] Remove unused fields
All checks were successful
DCO action / DCO (pull_request) Successful in 2m47s
Build / Build Components (1.21) (pull_request) Successful in 3m21s
Vulncheck / Vulncheck (pull_request) Successful in 3m33s
Build / Build Components (1.22) (pull_request) Successful in 5m25s
Tests and linters / gopls check (pull_request) Successful in 5m13s
Tests and linters / Staticcheck (pull_request) Successful in 6m13s
Tests and linters / Lint (pull_request) Successful in 7m2s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m2s
Tests and linters / Tests with -race (pull_request) Successful in 9m55s
Tests and linters / Tests (1.22) (pull_request) Successful in 10m11s
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-05-06 10:14:36 +03:00
ec2873caa7 [#1116] node: Fix writecache metrics
All checks were successful
DCO action / DCO (pull_request) Successful in 1m35s
Vulncheck / Vulncheck (pull_request) Successful in 3m30s
Build / Build Components (1.22) (pull_request) Successful in 4m40s
Build / Build Components (1.21) (pull_request) Successful in 4m46s
Tests and linters / gopls check (pull_request) Successful in 5m22s
Tests and linters / Staticcheck (pull_request) Successful in 6m19s
Tests and linters / Lint (pull_request) Successful in 7m7s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m36s
Tests and linters / Tests with -race (pull_request) Successful in 9m30s
Tests and linters / Tests (1.22) (pull_request) Successful in 9m46s
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-05-02 20:03:33 +03:00
71789676d5 [#1114] aclsvc: Add tests for request ownership
All checks were successful
DCO action / DCO (pull_request) Successful in 5m13s
Build / Build Components (1.21) (pull_request) Successful in 9m51s
Build / Build Components (1.22) (pull_request) Successful in 10m21s
Vulncheck / Vulncheck (pull_request) Successful in 12m9s
Tests and linters / Lint (pull_request) Successful in 16m50s
Tests and linters / gopls check (pull_request) Successful in 18m8s
Tests and linters / Staticcheck (pull_request) Successful in 19m1s
Tests and linters / Tests (1.22) (pull_request) Successful in 19m59s
Tests and linters / Tests with -race (pull_request) Successful in 20m5s
Tests and linters / Tests (1.21) (pull_request) Successful in 2m52s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-02 11:57:39 +03:00
c9efaa5819 [#966] node: Add path of the write_cache to metric labels
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-05-02 06:46:46 +00:00
4730ecfdb8 [#966] node: Refactor WriteCacheMetrics interface
Grouping common fields of methods will enhance the readability of the interface.

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-05-02 06:46:46 +00:00
411a8d0245 [#1004] blobovnicza: Use TTL for blobovnicza tree cache
All checks were successful
DCO action / DCO (pull_request) Successful in 12m46s
Vulncheck / Vulncheck (pull_request) Successful in 14m10s
Build / Build Components (1.22) (pull_request) Successful in 15m23s
Build / Build Components (1.21) (pull_request) Successful in 15m38s
Tests and linters / Staticcheck (pull_request) Successful in 3m36s
Tests and linters / gopls check (pull_request) Successful in 4m0s
Tests and linters / Lint (pull_request) Successful in 5m8s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m41s
Tests and linters / Tests with -race (pull_request) Successful in 7m46s
Tests and linters / Tests (1.22) (pull_request) Successful in 7m58s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-26 19:54:29 +03:00
112a7c690f [#1103] node: Implement Get\Head requests for EC object
All checks were successful
DCO action / DCO (pull_request) Successful in 1m44s
Vulncheck / Vulncheck (pull_request) Successful in 3m3s
Build / Build Components (1.21) (pull_request) Successful in 4m0s
Build / Build Components (1.22) (pull_request) Successful in 3m57s
Tests and linters / Staticcheck (pull_request) Successful in 4m46s
Tests and linters / gopls check (pull_request) Successful in 4m48s
Tests and linters / Lint (pull_request) Successful in 5m45s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m57s
Tests and linters / Tests with -race (pull_request) Successful in 9m10s
Tests and linters / Tests (1.22) (pull_request) Successful in 9m20s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-24 18:15:53 +03:00
167c52a1a9 [#1103] node: Reduce amount of lines for method StorageEngine.head
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-24 16:31:04 +03:00
700e891b85 [#1103] Fix end of file and trim trailing whitespace
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-24 16:31:04 +03:00
10ee865e98 [#1096] tree: Make verifyClient fill ape request with user claim tags
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 2m52s
DCO action / DCO (pull_request) Successful in 2m51s
Build / Build Components (1.21) (pull_request) Successful in 3m44s
Build / Build Components (1.22) (pull_request) Successful in 3m48s
Tests and linters / Staticcheck (pull_request) Successful in 4m34s
Tests and linters / gopls check (pull_request) Successful in 4m48s
Tests and linters / Lint (pull_request) Successful in 5m42s
Tests and linters / Tests with -race (pull_request) Successful in 8m56s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m7s
Tests and linters / Tests (1.22) (pull_request) Successful in 9m15s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-16 15:12:46 +03:00
c21d72ac23 [#1096] object: Make ape middleware fill request with user claim tags
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-16 15:12:44 +03:00
6772976657 [#1096] container: Make ape middleware fill request with user claim tags
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-16 15:10:20 +03:00
3ea1d7b729 [#1089] control: Add USER and GROUP targets for local override storage
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-16 11:03:50 +00:00
0094186299 [#1089] control: Format proto files with clang-format
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-16 11:03:50 +00:00
91e79c98ba [#1089] ape: Provide request actor as an additional target
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-16 11:03:50 +00:00
e5e0542482 [#1085] log: Move storage log message to constants package
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-15 07:57:00 +00:00
5be36924e3 [#41] log: Log storage operations in only in Debug
They are mostly useless unless we need to _debug_ a specific issue.
The amount of logs we produce is too big.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-15 07:57:00 +00:00
6a46c6d229 [#1090] tree: Make workaround for APE checks
* Make `verifyClient` method perform APE check if a container
  was created with zero-filled basic ACL.
* Object verbs are used in APE, until tree verbs are introduced.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-15 07:45:45 +00:00
f4dcb418f2 [#1090] ape: Move ape request and resource implementations to common package
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-15 07:45:45 +00:00
40781b3a20 [#1086] engine: Change mode in case of errors async
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 12:29:43 +00:00
5ef5734c4e Reapply "[#972] Drop x/exp/slices dependency"
This reverts commit 946f2ec2bf.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-10 12:09:34 +00:00
669103a33e Reapply "[#972] Use slices.Sort* when useful"
This reverts commit 3359349acb.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-10 12:09:34 +00:00
2c4b50a71e Reapply "[#972] Use require.ElementsMatch() where possible"
This reverts commit 7627d08914.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-10 12:09:34 +00:00
3dc81cb4fc Reapply "[#972] Use min/max builtins"
This reverts commit dad56d2e98.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-10 12:09:34 +00:00
d864945961 Reapply "[#972] pilorama: Remove removeDuplicatesInPlace()"
This reverts commit 9f68305c2e.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-10 12:09:34 +00:00
1b17258c04 [#1029] metabase: Add refill metrics
All checks were successful
DCO action / DCO (pull_request) Successful in 1m22s
Vulncheck / Vulncheck (pull_request) Successful in 3m11s
Build / Build Components (1.21) (pull_request) Successful in 3m56s
Build / Build Components (1.20) (pull_request) Successful in 3m59s
Tests and linters / Staticcheck (pull_request) Successful in 5m31s
Tests and linters / gopls check (pull_request) Successful in 5m26s
Tests and linters / Lint (pull_request) Successful in 6m13s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m54s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m13s
Tests and linters / Tests with -race (pull_request) Successful in 9m30s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 13:05:44 +03:00
e3d9dd6ee8 [#1024] blobovnicza: Copy data on iterate
DB value is only valid while the tx is alive.
But handler may to run something in other goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:21:11 +03:00
57466594fb [#1024] shard: Resync metabase concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:21:10 +03:00
1005bf4f56 [#1024] shard: Add refill metabase benchmark
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:21:10 +03:00
76398c06b0 [#1080] metabase: Add StorageID metric
All checks were successful
DCO action / DCO (pull_request) Successful in 1m19s
Build / Build Components (1.21) (pull_request) Successful in 4m6s
Vulncheck / Vulncheck (pull_request) Successful in 4m35s
Build / Build Components (1.20) (pull_request) Successful in 7m32s
Tests and linters / gopls check (pull_request) Successful in 7m43s
Tests and linters / Lint (pull_request) Successful in 9m51s
Tests and linters / Staticcheck (pull_request) Successful in 9m53s
Tests and linters / Tests (1.20) (pull_request) Successful in 12m48s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m11s
Tests and linters / Tests with -race (pull_request) Successful in 14m4s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:00:08 +03:00
7b1adfed3e [#1080] metabase: Open bucket for container counter once
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-10 10:00:08 +03:00
e74bdaa5d5 [#1080] ape: Use value for APE request
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 18:42:03 +03:00
338d8cbebd [#1080] ape: Do not read object headers before Head/Get
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 15:27:40 +03:00
5b8200de88 [#984] blobovnicza: Do not fail rebuild on big objects
If blobovnicza contains objects larger than object size parameter
value, then rebuild fails with an error, because there is no such
bucket in database. This commit forces to create bucket on rebuild.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 11:51:18 +00:00
2b88361849 [#1062] object: Fix buffer allocation for PayloadRange
All checks were successful
DCO action / DCO (pull_request) Successful in 5m34s
Vulncheck / Vulncheck (pull_request) Successful in 5m14s
Build / Build Components (1.20) (pull_request) Successful in 8m56s
Build / Build Components (1.21) (pull_request) Successful in 8m56s
Tests and linters / gopls check (pull_request) Successful in 9m3s
Tests and linters / Staticcheck (pull_request) Successful in 9m39s
Tests and linters / Lint (pull_request) Successful in 10m9s
Tests and linters / Tests (1.20) (pull_request) Successful in 13m43s
Tests and linters / Tests (1.21) (pull_request) Successful in 14m4s
Tests and linters / Tests with -race (pull_request) Successful in 14m40s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-09 11:59:07 +03:00
f5b67c6735 [#1064] policer: Disable EC processing
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 07:08:53 +00:00
1c5e0f90aa [#1064] putsvc: Add EC put
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 07:08:53 +00:00
39da643354 [#1064] putsvc: Refactor distributed target
Extract object builder.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-09 07:08:53 +00:00
17f7adb640 [#1065] adm: Add support EC parameters
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-04-08 12:27:30 +03:00
9aa533e59a [#1072] node, ir: Add new config option kludge_compatibility_mode
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-04 11:05:25 +03:00
d614f04a0a [#1072] Fix gofumpt issues
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-04-03 22:21:14 +03:00
4738508ce2 [#1063] go.mod: Update SDK version
* Update frostfs-sdk and frostfs-api-go versions.
* Refactor depreacted method ReplicaNumberByIndex.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-04-03 10:33:51 +00:00
ff4c23f59a [#1070] services/tree: Fix fast listing depth processing
For unsorted `GetSubTree()` we return a single node for depth=1.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-02 14:41:31 +00:00
17af91619a [#1070] pilorama: Fix cycling behaviour for sorted listing
In case there are no items left, return empty slice.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-02 14:41:31 +00:00
d5194ab2a6 [#949] metabase: fix shard.UpdateID()
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m20s
DCO action / DCO (pull_request) Successful in 1m59s
Build / Build Components (1.21) (pull_request) Successful in 3m25s
Build / Build Components (1.20) (pull_request) Successful in 4m46s
Tests and linters / Staticcheck (pull_request) Successful in 6m5s
Tests and linters / gopls check (pull_request) Successful in 6m17s
Tests and linters / Lint (pull_request) Successful in 7m7s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m38s
Tests and linters / Tests with -race (pull_request) Successful in 8m51s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m56s
metabase.Open() now reports metabase mode metric. shard.UpdateID()
needs to read shard ID from metabase => needs to open metabase.
It caused reporting 'shard undefined' metrics. To avoid reporting
wrong metrics metabase.GetShardID() was added which also opens
metabase and does not report metrics.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-04-01 17:27:34 +03:00
81a0346a96 [#949] metabase: fix metabase mode metric
It used to always show CLOSED regardless of actual mode.
Now metric represents actual metabase mode of operations.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-04-01 17:27:34 +03:00
e12fcc041d [#1059] services/tree: Fast sorted listing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-01 12:37:34 +00:00
f23e38c285 Revert "[#446] engine: Move to read-only on blobstor errors"
All checks were successful
DCO action / DCO (pull_request) Successful in 2m14s
Build / Build Components (1.20) (pull_request) Successful in 4m7s
Vulncheck / Vulncheck (pull_request) Successful in 3m30s
Build / Build Components (1.21) (pull_request) Successful in 4m15s
Tests and linters / Staticcheck (pull_request) Successful in 5m41s
Tests and linters / Lint (pull_request) Successful in 6m6s
Tests and linters / gopls check (pull_request) Successful in 6m42s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m47s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m21s
Tests and linters / Tests with -race (pull_request) Successful in 8m20s
This reverts commit 69df0d21c2.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-04-01 12:48:30 +03:00
942d83611b [#874] engine: Revert Check object existance concurrently
This reverts commit f526f49995.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-04-01 08:42:34 +00:00
fd8cdb9671 [#1057] netmap: Do not iterate over external addresses in Node
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-29 20:46:24 +00:00
0990a9b0bd [#1055] blobstor: fix mode metric
It used to always show CLOSED after setting shard mode
to read-only regardless of actual mode.
Now metric represents actual blobstor mode of operations.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-03-29 20:44:47 +00:00
c7a12ca3d8 [#1054] network: Optimize IsTLSEnabled()
All checks were successful
DCO action / DCO (pull_request) Successful in 5m33s
Vulncheck / Vulncheck (pull_request) Successful in 5m54s
Build / Build Components (1.21) (pull_request) Successful in 10m3s
Tests and linters / gopls check (pull_request) Successful in 11m49s
Build / Build Components (1.20) (pull_request) Successful in 12m47s
Tests and linters / Staticcheck (pull_request) Successful in 13m21s
Tests and linters / Lint (pull_request) Successful in 14m21s
Tests and linters / Tests (1.20) (pull_request) Successful in 16m39s
Tests and linters / Tests with -race (pull_request) Successful in 16m46s
Tests and linters / Tests (1.21) (pull_request) Successful in 16m59s
No big deal, but it is called multiple times in sorting routine, this
easily results in 20 allocations per group traversal.

```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                    │     old     │                 new                 │
                    │   sec/op    │   sec/op     vs base                │
AddressTLSEnabled-8   184.6n ± 1%   103.3n ± 6%  -44.04% (p=0.000 n=10)

                    │    old     │                new                │
                    │    B/op    │   B/op    vs base                 │
AddressTLSEnabled-8   704.0 ± 0%   0.0 ± 0%  -100.00% (p=0.000 n=10)

                    │    old     │                 new                 │
                    │ allocs/op  │ allocs/op   vs base                 │
AddressTLSEnabled-8   1.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-03-21 18:48:35 +03:00
c09c701613 [#1048] metabase: Fix drop buckets during resync
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m35s
DCO action / DCO (pull_request) Successful in 2m12s
Build / Build Components (1.21) (pull_request) Successful in 3m57s
Build / Build Components (1.20) (pull_request) Successful in 4m45s
Tests and linters / Staticcheck (pull_request) Successful in 5m32s
Tests and linters / gopls check (pull_request) Successful in 5m33s
Tests and linters / Lint (pull_request) Successful in 6m18s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m8s
Tests and linters / Tests with -race (pull_request) Successful in 8m14s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m36s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-19 14:28:31 +03:00
6959e617c4 [#1047] object: Set container owner ID property to ape request
* Introduce ContainerOwner field in RequestContext.
* Set ContainerOwner in aclv2 middleware.
* Set PropertyKeyContainerOwnerID for object ape request.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-18 15:39:50 +00:00
d7be70e93f [#1040] object: Wrap CheckAPE errors to status errors
* All methods should wrap CheckAPE error, if it occurs, to
  status error.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-14 07:34:03 +00:00
5ee5f1df42 [#976] control: Introduce new method RemoveChainLocalOverridesByTarget
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-13 15:33:19 +03:00
2d595ec15f [#976] ape: Update dependency
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-13 14:54:19 +03:00
7ed07d2dfd [#976] morph: Implement missing methods for SwitchRPCGuardedActor
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-13 14:54:19 +03:00
17f5463389 [#1043] cli: Add reset evacuation status command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-13 10:29:45 +00:00
31e2396a5f [#1043] control: Add ResetEvacuationStatus implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-13 10:29:45 +00:00
926cdeb072 [#1043] services: Regenerate proto
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-13 10:29:45 +00:00
5c252c9193 [#1039] object: Skip APE check for certain request roles
All checks were successful
DCO action / DCO (pull_request) Successful in 1m31s
Vulncheck / Vulncheck (pull_request) Successful in 2m52s
Build / Build Components (1.21) (pull_request) Successful in 3m52s
Build / Build Components (1.20) (pull_request) Successful in 4m16s
Tests and linters / gopls check (pull_request) Successful in 11m54s
Tests and linters / Staticcheck (pull_request) Successful in 12m31s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m49s
Tests and linters / Tests (1.20) (pull_request) Successful in 13m8s
Tests and linters / Tests with -race (pull_request) Successful in 13m14s
Tests and linters / Lint (pull_request) Successful in 13m31s
* Skip APE check if a role is Container.
* Skip APE check if a role is IR and methods are get-like.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-03-12 16:15:20 +03:00
3195142d67 [#959] writecache: Avoid manipulation with cache in DEGRADED mode
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-11 18:35:41 +00:00
d433b49265 [#973] node: Resolve perfsprint linter
All checks were successful
DCO action / DCO (pull_request) Successful in 2m40s
Vulncheck / Vulncheck (pull_request) Successful in 3m41s
Build / Build Components (1.20) (pull_request) Successful in 4m27s
Build / Build Components (1.21) (pull_request) Successful in 5m6s
Tests and linters / Staticcheck (pull_request) Successful in 6m16s
Tests and linters / gopls check (pull_request) Successful in 6m23s
Tests and linters / Lint (pull_request) Successful in 6m48s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m4s
Tests and linters / Tests with -race (pull_request) Successful in 9m9s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m23s
`fmt.Errorf can be replaced with errors.New` and `fmt.Sprintf can be replaced with string addition`

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-11 17:55:50 +03:00
66a26b7775 [#973] node: Resolve revive: unused-parameter linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-11 17:11:49 +03:00
0882840bf5 [#634] shard: Add writecache inhume tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-06 13:12:34 +03:00
ae5bb87e70 Revert "[#866] Use TTL for blobovnicza tree cache"
All checks were successful
DCO action / DCO (pull_request) Successful in 2m7s
Vulncheck / Vulncheck (pull_request) Successful in 3m7s
Tests and linters / Staticcheck (pull_request) Successful in 5m12s
Tests and linters / Lint (pull_request) Successful in 6m23s
Tests and linters / Tests with -race (pull_request) Successful in 7m11s
Tests and linters / gopls check (pull_request) Successful in 7m27s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m5s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m22s
Build / Build Components (1.21) (pull_request) Successful in 1m41s
Build / Build Components (1.20) (pull_request) Successful in 1m56s
This reverts commit d9cbb16bd3.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-03-01 19:29:33 +03:00
d6534fd755 [#1016] frostfs-node: Fix gopls issues
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-03-01 12:13:43 +03:00
93bf9acbc2 [#898] control: Remove removed flag from RemoveChainLocalOverrideResponse
* Remove removed flag in service.proto for RemoveChainLocalOverrideResponse.
* Regenerate control API.
* Return error only if RemoveOverride returns non-NotFound code.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-28 19:07:07 +00:00
75a1a95c2c [#986] tree: Skip ACL checks if basicACL mask is unset
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-28 19:05:57 +00:00
b1d171c261 [#986] container: Interpret APE NoRuleFound as request deny
* If APE check returns NoRuleFound, then it is taken for request deny.
* Add more unit-test for ape container middleware.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-28 19:05:57 +00:00
7cc368e188 [#986] object: Introduce soft ape checks
* Soft APE check means that APE should allow request even
  it gets status NoRuleFound for a request. Otherwise,
  it is interpreted as Deny.
* Soft APE check is performed if basic ACL mask is not set.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-28 19:05:57 +00:00
918613546f [#1008] metabase: Do not update storageID on put
All checks were successful
DCO action / DCO (pull_request) Successful in 4m56s
Build / Build Components (1.21) (pull_request) Successful in 7m52s
Build / Build Components (1.20) (pull_request) Successful in 8m11s
Vulncheck / Vulncheck (pull_request) Successful in 8m9s
Tests and linters / Staticcheck (pull_request) Successful in 9m58s
Tests and linters / Lint (pull_request) Successful in 10m42s
Tests and linters / Tests (1.20) (pull_request) Successful in 12m14s
Tests and linters / Tests with -race (pull_request) Successful in 12m17s
Tests and linters / Tests (1.21) (pull_request) Successful in 12m34s
There may be a race condition between put an object and
flushing the writecache:
1. Put object to the writecache
2. Writecache flushes object to the blobstore and sets blobstore's
storageID
3. Put object to the metabase, set writecache's storageID

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-28 11:01:50 +03:00
2ad433dbcb [#1005] engine: Drop shards weights
All checks were successful
DCO action / DCO (pull_request) Successful in 4m7s
Vulncheck / Vulncheck (pull_request) Successful in 4m53s
Build / Build Components (1.21) (pull_request) Successful in 5m46s
Build / Build Components (1.20) (pull_request) Successful in 6m21s
Tests and linters / Staticcheck (pull_request) Successful in 7m45s
Tests and linters / Lint (pull_request) Successful in 8m44s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m1s
Tests and linters / Tests (1.20) (pull_request) Successful in 15m42s
Tests and linters / Tests with -race (pull_request) Successful in 16m10s
Unused.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-26 17:25:05 +03:00
4b13b85173 [#1000] morph: Fix batch size in TraverseIterator()
Initial prefetch size can be arbitrary an restricted only by VM/RPC
limits. For TraverseIterator() there is an explicit check on the
server-side, though.
Introduced in df055fead5.
Refs #931.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-23 06:24:40 +00:00
e18f0f5178 [#999] morph: Use Global scope for proxy contract
All checks were successful
DCO action / DCO (pull_request) Successful in 2m28s
Vulncheck / Vulncheck (pull_request) Successful in 2m52s
Build / Build Components (1.21) (pull_request) Successful in 4m5s
Build / Build Components (1.20) (pull_request) Successful in 4m12s
Tests and linters / Lint (pull_request) Successful in 6m10s
Tests and linters / Staticcheck (pull_request) Successful in 5m59s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m24s
Tests and linters / Tests with -race (pull_request) Successful in 8m38s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m58s
Proxy contract can now be used as an owner of NNS domains, thus we need
it not only to pay for the transaction but also to check domain
ownership. CalledByEntry is not enough, because we may register NNS
domains owned by proxy indirectly from the container contract.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-21 14:28:42 +03:00
7470c383dd [#997] metabase: Drop toMoveIt bucket
All checks were successful
DCO action / DCO (pull_request) Successful in 2m9s
Vulncheck / Vulncheck (pull_request) Successful in 3m3s
Build / Build Components (1.20) (pull_request) Successful in 3m56s
Build / Build Components (1.21) (pull_request) Successful in 3m54s
Tests and linters / Staticcheck (pull_request) Successful in 5m26s
Tests and linters / Lint (pull_request) Successful in 6m19s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m6s
Tests and linters / Tests with -race (pull_request) Successful in 8m20s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m39s
It is not used.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-21 10:06:05 +03:00
adf7ebab5b [#996] metabase: Speed up bucket creation
Most of the time it exits, e.g. when it is per-container and use on each
object PUT. Bbolt implementation first tries to create bucket and then
returns it if it exists. Create operation uses cursor and thus is not
very lightweight, we can avoid it.

```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                 │     old     │                new                 │
                 │   sec/op    │   sec/op     vs base               │
Put/parallel-8     174.4µ ± 3%   163.3µ ± 3%  -6.39% (p=0.000 n=10)
Put/sequential-8   263.3µ ± 2%   259.0µ ± 1%  -1.64% (p=0.000 n=10)
geomean            214.3µ        205.6µ       -4.05%

                 │     old      │                 new                 │
                 │     B/op     │     B/op      vs base               │
Put/parallel-8     275.3Ki ± 3%   281.1Ki ± 4%       ~ (p=0.063 n=10)
Put/sequential-8   413.0Ki ± 2%   426.6Ki ± 2%  +3.29% (p=0.003 n=10)
geomean            337.2Ki        346.3Ki       +2.70%

                 │     old     │                 new                 │
                 │  allocs/op  │  allocs/op   vs base                │
Put/parallel-8      678.0 ± 1%    524.5 ± 2%  -22.64% (p=0.000 n=10)
Put/sequential-8   1.329k ± 0%   1.183k ± 0%  -10.91% (p=0.000 n=10)
geomean             949.1         787.9       -16.98%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-20 15:42:58 +00:00
9f68305c2e Revert "[#972] pilorama: Remove removeDuplicatesInPlace()"
This reverts commit 45fd4e4ff1.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 15:36:01 +00:00
dad56d2e98 Revert "[#972] Use min/max builtins"
This reverts commit 89784b2e0a.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 15:36:01 +00:00
7627d08914 Revert "[#972] Use require.ElementsMatch() where possible"
This reverts commit 6d9707ff1f.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 15:36:01 +00:00
3359349acb Revert "[#972] Use slices.Sort* when useful"
This reverts commit b871d7a5e8.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 15:36:01 +00:00
946f2ec2bf Revert "[#972] Drop x/exp/slices dependency"
This reverts commit f3e50772fd.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 15:36:01 +00:00
13d5cd3e21 [#991] logger: Fix journald logger
All checks were successful
DCO action / DCO (pull_request) Successful in 1m7s
Vulncheck / Vulncheck (pull_request) Successful in 3m32s
Build / Build Components (1.22) (pull_request) Successful in 3m52s
Build / Build Components (1.21) (pull_request) Successful in 3m58s
Tests and linters / Lint (pull_request) Successful in 5m13s
Tests and linters / Tests (1.22) (pull_request) Successful in 7m22s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m34s
Tests and linters / Tests with -race (pull_request) Successful in 7m32s
Tests and linters / Staticcheck (pull_request) Successful in 2m22s
Allow to change logger level.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-19 16:18:13 +03:00
f3e50772fd [#972] Drop x/exp/slices dependency
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 13:13:09 +00:00
b871d7a5e8 [#972] Use slices.Sort* when useful
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 13:13:09 +00:00
6d9707ff1f [#972] Use require.ElementsMatch() where possible
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 13:13:09 +00:00
89784b2e0a [#972] Use min/max builtins
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 13:13:09 +00:00
45fd4e4ff1 [#972] pilorama: Remove removeDuplicatesInPlace()
Also, check that slices.CompareFunc() indeed passes all the tests before
removal.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 13:13:09 +00:00
15fc5bac26 [#972] keyer: Use UncompressedBytes() for marshaling
elliptic.Marshal() becomes deprecated in go1.21

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-19 13:13:09 +00:00
2680192ba0 [#988] objectSvc: Fix SetMarshalData for PutSingle
All checks were successful
DCO action / DCO (pull_request) Successful in 5m10s
Vulncheck / Vulncheck (pull_request) Successful in 5m24s
Build / Build Components (1.21) (pull_request) Successful in 7m40s
Build / Build Components (1.20) (pull_request) Successful in 7m50s
Tests and linters / Staticcheck (pull_request) Successful in 7m52s
Tests and linters / Lint (pull_request) Successful in 9m8s
Tests and linters / Tests with -race (pull_request) Successful in 10m54s
Tests and linters / Tests (1.20) (pull_request) Successful in 11m12s
Tests and linters / Tests (1.21) (pull_request) Successful in 11m7s
After api-go update it is required to pass marshal data
to `SetMarshalData`.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-15 17:21:08 +03:00
05b5f5ca85 [#959] writecache: Fix panic on Get when it is not initialized
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 2m41s
DCO action / DCO (pull_request) Successful in 2m43s
Tests and linters / Staticcheck (pull_request) Successful in 3m55s
Build / Build Components (1.21) (pull_request) Successful in 3m45s
Build / Build Components (1.20) (pull_request) Successful in 4m19s
Tests and linters / Lint (pull_request) Successful in 5m30s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m57s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m22s
Tests and linters / Tests with -race (pull_request) Successful in 8m39s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-14 16:10:33 +03:00
2429508ac5 [#959] shard: Skip rebuild in DEGRADED mode
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-14 15:39:28 +03:00
0bd030507e [#948] metrics: Set actual value for shard_id after restart
All checks were successful
DCO action / DCO (pull_request) Successful in 2m26s
Vulncheck / Vulncheck (pull_request) Successful in 3m56s
Tests and linters / Staticcheck (pull_request) Successful in 4m18s
Build / Build Components (1.21) (pull_request) Successful in 4m11s
Build / Build Components (1.20) (pull_request) Successful in 4m19s
Tests and linters / Lint (pull_request) Successful in 6m3s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m7s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m26s
Tests and linters / Tests with -race (pull_request) Successful in 9m20s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-13 09:43:21 +03:00
6a5769d1da [#948] Fix gofumpt issue
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-13 09:40:46 +03:00
962e5a9c19 [#736] logger: Add journald support
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-12 06:17:22 +00:00
b36a453238 [#970] fstree: Add build tag to enable generic version on linux
Unless tested, generic version can start gaining bugs. With a separate
build tag we can have the best of both worlds:
1. Use optimized implementation for linux by default.
2. Run tests or benchmarks for both. Note that they are not actually
   run automatically now, but this is at leas possible.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
abd502215f [#970] fstree: Move file locking to the generic writer
It is not a part of FSTree itself, but rather a way to solve concurrent
counter update on non-linux implementations. New linux implementations
is pretty simple: link fails when the file exists, unlink fails when the
file doesn't exist.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
fb74524ac7 [#970] fstree: Move delete implementation to a separate file
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
7f692409cf [#970] fstree: Handle unsupported O_TMPFILE
Metabase test relied on this behaviour, so fix the test too.

Cherry-picking was hard and did too many conflicts,
here is an original PR:
https://github.com/nspcc-dev/neofs-node/pull/2624

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
Roman Khimov
fc31b9c947 [#970] fstree: Add linux-specific file writer using O_TMPFILE
O_TMPFILE is implemented for all modern FSes and it's much easier and safer to
use. If application crashes in the middle of writing this file would be gone
and won't leave any garbage.

Notice that this implementation makes a different choice wrt EEXIST handling,
generic one always overwrites, while this one keeps the old data.

There is no real performance difference.

SSD (slow&old), XFS, Core i7-8565U:

Sync
```
name                                  old time/op    new time/op    delta
Put/size=1024,thread=1/fstree-8         1.74ms ± 3%    0.06ms ± 7%  -96.31%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8        10.0ms ±41%     1.1ms ±18%  -88.95%  (p=0.000 n=9+10)
Put/size=1024,thread=100/fstree-8       32.3ms ±60%     6.5ms ±14%  -79.97%  (p=0.000 n=10+10)
Put/size=1048576,thread=1/fstree-8      17.8ms ±90%     3.4ms ±70%  -81.08%  (p=0.000 n=10+10)
Put/size=1048576,thread=20/fstree-8     103ms ±174%    112ms ±158%     ~     (p=0.971 n=10+10)
Put/size=1048576,thread=100/fstree-8     949ms ±78%    583ms ±132%     ~     (p=0.089 n=10+10)

name                                  old alloc/op   new alloc/op   delta
Put/size=1024,thread=1/fstree-8         3.17kB ± 1%    1.96kB ± 0%  -38.09%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8        59.6kB ± 1%    39.2kB ± 1%  -34.30%  (p=0.000 n=8+10)
Put/size=1024,thread=100/fstree-8        299kB ± 0%     198kB ± 0%  -33.90%  (p=0.000 n=7+9)
Put/size=1048576,thread=1/fstree-8      3.38kB ± 1%    2.36kB ± 1%  -30.22%  (p=0.000 n=10+10)
Put/size=1048576,thread=20/fstree-8     65.7kB ± 4%    47.7kB ± 6%  -27.27%  (p=0.000 n=10+10)
Put/size=1048576,thread=100/fstree-8     351kB ± 8%     245kB ± 8%  -30.22%  (p=0.000 n=10+10)

name                                  old allocs/op  new allocs/op  delta
Put/size=1024,thread=1/fstree-8           30.3 ± 2%      21.0 ± 0%  -30.69%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8           554 ± 1%       413 ± 0%  -25.35%  (p=0.000 n=8+10)
Put/size=1024,thread=100/fstree-8        2.77k ± 0%     2.07k ± 0%  -25.27%  (p=0.000 n=7+10)
Put/size=1048576,thread=1/fstree-8        32.0 ± 0%      25.0 ± 0%  -21.88%  (p=0.000 n=9+8)
Put/size=1048576,thread=20/fstree-8        609 ± 5%       494 ± 6%  -18.93%  (p=0.000 n=10+10)
Put/size=1048576,thread=100/fstree-8     3.25k ± 9%     2.50k ± 8%  -23.21%  (p=0.000 n=10+10)
```

No sync
```
name                                  old time/op    new time/op    delta
Put/size=1024,thread=1/fstree-8         71.3µs ±10%    59.8µs ±10%  -16.21%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8        1.43ms ± 6%    1.22ms ±13%  -14.53%  (p=0.000 n=10+10)
Put/size=1024,thread=100/fstree-8       8.12ms ± 3%    6.36ms ± 2%  -21.67%  (p=0.000 n=8+9)
Put/size=1048576,thread=1/fstree-8      1.88ms ±70%    1.61ms ±78%     ~     (p=0.393 n=10+10)
Put/size=1048576,thread=20/fstree-8     32.7ms ±28%   34.2ms ±112%     ~     (p=0.968 n=9+10)
Put/size=1048576,thread=100/fstree-8     262ms ±56%     226ms ±34%     ~     (p=0.447 n=10+9)

name                                  old alloc/op   new alloc/op   delta
Put/size=1024,thread=1/fstree-8         2.89kB ± 0%    1.96kB ± 0%  -32.28%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8        58.2kB ± 0%    39.5kB ± 0%  -32.09%  (p=0.000 n=8+8)
Put/size=1024,thread=100/fstree-8        291kB ± 0%     198kB ± 0%  -32.19%  (p=0.000 n=9+9)
Put/size=1048576,thread=1/fstree-8      3.05kB ± 1%    2.13kB ± 1%  -30.16%  (p=0.000 n=10+9)
Put/size=1048576,thread=20/fstree-8     62.6kB ± 0%    44.3kB ± 0%  -29.23%  (p=0.000 n=9+9)
Put/size=1048576,thread=100/fstree-8     302kB ± 0%     210kB ± 1%  -30.39%  (p=0.000 n=9+9)

name                                  old allocs/op  new allocs/op  delta
Put/size=1024,thread=1/fstree-8           27.0 ± 0%      21.0 ± 0%  -22.22%  (p=0.000 n=10+10)
Put/size=1024,thread=20/fstree-8           539 ± 0%       415 ± 0%  -22.98%  (p=0.000 n=10+10)
Put/size=1024,thread=100/fstree-8        2.69k ± 0%     2.07k ± 0%  -23.09%  (p=0.000 n=9+9)
Put/size=1048576,thread=1/fstree-8        28.0 ± 0%      22.3 ± 3%  -20.36%  (p=0.000 n=8+10)
Put/size=1048576,thread=20/fstree-8        577 ± 0%       458 ± 0%  -20.72%  (p=0.000 n=9+9)
Put/size=1048576,thread=100/fstree-8     2.76k ± 0%     2.15k ± 0%  -22.05%  (p=0.000 n=9+8)
```

HDD (LVM), ext4, Ryzen 5 1600:

Sync
```
                                      │ fs.sync-generic │            fs.sync-linux            │
                                      │     sec/op      │    sec/op     vs base               │
Put/size=1024,thread=1/fstree-12           34.70m ± 19%   33.59m ± 16%       ~ (p=0.529 n=10)
Put/size=1024,thread=20/fstree-12          188.8m ±  8%   189.2m ± 16%       ~ (p=0.739 n=10)
Put/size=1024,thread=100/fstree-12         264.8m ± 22%   273.6m ± 28%       ~ (p=0.353 n=10)
Put/size=1048576,thread=1/fstree-12        54.90m ± 14%   47.08m ± 18%       ~ (p=0.063 n=10)
Put/size=1048576,thread=20/fstree-12       244.1m ± 14%   220.4m ± 22%       ~ (p=0.579 n=10)
Put/size=1048576,thread=100/fstree-12      847.2m ±  5%   893.6m ±  3%  +5.48% (p=0.000 n=10)
geomean                                    164.3m         158.9m        -3.29%

                                      │ fs.sync-generic │            fs.sync-linux             │
                                      │      B/op       │     B/op      vs base                │
Put/size=1024,thread=1/fstree-12           3.375Ki ± 1%   2.471Ki ± 1%  -26.80% (p=0.000 n=10)
Put/size=1024,thread=20/fstree-12          66.62Ki ± 6%   49.21Ki ± 6%  -26.15% (p=0.000 n=10)
Put/size=1024,thread=100/fstree-12         319.2Ki ± 1%   230.9Ki ± 2%  -27.64% (p=0.000 n=10)
Put/size=1048576,thread=1/fstree-12        3.457Ki ± 1%   2.559Ki ± 1%  -25.97% (p=0.000 n=10)
Put/size=1048576,thread=20/fstree-12       66.91Ki ± 1%   49.16Ki ± 1%  -26.52% (p=0.000 n=10)
Put/size=1048576,thread=100/fstree-12      338.8Ki ± 2%   252.3Ki ± 3%  -25.54% (p=0.000 n=10)
geomean                                    42.17Ki        31.02Ki       -26.44%

                                      │ fs.sync-generic │            fs.sync-linux            │
                                      │    allocs/op    │  allocs/op   vs base                │
Put/size=1024,thread=1/fstree-12             33.00 ± 0%    27.00 ± 0%  -18.18% (p=0.000 n=10)
Put/size=1024,thread=20/fstree-12            639.5 ± 1%    519.0 ± 2%  -18.84% (p=0.000 n=10)
Put/size=1024,thread=100/fstree-12          3.059k ± 1%   2.478k ± 2%  -18.99% (p=0.000 n=10)
Put/size=1048576,thread=1/fstree-12          33.50 ± 1%    28.00 ± 4%  -16.42% (p=0.000 n=10)
Put/size=1048576,thread=20/fstree-12         638.5 ± 1%    520.0 ± 1%  -18.56% (p=0.000 n=10)
Put/size=1048576,thread=100/fstree-12       3.209k ± 2%   2.655k ± 2%  -17.28% (p=0.000 n=10)
geomean                                      405.3         332.1       -18.05%
```

No sync
```
                                      │ fs.nosync-generic │             fs.nosync-linux              │
                                      │      sec/op       │    sec/op     vs base                    │
Put/size=1024,thread=1/fstree-12           148.2µ ± 20%     136.6µ ± 19%   -7.89% (p=0.029 n=10)
Put/size=1024,thread=20/fstree-12          1.140m ± 26%     1.364m ± 16%        ~ (p=0.143 n=10)
Put/size=1024,thread=100/fstree-12         11.93m ± 68%     26.89m ± 62%        ~ (p=0.123 n=10)
Put/size=1048576,thread=1/fstree-12        1.302m ±  3%     1.287m ±  5%        ~ (p=0.481 n=10)
Put/size=1048576,thread=20/fstree-12       77.52m ±  8%     74.07m ±  7%        ~ (p=0.278 n=10+9)
Put/size=1048576,thread=100/fstree-12      226.1m ±   ∞ ¹
geomean                                    5.986m           3.434m        +18.60%                  ²
¹ need >= 6 samples for confidence interval at level 0.95
² benchmark set differs from baseline; geomeans may not be comparable

                                      │ fs.nosync-generic │             fs.nosync-linux              │
                                      │       B/op        │     B/op      vs base                    │
Put/size=1024,thread=1/fstree-12           2.879Ki ± 0%     1.972Ki ± 0%  -31.51% (p=0.000 n=10)
Put/size=1024,thread=20/fstree-12          55.94Ki ± 1%     37.90Ki ± 1%  -32.25% (p=0.000 n=10)
Put/size=1024,thread=100/fstree-12         272.6Ki ± 0%     182.1Ki ± 9%  -33.21% (p=0.000 n=10)
Put/size=1048576,thread=1/fstree-12        3.158Ki ± 0%     2.259Ki ± 0%  -28.46% (p=0.000 n=10)
Put/size=1048576,thread=20/fstree-12       58.87Ki ± 0%     41.03Ki ± 0%  -30.30% (p=0.000 n=10+9)
Put/size=1048576,thread=100/fstree-12      299.8Ki ±  ∞ ¹
geomean                                    36.71Ki          16.60Ki       -31.17%                  ²
¹ need >= 6 samples for confidence interval at level 0.95
² benchmark set differs from baseline; geomeans may not be comparable

                                      │ fs.nosync-generic │            fs.nosync-linux            │
                                      │     allocs/op     │  allocs/op   vs base                  │
Put/size=1024,thread=1/fstree-12             28.00 ± 0%      22.00 ± 0%  -21.43% (p=0.000 n=10)
Put/size=1024,thread=20/fstree-12            530.0 ± 0%      407.5 ± 1%  -23.11% (p=0.000 n=10)
Put/size=1024,thread=100/fstree-12          2.567k ± 0%     1.956k ± 9%  -23.77% (p=0.000 n=10)
Put/size=1048576,thread=1/fstree-12          30.00 ± 0%      24.00 ± 0%  -20.00% (p=0.000 n=10)
Put/size=1048576,thread=20/fstree-12         553.5 ± 0%      434.0 ± 0%  -21.59% (n=10+9)
Put/size=1048576,thread=100/fstree-12       2.803k ±  ∞ ¹
geomean                                      347.9           178.8       -21.99%                ²
¹ need >= 6 samples for confidence interval at level 0.95
² benchmark set differs from baseline; geomeans may not be comparable
```

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
ff488b53a1 [#970] fstree: Move write functions to a separate file
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
9a622a750d [#970] fstree: Move temporary path handling in a separate function
Allow to easier test different implementations.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-09 16:12:11 +00:00
d19ade23c8 [#959] node: Set mode to shard's components when open it
Avoid opening database for `metabase` and `cache` in `Degraded` mode.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-09 14:04:01 +00:00
db67c21d55 [#947] engine: Evacuate trees to remote nodes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:33:15 +03:00
728150d1d2 [#947] engine: Evacuate trees to local shards
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:33:15 +03:00
15d853ea22 [#947] controlSvc: Return tree evacuation stat
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:20:39 +03:00
b3f3505ada [#947] cli: Allow to specify evacuation scope
It may be required to evacuate only objects or only tree or all, so
now it spossible to specify.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:20:38 +03:00
a6eb66bf9c [#947] evacuate: Refactor evacuate parameters
Drop methods to make it easier to extend.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:20:38 +03:00
8e2a0611f4 [#947] tree: Add method to list all trees
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-09 11:20:38 +03:00
edbe06e07e [#956] policer/test: Reuse testPool helper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 15:49:43 +00:00
cbfeb72466 [#956] policer: Remove WithMaxCapacity option
We already provide the pool and this argument is used only for
preallocation. No functional changes.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 15:49:43 +00:00
cfc5ce7853 [#964] metabase: Drop GC marks if object not found
GC inhumes expired locks and tombstones on all the shards.
So it could be GC mark without object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-08 07:54:39 +00:00
c3fa902780 [#969] policer: Restrict the number of remembered errors
All checks were successful
DCO action / DCO (pull_request) Successful in 4m23s
Vulncheck / Vulncheck (pull_request) Successful in 4m57s
Build / Build Components (1.21) (pull_request) Successful in 6m3s
Build / Build Components (1.20) (pull_request) Successful in 9m2s
Tests and linters / Staticcheck (pull_request) Successful in 10m4s
Tests and linters / Lint (pull_request) Successful in 10m51s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m44s
Tests and linters / Tests (1.20) (pull_request) Successful in 10m56s
Tests and linters / Tests with -race (pull_request) Successful in 10m57s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 10:10:41 +03:00
6010dfdf3d [#969] policer: Make error skip thread-safe
Introduces in afd2ba9a66.
Refs #914

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-08 10:10:41 +03:00
a6c9a337cd [#965] morph: Get rid of container.List invocations
ContainersOf() is better in almost every aspect, besides creating a
session when the containers number is between 1024 and 2048 (prefetch
script does limited unwrapping). Making List() private helps to ensure
it is no longer used and can be safely removed in future.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-07 08:56:27 +00:00
9ba48c582d [#917] engine: Allow to detach shards
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-06 14:49:47 +03:00
4358d3c423 [#917] controlSvc: Add DetachShards handler
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-02-06 14:47:52 +03:00
afd2ba9a66 [#110] Add check for repeated error log in policer
All checks were successful
DCO action / DCO (pull_request) Successful in 2m55s
Vulncheck / Vulncheck (pull_request) Successful in 3m22s
Tests and linters / Staticcheck (pull_request) Successful in 4m10s
Build / Build Components (1.20) (pull_request) Successful in 4m58s
Build / Build Components (1.21) (pull_request) Successful in 4m54s
Tests and linters / Lint (pull_request) Successful in 5m47s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m45s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m58s
Tests and linters / Tests with -race (pull_request) Successful in 7m42s
processObject() returns 3 types of errors: container not found errors,
could not get container error and placement vector building error. Every
error will occur for all objects in container simultaneously, so we can
log each error once and safely ignore the rest.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-02-06 00:56:41 +03:00
602ee11123 [#934] containersvc: Marhal public key in short format for APE
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-02-02 17:51:38 +00:00
c1a5b831b6 [#955] chainbase: Fix rule chain unmarshalling
All checks were successful
DCO action / DCO (pull_request) Successful in 4m41s
Vulncheck / Vulncheck (pull_request) Successful in 4m46s
Tests and linters / Staticcheck (pull_request) Successful in 6m30s
Tests and linters / Lint (pull_request) Successful in 7m36s
Build / Build Components (1.21) (pull_request) Successful in 8m57s
Build / Build Components (1.20) (pull_request) Successful in 9m8s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m32s
Tests and linters / Tests (1.20) (pull_request) Successful in 10m42s
Tests and linters / Tests with -race (pull_request) Successful in 11m23s
* Use correct way DecodeBytes instead unmarshalling by json.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-02 20:28:04 +03:00
9916598dfb [#922] control: Extend api with ListOverrideDefinedTargets
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-02 12:09:51 +00:00
95e15f499f [#922] Update files generated by protoc
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-02 12:09:51 +00:00
2cb04379a4 [#922] go.mod: Update APE
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-02 12:09:51 +00:00
a5446bc17d [#952] object: Pass namespace within context in ACL service
All checks were successful
DCO action / DCO (pull_request) Successful in 6m23s
Vulncheck / Vulncheck (pull_request) Successful in 7m3s
Build / Build Components (1.21) (pull_request) Successful in 8m21s
Build / Build Components (1.20) (pull_request) Successful in 8m31s
Tests and linters / Staticcheck (pull_request) Successful in 11m1s
Tests and linters / Lint (pull_request) Successful in 11m26s
Tests and linters / Tests (1.20) (pull_request) Successful in 12m51s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m14s
Tests and linters / Tests with -race (pull_request) Successful in 13m31s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-02 14:48:11 +03:00
d0eadf7ea2 [#799] engine: Skip put when object removed from shard
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-01 17:49:22 +00:00
6534252c22 [#799] policer: Refactor method processNodes
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-02-01 17:49:22 +00:00
5be2af881a [#934] container: Make container APE middleware read namespaces
* Those methods that can access already existing containers and thus
  can get container properties should read namespace from Zone
  property. If Zone is not set, take a namespace for root.
* Otherwise, define namespaces by owner ID via frostfs-id contract.
* Improve unit-tests, consider more cases.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-01 17:38:24 +00:00
4352bd0e8e [#934] ape: Transform empty namespace within chainbase
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-02-01 17:38:24 +00:00
483a67b170 [#937] ape: Validate chain resource name
All checks were successful
DCO action / DCO (pull_request) Successful in 2m13s
Build / Build Components (1.20) (pull_request) Successful in 3m38s
Vulncheck / Vulncheck (pull_request) Successful in 3m13s
Tests and linters / Staticcheck (pull_request) Successful in 6m15s
Build / Build Components (1.21) (pull_request) Successful in 7m11s
Tests and linters / Lint (pull_request) Successful in 9m35s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m56s
Tests and linters / Tests (1.20) (pull_request) Successful in 14m14s
Tests and linters / Tests with -race (pull_request) Successful in 6m2s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-31 11:34:35 +03:00
e3573de6db [#930] gc: Stop internal activity by context
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-31 08:30:34 +00:00
c441296592 [#930] policer: Release task pool when context cancelled
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-31 08:30:34 +00:00
675eec91f3 [#938] shard: Update only changed counters
All checks were successful
DCO action / DCO (pull_request) Successful in 5m40s
Vulncheck / Vulncheck (pull_request) Successful in 5m33s
Tests and linters / Staticcheck (pull_request) Successful in 7m12s
Build / Build Components (1.21) (pull_request) Successful in 7m25s
Build / Build Components (1.20) (pull_request) Successful in 7m29s
Tests and linters / Lint (pull_request) Successful in 8m26s
Tests and linters / Tests (1.20) (pull_request) Successful in 9m48s
Tests and linters / Tests with -race (pull_request) Successful in 9m56s
Tests and linters / Tests (1.21) (pull_request) Successful in 10m10s
If metric value hasn't changed, but we update metric, then
non existed metric will apear with zero value.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-30 12:37:48 +03:00
c681354afd [#938] engine: Fix container count removal
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-30 12:37:48 +03:00
df055fead5 [#931] morph: Provide batch size for container listing explicitly
Besides VM stack item limit we also have restrictions on the size of
JSON for a stackitem, which is 128k. This limit is much harder to
calculate, because JSON representation includes type and the encoding is
different for different items. Thus is makes no sense to invent our own
default, so use the one provided by neo-go. But for container listing we
know exactly what we process, so use big enough value, which is tested.

Introduced in be8607a1f6.

Refs #902
Refs https://github.com/nspcc-dev/neo-go/blob/v0.105.0/pkg/vm/stackitem/json.go#L353

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-01-29 14:00:11 +00:00
6e2cc32768 [#681] objsvc: Validate session token owner for local sessions
Previously, the check was in place only when session token was missing.
Format validator checks are applied only to fully-prepared object, so
this lead to the following situation:
1. Object is put locally with malformed token, because there are no
   checks.
2. Object cannot be replicated, because the token is malformed.

This is now fixed and token check is done before any payload receival.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-01-26 08:52:29 +00:00
b6fc3321c5 [#876] Fix linters
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-01-25 20:26:13 +03:00
1fe7736d92 [#925] morph: Introduce switch rpc guarded rpc actor
* Introduce switch rpc guarded rpc actor in morph client
  to prevent using invalidated rpc actor when RPC switch
  happens.
* Initialize NewContractStorage with SwitchRPCGuardedActor.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-25 13:24:13 +00:00
5ed330e436 [#927] metabase: Delete GC marks
All checks were successful
DCO action / DCO (pull_request) Successful in 6m34s
Vulncheck / Vulncheck (pull_request) Successful in 7m19s
Build / Build Components (1.20) (pull_request) Successful in 7m48s
Build / Build Components (1.21) (pull_request) Successful in 9m39s
Tests and linters / Staticcheck (pull_request) Successful in 10m36s
Tests and linters / Lint (pull_request) Successful in 11m26s
Tests and linters / Tests (1.20) (pull_request) Successful in 12m57s
Tests and linters / Tests (1.21) (pull_request) Successful in 13m8s
Tests and linters / Tests with -race (pull_request) Successful in 13m7s
`key` is changed inside `db.get`, so encode address again after get.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-24 18:51:16 +03:00
931a5e9aaf [#918] engine: Move shard to degraded mode if metabase open failed
All checks were successful
DCO action / DCO (pull_request) Successful in 1m55s
Build / Build Components (1.20) (pull_request) Successful in 2m14s
Vulncheck / Vulncheck (pull_request) Successful in 2m15s
Build / Build Components (1.21) (pull_request) Successful in 4m6s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m58s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m24s
Tests and linters / Staticcheck (pull_request) Successful in 6m11s
Tests and linters / Lint (pull_request) Successful in 6m37s
Tests and linters / Tests with -race (pull_request) Successful in 6m35s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-23 11:16:40 +03:00
f2f3294fc3 [#919] ape: Improve error messages in ape service
* Wrap all APE middleware errors in apeErr that
  makes errors more explicit with status AccessDenied.
* Use denyingRuleErr for denying status from chain router.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-23 08:11:24 +00:00
f526f49995 [#874] engine: Check object existance concurrently
All checks were successful
DCO action / DCO (pull_request) Successful in 1m36s
Vulncheck / Vulncheck (pull_request) Successful in 2m33s
Build / Build Components (1.21) (pull_request) Successful in 3m9s
Build / Build Components (1.20) (pull_request) Successful in 4m49s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m5s
Tests and linters / Staticcheck (pull_request) Successful in 6m6s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m39s
Tests and linters / Lint (pull_request) Successful in 7m0s
Tests and linters / Tests with -race (pull_request) Successful in 6m46s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-23 09:28:29 +03:00
f5160b27fc [#920] tests: Fix data races
All checks were successful
DCO action / DCO (pull_request) Successful in 2m2s
Build / Build Components (1.21) (pull_request) Successful in 3m35s
Vulncheck / Vulncheck (pull_request) Successful in 3m3s
Tests and linters / Staticcheck (pull_request) Successful in 5m23s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m33s
Tests and linters / Tests (1.21) (pull_request) Successful in 7m41s
Tests and linters / Tests with -race (pull_request) Successful in 7m47s
Tests and linters / Lint (pull_request) Successful in 8m12s
Build / Build Components (1.20) (pull_request) Successful in 1m32s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-19 14:06:05 +03:00
be33070550 [#910] control: Take empty namespace for 'root'
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-19 06:22:10 +00:00
63d3ed1ad8 [#904] tests: Close test engine after test
All checks were successful
DCO action / DCO (pull_request) Successful in 2m7s
Vulncheck / Vulncheck (pull_request) Successful in 2m37s
Build / Build Components (1.20) (pull_request) Successful in 4m0s
Build / Build Components (1.21) (pull_request) Successful in 3m56s
Tests and linters / Tests (1.21) (pull_request) Successful in 6m25s
Tests and linters / Staticcheck (pull_request) Successful in 6m22s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m43s
Tests and linters / Tests with -race (pull_request) Successful in 6m47s
Tests and linters / Lint (pull_request) Successful in 7m5s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 19:04:39 +03:00
57171907e3 [#904] metabase: Return if object was actuall inserted
This requires to count metrics properly.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 19:04:39 +03:00
c1a80235db [#904] metabase: Log Inhume operation
It will be very useful for troubleshooting.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-17 18:42:52 +03:00
96b020626f [#915] ape: Fix method name in getStreamBasicChecker
All checks were successful
DCO action / DCO (pull_request) Successful in 2m2s
Build / Build Components (1.21) (pull_request) Successful in 2m17s
Build / Build Components (1.20) (pull_request) Successful in 3m2s
Vulncheck / Vulncheck (pull_request) Successful in 2m39s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m54s
Tests and linters / Staticcheck (pull_request) Successful in 5m49s
Tests and linters / Tests (1.20) (pull_request) Successful in 6m11s
Tests and linters / Lint (pull_request) Successful in 6m44s
Tests and linters / Tests with -race (pull_request) Successful in 6m32s
* Replace incorrect MethodGetContainer by MethodGetObject constant.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-16 23:52:37 +03:00
c8baf76fae [#872] object: Introduce APE middlewar for object service
All checks were successful
DCO action / DCO (pull_request) Successful in 2m4s
Vulncheck / Vulncheck (pull_request) Successful in 3m12s
Build / Build Components (1.21) (pull_request) Successful in 4m1s
Build / Build Components (1.20) (pull_request) Successful in 4m13s
Tests and linters / Staticcheck (pull_request) Successful in 4m3s
Tests and linters / Lint (pull_request) Successful in 8m7s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m14s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m18s
Tests and linters / Tests with -race (pull_request) Successful in 8m24s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-12 18:41:35 +03:00
52ffa9f164 [#891] getSvc: Refactor Get service V2 creation
All checks were successful
DCO action / DCO (pull_request) Successful in 2m46s
Vulncheck / Vulncheck (pull_request) Successful in 2m53s
Build / Build Components (1.20) (pull_request) Successful in 4m31s
Build / Build Components (1.21) (pull_request) Successful in 4m26s
Tests and linters / Lint (pull_request) Successful in 6m9s
Tests and linters / Staticcheck (pull_request) Successful in 6m7s
Tests and linters / Tests (1.20) (pull_request) Successful in 8m27s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m4s
Tests and linters / Tests with -race (pull_request) Successful in 9m48s
Use arguments for mandatory fields.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-12 13:35:38 +03:00
394f086fe2 [#891] getSvc: Fix get range hash implementation
Get range can perform GET request, so this request must be done
from container node to not to get access denied error.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-12 13:35:38 +03:00
be8607a1f6 [#902] morph: Avoid creating session in TestInvokeIterator
When the number of items to iterate over is less than 2048, there is no
need to create a session and consume resources.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-01-12 10:25:56 +00:00
a2ab373a0a [#895] metabase: Do not delete GC mark for virtual objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
7166e77c2b [#895] test: Add logger to test shard
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
47dcfa20f3 [#895] test: Use t.Cleanup only for external resources
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
836818fb75 [#895] test: Use zaptest logger
Previous implementation mixes logs from different tests, now logs
are separeted by test.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
f1b2b8bffa [#895] test: Fix NewLogger arguments list
`debug` is always true.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-11 12:32:09 +00:00
a8e52ef7aa [#898] control: Fix codes for returning APE errors
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-11 12:31:31 +00:00
5d982976fd [#898] ape: Fix bug with creating type bucket in chainbase
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-01-11 12:31:31 +00:00
c19396d203 [#885] control: Make chain id bytes in grpc
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-01-11 07:24:22 +00:00
5c0a736a25 [#899] containerSvc: Fix invalid session token type
Some checks failed
DCO action / DCO (pull_request) Successful in 1m23s
Vulncheck / Vulncheck (pull_request) Successful in 3m29s
Tests and linters / Tests (1.21) (pull_request) Failing after 3m58s
Build / Build Components (1.21) (pull_request) Successful in 3m46s
Build / Build Components (1.20) (pull_request) Successful in 3m52s
Tests and linters / Lint (pull_request) Successful in 4m48s
Tests and linters / Staticcheck (pull_request) Successful in 5m5s
Tests and linters / Tests (1.20) (pull_request) Successful in 7m4s
Tests and linters / Tests with -race (pull_request) Successful in 8m36s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-10 18:37:54 +03:00
79bebe4a68 [#884] cli: Fix error message for undefined endpoint
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-01-10 10:02:16 +00:00
4b8b4da681 [#864] engine: Drop container count metric if container removed
All checks were successful
DCO action / DCO (pull_request) Successful in 1m30s
Build / Build Components (1.21) (pull_request) Successful in 3m29s
Build / Build Components (1.20) (pull_request) Successful in 3m53s
Tests and linters / Lint (pull_request) Successful in 4m31s
Tests and linters / Tests (1.20) (pull_request) Successful in 5m1s
Tests and linters / Staticcheck (pull_request) Successful in 4m51s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m13s
Tests and linters / Tests with -race (pull_request) Successful in 8m34s
Vulncheck / Vulncheck (pull_request) Successful in 58s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-10 10:45:32 +03:00
d75e7e9a21 [#864] engine: Drop container size metric if container deleted
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-10 10:44:54 +03:00
dfd62ca6b1 [#864] metabase: Refactor delete/inhume
Available -> Logic, Raw -> Phy for delete/inhume results.
Use single counter instead of vectors.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-01-09 09:59:42 +03:00
225fe2d4d5 [#894] blobovniczatree: Speedup rebuild test
Down from 3s to 300ms.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-12-29 16:28:54 +00:00