Commit graph

3180 commits

Author SHA1 Message Date
9c2c76ca32
[#1259] Move pkg/util/locode to frostfs-locode-db
Removed pkg/util/locode package, added
git.frostfs.info/TrueCloudLab/frostfs-locode-db/pkg/locode dependency.

Signed-off-by: George Bartolomey <george@bh4.ru>
2024-07-23 14:59:14 +03:00
9ee1bd3669
[#1265] *: Run gofumpt
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-07-23 10:36:32 +03:00
18182e578e [#1261] shard: Fix delete objects from FSTree
Replace nil storageID with empty like by shard.Get.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-22 14:38:09 +03:00
3119f2fbc3 [#1257] metabase: Delete EC gc marks and split info
EC parent and split gc marks should be deleted after last EC chunk delete.
Also should delete split info for EC parent and split parent.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-19 17:14:14 +03:00
eadcea8df0 [#1249] object: Remove all APE pre-checks in handlers
* Methods `Head`, `Get`, `GetRangeHash` should no longer use APE pre-checks
  as that leads only to incorrect rule chain processing for requests:
  1. Immediate return with `NoRuleFound` may be unexpected as some `Allow`
     rule is actually defined but can't be matched yet as it gets no object
     attributes;
  2. Immdediate return with `Allow` may be incorrect as some `Deny` rule
     is actually defined but can't bet matched yet as it gets no object
     attirbutes;
  3. Pre-check breaks compatibility for converted EACL-tables.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-18 13:52:43 +00:00
5e5ee545b6 [#1254] policer: Fix svacer warning
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-18 11:30:39 +00:00
27caa8a72f [#1256] metabase: Put split parent object ID for EC chunks
It is required to save split parent ID too, not only split ID.
Otherwise inhume operation works incorrect: shard with last part may be skipped
and parent object will be available.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-17 18:08:07 +03:00
e83d39e33f [#1253] deleteSvc: Use copy of common parameters
getSvc may change the values of some fields, so Head will affect Delete
or Put. In this case, the change is necessary so that the session token
is stored in the tombstone object (EC assemble calls `ForgetTokens`).

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-17 14:24:27 +03:00
fc383ea6ae [#1253] getSvc: Fix EC objects get
Now EC objects assembling is performed concurrently.
Also fixed issue with an error in case of getting
EC object via non-container node.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-17 14:24:27 +03:00
00a88b9936 [#1251] *: Run gofumpt
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-17 11:25:07 +03:00
3940bc17c1 [#1251] pilorama: Allow traversing multiple branches in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-17 11:25:07 +03:00
e5767c9002 [#1250] *: Reformat proto filets with clang-format
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 15:13:08 +00:00
286df198c9 [#1248] placement: Decouple ContainerNodes() cache from the placement builder.
Also, write tests.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 12:21:30 +00:00
3a48b282b6 [#1248] placement: Use epoch to track netmap versions
Previously we used pointer, this could have worked,
because most of the time, the netmap is cached.
This didn't work, however, because `lastNm` field was always nil.

Rework the mechanism completely:
1. Use epoch to track netmap versions, as it it simpler and
   is unrelated to the TTL of an underlying cache.
2. Fix a bug where the epoch could change while mutex was unlocked.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 12:21:30 +00:00
21431f22c0 [#1248] placement: Use cid.ID as key in the cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 12:21:30 +00:00
d5dc14c639 [#1243] object: Make APE checker set x-headers to request properties
* Update go.mod, go.sum;
* Add x-headers to request properties;
* Add a unit-test.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-16 07:28:42 +00:00
39866a957c [#1196] morph/client: Fix ArrayFromStackItem() description
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-07-15 16:24:58 +03:00
d02a7c2d38 [#1196] morph/client: Remove duplicate utility functions
* We used several utility functions to parse frostfsid client
  subject and extended subject. However, following the changes
  in TrueCloudLab/frostfs-contract#97, these utility functions
  have become public. So there is no more need to have them here.

* There was a mismatch of slice parameter required length between
  frostfs-node's and frostfs-contract's utility functions,
  `checkStackItem()` solves this problem.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-07-15 16:24:48 +03:00
3bf6e6dde6 [#1238] engine: Add non-blocking send in the shard's notification channel
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-07-10 11:37:11 +03:00
b027a7f91e [#1234] pilorama: Fix GetByPath() on duplicate directories
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-10 06:11:38 +00:00
6ace2f597e [#1234] pilorama: Add test for duplicate directory behaviour
When AddByPath() is called concurrently on 2 different nodes,
internal path components may be created twice. This violates some
of our assumptions in GetByPath() and, indirectly, in S3 handling of
GetSubTree() results.

Add a test for the correct behaviour, fixes will follow.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-10 06:11:38 +00:00
84ecd61dfd [#1233] putSvc: Try to put EC chunk to any node
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-09 07:54:29 +00:00
d90aab5454 [#1229] util: Fix session token expiration check
* Make session token expired at `current_epoch + 1` but
  not at `current_epoch` when it's still valid.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-08 08:15:56 +00:00
0c2b6f3dac [#1216] ape: Make services use bearer chains fed router
* Refactor object and tree service - they should instantiate
  chain router cheking the bearer token. If there are no bearer
  token rules, then defaul chain router is used.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-05 18:26:48 +00:00
47bcd346d3 [#1216] ape: Introduce BearerChainFeedRouter
* Unlike default chain router, `BearerChainFedRouter` performs checks for
  overrides defined in the bearer token;
* Add unit-test for the introduced router.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-05 18:26:48 +00:00
8eb591d668 [#1231] policer: Add EC node-off unit test
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-05 12:11:36 +03:00
cfd5e3d403 [#1227] morph/event: Release ants.Pool on listener stopping
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-04 10:55:05 +00:00
62cbb72a5e [#1226] blobovniczatree: Delete fix db extensions in Init()
Since several releases have been released, this code is not relevant.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 12:22:06 +03:00
78b1d9b18d [#1226] blobovniczatree: Drop leaf width limitation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 12:22:06 +03:00
40c9ddb6ba [#1226] blobovniczatree: Drop init in advance option
To make blobovniczatree unlimited.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 12:22:06 +03:00
3a797e4682 [#1222] engine: Fix tree evacuation
Do not fail if it is unable to evacuate tree to other node.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 10:38:10 +03:00
2bac82cd6f [#1222] engine: Fix object evacuation
Do not fail evacuation if it unable to evacuate object to other node.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 10:38:10 +03:00
bbe95dac8b [#1225] engine: Log the error when check object existence
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-07-04 07:22:54 +00:00
80d7459560 Revert "[#1196] morph/client: Remove duplicate utility functions"
This reverts commit 259007540f.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-03 15:44:35 +03:00
4bd4667791 [#1218] tree: Fix bearer token validation
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-03 07:22:11 +00:00
f3a861806e [#1218] object: Fix bearer token validation
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-03 07:22:11 +00:00
a378ff9cf6 [#1218] object: Pass container owner for backward get method check
* `getStreamBasicChecker` must define `containerOwner` for backward checks,
  otherwise bearer token cannot be validated for the token issuer.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-03 07:22:11 +00:00
91bed3b0ba [#1219] Remove Container.SetEACL method
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-02 13:05:40 +00:00
259007540f [#1196] morph/client: Remove duplicate utility functions
We used several utility functions to parse frostfsid client
subject and extended subject. However, following the changes
in TrueCloudLab/frostfs-contract#97, these utility functions
have become public. So there is no more need to have them here.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-07-02 15:19:59 +03:00
56eeb630b6 [#1217] Fix grammar mistakes and misspelling
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-07-01 19:14:25 +03:00
7a8ac4907a [#1213] engine: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-01 06:49:35 +00:00
dc2867682f [#1213] deleteSvc: Do not allow to delete EC chunks
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-01 06:49:35 +00:00
dc6778f385 [#1213] fmt: Drop unused interfaces
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-01 06:49:35 +00:00
7085723c6b [#1074] pilorama: Allow empty filenames in SortedByFilename()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-28 17:46:24 +03:00
4c7ff159ec [#1201] writecache: Cancel background flush without lock
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-28 09:02:36 +03:00
4f7d76c9ef [#1206] audit: Drop not required events
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-27 10:54:31 +00:00
4951babd5f [#1208] blobstor: Fix delete without storage id
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-27 11:59:08 +03:00
81ea91de52 [#451] metrics: Move to internal
`metrics` don't look like something others want to import.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-25 08:52:37 +00:00
11a38a0a84 [#1190] tree: GroupIDs must also be target of APE checks
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-25 08:49:20 +00:00
0b87388c18 [#1190] object: GroupIDs must also be target of APE checks
* Also add new test case for ape middleware in container service.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-25 08:49:20 +00:00