Commit graph

1252 commits

Author SHA1 Message Date
54fe8383a4 [#1374] tree: Use NewClient to create grpc connection for sync
Created connection will be used to sync trees, so it is ok to defer
dial to the first RPC call.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-13 15:59:26 +03:00
99be4c83a7
[#1368] *: Run gofumpt
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-12 10:00:28 +03:00
a812932984 [#1362] ape: Move common APE check logic to separate package
* Tree and object service have the same log for checking APE. So,
  this check should be moved to common package.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-10 12:40:34 +00:00
6b6eabe41c [#1337] cli: Add control shards rebuild command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 13:57:27 +03:00
f652518c24 [#1357] go: Fix panic caused by using range over slice len
If slice is altered in `for` loop, we cannot use range over its
length: it may cause panic if slice gets shorter.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-06 13:09:58 +03:00
273980cfb9 [#1310] object: Remove irrelevant comments
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-05 16:40:32 +03:00
108e4e07be [#1349] node: Evacuate objects without setting mode to MAINTENANCE
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-05 16:08:27 +03:00
b3deb893ba [#1310] object: Move target initialization to separate package
* Split the logic of write target initialization to different packages;
* Refactor patch and put services: since both service initialize the target
  themselves.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-05 13:03:58 +00:00
a4fb7f085b
[#1348] go.mod: Update api-go and sdk-go
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-04 10:47:26 +03:00
a685fcdc96 [#1317] go.mod: Use range over int
Since Go 1.22 a "for" statement with a "range" clause is able
to iterate through integer values from zero to an upper limit.

gopatch script:
@@
var i, e expression
@@
-for i := 0; i <= e - 1; i++ {
+for i := range e {
    ...
}

@@
var i, e expression
@@
-for i := 0; i <= e; i++ {
+for i := range e + 1 {
    ...
}

@@
var i, e expression
@@
-for i := 0; i < e; i++ {
+for i := range e {
    ...
}

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-03 13:00:54 +03:00
98fe24cdb7 [#1343] go.mod: Update api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-30 08:45:24 +00:00
03976c6ed5 [#1341] .golangci.yml: Replace exportloopref with copyloopvar
exportloopref is deprecated.
gopatch:
```
@@
var index, value identifier
var slice expression
@@
for index, value := range slice {
...
-value := value
...
}

@@
var index, value identifier
var slice expression
@@
for index, value := range slice {
...
-index := index
...
}

@@
var value identifier
var channel expression
@@
for value := range channel {
...
-value := value
...
}
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-28 15:44:41 +00:00
7abbdca064 [#1340] getSvc: Fix access denied error handling
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-28 14:02:28 +03:00
6488ddee88 [#1338] object: Fix range provider in Patch handler
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-27 16:51:12 +03:00
d6b42972a8 [#1338] object: Fix audit patch stream
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-27 16:18:29 +03:00
5e9a97fd3e [#1336] go.mod: Update api-go and sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-27 14:48:43 +03:00
bd24beecf8 [#1329] putSvc: Reset SuccessAfter for non-EC objects in EC container broadcasting
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-23 13:51:16 +03:00
7bca428db0 [#1322] Use new protobuf marshaler
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-22 07:17:41 +00:00
8319b59238 [#1318] Fix gofumpt issue
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-20 14:34:54 +03:00
b7acb34fa4 [#1319] treeSvc: Do not wrap error from APE
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-19 18:38:27 +03:00
eeab417dcf [#1307] object: Add APE check for Patch handler
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
e890f1b4b1 [#1307] object: Implement Patch method
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
a4a1c3f18b [#1307] go.mod: Bump frostfs-sdk-go/frostfs-api-go/v2 versions
* Also, resolve dependencies and conflicts for object service
  by creating stub for `Patch` method.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
d055168e2a [#1135] ir: Add healthstatus RECONFIGURING
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-15 16:54:45 +03:00
93d63e1632 [#1284] writecache: Allow to seal writecache async
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 13:23:35 +03:00
c985b1198f [#1302] putSvc: Override SuccessAfter for non-regular objects in EC containers
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 10:42:55 +03:00
36efccd862 [#1298] writecache: Add shrink flag for Seal command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-08 16:32:29 +03:00
5c01bd5be8 [#1298] writecache: Add restore-mode flag for Seal command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-07 11:20:14 +03:00
8e51d7849a [#1295] getSvc: Assemble complex EC object headers without linking object
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-06 16:48:12 +03:00
8021bacc43 [#1288] putSvc: Respect TTL for EC put
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-02 13:01:42 +00:00
7e04083c27 [#1278] containerSvc: Validate FrostFSID subject exitence on Put
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-29 16:01:37 +03:00
8377372a40 [#1276] go.mod: Update api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-26 16:44:19 +03:00
7fd7961dfa [#1271] getSvc: Fix local EC chunk get
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-24 08:52:16 +03:00
8398a8b4b3 [#1271] getSvc: Fix head --raw assemble for EC
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-24 08:52:08 +03:00
eadcea8df0 [#1249] object: Remove all APE pre-checks in handlers
* Methods `Head`, `Get`, `GetRangeHash` should no longer use APE pre-checks
  as that leads only to incorrect rule chain processing for requests:
  1. Immediate return with `NoRuleFound` may be unexpected as some `Allow`
     rule is actually defined but can't be matched yet as it gets no object
     attributes;
  2. Immdediate return with `Allow` may be incorrect as some `Deny` rule
     is actually defined but can't bet matched yet as it gets no object
     attirbutes;
  3. Pre-check breaks compatibility for converted EACL-tables.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-18 13:52:43 +00:00
5e5ee545b6 [#1254] policer: Fix svacer warning
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-18 11:30:39 +00:00
e83d39e33f [#1253] deleteSvc: Use copy of common parameters
getSvc may change the values of some fields, so Head will affect Delete
or Put. In this case, the change is necessary so that the session token
is stored in the tombstone object (EC assemble calls `ForgetTokens`).

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-17 14:24:27 +03:00
fc383ea6ae [#1253] getSvc: Fix EC objects get
Now EC objects assembling is performed concurrently.
Also fixed issue with an error in case of getting
EC object via non-container node.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-17 14:24:27 +03:00
00a88b9936 [#1251] *: Run gofumpt
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-17 11:25:07 +03:00
3940bc17c1 [#1251] pilorama: Allow traversing multiple branches in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-17 11:25:07 +03:00
e5767c9002 [#1250] *: Reformat proto filets with clang-format
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 15:13:08 +00:00
286df198c9 [#1248] placement: Decouple ContainerNodes() cache from the placement builder.
Also, write tests.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 12:21:30 +00:00
3a48b282b6 [#1248] placement: Use epoch to track netmap versions
Previously we used pointer, this could have worked,
because most of the time, the netmap is cached.
This didn't work, however, because `lastNm` field was always nil.

Rework the mechanism completely:
1. Use epoch to track netmap versions, as it it simpler and
   is unrelated to the TTL of an underlying cache.
2. Fix a bug where the epoch could change while mutex was unlocked.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 12:21:30 +00:00
21431f22c0 [#1248] placement: Use cid.ID as key in the cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 12:21:30 +00:00
d5dc14c639 [#1243] object: Make APE checker set x-headers to request properties
* Update go.mod, go.sum;
* Add x-headers to request properties;
* Add a unit-test.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-16 07:28:42 +00:00
84ecd61dfd [#1233] putSvc: Try to put EC chunk to any node
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-09 07:54:29 +00:00
d90aab5454 [#1229] util: Fix session token expiration check
* Make session token expired at `current_epoch + 1` but
  not at `current_epoch` when it's still valid.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-08 08:15:56 +00:00
0c2b6f3dac [#1216] ape: Make services use bearer chains fed router
* Refactor object and tree service - they should instantiate
  chain router cheking the bearer token. If there are no bearer
  token rules, then defaul chain router is used.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-05 18:26:48 +00:00
8eb591d668 [#1231] policer: Add EC node-off unit test
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-05 12:11:36 +03:00
3a797e4682 [#1222] engine: Fix tree evacuation
Do not fail if it is unable to evacuate tree to other node.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 10:38:10 +03:00
2bac82cd6f [#1222] engine: Fix object evacuation
Do not fail evacuation if it unable to evacuate object to other node.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-04 10:38:10 +03:00
4bd4667791 [#1218] tree: Fix bearer token validation
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-03 07:22:11 +00:00
f3a861806e [#1218] object: Fix bearer token validation
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-03 07:22:11 +00:00
a378ff9cf6 [#1218] object: Pass container owner for backward get method check
* `getStreamBasicChecker` must define `containerOwner` for backward checks,
  otherwise bearer token cannot be validated for the token issuer.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-03 07:22:11 +00:00
91bed3b0ba [#1219] Remove Container.SetEACL method
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-02 13:05:40 +00:00
dc2867682f [#1213] deleteSvc: Do not allow to delete EC chunks
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-01 06:49:35 +00:00
7085723c6b [#1074] pilorama: Allow empty filenames in SortedByFilename()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-28 17:46:24 +03:00
4f7d76c9ef [#1206] audit: Drop not required events
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-27 10:54:31 +00:00
11a38a0a84 [#1190] tree: GroupIDs must also be target of APE checks
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-25 08:49:20 +00:00
0b87388c18 [#1190] object: GroupIDs must also be target of APE checks
* Also add new test case for ape middleware in container service.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-25 08:49:20 +00:00
621dbf58ab [#1190] container: GroupIDs must also be target of APE checks
* Also add new test case for ape middleware in container service.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-25 08:49:20 +00:00
a83eeddb1d [#60] control: Add GetNetmapStatus method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-20 16:28:42 +03:00
fd28461def [#1184] ir: Add grpc middleware for control service
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-19 16:05:53 +03:00
ecd1ed7a5e [#1184] node: Add audit middleware for grpc services
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-06-19 16:05:53 +03:00
5b100699d7 [#566] policer: Move isClientErrMaintenance to frostfs-sdk-go
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-06-18 10:20:45 +03:00
239323eeef [#1157] tree: Make tree service use Bearer token's APE overrides
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-07 12:11:11 +00:00
04a3f891fd [#1157] object: Make APE checker use Bearer-token's APE overrides
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-06-07 12:11:11 +00:00
a90310335d [#1156] ape: Return not found when removing local overrides
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-06-07 12:10:57 +00:00
a849236b68 [#1161] node: Remove notification functionality
It is unused and will be reworked in future.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-06-07 12:10:51 +00:00
5aacb8fc86 [#1144] metabase: Save parent attributes for ec-chunks
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 19:55:32 +03:00
0b367007fc [#1152] go.mod: Update api-go and sdk versions
* Resolve conflicts for apemanager since api-go
  contains ape and apemanager packages and SDK only
  ape package.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-31 15:39:09 +03:00
c1af13b47e [#1147] node: Fix issue from gopls
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
6130650bb6 [#1147] node: Implement Lock\Delete requests for EC object
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
d355274cd0 [#1147] object: Use methods on pointer for searchsvc.execCtx
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
3555c73225 [#1147] object: Use methods on pointer for deletesvc.execCtx
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
e43e7bec3a [#1147] log: Remove redundant address field from log
Filled when logger created for `request` object from package `getsvc`.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
50923ed81c [#1147] Fix gofumpt issue
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-30 08:13:04 +00:00
3627b44e92 [#1142] tree: Fill APE-request with source IP property
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-27 10:17:17 +00:00
482c5129ac [#1142] object: Fill APE-request with source IP property
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-27 10:17:17 +00:00
43625e7536 [#1142] container: Fill APE-request property with source IP
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-27 10:17:17 +00:00
542d3adcb2 [#1105] apemanager: Implement apemanager service
* Introduce grpc server for apemanager service and
  its implementation in `pkg/services/apemanager`.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-27 09:34:21 +00:00
8fd678e269 [#1141] go.mod: Update frostfs-sdk-go and frostfs-api-go/v2
* Also fix unit-test.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-20 13:22:48 +03:00
436c9f5558 [#1129] policer: Restore EC object
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-17 14:36:18 +03:00
44f2e8f27f [#1129] putSvc: Allow to put single unprepared object to EC container
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:49 +03:00
1cd8562db8 [#1129] policer: Refactor shortage
Drop override inside method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:49 +03:00
4ab6c404f7 [#1129] policer: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:49 +03:00
cbe9757490 [#1129] policer: Pull required EC chunks
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:49 +03:00
d45d086acd [#1129] policer: Add EC chunk replication
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-16 16:28:48 +03:00
b078fe5ba1 [#1092] control: Move SignMessage to separate package
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-05-16 12:14:01 +03:00
b3eaa8a9bc [#1083] objsvc/v2: Check response status in RANGE_HASH forwarder
Fixes #1083

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-15 12:04:21 +03:00
0924b62a95 [#1083] objsvc/v2: Unify response verification after forwarding
1. Use the same routine for HEAD/GET_RANGE methods.
2. Make error message similar.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-15 12:04:06 +03:00
300654b045 [#1083] objsvc/v2: Properly check response status after forwarding
Previously we had cryptic error:
```
debug   get/remote.go:38        remote call failed      {"component": "Object.Get service", "request": "HEAD", "address": "9sTxoVrhJ7WBtXQfK2NJ7zDV5yCF7BPLKK1XTxYPdGsP/BbHV4KZZ8y2BPqAT5kyjdHRLkfbtY2xf5uYoMVqxACn1", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "unexpected header type <nil>"}
```
Now we have and expected error:
```
debug   get/remote.go:38        remote call failed      {"component": "Object.Get service", "request": "HEAD", "address": "D2rqaMG4D2VHdv3HKky8UYSYmwQFH2v9oXXqtyRZPTMy/BbHV4KZZ8y2BPqAT5kyjdHRLkfbtY2xf5uYoMVqxACn1", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-15 12:04:06 +03:00
952d13cd2b [#1124] cli: Improve APE rule parsing
* Make APE rule parser to read condition's kind in unambiguous using lexemes
`ResourceCondition`, `RequestCondition` instead confusing `Object.Request`, `Object.Resource`.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-05-14 12:23:26 +03:00
0144117cc9 [#1125] objectSvc: Add EC header APE check
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-08 16:25:55 +03:00
ada1b9f737 [#1120] objectSvc: Fix EC put placement
Use parent object ID to compute placement.
Fix too many copies saving.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-05-08 15:23:57 +03:00
fe2c1c926f [#1112] node: Fix race warning for GetObjectAndWritePayload
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-07 14:47:21 +03:00
3e782527b8 [#1112] node: Add test for Range request for EC object
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-07 14:47:21 +03:00
21a490da8f [#1112] Fix issue from gofumpt
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-07 14:47:21 +03:00
93c0ccad4f [#1077] objectsvc: Fix possible panic in GetRange()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-05-07 14:47:21 +03:00
00b2b77b26 [#1112] node: Implement Range\RangeHash requests for EC object
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-05-07 14:47:21 +03:00