achuprov/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Pull requests Activity
4723 commits 45 branches 27 tags 32 MiB
Commit graph

3104 commits

Author SHA1 Message Date
Alexander Chuprov
d61f78ec1f [#1121] node: Change mode of shard components 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2024-06-04 16:30:04 +03:00
Alexander Chuprov
0b0d829346 [#1121] node: Refactor mods of shard 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2024-06-04 16:30:04 +03:00
Airat Arifullin
5aacb8fc86 [#1144] metabase: Save parent attributes for ec-chunks 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-31 19:55:32 +03:00
Airat Arifullin
f8e33f8e3a [#1144] metabase: Proprely choose root OID for EC-splitted objects 
* If EC-parent is a part of Split itself, then save to root bucket
  its parent;
* If EC-parent is not a part of Split itself, then save to root bucket
  OID of this EC-parent.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-31 19:53:32 +03:00
Airat Arifullin
f0edebea18 [#1144] metabase: Support ec parent filter for Search 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-31 19:53:32 +03:00
Airat Arifullin
0b367007fc [#1152] go.mod: Update api-go and sdk versions 
* Resolve conflicts for apemanager since api-go
  contains ape and apemanager packages and SDK only
  ape package.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-31 15:39:09 +03:00
Anton Nikiforov
92e19feb57 [#1147] node: Use public fields for shard.ExistsPrm 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-30 08:13:04 +00:00
Anton Nikiforov
c1af13b47e [#1147] node: Fix issue from gopls 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-30 08:13:04 +00:00
Anton Nikiforov
6130650bb6 [#1147] node: Implement Lock\Delete requests for EC object 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-30 08:13:04 +00:00
Anton Nikiforov
a82c8cc5b8 [#1147] gc: Execute callback for expired tombstones when they exists 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-30 08:13:04 +00:00
Anton Nikiforov
d355274cd0 [#1147] object: Use methods on pointer for searchsvc.execCtx 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-30 08:13:04 +00:00
Anton Nikiforov
3555c73225 [#1147] object: Use methods on pointer for deletesvc.execCtx 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-30 08:13:04 +00:00
Anton Nikiforov
e43e7bec3a [#1147] log: Remove redundant address field from log 
Filled when logger created for `request` object from package `getsvc`.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-30 08:13:04 +00:00
Anton Nikiforov
50923ed81c [#1147] Fix gofumpt issue 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-30 08:13:04 +00:00
Alex Vanin
4a34d0d40e [#1149] go.mod: Bump neo-go up to v0.106.0 
Required to work with neo-go v0.106.0 node
with default hardfork configuration. Without
neo-go client version bump, it throws error.

  failed to get network magic: unexpected hardfork: Cockatrice

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2024-05-30 08:11:58 +00:00
Airat Arifullin
3627b44e92 [#1142] tree: Fill APE-request with source IP property 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-27 10:17:17 +00:00
Airat Arifullin
482c5129ac [#1142] object: Fill APE-request with source IP property 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-27 10:17:17 +00:00
Airat Arifullin
43625e7536 [#1142] container: Fill APE-request property with source IP 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-27 10:17:17 +00:00
Airat Arifullin
542d3adcb2 [#1105] apemanager: Implement apemanager service 
* Introduce grpc server for apemanager service and
  its implementation in `pkg/services/apemanager`.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-27 09:34:21 +00:00
Airat Arifullin
51ade979e8 [#1105] ape: Introduce contract storage with proxy contract verification 
* `ProxyVerificationContractStorage` uses Proxy contract as a cosigner.
* `ProxyVerificationContractStorage` recreates a contract storage for each handler
  invocation because of an issue: rpc-actor from morph client may be expired. This
  way won't create a bottlenecks because it is expected that this contract storage
  implementation will be used not so often.
* Make morph client return `RPCActor` (that is websocket client in fact).
* Make `SwitchRPCGuardedActor` return `RPCActor` as it will be used for
  `ProxyVerificationContractStorage`.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-27 09:34:21 +00:00
Airat Arifullin
40b04c00ef [#1141] metabase: Fix IsUserObject method 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-20 13:24:15 +03:00
Airat Arifullin
89a80e9a0f [#1141] metabase: Fix putUniqueIndexItem 
* `GetECHeader` is not correct way to determine if an object's got
  EC-header: `ECHeader` must be used for that.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-20 13:24:08 +03:00
Airat Arifullin
8fd678e269 [#1141] go.mod: Update frostfs-sdk-go and frostfs-api-go/v2 
* Also fix unit-test.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-20 13:22:48 +03:00
Dmitrii Stepanov
436c9f5558 [#1129] policer: Restore EC object 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-17 14:36:18 +03:00
Dmitrii Stepanov
44f2e8f27f [#1129] putSvc: Allow to put single unprepared object to EC container 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-16 16:28:49 +03:00
Dmitrii Stepanov
0e42126ddc [#1129] object: Fix check owner for EC part 
Do not validate EC part owner if request from container node.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-16 16:28:49 +03:00
Dmitrii Stepanov
1cd8562db8 [#1129] policer: Refactor shortage 
Drop override inside method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-16 16:28:49 +03:00
Dmitrii Stepanov
4ab6c404f7 [#1129] policer: Drop unused 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-16 16:28:49 +03:00
Dmitrii Stepanov
cbe9757490 [#1129] policer: Pull required EC chunks 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-16 16:28:49 +03:00
Dmitrii Stepanov
d45d086acd [#1129] policer: Add EC chunk replication 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-16 16:28:48 +03:00
Alexander Chuprov
b078fe5ba1 [#1092] control: Move SignMessage to separate package 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2024-05-16 12:14:01 +03:00
Evgenii Stratonikov
f3e09cb09b [#1135] sdnotify: Send MONOTONIC_USEC on reload 
Fixes #1135

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-15 12:52:52 +00:00
Anton Nikiforov
5c582e96fd [#1136] metabase: Fix creation of ECInfoError 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-15 11:04:27 +00:00
Evgenii Stratonikov
b3eaa8a9bc [#1083] objsvc/v2: Check response status in RANGE_HASH forwarder 
Fixes #1083

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-15 12:04:21 +03:00
Evgenii Stratonikov
0924b62a95 [#1083] objsvc/v2: Unify response verification after forwarding 
1. Use the same routine for HEAD/GET_RANGE methods.
2. Make error message similar.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-15 12:04:06 +03:00
Evgenii Stratonikov
300654b045 [#1083] objsvc/v2: Properly check response status after forwarding 
Previously we had cryptic error:
```
debug   get/remote.go:38        remote call failed      {"component": "Object.Get service", "request": "HEAD", "address": "9sTxoVrhJ7WBtXQfK2NJ7zDV5yCF7BPLKK1XTxYPdGsP/BbHV4KZZ8y2BPqAT5kyjdHRLkfbtY2xf5uYoMVqxACn1", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "unexpected header type <nil>"}
```
Now we have and expected error:
```
debug   get/remote.go:38        remote call failed      {"component": "Object.Get service", "request": "HEAD", "address": "D2rqaMG4D2VHdv3HKky8UYSYmwQFH2v9oXXqtyRZPTMy/BbHV4KZZ8y2BPqAT5kyjdHRLkfbtY2xf5uYoMVqxACn1", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-15 12:04:06 +03:00
Evgenii Stratonikov
6e71ae3bda [#1130] fstree: Remove useless Stat() call 
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                                           │     old     │                new                 │
                                           │   sec/op    │   sec/op     vs base               │
SubstorageReadPerf/fstree_nosync-seq100-8    2.689µ ± 2%   2.428µ ± 4%  -9.72% (p=0.000 n=10)
SubstorageReadPerf/fstree_nosync-rand100-8   2.727µ ± 1%   2.497µ ± 2%  -8.42% (p=0.000 n=10)
geomean                                      2.708µ        2.462µ       -9.07%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-14 16:05:45 +03:00
Evgenii Stratonikov
bf9bdde8ea [#1128] util/test: Remove unused package 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-14 12:48:13 +00:00
Airat Arifullin
952d13cd2b [#1124] cli: Improve APE rule parsing 
* Make APE rule parser to read condition's kind in unambiguous using lexemes
`ResourceCondition`, `RequestCondition` instead confusing `Object.Request`, `Object.Resource`.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-14 12:23:26 +03:00
aarifullin
20baf6e112 [#1108] ape: Update policy-engine version for listing by iteration 
* Update go.mod with a new version of policy-engine pacakge.
* Adapt SwitchRPCGuardedActor to ContractStorage interface.
* Fix `frostfs-adm` util.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-14 12:17:56 +03:00
Dmitrii Stepanov
0144117cc9 [#1125] objectSvc: Add EC header APE check 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-08 16:25:55 +03:00
Dmitrii Stepanov
ada1b9f737 [#1120] objectSvc: Fix EC put placement 
Use parent object ID to compute placement.
Fix too many copies saving.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-08 15:23:57 +03:00
Anton Nikiforov
fe2c1c926f [#1112] node: Fix race warning for GetObjectAndWritePayload 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-07 14:47:21 +03:00
Anton Nikiforov
3e782527b8 [#1112] node: Add test for Range request for EC object 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-07 14:47:21 +03:00
Anton Nikiforov
21a490da8f [#1112] Fix issue from gofumpt 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-07 14:47:21 +03:00
Evgenii Stratonikov
93c0ccad4f [#1077] objectsvc: Fix possible panic in GetRange() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-07 14:47:21 +03:00
Anton Nikiforov
00b2b77b26 [#1112] node: Implement Range\RangeHash requests for EC object 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-07 14:47:21 +03:00
aarifullin
b60a51b862 [#1117] ape: Introduce FormFrostfsIDRequestProperties method 
* `FormFrostfsIDRequestProperties` gets user claim tags and group id and sets them
  as ape request properties.
* Make tree, container and object service use the method.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-07 10:01:21 +00:00
aarifullin
6c76c9b457 [#1117] core: Introduce SubjectProvider interface for FrostfsID 
* Make tree, object and container services use SubjectProvider interface.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-07 10:01:21 +00:00
aarifullin
45f4e6939d [#1117] morph: Make frostfsid client provide GetSubjectExtended method 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-07 10:01:21 +00:00