aarifullin/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
4750 commits 137 branches 26 tags 62 MiB
Commit graph

4750 commits

This branch
This branch
All branches
Author SHA1 Message Date
Airat Arifullin
43625e7536 [#1142] container: Fill APE-request property with source IP 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-27 10:17:17 +00:00
Airat Arifullin
2b02f52cd9 [#1105] cli: Add apemanager commands 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-27 09:34:21 +00:00
Airat Arifullin
542d3adcb2 [#1105] apemanager: Implement apemanager service 
* Introduce grpc server for apemanager service and
  its implementation in `pkg/services/apemanager`.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-27 09:34:21 +00:00
Airat Arifullin
51ade979e8 [#1105] ape: Introduce contract storage with proxy contract verification 
* `ProxyVerificationContractStorage` uses Proxy contract as a cosigner.
* `ProxyVerificationContractStorage` recreates a contract storage for each handler
  invocation because of an issue: rpc-actor from morph client may be expired. This
  way won't create a bottlenecks because it is expected that this contract storage
  implementation will be used not so often.
* Make morph client return `RPCActor` (that is websocket client in fact).
* Make `SwitchRPCGuardedActor` return `RPCActor` as it will be used for
  `ProxyVerificationContractStorage`.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-27 09:34:21 +00:00
Airat Arifullin
2697d4d1fe [#1143] node: Fix frostfsid cache 
* Fix `subjectFromSubjectExtended` converter.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-20 18:49:52 +03:00
Airat Arifullin
40b04c00ef [#1141] metabase: Fix IsUserObject method 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-20 13:24:15 +03:00
Airat Arifullin
89a80e9a0f [#1141] metabase: Fix putUniqueIndexItem 
* `GetECHeader` is not correct way to determine if an object's got
  EC-header: `ECHeader` must be used for that.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-20 13:24:08 +03:00
Airat Arifullin
8fd678e269 [#1141] go.mod: Update frostfs-sdk-go and frostfs-api-go/v2 
* Also fix unit-test.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-20 13:22:48 +03:00
Dmitrii Stepanov
436c9f5558 [#1129] policer: Restore EC object 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-17 14:36:18 +03:00
Dmitrii Stepanov
44f2e8f27f [#1129] putSvc: Allow to put single unprepared object to EC container 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-16 16:28:49 +03:00
Dmitrii Stepanov
0e42126ddc [#1129] object: Fix check owner for EC part 
Do not validate EC part owner if request from container node.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-16 16:28:49 +03:00
Dmitrii Stepanov
1cd8562db8 [#1129] policer: Refactor shortage 
Drop override inside method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-16 16:28:49 +03:00
Dmitrii Stepanov
4ab6c404f7 [#1129] policer: Drop unused 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-16 16:28:49 +03:00
Dmitrii Stepanov
cbe9757490 [#1129] policer: Pull required EC chunks 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-16 16:28:49 +03:00
Dmitrii Stepanov
d45d086acd [#1129] policer: Add EC chunk replication 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-16 16:28:48 +03:00
Evgenii Stratonikov
af57d5a6a1 [#1138] adm/test: Use --size=1 for negative tests 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-16 10:40:59 +00:00
Evgenii Stratonikov
2f04ce2f79 [#1138] adm: Parallelize generate-alphabet command 
This is the longest test in our suite, try to help CI a bit.

Before:
```
$ go test -run=TestGenerateAlphabet -count=10 .
ok      git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/generate      45.400s
```

After:
```
$ go test -run=TestGenerateAlphabet -count=10 .
ok      git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/generate      33.267s
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-16 10:40:59 +00:00
Alexander Chuprov
b078fe5ba1 [#1092] control: Move SignMessage to separate package 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2024-05-16 12:14:01 +03:00
Evgenii Stratonikov
f3e09cb09b [#1135] sdnotify: Send MONOTONIC_USEC on reload 
Fixes #1135

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-15 12:52:52 +00:00
Anton Nikiforov
5c582e96fd [#1136] metabase: Fix creation of ECInfoError 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-15 11:04:27 +00:00
Evgenii Stratonikov
b3eaa8a9bc [#1083] objsvc/v2: Check response status in RANGE_HASH forwarder 
Fixes #1083

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-15 12:04:21 +03:00
Evgenii Stratonikov
0924b62a95 [#1083] objsvc/v2: Unify response verification after forwarding 
1. Use the same routine for HEAD/GET_RANGE methods.
2. Make error message similar.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-15 12:04:06 +03:00
Evgenii Stratonikov
300654b045 [#1083] objsvc/v2: Properly check response status after forwarding 
Previously we had cryptic error:
```
debug   get/remote.go:38        remote call failed      {"component": "Object.Get service", "request": "HEAD", "address": "9sTxoVrhJ7WBtXQfK2NJ7zDV5yCF7BPLKK1XTxYPdGsP/BbHV4KZZ8y2BPqAT5kyjdHRLkfbtY2xf5uYoMVqxACn1", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "unexpected header type <nil>"}
```
Now we have and expected error:
```
debug   get/remote.go:38        remote call failed      {"component": "Object.Get service", "request": "HEAD", "address": "D2rqaMG4D2VHdv3HKky8UYSYmwQFH2v9oXXqtyRZPTMy/BbHV4KZZ8y2BPqAT5kyjdHRLkfbtY2xf5uYoMVqxACn1", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-15 12:04:06 +03:00
Evgenii Stratonikov
b4cfc80579 [#1133] node: Improve tests for shards.default config section 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-15 09:48:21 +03:00
Evgenii Stratonikov
d0f64c23a5 Release v0.39.0 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-14 17:02:00 +03:00
Evgenii Stratonikov
6e71ae3bda [#1130] fstree: Remove useless Stat() call 
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                                           │     old     │                new                 │
                                           │   sec/op    │   sec/op     vs base               │
SubstorageReadPerf/fstree_nosync-seq100-8    2.689µ ± 2%   2.428µ ± 4%  -9.72% (p=0.000 n=10)
SubstorageReadPerf/fstree_nosync-rand100-8   2.727µ ± 1%   2.497µ ± 2%  -8.42% (p=0.000 n=10)
geomean                                      2.708µ        2.462µ       -9.07%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-14 16:05:45 +03:00
Evgenii Stratonikov
bf9bdde8ea [#1128] util/test: Remove unused package 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-14 12:48:13 +00:00
Airat Arifullin
952d13cd2b [#1124] cli: Improve APE rule parsing 
* Make APE rule parser to read condition's kind in unambiguous using lexemes
`ResourceCondition`, `RequestCondition` instead confusing `Object.Request`, `Object.Resource`.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-14 12:23:26 +03:00
aarifullin
20baf6e112 [#1108] ape: Update policy-engine version for listing by iteration 
* Update go.mod with a new version of policy-engine pacakge.
* Adapt SwitchRPCGuardedActor to ContractStorage interface.
* Fix `frostfs-adm` util.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-14 12:17:56 +03:00
Anoke
74135776c7 [#1067] adm: Fix panic on negative value 
Signed-off-by: Anoke <rustamgta1011@gmail.com>
 2024-05-13 16:44:04 +00:00
Dmitrii Stepanov
0144117cc9 [#1125] objectSvc: Add EC header APE check 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-08 16:25:55 +03:00
Dmitrii Stepanov
368218f0cc [#1120] cli: Edit object nodes output 
Print detailed information only.
Allow to output to JSON.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-08 15:23:58 +03:00
Dmitrii Stepanov
a45b548a6f [#1120] cli: Add explain to object nodes 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-08 15:23:58 +03:00
Dmitrii Stepanov
654384990c [#1120] cli: Fix object nodes for linking objects 
Do not use linking objects to get placement for complex object.
Linking objects should be stored on all container nodes, also they are not required.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-08 15:23:57 +03:00
Dmitrii Stepanov
ada1b9f737 [#1120] objectSvc: Fix EC put placement 
Use parent object ID to compute placement.
Fix too many copies saving.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-08 15:23:57 +03:00
Dmitrii Stepanov
5c730de96e [#1120] cli: Add EC support to object nodes command 
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
 2024-05-08 15:23:57 +03:00
Alexander Chuprov
854200a874 [#1115] node: Remove unused const 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2024-05-07 19:11:21 +03:00
Anton Nikiforov
fe2c1c926f [#1112] node: Fix race warning for GetObjectAndWritePayload 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-07 14:47:21 +03:00
Anton Nikiforov
3e782527b8 [#1112] node: Add test for Range request for EC object 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-07 14:47:21 +03:00
Anton Nikiforov
21a490da8f [#1112] Fix issue from gofumpt 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-07 14:47:21 +03:00
Evgenii Stratonikov
93c0ccad4f [#1077] objectsvc: Fix possible panic in GetRange() 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-07 14:47:21 +03:00
Anton Nikiforov
00b2b77b26 [#1112] node: Implement Range\RangeHash requests for EC object 
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
 2024-05-07 14:47:21 +03:00
Evgenii Stratonikov
a23d53b2d4 [#1118] .forgejo: Add pre-commit action 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-07 11:36:11 +00:00
Evgenii Stratonikov
fc7b07f314 [#1118] docs: Fix pre-commit warnings 
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
 2024-05-07 11:36:11 +00:00
Airat Arifullin
9cc51f86b7 [#1117] node: Introduce cache for frostfsid contract client 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-07 10:01:21 +00:00
aarifullin
b60a51b862 [#1117] ape: Introduce FormFrostfsIDRequestProperties method 
* `FormFrostfsIDRequestProperties` gets user claim tags and group id and sets them
  as ape request properties.
* Make tree, container and object service use the method.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-07 10:01:21 +00:00
aarifullin
6c76c9b457 [#1117] core: Introduce SubjectProvider interface for FrostfsID 
* Make tree, object and container services use SubjectProvider interface.
* Fix unit-tests.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-07 10:01:21 +00:00
aarifullin
45f4e6939d [#1117] morph: Make frostfsid client provide GetSubjectExtended method 
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
 2024-05-07 10:01:21 +00:00
Ekaterina Lebedeva
e07869a8cf [#1100] Remove unused fields 
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
 2024-05-06 10:14:36 +03:00
Alexander Chuprov
ec2873caa7 [#1116] node: Fix writecache metrics 
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
 2024-05-02 20:03:33 +03:00