fyrchik/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
3840 commits 17 branches 26 tags 22 MiB
Commit graph

462 commits

Author SHA1 Message Date
Alex Vanin
d2009c8731 [#59] Add grpc payload splitter in object service chain 
GRPC has default message limit of 4MiB. Since every transmitted
neofs message has to be signed, then original message should
be split into transfer fit structures before signature service.

This commit introduce transport payload splitter for object
service pipeline. This splitter works with stream response
for methods:

  - object.Get
  - object.Range
  - object.Search

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
e158497560 [#43] cmd/neofs-node: Support hostnames with dns, ipv4 and ipv6 addresses 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
f930993e3a [#43] pkg/network: Do not panic at multiaddr to net.Addr conversion 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
6824a6f67b [#61] object/search: Support non-root and non-leaf filters 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
51e373c3f0 [#61] object/search: Support latest search filters 
Refactor query to match object and its parents in a single call. Support
KeyRoot and KeyLeaf filters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
f89c848e84 [#61] object/search: Filter objects by container ID from request 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
1654df4d97 [#61] Update to latest neofs-api-go changes 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
f251645def [#58] object/delete: Process linking object in Delete service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
16252ad09a [#58] object/search: Add object-with-children filter 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
624e8cd3cb [#58] object/search: Refactor RelationSearcher implementation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
4bcfed37ca [#58] object/head: Generalize RelationSearcher interface method 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
6eb353c804 [#58] object/put: Validate payload content after filling 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
017afbf0e3 [#58] services/object: Implement Delete service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
39ddb3a3f4 [#45] object/search: Fix double write of local result 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
08b9ae547a [#45] object/search: Add filtering parent objects 
In previous implementation object.Search services allowed to search only
physically stored objects. This limitation did not allow getting the ID of
the split object.

Extend search execution logic with parent object filtering. Parent objects
that passed filters are now included in the result

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
88459963fb [#57] services/object: Sign requests with session key 
Use key storage in object services in order to sign requests with private
session key within user session.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
be322835af [#57] services/object: Implement private key storage 
Implement storage that provides access to local node key and session keys
through session token.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
2da323c4b9 [#57] services/object: Add session token to common parameters 
Add session token field to CommonPrm. Remove session token field from Put
parameters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
39c17253be [#57] services/object: Combine common service parameters 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
8cddbe58a6 [#56] object/transformer: Write session token to object body 
Add session token argument to object formatter constructor which is written
to the object. Pass session token from trusted object Put.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
a4b9560ef6 [#56] object/put: Validate object format in untrusted Put 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
fc74e9b40c [#32] Remove recover from basic ACL checks 
Basic ACL checker gets request field via getters that are
NPE-free, therefore we don't need to worry about function
invocations on nil structures.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
4a8de3263d [#32] Use less v2 specific structures in basic ACL checker 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
c5a44e0a05 [#32] Add tests for basic ACL helper 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
91fef72bb6 [#32] Make basic ACL check in all object request 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
49ee9a14a1 [#32] Add basic ACL helper 
Basic ACL helper provides functions for simple access to
bit fields of basic ACL.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
f6904db84f [#32] Use pkg/core interfaces to fetch container and netmap 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
ad36a2cd8f [#32] Use classifier in basic ACL check 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
5045b0c3d4 [#32] Add request sender classifier 
ACL has to classify request senders by roles:
- owner of the container,
- request from container or inner ring node,
- any other request.

According to this roles ACL checker use different
bits of basic ACL to grant or deny access.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
ab565b1862 [#32] Add basis of basic ACL check service 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
867f1d772d [#53] Revert "[#51] object/put: Fix incorrect splited object streaming" 
Revert commit 803ea345 that which spawned a duplication of the payload of
objects.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
0a51263e72 [#53] services/object: Implement GetRangeHash service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
834a8597c5 [#53] object/util: Add seek range function to range traverser 
Add SetSeekRange method to RangeTraverser that switches traverser to work
with provided object payload range.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
8791c4f0a5 [#53] object/range: Share object range traverser between services 
Replace object range chain utilities to object/util package in order to
reuse it in other services.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
759605410a [#53] object/range: Do not add chain element without the need 
In previous implementation one excess element could be added to the chain.
Add previous sibling presence check to prevent this.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
bf2c33d7a9 [#52] services/object: Implement Get service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
3880315a3f [#52] object/range: Add full range option to parameters 
Add FullRange option to get range operation parameters that allows to get
payload range [0:object_size] w/o the actual knowledge of the object size.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
abf9ad3573 [#52] object/range: Attach received object header to result 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
0490107165 [#50] services/object: Implement GetRange service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
dd16f568c3 [#50] object/head: Add right child to Head operation result 
Head service receives right child of the processing object in some cases.
Add right child to Head result in order to use it as needed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
3692f708ca [#51] object/put: Fix incorrect splited object streaming 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
6f4b477195 [#44] services/object: Refactor for minor optimization 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
3692032e30 [#44] object/head: Fix incorrect placement traversal 
Add object identifier option to placement traverser of object.Head service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
d6a9c06c25 [#39] service/object: Complicate Head service logic 
Add a header recovery step through finding and getting the header of the
rightmost child.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
753a6a2de5 [#39] service/object: Implement right child searcher on Search service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
39c324bd6d [#39] service/object: Add query filters to search right child 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
05f3963975 [#38] service/object: Implement simplified object Head service 
Implement Head service w/o linking object processing and restoration from
split-chain.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
09084a7bff [#34] service/object: Implement object Search distributed service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
a5ebdd1891 [#34] services/object: Replace local placement to util 
Replace local placement tool in order to reuse it in different object
services.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
57f8d3745d [#33] service/object: Implement object Put distributed service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
b7702349dc [#13] services/util: Support client-side stream in SignService 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
d54a5d4f66 [#13] services/util: Rename UnarySignService to SingService 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
75162d1f8d [#13] services/util: Support server-side stream in UnarySignService 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
6b4fb3a0aa [#13] services: Refactor UnarySignService 
Replace UnaryHandler from structure to method arguments in order to reuse
single instance for different service methods.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
0c83e84613 [#13] services/object: Implement GetRangeHash handlers 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
7a6b62c552 [#13] services/object: Implement Delete handlers 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
881f49814a [#13] services/object: Implement Head handlers 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
652f64ae85 [#13] services/object: Implement GetRange handlers 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
62d0c50b15 [#13] services/object: Implement Put handlers 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
f14d1fd65f [#13] services/object: Implement Get handlers 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
486aeef5df [#13] services/object: Define and implement Object executor 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
276ec2e1e0 [#13] services: Define signing Object service and implement Search 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00