TrueCloudLab/frostfs-node
21
1
Fork 27
Code Issues 108 Pull requests 3 Projects Releases 1 Activity Actions
4970 commits 9 branches 94 tags 50 MiB
Commit graph

4970 commits

This branch
This branch
All branches
Author SHA1 Message Date
Leonard Lyubich
0d5495e997 [#70] object manager: Implement an example object garbage collector 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Leonard Lyubich
798fca9354 [#70] core/object: Process a delete group at tombstone 
Send object group to delete queue processor after tombstone content
validation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Leonard Lyubich
276ed6c04b [#70] cmd/neofs-node: Implement Del method on in-memory bucket 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Leonard Lyubich
9a604a50b9 [#70] cmd/neofs-node: Start asynchronous workers on app launch 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Leonard Lyubich
2b16edebc9 [#70] object/put: Fix NPE caused by nil FormatValidator 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Alex Vanin
6fe6005e71 [#66] Check basic ACL in object service pipeline 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 19:47:49 +03:00
Alex Vanin
801999c577 [#66] Impersonate object service verb from session token 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 19:47:49 +03:00
Alex Vanin
afeebd310c [#66] Use session token of object header at put ACL check 
Owner of the request is stored in session token most of the times.
Put request contains session token in the object body, so we have
to fetch it from there.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 19:47:49 +03:00
Alex Vanin
2ee24998ba [#68] Add go pprof to neofs-storage 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 19:47:08 +03:00
Leonard Lyubich
69a69cdbee [#67] object/eacl: Implement eACL validator 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 19:46:45 +03:00
Leonard Lyubich
44fcd2f212 [#64] object/delete: Change the formation of tombstone 
Make delete service to write list of child object addresses to tombstone
payload.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 19:46:27 +03:00
Leonard Lyubich
33ca88f85f [#64] core/object: Claim TombstoneContent in tombstone payload 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 19:46:27 +03:00
Leonard Lyubich
21bdc82fb5 [#64] core/object: Implement tombstone content messages 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 19:46:27 +03:00
Alex Vanin
6ee908c2db [#65] Support FS bucket and BoltDB bucket 
These buckets can be used to store blobs and metadata.
They will be removed as enhanced blob storage will be
implemented for neofs-node. To setup storage type
use `storage.object.type` and `storage.meta.type` params.

Available options:
  - inmemory (default)
  - boltdb
  - filesystem

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 14:48:00 +03:00
Alex Vanin
861bac3892 [#59] Use max msg size in transport server and splitter 
For GRPC it is about 4 MiB.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
d2009c8731 [#59] Add grpc payload splitter in object service chain 
GRPC has default message limit of 4MiB. Since every transmitted
neofs message has to be signed, then original message should
be split into transfer fit structures before signature service.

This commit introduce transport payload splitter for object
service pipeline. This splitter works with stream response
for methods:

  - object.Get
  - object.Range
  - object.Search

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
64691e6248 [#62] object/transformer: Fix incorrect relation init stage 
In previous implementation InitRelations call in payload size limiter was
called in write chunk method. This provoked clearing the split header in
children starting from the second.

Replace InitRelations call to the 1st child allocating stage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
13080aa7a1 [#43] cmd/neofs-node: Fetch max object size from config 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
e158497560 [#43] cmd/neofs-node: Support hostnames with dns, ipv4 and ipv6 addresses 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
f930993e3a [#43] pkg/network: Do not panic at multiaddr to net.Addr conversion 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
8585e147d0 [#43] cmd/neofs-node: Use morph wrappers as data sources 
Morph wrappers are being uses as sources for containers and
snapshots of network map.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
1914b4dbad [#43] cmd/neofs-node: Use object service in application 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
ea511d559c [#43] cmd/neofs-node: Refactor mocked object service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
a5ae128f34 [#43] cmd/neofs-node: Implement mocked max object size source 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
8dced272d2 [#43] cmd/neofs-node: Implement in-memory bucket 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
480362b02f [#43] cmd/neofs-node: Make cfg provide network address source interface 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
caedef82af [#43] cmd/neofs-node: Change object executor mocked calls 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
6824a6f67b [#61] object/search: Support non-root and non-leaf filters 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
51e373c3f0 [#61] object/search: Support latest search filters 
Refactor query to match object and its parents in a single call. Support
KeyRoot and KeyLeaf filters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
f89c848e84 [#61] object/search: Filter objects by container ID from request 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
1654df4d97 [#61] Update to latest neofs-api-go changes 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
f251645def [#58] object/delete: Process linking object in Delete service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
16252ad09a [#58] object/search: Add object-with-children filter 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
624e8cd3cb [#58] object/search: Refactor RelationSearcher implementation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
4bcfed37ca [#58] object/head: Generalize RelationSearcher interface method 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
6eb353c804 [#58] object/put: Validate payload content after filling 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
2abb03dbd1 [#58] core/object: Decouple the payload content validation method 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
107f3097e4 [#58] core/object: Add content validation to FormatValidator 
Add content validation step to FormatValidator. Check tombstone payload
correctness.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
017afbf0e3 [#58] services/object: Implement Delete service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
b24adeae89 [#58] object/transformer: Inherit type of parent object 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
39ddb3a3f4 [#45] object/search: Fix double write of local result 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
08b9ae547a [#45] object/search: Add filtering parent objects 
In previous implementation object.Search services allowed to search only
physically stored objects. This limitation did not allow getting the ID of
the split object.

Extend search execution logic with parent object filtering. Parent objects
that passed filters are now included in the result

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
88459963fb [#57] services/object: Sign requests with session key 
Use key storage in object services in order to sign requests with private
session key within user session.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
68178b8d74 [#57] core/object: Fix validation of linking object 
Linking object carries only identifier of the parent object. Fix Validate
method of FormatValidator to not validate parent object of the linking
object.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
be322835af [#57] services/object: Implement private key storage 
Implement storage that provides access to local node key and session keys
through session token.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
2da323c4b9 [#57] services/object: Add session token to common parameters 
Add session token field to CommonPrm. Remove session token field from Put
parameters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
39c17253be [#57] services/object: Combine common service parameters 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
8cddbe58a6 [#56] object/transformer: Write session token to object body 
Add session token argument to object formatter constructor which is written
to the object. Pass session token from trusted object Put.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
a4b9560ef6 [#56] object/put: Validate object format in untrusted Put 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
ede033256d [#56] core/object: Implement format validator 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00