Commit graph

604 commits

Author SHA1 Message Date
Leonard Lyubich
52b0c1f19a [#216] blobovnicza: Add configuration values
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
bce16de72a [#216] blobstor: Classify objects by size according to binary format
In previous implementation objects were classified by size according to
payload size. From now they are classified by the size of their binary
representation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
40b09f9266 [#216] blobovnicza: Define Blobovnicza type
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
b9054e2ee0 [#211] blobstor: Add a branch for processing objects by size in Put
Implement "big or small" property classifier (only the size of the payload
is temporarily considered). Save "big" objects in shallow dir. Save "small"
objects in shallow dir until the moment of implementation of blobovnicza.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
cfc770c3fe [#211] blobstor: Add SmallSizeLimit parameter
Add numeric parameter of maximum size of the "small" objects.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
3028718691 [#211] blobovnicza: Refactor ID implementation
Replace UUID implementation of ID with byte slice.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
53b114cf8b [#211] blobstor: Remove global lock
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
59f7cf9873 [#211] blobstor: Implement Exists method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1ba556f5e7 [#211] blobstor: Define Iterate operation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
3c0ef270f4 [#211] blobstor: Define Exists operation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
915243c69e [#211] blobstor: Define DeleteSmall operation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
65da2025e5 [#211] blobstor: Rename Delete to DeleteBig
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
b605ea3678 [#211] blobstor: Refactor Put operation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
7afcde659d [#211] blobstor: Define GetRangeSmall operation interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
0b130150ae [#211] blobstor: Rename GetRange to GetRangeBig
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2ba4ebc8c9 [#211] blobstor: Implement no-op GetSmall operation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2f61d8d63c [#211] localstorage: Define blobovnicza ID
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
6813f40665 [#211] blobstor: Refactor GetBig parameters and result
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
fb6857a1cb [#211] blobstor: Rename Get operation to GetBig
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4e8cc51e7b [#189] shard: Extend Info with WeightValues
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4f16a10235 [#189] localstorage: Implement DumpInfo method
Implement method to get the information about the local storage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
8fbda2a588 [#189] shard: Implement DumpInfo method
Implement method to get the information about the shard.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
24cf86e269 [#189] metabase: Implement DumpInfo method
Implement method to get the information about the metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
eaa7068e3c [#189] blobstor: Implement DumpInfo method
Implement method to get the information about the BLOB storage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
dae8d3de5b [#189] shard: Fix missing return statement in SetMode method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
cba87a1300 [#188] localstorage: Implement SetShardMode method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
3fa3661cad [#188] shard: Implement SetMode method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2fb379b7dd [#188] shard: Define the enumeration of shard modes
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
0b515837ca [#186] localstorage: Fix address forwarding in Delete method
In previous implementation storage engine not forwarded object address to
shards.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
953387a1e5 [#186] object/acl: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
046206f670 [#186] object/head: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2be8f154a0 [#186] object/put: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
14442a0801 [#186] object/range: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
9b32f131c0 [#186] object/rangehash: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1bce2a4e55 [#186] object/search: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
aa9eb2eaf2 [#186] policer: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1b867e53d7 [#186] replicator: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
9a4cd01eb8 [#186] localstorage: Implement utility functions over StorageEngine
Implement Put/Get/GetRange/Select/SelectAll functions over storage
engine. These functions are going to be used by Object service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
70a65c5aff [#176] blobstor: Use 1 byte names of sub-directories in FS tree
Using 2 bytes length for directories in fs tree lead to large growth
in the number of directories.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2d917a024f [#176] blobstor: Fix missing mutex in constructor
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
b1718a60e4 [#176] shard: Fix BlobStor access when data range is zero
In previous implementation Shard accessed the BlobStor to get the
object header. However, the shard must take headers from the metabase.
From now zero length of the requested payload range seens as object
header request. In this case shard calls metabase to get the header.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
796bba3793 [#176] shard: Finalize the implementation of Exists method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4518357ad7 [#176] localstore: Fix linter's remarks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
f194c840d7 [#176] blobstor: Handle error of zstd (de)compressor creation
In previous implementation WithCompressObjects returned Option
than could panic if zstd (de)compressor creation failed with error.
From now errors with (de)compressor creation result in an option
without using data compression. In this case, the error is written
to the log passed to the option constructor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
6f8c45d61b [#176] localstore: Change multiple access sync
In previous implementation each operation on local storage
locked engine mutex. This was done under the assumption that
the weights of the shards change as a result of write operations.
With the transition to static weights of shards, it is no longer
necessary to lock the global mutex during the execution of operations.
However, since the set of engine shards is dynamic, there is still a
need to control multiple access to this set. The same mutex is used
for synchronization.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
df558cbe6b [#176] shard: Get rid of using the global RW lock
In previous implementation each shard operation locked
RW shard mutex. With this approach RW operations were executed
one-by-one and blocked the execution of RO operations.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
9ab371aafb [#176] shard: Make ID stringer to return base58 encoded string
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
86385af67d [#176] blobstor: Fix maximum tree depth
Correct the calculation of maximum value of fs tree depth. Fix check
of the max depth overflow in WithShallowDepth function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
f7116f27ec [#176] localstore/shard: Change signature of options
Make WithBlobStorOptions/WithMetaBaseOptions functions to accept
variadic parameter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
73494c4bbc [#176] localstore: Fix Get with non-zero length
GetPrm has WithPayloadRange option to specify the requested
payload range. In previous implementation StorageEngine.Get
method ignored this option. From now zero length matches
full payload request.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
46e455dcae [#176] localstore: Implement shard methods via Metabase and BlobStor
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
b127607ac6 [#176] localstore: Implement primary BlobStor
Implement primary local BLOB storage based on filesystem tree.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
09750484f9 [#176] localstore: Draft storage engine structure and ops
Implement the primary structure and operation of the local object
storage engine.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
383c483be7 [#176] localstore: Draft storage engine shard structure and ops
Implement the primary structure and operation of the local object
storage shard.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
1486cc0e29 [#170] morph/client: Print reversed hash in logs
JSON-RPC method takes reversed hash as an input so it is
better to output reveresed hash in logs.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
e2547f5bf8 [#170] Invoke all alphabet contracts for voting
Now alphabet contracts use signature collection to make
a vote. All inner ring nodes votes for some list of validators
the same way as they vote fore new epoch or new container.
As soon as list is accepted by alphabet contracts, each of them
votes for one candidate from the list.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
49666f87f1 [#170] Implement --vote argument in neofs-ir
With this argument neofs-ir will be run as an CLI
application that initialize inner ring server, invokes
`vote` method on corresponding alphabet contract and
exits.

User can provide a comma-separated list of validator
public keys.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
0b65c64e98 [#170] Vote for validators on inner ring startup
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
2240cde826 [#170] Implement validator voting function in inner ring
This function calculates validator key based on inner ring index
and invokes `Vote` method of corresponding alphabet contract.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
d02c24db5c [#170] Add alphabet vote invocation method
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Leonard Lyubich
6cddc91cbf [#227] transformer: Inherit parent header in linking object
In previous implementation linking object inherited only ID of the parent
object. From now it inherits full header like last child.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-02 14:53:04 +03:00
Leonard Lyubich
b8e8e1e80d [#227] transformer: Do not inherit attribute in generated objects
In previous implementation child objects inherited parent attributes after
split-transformation, which was redundant. From now attributes are not
inherited.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-02 14:53:04 +03:00
Leonard Lyubich
5470d94416 [#223] placement: Fix local and single-success placement traversal
In previous implementation placement traverser processed incorrectly with
local placement build. Also entity incorrectly traversed the placement
vectors for fixed number read operations until success. The erroneous
behavior was due to the use of a vector number of successes instead of
a scalar number in these scenarios.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-01 19:36:46 +03:00
Leonard Lyubich
d94a0eb25e [#221] eacl: Get rid of deprecated methods usage
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-01 11:08:47 +03:00
Alex Vanin
f96630581a [#208] Set unique splitID in children
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-30 10:44:15 +03:00
Alex Vanin
6280d075b9 [#208] Remove childfree search attribute
With updated specification of object related operation
we don't have this search attribute any more and we
should not use functions related to this attribute.

This commit breaks object service logic, however it will
be fixed later.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-30 10:44:15 +03:00
Leonard Lyubich
706bdf736e [#209] eacl: Support object ID filter in eACL mechanism
Add object ID header to the list of processing object headers in eACL
validation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-25 11:59:15 +03:00
Leonard Lyubich
fddc50fd85 [#203] Replace ErrEACLNotFound to core library
ErrEACLNotFound error was defined in implementation package. EACL validator
checked this error after the call of eACL storage interface method. Replace
ErrEACLNotFound to core container library. in order to: on the one hand not
use an implementation error, on the other hand, to be able to reuse a
generic type error (404).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 20:19:20 +03:00
Leonard Lyubich
9148980bd0 [#193] services/object: Support client options in all Object services
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 15:34:30 +03:00
Leonard Lyubich
27fc4d6e01 [#193] network/cache: Support client options in ClientCache.Get
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 15:34:30 +03:00
Alex Vanin
cf1ea983e5 [#203] Do not fail eACL check on EACLNotFound error
Now morph library returns error if there is not eACL in
sidechain storage. However in this case eACL check should
be passed since it is the same as having empty eACL table.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-24 15:32:26 +03:00
Alex Vanin
8cedb93cea [#203] Make eACL not found error public
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-24 15:32:26 +03:00
Leonard Lyubich
1f6ad5ccd8 [#201] morph/eacl: Return error if container eACL is not set
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:56:03 +03:00
Leonard Lyubich
46dab77705 [#195] services/object: Write debug log messages on worker pool errors
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
fa6e4a3ca4 [#195] services/object: Write debug log messages on internal service errors
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
6a5c37d592 [#195] object/search: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
cb46e4b154 [#195] object/rangehash: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
af6484e3b1 [#195] object/range: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
cf2dc37a42 [#195] object/put: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
9fbfc0b5e4 [#195] object/head: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
e1e5a590e9 [#195] object/delete: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
ffeea2c003 [#177] metabase: Add index by object ID
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-23 09:54:59 +03:00
Alex Vanin
dbf6c9efef [#190] Use request sender owner in sticky bit check
Sticky bit checks if object owner and request owner are the
same. Container owner should not used in this check.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 19:07:16 +03:00
Alex Vanin
2148e282ec [#190] Rename owner to cnrOwner in object request info
`owner` field may be misused as request sender owner, however
it is a owner of a container for that request. New naming
should be clear.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 19:07:16 +03:00
Alex Vanin
f0537b35c1 [#190] Add isOwnerFromKey helper function in ACL
This function takes public key and returns true if
owner id was produced by this key.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 19:07:16 +03:00
Alex Vanin
a14bb6292b [#182] Reuse search filter in policer
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 17:59:46 +03:00
Alex Vanin
2e605b2435 [#182] Limit policer object filter to physical stored objects
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 17:59:46 +03:00
Alex Vanin
cf85fa9fab [#180] Return isInnerRing flag in request classifier
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 15:17:18 +03:00
Alex Vanin
6f841e319d [#180] Make separate basic ACL check for inner ring requests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 15:17:18 +03:00
Alex Vanin
e8fe07edd0 [#184] Use SDK client cache in object.Rangehash
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
f85e88c4f8 [#184] Use SDK client cache in object.Range
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
d485a5967d [#184] Use SDK client cache in object.Search
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
7ba95dd5fc [#184] Use SDK client cache in object.Put
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
e9a6365333 [#184] Use SDK client cache in object.Head
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
fb90827532 [#184] Add SDK client cache package
With this package we can reuse already created connections.
Later on neofs-api-go will support checking connection health
and `Close` operation, so this cache could run worker and remove
unhealthy clients.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Leonard Lyubich
1caf15463e [#174] Update to neofs-api-go v1.20.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Leonard Lyubich
58fcb35fb0 [#174] Use Marshal(JSON)/Unmarshal(JSON) methods for encoding/decoding
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Leonard Lyubich
3de8febe57 [#174] Update to latest neofs-api-go changes
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Alex Vanin
ea41dec23f [#160] Store root and phy indexes as fake bucket tree
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
f1a3acd27a [#160] Fix metabase test for virtual object
Parent should not have split header.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
32219bb294 [#160] Remove query match function
This function duplicates query processing that
is done in meta-storage now.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
03fed8ca59 [#160] Update metabase with new root and phy flags processing
Now root and phy (leaf) filters work like flags. They work with
any matcher and any value. So meta-storage sets `true` value for
all root and phy objects and puts them into separate bucket.

We also do not work with inversion anymore, so it either added
to the bucket or not. We don't need to store both options.
This is the reason `selectAll` function is changed a bit. Now
it performs some low-level parsing from primary bucket and root
bucket.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
2c50032831 [#160] Classify only regular objects as root
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
b59af19dff [#171] Increase extra gas on ApprovePeer method
With latest changes in neo's execution costs, we need more
gas to add storage node to netmap.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-13 17:12:46 +03:00
Alex Vanin
ded45e1fbc [#159] Add keyer to neofs-cli
Keyer prints information about private key, public key,
NEO3 Wallet, scripthash. It can generate new private key
or generate multisig address.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-09 18:59:21 +03:00
Leonard Lyubich
fcb35d82cf [#165] Revert "[#142] metabase: Store header value index in a tree leaf"
Revert commit 0faa40e4 to increase the disk space consumed by the
metabase in favor of the speed of index updates.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 18:58:58 +03:00
Leonard Lyubich
3c42f5b452 [#161] object/head: Inherit common parameters in HeadRelation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 17:19:34 +03:00
Leonard Lyubich
d38633e047 [#161] object/delete: Add address from request body to tombstone content
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 15:53:58 +03:00
Leonard Lyubich
49470ed613 [#158] metabase: Log errors inside DeleteObjects method implementation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
36d5af812e [#158] metabase: Add logger to DB configuration
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
60e4b5ddff [#158] metabase: Construct DB using options
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
0cd05fdca5 [#158] metabase: Implement operation of deleting a group of objects
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
200fdbd361 [#149] metabase: Do not write virtual objects to the primary index
In the previous implementation of the metabase, it was necessary to write
virtual objects to the primary index to be able to select them. In this
approach, virtual objects can be obtained directly using Head operation.
This has a side effect in handling object operations that do not expect to
receive a virtual object header in a single operation. With recent changes,
it is no longer necessary to have records of virtual objects in the primary
index, so this no longer happens for system integrity.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 14:01:01 +03:00
Leonard Lyubich
5ad013c10b [#149] object/search: Return fixed error if relation not found
Define ErrRelationNotFound error in searchsvc package. Return
ErrRelationNotFound from RelationSearcher.SearchRelation method if search
result is empty.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 14:01:01 +03:00
Leonard Lyubich
2913aa0fd1 [#149] metabase: Add parent ID index
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 14:01:01 +03:00
Alex Vanin
65be09d3db [#155] Update neofs-api-go with refactored pkg/netmap
Refactored pkg/netmap package provides JSON converters for
NodeInfo and PlacementPolicy structures, that has been used
by client applications.

It also updates Node structure itself so it is a part of
grpc <-> v2 <-> pkg conversion chain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 09:55:05 +03:00
Alex Vanin
9f8eb0fd51 [#152] Use ID in SetConfig contract method invocation.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 09:26:00 +03:00
Alex Vanin
7c1776a281 [#152] Add ID field to update state event
NeoFS contract produces event with three arguments:
  - update ID,
  - config key,
  - config value.

Update ID is a unique shared by inner ring holders
byte sequence that is used to update NeoFS runtime
configuration.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 09:26:00 +03:00
Leonard Lyubich
62bd22a379 [#142] metabase: Fix false-positive select in absence of filtered header
Fix a bug in the selection when the object without some filtered header
added to the final result.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
3c39c5a90c [#142] metabase: Fix selection emptying due to deleted object
Fix a bug in the selection when removed object that matches search query
provoked the return of an empty result.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
7a8f322d59 [#142] metabase: Fix false-positive select
Fix a bug in the selection when an object could be added to the result after
a mismatch in the previous filter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
77e80f517f [#142] metabase: Store header value index in a tree leaf
In the previous implementation of the metabase, the unique value of the
header was assigned a bucket, the elements of which were leaves with a
key-address and an empty value. This approach was relatively efficient in
terms of write speed. However, a large number of buckets led to a rapid
increase in the database volume (~4GB for 100K objects with unique
attributes). An approach is presented with storing indexes on the value of
headers in the leaves of the tree, where the keys are the unique values ​​of
the header, and the values ​​are a serialized list of addresses (gob
encoding is temporarily used for serialization).

The new approach gave a good result in saving space (~350MB), however, it
significantly reduced the write speed with an increase in the number of
objects (~ 80x after 100K objects).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
e48f8a189e [#142] metabase: Replace exclusive select with the inclusive one
The previous metabase implementation took an exclusionary approach: filters
narrowed the set of all objects to those that match all filters. An
inclusive approach is presented. In it, when traversing the indexed headers,
the object becomes a candidate for selection. If at least one of the
subsequent filters is not passed, the object ceases to be a candidate. At
the end of the traversal, the remaining candidates are added to the
resulting sample. The borderline case of no filters is handled in a special
way: all stored objects are added to the resulting selection.

Presented inclusive approach showed better performance in most scenarios
(although not all).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
3bbf1d6a60 [#137] metabase: Do not select objects if there is no filterable attribute
In previous metabase implementation the absence of an attribute presented in
the search filter did not exclude the object from the result. Change this
behavior to exclude the object from the result.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
d9a5007c72 [#137] metabase: Add unit test for a nonexistent attribute
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
7704811654 [#137] metabase: Implement and useful test funcs for work with DB
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
fc2038e929 [#137] metabase: Implement Path method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
8125b544b4 [#136] cmd/neofs-node: Use new metabase in app
Remove BoltDB bucket package. Construct meta.DB instance in node app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
c0aa892161 [#136] localstorage: Make local storage to use new metabase
Replace meta Bucket with meta.DB instance in local storage implementation.
Adopt all dependent components to new local storage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
a61f8d44d1 [#135] metabase: Implement benchmarking tests
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
1db01725c9 [#131] metabase: Implement indexing by object properties
Process parent objects in Put method. Headers of parent object are stored as
regular leaf objects in metabase from now. Build indexes for ROOT, LEAF and
CHILDFREE properties.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Alex Vanin
97077294fc [#141] Fix double sidechain GAS emission on asset mint
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-03 16:28:01 +03:00
Alex Vanin
0c49c08609 [#139] Use storage node GAS emission in inner ring app
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-03 16:20:41 +03:00
Alex Vanin
d193f1087c [#139] Emit gas to the storage nodes at emission tick
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-03 16:20:41 +03:00
Leonard Lyubich
a8e9d15f3f [#85] inner-ring/container: Check container format in process Put
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 14:14:38 +03:00
Leonard Lyubich
766eea4c8c [#85] services/container: Check container format in Put
Call CheckFormat function in container.Put handler for conducting initial
checks of the structure that are not performed by the smart contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 14:14:38 +03:00
Leonard Lyubich
a3e0a9f74c [#85] core/container: Implement CheckFormat function
Implement function that conducts sanity checks of the container data.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 14:14:38 +03:00
Alex Vanin
83119c00ec [#127] Remove magic number from config
With neo-project/neo-modules#358 nodes support RPC to
return used magic number. Therefore client doesn't need
that configuration value any more.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 13:54:31 +03:00
Alex Vanin
174efc9df3 [#124] Update neo-go to pre-preview4 version
Neo-go does not use smartcontract.Parameter to return values
anymore, so it's convertes partly removed from neofs-node.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 13:54:31 +03:00
Alex Vanin
c7975dbe87 [#132] Use SDK defined netmap structure in cleanup table
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 10:52:52 +03:00
Alex Vanin
976ba06300 [#132] Use netmap cleaner in inner ring server
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 10:52:52 +03:00
Alex Vanin
94957dd38c [#132] Generate and handle netmap cleanup event
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 10:52:52 +03:00
Alex Vanin
5c7de27546 [#132] Add netmap cleanup event processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 10:52:52 +03:00
Alex Vanin
6fcaa13e0b [#132] Flag all gracefully shutdown nodes in cleaner cache
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 10:52:52 +03:00
Alex Vanin
118f7fb309 [#132] Put new storage peers in netmap cleaner cache
There is an optimization to send ApprovePeer tx only in
case if `touch` returns false. It returns false if node
is not in the cache or it was flagged to removal.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 10:52:52 +03:00
Alex Vanin
0c42db2c3c [#132] Add cleanup table in netmap processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 10:52:52 +03:00
Alex Vanin
9b97e95f82 [#132] Add internal event to process cleanup
This event will be produced by new epoch handler. All
cleanups should be synchronous because smart contract can't
store inner ring votes forever. Therefore voting should be done
in similar time interval for all inner ring nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 10:52:52 +03:00
Alex Vanin
dfc2e81717 [#132] Add cleanup table in inner ring netmap processor
Cleanup table is a cache for inner ring node to look for netmap
snapshot. It updates access time of bootstrapped nodes and will
be used to send `updateState` txs to clean netmap from unresponsive
nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 10:52:52 +03:00
Alex Vanin
c80bce9438 [#132] Implement netmap snapshot getter for inner ring
Later it should be merged with morph/client getters.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 10:52:52 +03:00
Leonard Lyubich
b48a4ede02 [#125] services/eacl: Use latest object header keys
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-29 19:25:54 +03:00
Leonard Lyubich
8d931b81a6 [#125] object/search: Use latest search filter keys
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-29 19:25:54 +03:00
Leonard Lyubich
f34ad9e730 [#125] services/eacl: Fix undefined method usage
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-29 19:25:54 +03:00
Leonard Lyubich
20ed7c0d61 [#128] metabase: Implement Delete method
Implement Delete method on DB structure that adds deleted addresses to
tombstone index. Do not attach addresses from tombstone index to Select
result. Return error from Get method if address is presented in tombstone
index.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-29 17:34:41 +03:00
Leonard Lyubich
2d319aa29c [#128] metabase: Use static byte prefix for calculating non-empty keys
It is no longer necessary to make different prefixes to generate non-empty
keys for buckets.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-29 17:34:41 +03:00
Leonard Lyubich
5f78a18a4f [#128] metabase: Do not create leaves in indexed header bucket
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-29 17:34:41 +03:00
Leonard Lyubich
85aacbbb10 [#128] localstorage: Implement primary object metabase
Implement bolt-based metabase that is going to be used in local object
storage. Implement Put/Get/Select methods.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-29 17:34:41 +03:00
Alex Vanin
d08c1c76c1 [#122] Reduce precision from balance contract to Fixed8.
Fixed8 won't overflow int64 for values less than 92 billion
that is suitable for GAS.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-29 10:14:59 +03:00
Alex Vanin
03f52bca01 [#122] Return big.Int in balanceOf function
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-29 10:14:59 +03:00
Alex Vanin
e3c060b739 [#122] Add BigIntFromStackItem
Neo-go can return big.Int values on stack after contract execution.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-29 10:14:59 +03:00
Alex Vanin
330c4b42d9 [#122] Use balance precision converter in inner ring
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-28 13:36:59 +03:00
Alex Vanin
90984ee219 [#122] Add converter interface in balance and neofs processors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-28 13:36:59 +03:00
Alex Vanin
b6012977bc [#122] Implement balance precision getter for inner ring
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-28 13:36:59 +03:00
Alex Vanin
7db40eafe0 [#122] Add precision converter
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-28 13:36:59 +03:00
Leonard Lyubich
f66c7958e7 [#109] services/policer: Assign tasks to Replicator
Make Policer to call AddTask method of Replicator when an insufficient
number of copies of an object is detected in the container.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 15:23:22 +03:00
Leonard Lyubich
2d46baa4a5 [#109] services: Implement Replicator service
Implement Replicator service that performs background work to replicate
local object to remote nodes in the container. Replicator is going to be
used by Policer.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 15:23:22 +03:00
Leonard Lyubich
53efa18e14 [#109] object/put: Implement remote object sender
Define RemoteSender structure with PutObject method that puts object to a
remote node locally.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 15:23:22 +03:00
Leonard Lyubich
968033deed [#40] object/put: Assign zero return of MaxObjectSize invalid
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 14:03:25 +03:00
Leonard Lyubich
43533c991e [#40] morph/netmap: Implement MaxObjectSize method on wrapper
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 14:03:25 +03:00
Leonard Lyubich
90c38fc5e5 [#40] morph/client: Implement read config method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 14:03:25 +03:00
Leonard Lyubich
7fdb14cf8a [#83] services/response: Set epoch number from network state
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 10:54:48 +03:00
Leonard Lyubich
19f9c7eacb [#83] services: Remove setting of meta header from executing services
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 10:54:48 +03:00
Leonard Lyubich
0341773318 [#83] services: Implement response sub-service for each service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 10:54:48 +03:00
Leonard Lyubich
6bede7d836 [#83] services/util: Implement response service
Create response package. Implement response Service that sets values of
response meta header.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 10:54:48 +03:00
Leonard Lyubich
1cc7983c4e [#83] services/util: Add meta header methods to ResponseMessage
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 10:54:48 +03:00
Leonard Lyubich
71a06f9e01 [#83] services/util: Define type of response message interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 10:54:48 +03:00
Alex Vanin
7464254680 [#106] Put simplest bearer token check first
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-22 18:02:11 +03:00
Alex Vanin
23ec33e821 [#106] Check bearer token lifetime
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-22 18:02:11 +03:00
Alex Vanin
bb455af05f [#106] Ignore bearer token if basic ACL restrict it
There is a bit to allow or deny bearer token check for
each object service method. If this bit is not set then
ignore bearer token and use extended ACL table from
sidechain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-22 18:02:11 +03:00
Alex Vanin
89cd2ad463 [#106] Process bearer token in ACL service
If bearer token is presented in the request then check
if it is a valid one and then use it to process extended
ACL checks.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-22 18:02:11 +03:00
Alex Vanin
094248690b [#115] Make ACL classifier errors transparent for client
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-22 11:55:28 +03:00
Alex Vanin
ca552f53c6 [#115] Check session token validity
Malicious user can stole public session key and use
it by sending request from it's own scope. To prevent
this each session token is signed and signature private
key must be corresponded with owner id in token. Therefore
malicious node cannot impersonate request without private
key to sign token.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-22 11:55:28 +03:00
Leonard Lyubich
dae94aa230 [#60] morph/netmap: Implement Epoch method on netmap client wrapper
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-22 11:54:08 +03:00
Leonard Lyubich
20fb8547b6 [#60] morph/netmap: Implement Epoch method on Client
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-22 11:54:08 +03:00
Leonard Lyubich
16a5107ef1 [#60] object/put: Provide network State interface to formatter
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-22 11:54:08 +03:00
Leonard Lyubich
b627814dd8 [#60] object/transformer: Set creation epoch number in new objects
Set value of CreationEpoch object field to the value from network State.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-22 11:54:08 +03:00
Leonard Lyubich
7c081e4f15 [#60] core/netmap: Define interface of current network properties
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-22 11:54:08 +03:00
Leonard Lyubich
4a56f82571 [#60] object/transformer: Group parameters of NewFormatTarget func
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-22 11:54:08 +03:00
Leonard Lyubich
2541ed4b8f [#88] object/eacl: Use String() methods to calculate ID values
Replace hex encoding of IDs with String() call (base58) in eACL processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-21 18:53:04 +03:00
Leonard Lyubich
5318abcf38 [#88] object/search: Use String() methods to calculate ID values
Replace hex encoding of IDs with String() call (base58) in search query
processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-21 18:53:04 +03:00
Leonard Lyubich
0dab4b7581 [#108] services: Implement Policer service
Implement Policer service that performs background work to check compliance
with the placement policy for local objects in the container. In the initial
implementation, the selection of the working queue of objects is
simplified, and there is no transfer of the result to the replicator.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-21 14:42:51 +03:00
Leonard Lyubich
f6e56aa956 [#108] placement: Implement Builder from netmap source
Implement placement.Builder interface on netmap.Source wrapper.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-21 14:42:51 +03:00
Leonard Lyubich
5017ff0e4a [#108] object/head: Export remote header retrieval utility
Export remote head functionality in headsvc package. Refactor head service
to use RemoteHeader.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-21 14:42:51 +03:00
Leonard Lyubich
5ad0df7794 [#108] object/head: Return 404 error if header was not found
Define ErrNotFound error in headsvc package. Return ErrNotFound from Head
method if the header was not found in the container.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-21 14:42:51 +03:00
Alex Vanin
ae0dd9e051 [#106] Pass bearer token through generated requests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-20 18:05:29 +03:00
Alex Vanin
e6f04f7785 [#104] Update neofs-api-go with new protobuf API
Also update contains JSON converters for neofs-cli
and fixes bug in container.set-acl command of SDK.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-20 17:31:59 +03:00
Alex Vanin
9e08b41a6f [#102] Set split header in left object
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-20 09:59:09 +03:00
Leonard Lyubich
54bdeb60a1 [#22] Support string type in stack parameter converter
Set type of stack parameter to StringType in type-switch statement of
toStackParameter function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-19 12:27:56 +03:00
Alex Vanin
643e81254c [#25] Add placement policy QL encoder
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-17 13:16:59 +03:00
Evgenii Stratonikov
d7c53debb5 [#46] Make SELECT attribute optional
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2020-10-16 16:15:27 +03:00
Alex Vanin
719075ca97 [#99] Fix no-root search matcher
Wrong boolean operation order made matcher return false
on `non-root` search query with non-regular objects. Instead
it should return true for `non-root` query and false for `root`
query.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-16 13:45:35 +03:00
Alex Vanin
1332a6d3a8 [#92] Provide session token to all produced requests
If object service produces new request, the should contain
session token. This is the only way for node to grant access
for a private container.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-15 10:20:10 +03:00
Alex Vanin
dd48666357 [#72] Shutdown inner ring app if RPC node has been terminated
Adopt error channel from Listener interface.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-14 09:37:48 +03:00
Alex Vanin
ccbb9ce6ab [#72] Add ListenWithError method in Listener interface
Listen and ListenWithError methods check if subscriber channel
has been closed. If so, ListenWithError passes error message
into provided channel.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-14 09:37:48 +03:00
Alex Vanin
ca006245d2 [#72] Close subscription channel if RPC was terminated
RPC node closes websocket notification channel if it was terminated
or something wrong happened. Subscriber has to check this condition
and alert about this in upper context by closing it's own channel.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-14 09:37:48 +03:00
Alex Vanin
d3d8f00757 [#72] Init inner ring configuration before run
This way we can re-initialize application in any given
moment, which is useful for reconnects.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-14 09:37:48 +03:00
Alex Vanin
ade185191b [#47] Use less extra gas on netmap contract methods
* 1.0 extra gas for new epoch vote
* 0.5 extra gas for new peer approve

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-12 14:43:33 +03:00
Alex Vanin
b7ee05088c [#86] Use alphabet processor in inner ring app
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-12 14:43:33 +03:00
Alex Vanin
f1f20b49db [#86] Add alphabet contract processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-12 14:43:33 +03:00
Alex Vanin
4308a6f522 [#86] Run timer for gas emission event
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-12 14:43:33 +03:00
Alex Vanin
6bc787bb19 [#86] Add alphabet contract configuration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-12 14:43:33 +03:00
Alex Vanin
dcb384d551 [#86] Specify inner ring list index in global state
Inner ring index will be used to access alphabet contracts.
First seven inner ring nods communicating with one alphabet
contract, depending on their index.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-12 14:43:33 +03:00
Alex Vanin
2d5cb378a7 [#84] Add netmap service executor and signer
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-09 09:15:18 +03:00
Alex Vanin
f92dc5b27c [#84] Add GRPC layer of netmap service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-09 09:15:18 +03:00
Alex Vanin
0e7e0bd2d6 [#84] Remove mocks and debug code from neofs-node services
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-09 09:15:18 +03:00
Alex Vanin
b5aef1011b [#74] Update test of netmap.UpdateState event parser
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-09 09:15:18 +03:00
Alex Vanin
e3f8e350f9 [#82] Fix conversion of container id values from smart-contract
Smart-contract stores container ids as a raw bytes, not marshaled
protobuf structures.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-08 11:22:50 +03:00
Alex Vanin
7d51688d2c [#82] Return length check of serialized container
In #37 we've decided to remove length check, because smart contract would
fail on casting `nil` value from storage to `[]byte` producing FAULT state.
Apparently it does not fail, so we have to check length explicitly.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-08 11:22:50 +03:00
Alex Vanin
87fc4f5df7 [#82] Use morph wrapper in container service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-08 11:22:50 +03:00
Alex Vanin
204eaab5a9 [#75] Fix order of update peer state method arguments
This method has node status first and public key second.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-05 18:58:35 +03:00
Alex Vanin
f40f1ca0c0 [#74] Fix order of netmap.UpdateState event arguments
This event has node status first and public key second.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-05 18:31:13 +03:00
Alex Vanin
cd34145969 [#73] Use request owner public key in eACL check
Classifier fetches public key of the request owner
and owner itself. Extended ACL check should rely on
this public key, because it might be extracted from
session token.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-05 18:02:37 +03:00
Alex Vanin
7a2654719e [#71] Return only regular objects in root object search
Root search applies for user objects, so it should not
return tombstones and storage groups.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-05 14:03:55 +03:00
Alex Vanin
11262bed4a [#71] Broadcast tombstone to container
With one tombstone for split objects we can't simply
place it in container. We should inform all nodes that
store split objects of removed original object.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-05 14:03:55 +03:00
Leonard Lyubich
9cdf7d3896 [#69] object/acl: Check eACL rules in ACL service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 14:02:14 +03:00
Leonard Lyubich
1d676fcfb2 [#69] object/acl: Add eACL components to service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 14:02:14 +03:00
Leonard Lyubich
a7782cf1f9 [#69] object/acl: Extended requestInfo structure
Add container identifier field. Add send public key field.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 14:02:14 +03:00
Leonard Lyubich
6c3c872ee4 [#69] object/acl: Define access denied error
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 14:02:14 +03:00
Leonard Lyubich
0f52444ae9 [#69] object/acl: Change basic ACL type in requestInfo
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 14:02:14 +03:00
Leonard Lyubich
30e6912c7b [#69] object/acl: Construct service from options
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 14:02:14 +03:00
Leonard Lyubich
e5898c9ca8 [#69] object/acl: Rename BasicChecker to Service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 14:02:14 +03:00
Leonard Lyubich
0d5495e997 [#70] object manager: Implement an example object garbage collector
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 09:36:29 +03:00
Leonard Lyubich
798fca9354 [#70] core/object: Process a delete group at tombstone
Send object group to delete queue processor after tombstone content
validation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 09:36:29 +03:00
Leonard Lyubich
2b16edebc9 [#70] object/put: Fix NPE caused by nil FormatValidator
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 09:36:29 +03:00
Alex Vanin
801999c577 [#66] Impersonate object service verb from session token
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 19:47:49 +03:00
Alex Vanin
afeebd310c [#66] Use session token of object header at put ACL check
Owner of the request is stored in session token most of the times.
Put request contains session token in the object body, so we have
to fetch it from there.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 19:47:49 +03:00
Leonard Lyubich
69a69cdbee [#67] object/eacl: Implement eACL validator
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 19:46:45 +03:00
Leonard Lyubich
44fcd2f212 [#64] object/delete: Change the formation of tombstone
Make delete service to write list of child object addresses to tombstone
payload.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 19:46:27 +03:00
Leonard Lyubich
33ca88f85f [#64] core/object: Claim TombstoneContent in tombstone payload
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 19:46:27 +03:00
Leonard Lyubich
21bdc82fb5 [#64] core/object: Implement tombstone content messages
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 19:46:27 +03:00
Alex Vanin
6ee908c2db [#65] Support FS bucket and BoltDB bucket
These buckets can be used to store blobs and metadata.
They will be removed as enhanced blob storage will be
implemented for neofs-node. To setup storage type
use `storage.object.type` and `storage.meta.type` params.

Available options:
  - inmemory (default)
  - boltdb
  - filesystem

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 14:48:00 +03:00
Alex Vanin
861bac3892 [#59] Use max msg size in transport server and splitter
For GRPC it is about 4 MiB.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
d2009c8731 [#59] Add grpc payload splitter in object service chain
GRPC has default message limit of 4MiB. Since every transmitted
neofs message has to be signed, then original message should
be split into transfer fit structures before signature service.

This commit introduce transport payload splitter for object
service pipeline. This splitter works with stream response
for methods:

  - object.Get
  - object.Range
  - object.Search

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
64691e6248 [#62] object/transformer: Fix incorrect relation init stage
In previous implementation InitRelations call in payload size limiter was
called in write chunk method. This provoked clearing the split header in
children starting from the second.

Replace InitRelations call to the 1st child allocating stage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
e158497560 [#43] cmd/neofs-node: Support hostnames with dns, ipv4 and ipv6 addresses
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
f930993e3a [#43] pkg/network: Do not panic at multiaddr to net.Addr conversion
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
6824a6f67b [#61] object/search: Support non-root and non-leaf filters
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
51e373c3f0 [#61] object/search: Support latest search filters
Refactor query to match object and its parents in a single call. Support
KeyRoot and KeyLeaf filters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
f89c848e84 [#61] object/search: Filter objects by container ID from request
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
1654df4d97 [#61] Update to latest neofs-api-go changes
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
f251645def [#58] object/delete: Process linking object in Delete service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00