Commit graph

519 commits

Author SHA1 Message Date
Alex Vanin
b45d523125 [] Split IsSmall check in metabase
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
5bfae833fe [] Update blobovniczaID on existed objects in metabase
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
54cd91adff [] Simplify tests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
a8d76f2ebb [] Implement ObjectID selection filter in metabase
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
700bd7de01 [] Get virtual objects from metabase
Virtual objects are obtained differently with
relative lookup in parent bucket.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
4e7d49791b [] Index parent first in metabase
With exist check we should index parent first, because
as soon as child will be added to metabase, exist on
parent will return true even if it was not indexed yet.

Also this commit makes one db.Update instead of two for
parent and child.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
f6387a9b94 [] Make exist check on all primary buckets in metabase
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
e478c0d024 [] Refactor metabase internal structure
Accoring to MetaBase-Plan-B storage engine specification.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1c81d507fd [] blobstor: Inherit the root path to Blobovnicza from BlobStor
Place the root of blobovnicza tree in a subdirectory of BlobStor with same
permissions. Abolish WithBlobovniczaRootPath and WithBlobovniczaPersmissions
options.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4c3d231e82 [] blobstor: Implement blobovnicza tree
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
3acbd3011b [] blobovnicza: Сreate at least one range bucket
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1b0d2c67d1 [] blobstor: Add blobovnicza ID value to DeleteSmallPrm
Blobovnicza ID parameter provides the ability to specify particular
blobovnicza to delete object from. In this case only specified blobovnicza
is processed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
a97ab242eb [] blobstor: Add config values of blobovnicza tree
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
35ccda13a8 [] blobovnicza: Change ID stringer
Replace hex encoding with slice to string conversion.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
07e998d846 [] blobovnicza: Implement main methods and logic
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
52b0c1f19a [] blobovnicza: Add configuration values
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
bce16de72a [] blobstor: Classify objects by size according to binary format
In previous implementation objects were classified by size according to
payload size. From now they are classified by the size of their binary
representation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
40b09f9266 [] blobovnicza: Define Blobovnicza type
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
b9054e2ee0 [] blobstor: Add a branch for processing objects by size in Put
Implement "big or small" property classifier (only the size of the payload
is temporarily considered). Save "big" objects in shallow dir. Save "small"
objects in shallow dir until the moment of implementation of blobovnicza.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
cfc770c3fe [] blobstor: Add SmallSizeLimit parameter
Add numeric parameter of maximum size of the "small" objects.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
3028718691 [] blobovnicza: Refactor ID implementation
Replace UUID implementation of ID with byte slice.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
53b114cf8b [] blobstor: Remove global lock
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
59f7cf9873 [] blobstor: Implement Exists method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1ba556f5e7 [] blobstor: Define Iterate operation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
3c0ef270f4 [] blobstor: Define Exists operation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
915243c69e [] blobstor: Define DeleteSmall operation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
65da2025e5 [] blobstor: Rename Delete to DeleteBig
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
b605ea3678 [] blobstor: Refactor Put operation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
7afcde659d [] blobstor: Define GetRangeSmall operation interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
0b130150ae [] blobstor: Rename GetRange to GetRangeBig
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2ba4ebc8c9 [] blobstor: Implement no-op GetSmall operation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2f61d8d63c [] localstorage: Define blobovnicza ID
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
6813f40665 [] blobstor: Refactor GetBig parameters and result
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
fb6857a1cb [] blobstor: Rename Get operation to GetBig
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4e8cc51e7b [] shard: Extend Info with WeightValues
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4f16a10235 [] localstorage: Implement DumpInfo method
Implement method to get the information about the local storage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
8fbda2a588 [] shard: Implement DumpInfo method
Implement method to get the information about the shard.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
24cf86e269 [] metabase: Implement DumpInfo method
Implement method to get the information about the metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
eaa7068e3c [] blobstor: Implement DumpInfo method
Implement method to get the information about the BLOB storage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
dae8d3de5b [] shard: Fix missing return statement in SetMode method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
cba87a1300 [] localstorage: Implement SetShardMode method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
3fa3661cad [] shard: Implement SetMode method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2fb379b7dd [] shard: Define the enumeration of shard modes
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
0b515837ca [] localstorage: Fix address forwarding in Delete method
In previous implementation storage engine not forwarded object address to
shards.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
953387a1e5 [] object/acl: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
046206f670 [] object/head: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2be8f154a0 [] object/put: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
14442a0801 [] object/range: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
9b32f131c0 [] object/rangehash: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1bce2a4e55 [] object/search: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
aa9eb2eaf2 [] policer: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1b867e53d7 [] replicator: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
9a4cd01eb8 [] localstorage: Implement utility functions over StorageEngine
Implement Put/Get/GetRange/Select/SelectAll functions over storage
engine. These functions are going to be used by Object service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
70a65c5aff [] blobstor: Use 1 byte names of sub-directories in FS tree
Using 2 bytes length for directories in fs tree lead to large growth
in the number of directories.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2d917a024f [] blobstor: Fix missing mutex in constructor
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
b1718a60e4 [] shard: Fix BlobStor access when data range is zero
In previous implementation Shard accessed the BlobStor to get the
object header. However, the shard must take headers from the metabase.
From now zero length of the requested payload range seens as object
header request. In this case shard calls metabase to get the header.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
796bba3793 [] shard: Finalize the implementation of Exists method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4518357ad7 [] localstore: Fix linter's remarks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
f194c840d7 [] blobstor: Handle error of zstd (de)compressor creation
In previous implementation WithCompressObjects returned Option
than could panic if zstd (de)compressor creation failed with error.
From now errors with (de)compressor creation result in an option
without using data compression. In this case, the error is written
to the log passed to the option constructor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
6f8c45d61b [] localstore: Change multiple access sync
In previous implementation each operation on local storage
locked engine mutex. This was done under the assumption that
the weights of the shards change as a result of write operations.
With the transition to static weights of shards, it is no longer
necessary to lock the global mutex during the execution of operations.
However, since the set of engine shards is dynamic, there is still a
need to control multiple access to this set. The same mutex is used
for synchronization.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
df558cbe6b [] shard: Get rid of using the global RW lock
In previous implementation each shard operation locked
RW shard mutex. With this approach RW operations were executed
one-by-one and blocked the execution of RO operations.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
9ab371aafb [] shard: Make ID stringer to return base58 encoded string
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
86385af67d [] blobstor: Fix maximum tree depth
Correct the calculation of maximum value of fs tree depth. Fix check
of the max depth overflow in WithShallowDepth function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
f7116f27ec [] localstore/shard: Change signature of options
Make WithBlobStorOptions/WithMetaBaseOptions functions to accept
variadic parameter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
73494c4bbc [] localstore: Fix Get with non-zero length
GetPrm has WithPayloadRange option to specify the requested
payload range. In previous implementation StorageEngine.Get
method ignored this option. From now zero length matches
full payload request.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
46e455dcae [] localstore: Implement shard methods via Metabase and BlobStor
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
b127607ac6 [] localstore: Implement primary BlobStor
Implement primary local BLOB storage based on filesystem tree.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
09750484f9 [] localstore: Draft storage engine structure and ops
Implement the primary structure and operation of the local object
storage engine.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
383c483be7 [] localstore: Draft storage engine shard structure and ops
Implement the primary structure and operation of the local object
storage shard.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
1486cc0e29 [] morph/client: Print reversed hash in logs
JSON-RPC method takes reversed hash as an input so it is
better to output reveresed hash in logs.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
e2547f5bf8 [] Invoke all alphabet contracts for voting
Now alphabet contracts use signature collection to make
a vote. All inner ring nodes votes for some list of validators
the same way as they vote fore new epoch or new container.
As soon as list is accepted by alphabet contracts, each of them
votes for one candidate from the list.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
49666f87f1 [] Implement --vote argument in neofs-ir
With this argument neofs-ir will be run as an CLI
application that initialize inner ring server, invokes
`vote` method on corresponding alphabet contract and
exits.

User can provide a comma-separated list of validator
public keys.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
0b65c64e98 [] Vote for validators on inner ring startup
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
2240cde826 [] Implement validator voting function in inner ring
This function calculates validator key based on inner ring index
and invokes `Vote` method of corresponding alphabet contract.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
d02c24db5c [] Add alphabet vote invocation method
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Leonard Lyubich
6cddc91cbf [] transformer: Inherit parent header in linking object
In previous implementation linking object inherited only ID of the parent
object. From now it inherits full header like last child.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-02 14:53:04 +03:00
Leonard Lyubich
b8e8e1e80d [] transformer: Do not inherit attribute in generated objects
In previous implementation child objects inherited parent attributes after
split-transformation, which was redundant. From now attributes are not
inherited.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-02 14:53:04 +03:00
Leonard Lyubich
5470d94416 [] placement: Fix local and single-success placement traversal
In previous implementation placement traverser processed incorrectly with
local placement build. Also entity incorrectly traversed the placement
vectors for fixed number read operations until success. The erroneous
behavior was due to the use of a vector number of successes instead of
a scalar number in these scenarios.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-01 19:36:46 +03:00
Leonard Lyubich
d94a0eb25e [] eacl: Get rid of deprecated methods usage
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-01 11:08:47 +03:00
Alex Vanin
f96630581a [] Set unique splitID in children
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-30 10:44:15 +03:00
Alex Vanin
6280d075b9 [] Remove childfree search attribute
With updated specification of object related operation
we don't have this search attribute any more and we
should not use functions related to this attribute.

This commit breaks object service logic, however it will
be fixed later.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-30 10:44:15 +03:00
Leonard Lyubich
706bdf736e [] eacl: Support object ID filter in eACL mechanism
Add object ID header to the list of processing object headers in eACL
validation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-25 11:59:15 +03:00
Leonard Lyubich
fddc50fd85 [] Replace ErrEACLNotFound to core library
ErrEACLNotFound error was defined in implementation package. EACL validator
checked this error after the call of eACL storage interface method. Replace
ErrEACLNotFound to core container library. in order to: on the one hand not
use an implementation error, on the other hand, to be able to reuse a
generic type error (404).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 20:19:20 +03:00
Leonard Lyubich
9148980bd0 [] services/object: Support client options in all Object services
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 15:34:30 +03:00
Leonard Lyubich
27fc4d6e01 [] network/cache: Support client options in ClientCache.Get
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 15:34:30 +03:00
Alex Vanin
cf1ea983e5 [] Do not fail eACL check on EACLNotFound error
Now morph library returns error if there is not eACL in
sidechain storage. However in this case eACL check should
be passed since it is the same as having empty eACL table.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-24 15:32:26 +03:00
Alex Vanin
8cedb93cea [] Make eACL not found error public
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-24 15:32:26 +03:00
Leonard Lyubich
1f6ad5ccd8 [] morph/eacl: Return error if container eACL is not set
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:56:03 +03:00
Leonard Lyubich
46dab77705 [] services/object: Write debug log messages on worker pool errors
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
fa6e4a3ca4 [] services/object: Write debug log messages on internal service errors
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
6a5c37d592 [] object/search: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
cb46e4b154 [] object/rangehash: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
af6484e3b1 [] object/range: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
cf2dc37a42 [] object/put: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
9fbfc0b5e4 [] object/head: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
e1e5a590e9 [] object/delete: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
ffeea2c003 [] metabase: Add index by object ID
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-23 09:54:59 +03:00
Alex Vanin
dbf6c9efef [] Use request sender owner in sticky bit check
Sticky bit checks if object owner and request owner are the
same. Container owner should not used in this check.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 19:07:16 +03:00
Alex Vanin
2148e282ec [] Rename owner to cnrOwner in object request info
`owner` field may be misused as request sender owner, however
it is a owner of a container for that request. New naming
should be clear.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 19:07:16 +03:00
Alex Vanin
f0537b35c1 [] Add isOwnerFromKey helper function in ACL
This function takes public key and returns true if
owner id was produced by this key.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 19:07:16 +03:00
Alex Vanin
a14bb6292b [] Reuse search filter in policer
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 17:59:46 +03:00
Alex Vanin
2e605b2435 [] Limit policer object filter to physical stored objects
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 17:59:46 +03:00
Alex Vanin
cf85fa9fab [] Return isInnerRing flag in request classifier
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 15:17:18 +03:00
Alex Vanin
6f841e319d [] Make separate basic ACL check for inner ring requests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 15:17:18 +03:00
Alex Vanin
e8fe07edd0 [] Use SDK client cache in object.Rangehash
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
f85e88c4f8 [] Use SDK client cache in object.Range
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
d485a5967d [] Use SDK client cache in object.Search
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
7ba95dd5fc [] Use SDK client cache in object.Put
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
e9a6365333 [] Use SDK client cache in object.Head
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
fb90827532 [] Add SDK client cache package
With this package we can reuse already created connections.
Later on neofs-api-go will support checking connection health
and `Close` operation, so this cache could run worker and remove
unhealthy clients.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Leonard Lyubich
1caf15463e [] Update to neofs-api-go v1.20.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Leonard Lyubich
58fcb35fb0 [] Use Marshal(JSON)/Unmarshal(JSON) methods for encoding/decoding
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Leonard Lyubich
3de8febe57 [] Update to latest neofs-api-go changes
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Alex Vanin
ea41dec23f [] Store root and phy indexes as fake bucket tree
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
f1a3acd27a [] Fix metabase test for virtual object
Parent should not have split header.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
32219bb294 [] Remove query match function
This function duplicates query processing that
is done in meta-storage now.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
03fed8ca59 [] Update metabase with new root and phy flags processing
Now root and phy (leaf) filters work like flags. They work with
any matcher and any value. So meta-storage sets `true` value for
all root and phy objects and puts them into separate bucket.

We also do not work with inversion anymore, so it either added
to the bucket or not. We don't need to store both options.
This is the reason `selectAll` function is changed a bit. Now
it performs some low-level parsing from primary bucket and root
bucket.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
2c50032831 [] Classify only regular objects as root
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
b59af19dff [] Increase extra gas on ApprovePeer method
With latest changes in neo's execution costs, we need more
gas to add storage node to netmap.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-13 17:12:46 +03:00
Alex Vanin
ded45e1fbc [] Add keyer to neofs-cli
Keyer prints information about private key, public key,
NEO3 Wallet, scripthash. It can generate new private key
or generate multisig address.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-09 18:59:21 +03:00
Leonard Lyubich
fcb35d82cf [] Revert "[] metabase: Store header value index in a tree leaf"
Revert commit 0faa40e4 to increase the disk space consumed by the
metabase in favor of the speed of index updates.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 18:58:58 +03:00
Leonard Lyubich
3c42f5b452 [] object/head: Inherit common parameters in HeadRelation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 17:19:34 +03:00
Leonard Lyubich
d38633e047 [] object/delete: Add address from request body to tombstone content
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 15:53:58 +03:00
Leonard Lyubich
49470ed613 [] metabase: Log errors inside DeleteObjects method implementation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
36d5af812e [] metabase: Add logger to DB configuration
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
60e4b5ddff [] metabase: Construct DB using options
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
0cd05fdca5 [] metabase: Implement operation of deleting a group of objects
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
200fdbd361 [] metabase: Do not write virtual objects to the primary index
In the previous implementation of the metabase, it was necessary to write
virtual objects to the primary index to be able to select them. In this
approach, virtual objects can be obtained directly using Head operation.
This has a side effect in handling object operations that do not expect to
receive a virtual object header in a single operation. With recent changes,
it is no longer necessary to have records of virtual objects in the primary
index, so this no longer happens for system integrity.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 14:01:01 +03:00
Leonard Lyubich
5ad013c10b [] object/search: Return fixed error if relation not found
Define ErrRelationNotFound error in searchsvc package. Return
ErrRelationNotFound from RelationSearcher.SearchRelation method if search
result is empty.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 14:01:01 +03:00
Leonard Lyubich
2913aa0fd1 [] metabase: Add parent ID index
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 14:01:01 +03:00
Alex Vanin
65be09d3db [] Update neofs-api-go with refactored pkg/netmap
Refactored pkg/netmap package provides JSON converters for
NodeInfo and PlacementPolicy structures, that has been used
by client applications.

It also updates Node structure itself so it is a part of
grpc <-> v2 <-> pkg conversion chain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 09:55:05 +03:00
Alex Vanin
9f8eb0fd51 [] Use ID in SetConfig contract method invocation.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 09:26:00 +03:00
Alex Vanin
7c1776a281 [] Add ID field to update state event
NeoFS contract produces event with three arguments:
  - update ID,
  - config key,
  - config value.

Update ID is a unique shared by inner ring holders
byte sequence that is used to update NeoFS runtime
configuration.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 09:26:00 +03:00
Leonard Lyubich
62bd22a379 [] metabase: Fix false-positive select in absence of filtered header
Fix a bug in the selection when the object without some filtered header
added to the final result.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
3c39c5a90c [] metabase: Fix selection emptying due to deleted object
Fix a bug in the selection when removed object that matches search query
provoked the return of an empty result.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
7a8f322d59 [] metabase: Fix false-positive select
Fix a bug in the selection when an object could be added to the result after
a mismatch in the previous filter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
77e80f517f [] metabase: Store header value index in a tree leaf
In the previous implementation of the metabase, the unique value of the
header was assigned a bucket, the elements of which were leaves with a
key-address and an empty value. This approach was relatively efficient in
terms of write speed. However, a large number of buckets led to a rapid
increase in the database volume (~4GB for 100K objects with unique
attributes). An approach is presented with storing indexes on the value of
headers in the leaves of the tree, where the keys are the unique values ​​of
the header, and the values ​​are a serialized list of addresses (gob
encoding is temporarily used for serialization).

The new approach gave a good result in saving space (~350MB), however, it
significantly reduced the write speed with an increase in the number of
objects (~ 80x after 100K objects).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
e48f8a189e [] metabase: Replace exclusive select with the inclusive one
The previous metabase implementation took an exclusionary approach: filters
narrowed the set of all objects to those that match all filters. An
inclusive approach is presented. In it, when traversing the indexed headers,
the object becomes a candidate for selection. If at least one of the
subsequent filters is not passed, the object ceases to be a candidate. At
the end of the traversal, the remaining candidates are added to the
resulting sample. The borderline case of no filters is handled in a special
way: all stored objects are added to the resulting selection.

Presented inclusive approach showed better performance in most scenarios
(although not all).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
3bbf1d6a60 [] metabase: Do not select objects if there is no filterable attribute
In previous metabase implementation the absence of an attribute presented in
the search filter did not exclude the object from the result. Change this
behavior to exclude the object from the result.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
d9a5007c72 [] metabase: Add unit test for a nonexistent attribute
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
7704811654 [] metabase: Implement and useful test funcs for work with DB
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
fc2038e929 [] metabase: Implement Path method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
8125b544b4 [] cmd/neofs-node: Use new metabase in app
Remove BoltDB bucket package. Construct meta.DB instance in node app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
c0aa892161 [] localstorage: Make local storage to use new metabase
Replace meta Bucket with meta.DB instance in local storage implementation.
Adopt all dependent components to new local storage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
a61f8d44d1 [] metabase: Implement benchmarking tests
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
1db01725c9 [] metabase: Implement indexing by object properties
Process parent objects in Put method. Headers of parent object are stored as
regular leaf objects in metabase from now. Build indexes for ROOT, LEAF and
CHILDFREE properties.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Alex Vanin
97077294fc [] Fix double sidechain GAS emission on asset mint
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-03 16:28:01 +03:00
Alex Vanin
0c49c08609 [] Use storage node GAS emission in inner ring app
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-03 16:20:41 +03:00
Alex Vanin
d193f1087c [] Emit gas to the storage nodes at emission tick
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-03 16:20:41 +03:00
Leonard Lyubich
a8e9d15f3f [] inner-ring/container: Check container format in process Put
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 14:14:38 +03:00