Commit graph

5213 commits

Author SHA1 Message Date
Leonard Lyubich
2ba4ebc8c9 [#211] blobstor: Implement no-op GetSmall operation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2f61d8d63c [#211] localstorage: Define blobovnicza ID
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
6813f40665 [#211] blobstor: Refactor GetBig parameters and result
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
fb6857a1cb [#211] blobstor: Rename Get operation to GetBig
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4e8cc51e7b [#189] shard: Extend Info with WeightValues
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4f16a10235 [#189] localstorage: Implement DumpInfo method
Implement method to get the information about the local storage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
8fbda2a588 [#189] shard: Implement DumpInfo method
Implement method to get the information about the shard.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
24cf86e269 [#189] metabase: Implement DumpInfo method
Implement method to get the information about the metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
eaa7068e3c [#189] blobstor: Implement DumpInfo method
Implement method to get the information about the BLOB storage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
dae8d3de5b [#189] shard: Fix missing return statement in SetMode method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
cba87a1300 [#188] localstorage: Implement SetShardMode method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
3fa3661cad [#188] shard: Implement SetMode method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2fb379b7dd [#188] shard: Define the enumeration of shard modes
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
0b515837ca [#186] localstorage: Fix address forwarding in Delete method
In previous implementation storage engine not forwarded object address to
shards.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4ca1035aa4 [#186] cmd/neofs-node: Integrate new storage engine into application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
953387a1e5 [#186] object/acl: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
046206f670 [#186] object/head: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2be8f154a0 [#186] object/put: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
14442a0801 [#186] object/range: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
9b32f131c0 [#186] object/rangehash: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1bce2a4e55 [#186] object/search: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
aa9eb2eaf2 [#186] policer: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1b867e53d7 [#186] replicator: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
9a4cd01eb8 [#186] localstorage: Implement utility functions over StorageEngine
Implement Put/Get/GetRange/Select/SelectAll functions over storage
engine. These functions are going to be used by Object service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
70a65c5aff [#176] blobstor: Use 1 byte names of sub-directories in FS tree
Using 2 bytes length for directories in fs tree lead to large growth
in the number of directories.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2d917a024f [#176] blobstor: Fix missing mutex in constructor
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
b1718a60e4 [#176] shard: Fix BlobStor access when data range is zero
In previous implementation Shard accessed the BlobStor to get the
object header. However, the shard must take headers from the metabase.
From now zero length of the requested payload range seens as object
header request. In this case shard calls metabase to get the header.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
796bba3793 [#176] shard: Finalize the implementation of Exists method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4518357ad7 [#176] localstore: Fix linter's remarks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
f194c840d7 [#176] blobstor: Handle error of zstd (de)compressor creation
In previous implementation WithCompressObjects returned Option
than could panic if zstd (de)compressor creation failed with error.
From now errors with (de)compressor creation result in an option
without using data compression. In this case, the error is written
to the log passed to the option constructor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
6f8c45d61b [#176] localstore: Change multiple access sync
In previous implementation each operation on local storage
locked engine mutex. This was done under the assumption that
the weights of the shards change as a result of write operations.
With the transition to static weights of shards, it is no longer
necessary to lock the global mutex during the execution of operations.
However, since the set of engine shards is dynamic, there is still a
need to control multiple access to this set. The same mutex is used
for synchronization.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
df558cbe6b [#176] shard: Get rid of using the global RW lock
In previous implementation each shard operation locked
RW shard mutex. With this approach RW operations were executed
one-by-one and blocked the execution of RO operations.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
9ab371aafb [#176] shard: Make ID stringer to return base58 encoded string
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
86385af67d [#176] blobstor: Fix maximum tree depth
Correct the calculation of maximum value of fs tree depth. Fix check
of the max depth overflow in WithShallowDepth function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
f7116f27ec [#176] localstore/shard: Change signature of options
Make WithBlobStorOptions/WithMetaBaseOptions functions to accept
variadic parameter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
73494c4bbc [#176] localstore: Fix Get with non-zero length
GetPrm has WithPayloadRange option to specify the requested
payload range. In previous implementation StorageEngine.Get
method ignored this option. From now zero length matches
full payload request.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
46e455dcae [#176] localstore: Implement shard methods via Metabase and BlobStor
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
b127607ac6 [#176] localstore: Implement primary BlobStor
Implement primary local BLOB storage based on filesystem tree.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
09750484f9 [#176] localstore: Draft storage engine structure and ops
Implement the primary structure and operation of the local object
storage engine.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
383c483be7 [#176] localstore: Draft storage engine shard structure and ops
Implement the primary structure and operation of the local object
storage shard.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
1486cc0e29 [#170] morph/client: Print reversed hash in logs
JSON-RPC method takes reversed hash as an input so it is
better to output reveresed hash in logs.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
e2547f5bf8 [#170] Invoke all alphabet contracts for voting
Now alphabet contracts use signature collection to make
a vote. All inner ring nodes votes for some list of validators
the same way as they vote fore new epoch or new container.
As soon as list is accepted by alphabet contracts, each of them
votes for one candidate from the list.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
49666f87f1 [#170] Implement --vote argument in neofs-ir
With this argument neofs-ir will be run as an CLI
application that initialize inner ring server, invokes
`vote` method on corresponding alphabet contract and
exits.

User can provide a comma-separated list of validator
public keys.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
0b65c64e98 [#170] Vote for validators on inner ring startup
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
2240cde826 [#170] Implement validator voting function in inner ring
This function calculates validator key based on inner ring index
and invokes `Vote` method of corresponding alphabet contract.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
d02c24db5c [#170] Add alphabet vote invocation method
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Leonard Lyubich
6cddc91cbf [#227] transformer: Inherit parent header in linking object
In previous implementation linking object inherited only ID of the parent
object. From now it inherits full header like last child.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-02 14:53:04 +03:00
Leonard Lyubich
b8e8e1e80d [#227] transformer: Do not inherit attribute in generated objects
In previous implementation child objects inherited parent attributes after
split-transformation, which was redundant. From now attributes are not
inherited.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-02 14:53:04 +03:00
Leonard Lyubich
5470d94416 [#223] placement: Fix local and single-success placement traversal
In previous implementation placement traverser processed incorrectly with
local placement build. Also entity incorrectly traversed the placement
vectors for fixed number read operations until success. The erroneous
behavior was due to the use of a vector number of successes instead of
a scalar number in these scenarios.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-01 19:36:46 +03:00
Leonard Lyubich
f6e5644b85 Update to latest neofs-api-go
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-01 13:35:53 +03:00