Commit graph

1985 commits

Author SHA1 Message Date
Leonard Lyubich
d938c7267b [#789] node/config: Add Shard's refill_metabase config to examples
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
b5fff810f4 [#789] cmd/node: Add refill_metabase config to shard section
Implement `RefillMetabase` method to `shardconfig.Config` type which reads
`refill_metabase` config value. Pass the result to `WithRefillMetabase`
option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
6bf7a00cfe [#789] shard: Add option to refill metabase on initialization
Add `WithRefillMetabase` option constructor which allows to set flag to
refill metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
b9c22e21b1 [#789] shard: Add method to refill the metabase
There is a need to refill Metabase data with the objects from BlobStor.

Implement `refillMetabase` method which iterates over all objects from
BlobStor and saves them in Metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
8d016d2529 [#789] blobstor: Implement iterator
There is a need to be able to process all objects saved in `BlobStor`.

Implement `BlobStor.Iterate` method which iterates over all objects.
Implement `IterateBinaryObjects` and `IterateObjects` helper functions to
simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
b618a44d69 [#789] blobovnicza: Implement iterator
There is a need to be able to process all stored objects saved in
`Blobovnicza`.

Implement `Blobovnicza.Iterate` method which iterates over all objects.
Implement `IterateObjects` helper function to simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
55c94a0152 [#789] metabase: Implement Reset method
In the previous implementation of the metabase, there was no possibility of
reinitializing the metabase: clearing information about existing objects and
bringing it back to its initial state. This operation can be useful in
cases when the stored metadata about objects has lost (or possibly lost)
relevance, and you need to generate data from scratch. Also at the
initialization stage, static resources of the base were not created -
container-independent buckets.

Make `Metabase.Init` method to allocate graveyard, container-size and
to-move-it buckets in underlying BoltDB instance. Implement `Metabase.Reset`
method: it works like `Init` but clean up all static buckets and removes
other ones. Due to the logical similarity, the methods share a single piece
of code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Pavel Karpy
7a15e649ba [#825] event/notaryPreparator: Add unit test for incorrect Main TX script
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Pavel Karpy
1b4f8f7e9f [#825] event/notaryPreparator: Add unit test for incorrect NR
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Pavel Karpy
5d791a4366 [#825] morph/event: Fix commentary
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Pavel Karpy
f0beb4abc9 [#825] event/notaryPreparator: Add args checking unit test
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Leonard Lyubich
368e280413 [#776] cmd/node: Configure write-cache size limit
Add `size_limit` config value to write-cache section. Add `SizeLimit` method
of `writecache.Config` type. Pass its value to `WithMaxCacheSize` option on
app construction.

Reflect config update in examples. Cover new value in unit test.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:07:36 +03:00
Leonard Lyubich
a1696a81b6 [#776] writecache: Limit size of used disk space
There is a need to limit disk space used by write-cache. It is almost
impossible to calculate the value exactly. It is proposed to estimate the
size of the cache by the number of objects stored in it.

Track amounts of objects saved in DB and FSTree separately. To do this,
`ObjectCounters` interface is defined. It is generalized to a store of
numbers that can be made persistent (new option `WithObjectCounters`). By
default DB number is calculated as key number in default bucket, and FS
number is set same to DB since it is currently hard to read the actual value
from `FSTree` instance. Each PUT/DELETE operation to DB or FS
increases/decreases corresponding counter. Before each PUT op an overflow
check is performed with the following formula for evaluating the occupied
space: `NumDB * MaxDBSize + NumFS * MaxFSSize`. If next PUT can cause
write-cache overflow, object is written to the main storage.

By default maximum write-cache size is set to 1GB.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:07:36 +03:00
Stanislav Bogatyrev
0a130177d6 [#725] Simplify Makefile a little
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2021-09-15 17:58:58 +03:00
Stanislav Bogatyrev
03b5aad2e4 [#785] Remove unused layer from builder images
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
e950dfe44f [#785] docker: Do not use Alpine image for building bins
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
71df291cda [#785] docs: Add updating Go instructions
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
80df695a25 [#785] make: Add image-storage-testnet to images target deps
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
654a3a6e60 [#785] docs: Actualize release instructions
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
94f3937c0e [#785] make: Add preparing release binaries
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
9fa4a254b2 [#785] make: Add docker/* target wrapper
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Leonard Lyubich
3258d9c616 [#790] writecache: Log each writing operation
Call `storagelog.Write` in all places after the successful writing op.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Leonard Lyubich
4f73c00776 [#790] blobstor: Log each writing operation
Call `storagelog.Write` in all places after the successful writing op.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Leonard Lyubich
3cfb58aabd [#790] storage engine: Add common template of log messages
There is a need to keep track of each local storage change. Log messages are
the most convenient way to do it.

Implement function which writes log message about the completed writing
operation in storage engine.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Alex Vanin
d840627816 [#786] cmd/neofs-node: Use NNS to find contract script hashes missing in config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin
b6dfa6c118 [#786] pkg/innerring: Use NNS to find contract script hashes missing in config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin
f43f389399 [#786] morph/client: Add NNS contract address getter
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin
affae68061 [#812] cmd/neofs-node: Support notary notifications in reputation service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
dc26a09ec3 [#812] pkg/innerring: Support notary notifications in reputation processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
a7f6a3df78 [#812] morph/client: Add underline client getter in reputation wrapper
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
e2c2e27c60 [#812] morph/event: Expand reputation put event with notary notification
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
90823a7d05 [#812] cmd/neofs-node: Fix reputation wrapper import typo
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Pavel Karpy
288a8784d0 [#815] node: Make notary deposit before bootstrap
Bootstrap process involves `addPeer` contract
call => it is necessary to make deposit before
booting up.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Pavel Karpy
b303e49408 [#815] morph/event/netmap: Add updateState notary notification support
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Pavel Karpy
74afcbe409 [#815] pkg/morph/netmap: Add Morph() method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Pavel Karpy
00caed8d3d [#815] morph/event/netmap: Add addPeer notary notification support
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Alex Vanin
01df4ffa61 [#811] service/container: Hide cache invalidation logic in Writer interface
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Alex Vanin
49b6b5b49d [#811] cmd/node: Use cache invalidator in container service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Alex Vanin
4a1f0de8f4 [#811] service/container: Invalidate cache records on update requests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Pavel Karpy
a8d41f596d [#822] event/notaryPreparator: Add unit test for correct NR
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 10:32:43 +03:00
Pavel Karpy
3b9ef4f63c [#822] event/notaryPreparator: Do not pass PACK opcode
Do not pass high level `PACK` opcode to
notary parsers. Add opcode amount check.
Delete `PACK` cases in notary parsers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 10:32:43 +03:00
Pavel Karpy
4f3de1a9af [#822] moprh/event: Add parsers from Op struct
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 10:32:43 +03:00
Leonard Lyubich
61b4baf736 [#778] services/object: Wrap last client's error into errIncompletePut
Make `errIncompletePut` to be a structure which wraps single client error.
Wrap error of the last client into `errIncompletePut` during placement
execution.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 16:49:37 +03:00
Leonard Lyubich
02f2a98bcc [#778] services/object: Return the composite error of PUT operation
In previous implementation Object service's handler returned const error in
case of failure (full or partial) of PUT operation. This did not even allow
us to roughly guess what the reason is. Not as a complete solution, but to
alleviate some cases where all nodes in a container return the same error,
it is suggested to return the error of the last server that responded.

Return latest server error from placement loop of `iteratePlacement` method
of `distributedTarget` type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 16:49:37 +03:00
Evgenii Stratonikov
5f86d54721 [#816] object/acl: fix eACL target processing
Ignore role if public keys are present.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-09-10 15:15:36 +03:00
Leonard Lyubich
7a13053fab [#818] object/acl: Fit sticky bit specification
In previous implementation sticky bit could disrupt access of container
nodes to replication. According to NeoFS specification sticky bit should not
affect the requests sent by nodes from SYSTEM group.

Add role check to `stickyBitCheck`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 15:14:59 +03:00
Pavel Karpy
ba77bb44e4 [#807] ir: Merge ContractProcessor and NotaryContractProcessor interfaces
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
539da27ccb [#807] morph/event/container: Add setEACL notary support
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
e03b44ffc1 [#807] morph/event/container: Add delete notary support
Includes:
- `delete` container notary notification;
- refactor in processor bindings;
- fixed conflicts in `put` notifications.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
19e97e4d7c [#807] node: Fix notary container invocation
Create non-notary container wrapper and
use it in load announcing since it shouldn't
be invoked as notary request.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00