Commit Graph

1793 Commits (264ab489bb563fcac5685cefcbc717cb31375318)

Author SHA1 Message Date
Leonard Lyubich 264ab489bb [#848] ir/container: Prevent potential NPE in key conversion
Set `Curve` field in `ecdsa.PublicKey` instance from `keys.PublicKey` one in
`checkKeyOwnership` method of container processor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-24 18:57:00 +03:00
Alex Vanin 3e5c7e0ade [#836] neofs-adm: Fix proxy contract deploy arguments
Proxy contract does not include notary flag because
contract is useless without notary subsystem.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-23 09:55:46 +03:00
Alex Vanin 4c30757439 [#836] neofs-adm: Do not define contract owner
Contract owners are removed in neofs-contract v0.11.0.
Now side chain committee has rights to update contracts.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-23 09:55:46 +03:00
Alex Vanin ce8a906bb5 [#834] neofs-adm: Update NNS contract during contract update
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-22 18:23:59 +03:00
Alex Vanin d996004d80 [#834] neofs-adm: Use committee signature when updating alphabet contract
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-22 18:23:59 +03:00
Alex Vanin 6b3a9e6fcc [#834] neofs-adm: Update contract update if it exists
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-22 18:23:59 +03:00
Alex Vanin 60636d4c1d [#834] neofs-adm: Throw error if deploy produce non-HALT state
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-22 18:23:59 +03:00
Leonard Lyubich 504abdda06 [#789] shard: Exclude object.ErrAlreadyRemoved errors in refillMetabase
Tombstone and "alive" objects can be both stored in BlobStor. They can
appear during iterating in different order. Metabase returns
`ErrAlreadyRemoved` error if object is inhumed.

Ignore `object.ErrAlreadyRemoved` errors of `metabase.Put`in Shard's
`refillMetabase` operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich d938c7267b [#789] node/config: Add Shard's `refill_metabase` config to examples
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich b5fff810f4 [#789] cmd/node: Add `refill_metabase` config to `shard` section
Implement `RefillMetabase` method to `shardconfig.Config` type which reads
`refill_metabase` config value. Pass the result to `WithRefillMetabase`
option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich 6bf7a00cfe [#789] shard: Add option to refill metabase on initialization
Add `WithRefillMetabase` option constructor which allows to set flag to
refill metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich b9c22e21b1 [#789] shard: Add method to refill the metabase
There is a need to refill Metabase data with the objects from BlobStor.

Implement `refillMetabase` method which iterates over all objects from
BlobStor and saves them in Metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich 8d016d2529 [#789] blobstor: Implement iterator
There is a need to be able to process all objects saved in `BlobStor`.

Implement `BlobStor.Iterate` method which iterates over all objects.
Implement `IterateBinaryObjects` and `IterateObjects` helper functions to
simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich b618a44d69 [#789] blobovnicza: Implement iterator
There is a need to be able to process all stored objects saved in
`Blobovnicza`.

Implement `Blobovnicza.Iterate` method which iterates over all objects.
Implement `IterateObjects` helper function to simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich 55c94a0152 [#789] metabase: Implement Reset method
In the previous implementation of the metabase, there was no possibility of
reinitializing the metabase: clearing information about existing objects and
bringing it back to its initial state. This operation can be useful in
cases when the stored metadata about objects has lost (or possibly lost)
relevance, and you need to generate data from scratch. Also at the
initialization stage, static resources of the base were not created -
container-independent buckets.

Make `Metabase.Init` method to allocate graveyard, container-size and
to-move-it buckets in underlying BoltDB instance. Implement `Metabase.Reset`
method: it works like `Init` but clean up all static buckets and removes
other ones. Due to the logical similarity, the methods share a single piece
of code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Pavel Karpy 7a15e649ba [#825] event/notaryPreparator: Add unit test for incorrect Main TX script
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Pavel Karpy 1b4f8f7e9f [#825] event/notaryPreparator: Add unit test for incorrect NR
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Pavel Karpy 5d791a4366 [#825] morph/event: Fix commentary
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Pavel Karpy f0beb4abc9 [#825] event/notaryPreparator: Add args checking unit test
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Leonard Lyubich 368e280413 [#776] cmd/node: Configure write-cache size limit
Add `size_limit` config value to write-cache section. Add `SizeLimit` method
of `writecache.Config` type. Pass its value to `WithMaxCacheSize` option on
app construction.

Reflect config update in examples. Cover new value in unit test.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:07:36 +03:00
Leonard Lyubich a1696a81b6 [#776] writecache: Limit size of used disk space
There is a need to limit disk space used by write-cache. It is almost
impossible to calculate the value exactly. It is proposed to estimate the
size of the cache by the number of objects stored in it.

Track amounts of objects saved in DB and FSTree separately. To do this,
`ObjectCounters` interface is defined. It is generalized to a store of
numbers that can be made persistent (new option `WithObjectCounters`). By
default DB number is calculated as key number in default bucket, and FS
number is set same to DB since it is currently hard to read the actual value
from `FSTree` instance. Each PUT/DELETE operation to DB or FS
increases/decreases corresponding counter. Before each PUT op an overflow
check is performed with the following formula for evaluating the occupied
space: `NumDB * MaxDBSize + NumFS * MaxFSSize`. If next PUT can cause
write-cache overflow, object is written to the main storage.

By default maximum write-cache size is set to 1GB.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:07:36 +03:00
Stanislav Bogatyrev 0a130177d6 [#725] Simplify Makefile a little
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2021-09-15 17:58:58 +03:00
Stanislav Bogatyrev 03b5aad2e4 [#785] Remove unused layer from builder images
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy e950dfe44f [#785] docker: Do not use Alpine image for building bins
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy 71df291cda [#785] docs: Add updating Go instructions
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy 80df695a25 [#785] make: Add `image-storage-testnet` to `images` target deps
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy 654a3a6e60 [#785] docs: Actualize release instructions
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy 94f3937c0e [#785] make: Add preparing release binaries
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy 9fa4a254b2 [#785] make: Add `docker/*` target wrapper
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Leonard Lyubich 3258d9c616 [#790] writecache: Log each writing operation
Call `storagelog.Write` in all places after the successful writing op.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Leonard Lyubich 4f73c00776 [#790] blobstor: Log each writing operation
Call `storagelog.Write` in all places after the successful writing op.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Leonard Lyubich 3cfb58aabd [#790] storage engine: Add common template of log messages
There is a need to keep track of each local storage change. Log messages are
the most convenient way to do it.

Implement function which writes log message about the completed writing
operation in storage engine.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Alex Vanin d840627816 [#786] cmd/neofs-node: Use NNS to find contract script hashes missing in config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin b6dfa6c118 [#786] pkg/innerring: Use NNS to find contract script hashes missing in config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin f43f389399 [#786] morph/client: Add NNS contract address getter
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin affae68061 [#812] cmd/neofs-node: Support notary notifications in reputation service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin dc26a09ec3 [#812] pkg/innerring: Support notary notifications in reputation processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin a7f6a3df78 [#812] morph/client: Add underline client getter in reputation wrapper
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin e2c2e27c60 [#812] morph/event: Expand reputation put event with notary notification
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin 90823a7d05 [#812] cmd/neofs-node: Fix reputation wrapper import typo
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Pavel Karpy 288a8784d0 [#815] node: Make notary deposit before bootstrap
Bootstrap process involves `addPeer` contract
call => it is necessary to make deposit before
booting up.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Pavel Karpy b303e49408 [#815] morph/event/netmap: Add `updateState` notary notification support
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Pavel Karpy 74afcbe409 [#815] pkg/morph/netmap: Add `Morph()` method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Pavel Karpy 00caed8d3d [#815] morph/event/netmap: Add `addPeer` notary notification support
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Alex Vanin 01df4ffa61 [#811] service/container: Hide cache invalidation logic in Writer interface
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Alex Vanin 49b6b5b49d [#811] cmd/node: Use cache invalidator in container service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Alex Vanin 4a1f0de8f4 [#811] service/container: Invalidate cache records on update requests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Pavel Karpy a8d41f596d [#822] event/notaryPreparator: Add unit test for correct NR
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 10:32:43 +03:00
Pavel Karpy 3b9ef4f63c [#822] event/notaryPreparator: Do not pass `PACK` opcode
Do not pass high level `PACK` opcode to
notary parsers. Add opcode amount check.
Delete `PACK` cases in notary parsers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 10:32:43 +03:00
Pavel Karpy 4f3de1a9af [#822] moprh/event: Add parsers from Op struct
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 10:32:43 +03:00