Commit graph

2202 commits

Author SHA1 Message Date
Leonard Lyubich
14329ab565 [#840] metabase: Distinguish objects with tombstones and GC marks
Each object from graveyard has tombstone or GC mark. If object has
tombstone, metabase should return `ErrAlreadyRemoved` on object requests.
This is the case when user clearly removed the object from container. GC
marks are used for physical removal which can appear even if object is still
presented in container (Control service, Policer job, etc.). In this case
metabase should return 404 error on object requests.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Leonard Lyubich
02e6df683a [#840] shard: Return only physically stored objects from List
`List` method of `Shard` must return only physically stored objects.

Use `AddPhyFilter` to select only phy objects.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Alex Vanin
b97cdbea9e [#839] service/object: Add high level functions to control request forwarding
Applicable for object/get execution context.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-27 11:10:50 +03:00
Alex Vanin
b3464e8140 [#839] service/object: Do not use request forwarding when assembling object
Forwarding mechanism resends original request. During split object chain traversal,
storage node performs multiple `object.Head` requests on each child. If request
forwarding happens, then `object.Head` returns object ID of the original request.
This produces infinite assembly loop.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-27 11:10:50 +03:00
Pavel Karpy
057d3ac06b [#846] innerring: Reuse original TX nonce from SN's NR
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-27 11:10:15 +03:00
Pavel Karpy
5702349cb2 [#846] innerring: Use epoch as nonce in cleanup netmap process
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-27 11:10:15 +03:00
Pavel Karpy
e3c0288e50 [#846] morph/notary: Add nonce parameter to notary invocation method
This prevents notary requests collisions
for TXs that contains equals hashable fields.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-27 11:10:15 +03:00
Pavel Karpy
3a8f0edac1 [#846] morph/event: Add notary request to addPeer event
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-27 11:10:15 +03:00
Pavel Karpy
5e74830c38 [#837] morph: Add WithSingleClient client constructor option
`WithSingleClient` allows Morph client
creation with existing raw neo-go client.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-24 19:14:28 +03:00
Leonard Lyubich
264ab489bb [#848] ir/container: Prevent potential NPE in key conversion
Set `Curve` field in `ecdsa.PublicKey` instance from `keys.PublicKey` one in
`checkKeyOwnership` method of container processor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-24 18:57:00 +03:00
Alex Vanin
3e5c7e0ade [#836] neofs-adm: Fix proxy contract deploy arguments
Proxy contract does not include notary flag because
contract is useless without notary subsystem.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-23 09:55:46 +03:00
Alex Vanin
4c30757439 [#836] neofs-adm: Do not define contract owner
Contract owners are removed in neofs-contract v0.11.0.
Now side chain committee has rights to update contracts.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-23 09:55:46 +03:00
Alex Vanin
ce8a906bb5 [#834] neofs-adm: Update NNS contract during contract update
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-22 18:23:59 +03:00
Alex Vanin
d996004d80 [#834] neofs-adm: Use committee signature when updating alphabet contract
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-22 18:23:59 +03:00
Alex Vanin
6b3a9e6fcc [#834] neofs-adm: Update contract update if it exists
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-22 18:23:59 +03:00
Alex Vanin
60636d4c1d [#834] neofs-adm: Throw error if deploy produce non-HALT state
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-22 18:23:59 +03:00
Leonard Lyubich
504abdda06 [#789] shard: Exclude object.ErrAlreadyRemoved errors in refillMetabase
Tombstone and "alive" objects can be both stored in BlobStor. They can
appear during iterating in different order. Metabase returns
`ErrAlreadyRemoved` error if object is inhumed.

Ignore `object.ErrAlreadyRemoved` errors of `metabase.Put`in Shard's
`refillMetabase` operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
d938c7267b [#789] node/config: Add Shard's refill_metabase config to examples
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
b5fff810f4 [#789] cmd/node: Add refill_metabase config to shard section
Implement `RefillMetabase` method to `shardconfig.Config` type which reads
`refill_metabase` config value. Pass the result to `WithRefillMetabase`
option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
6bf7a00cfe [#789] shard: Add option to refill metabase on initialization
Add `WithRefillMetabase` option constructor which allows to set flag to
refill metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
b9c22e21b1 [#789] shard: Add method to refill the metabase
There is a need to refill Metabase data with the objects from BlobStor.

Implement `refillMetabase` method which iterates over all objects from
BlobStor and saves them in Metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
8d016d2529 [#789] blobstor: Implement iterator
There is a need to be able to process all objects saved in `BlobStor`.

Implement `BlobStor.Iterate` method which iterates over all objects.
Implement `IterateBinaryObjects` and `IterateObjects` helper functions to
simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
b618a44d69 [#789] blobovnicza: Implement iterator
There is a need to be able to process all stored objects saved in
`Blobovnicza`.

Implement `Blobovnicza.Iterate` method which iterates over all objects.
Implement `IterateObjects` helper function to simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
55c94a0152 [#789] metabase: Implement Reset method
In the previous implementation of the metabase, there was no possibility of
reinitializing the metabase: clearing information about existing objects and
bringing it back to its initial state. This operation can be useful in
cases when the stored metadata about objects has lost (or possibly lost)
relevance, and you need to generate data from scratch. Also at the
initialization stage, static resources of the base were not created -
container-independent buckets.

Make `Metabase.Init` method to allocate graveyard, container-size and
to-move-it buckets in underlying BoltDB instance. Implement `Metabase.Reset`
method: it works like `Init` but clean up all static buckets and removes
other ones. Due to the logical similarity, the methods share a single piece
of code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Pavel Karpy
7a15e649ba [#825] event/notaryPreparator: Add unit test for incorrect Main TX script
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Pavel Karpy
1b4f8f7e9f [#825] event/notaryPreparator: Add unit test for incorrect NR
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Pavel Karpy
5d791a4366 [#825] morph/event: Fix commentary
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Pavel Karpy
f0beb4abc9 [#825] event/notaryPreparator: Add args checking unit test
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 18:07:50 +03:00
Leonard Lyubich
368e280413 [#776] cmd/node: Configure write-cache size limit
Add `size_limit` config value to write-cache section. Add `SizeLimit` method
of `writecache.Config` type. Pass its value to `WithMaxCacheSize` option on
app construction.

Reflect config update in examples. Cover new value in unit test.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:07:36 +03:00
Leonard Lyubich
a1696a81b6 [#776] writecache: Limit size of used disk space
There is a need to limit disk space used by write-cache. It is almost
impossible to calculate the value exactly. It is proposed to estimate the
size of the cache by the number of objects stored in it.

Track amounts of objects saved in DB and FSTree separately. To do this,
`ObjectCounters` interface is defined. It is generalized to a store of
numbers that can be made persistent (new option `WithObjectCounters`). By
default DB number is calculated as key number in default bucket, and FS
number is set same to DB since it is currently hard to read the actual value
from `FSTree` instance. Each PUT/DELETE operation to DB or FS
increases/decreases corresponding counter. Before each PUT op an overflow
check is performed with the following formula for evaluating the occupied
space: `NumDB * MaxDBSize + NumFS * MaxFSSize`. If next PUT can cause
write-cache overflow, object is written to the main storage.

By default maximum write-cache size is set to 1GB.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:07:36 +03:00
Stanislav Bogatyrev
0a130177d6 [#725] Simplify Makefile a little
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2021-09-15 17:58:58 +03:00
Stanislav Bogatyrev
03b5aad2e4 [#785] Remove unused layer from builder images
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
e950dfe44f [#785] docker: Do not use Alpine image for building bins
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
71df291cda [#785] docs: Add updating Go instructions
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
80df695a25 [#785] make: Add image-storage-testnet to images target deps
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
654a3a6e60 [#785] docs: Actualize release instructions
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
94f3937c0e [#785] make: Add preparing release binaries
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Pavel Karpy
9fa4a254b2 [#785] make: Add docker/* target wrapper
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-15 17:58:58 +03:00
Leonard Lyubich
3258d9c616 [#790] writecache: Log each writing operation
Call `storagelog.Write` in all places after the successful writing op.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Leonard Lyubich
4f73c00776 [#790] blobstor: Log each writing operation
Call `storagelog.Write` in all places after the successful writing op.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Leonard Lyubich
3cfb58aabd [#790] storage engine: Add common template of log messages
There is a need to keep track of each local storage change. Log messages are
the most convenient way to do it.

Implement function which writes log message about the completed writing
operation in storage engine.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Alex Vanin
d840627816 [#786] cmd/neofs-node: Use NNS to find contract script hashes missing in config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin
b6dfa6c118 [#786] pkg/innerring: Use NNS to find contract script hashes missing in config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin
f43f389399 [#786] morph/client: Add NNS contract address getter
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin
affae68061 [#812] cmd/neofs-node: Support notary notifications in reputation service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
dc26a09ec3 [#812] pkg/innerring: Support notary notifications in reputation processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
a7f6a3df78 [#812] morph/client: Add underline client getter in reputation wrapper
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
e2c2e27c60 [#812] morph/event: Expand reputation put event with notary notification
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
90823a7d05 [#812] cmd/neofs-node: Fix reputation wrapper import typo
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Pavel Karpy
288a8784d0 [#815] node: Make notary deposit before bootstrap
Bootstrap process involves `addPeer` contract
call => it is necessary to make deposit before
booting up.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00