- Delete objects physically on tombstone's arrival;
- Store information about tombstones in the Graveyard;
- Clear Graveyard every epoch based on the information about TS in the
network.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
The service fetches tombstones from the network via object service, every
request is handled in the following order:
1. checks local LRU cache;
2. checks local storage engine;
3. tries to find object in the placement nodes.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Add offset element to the iterations over deleted objects (both the
Graveyard and the Garbage buckets).
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
It allows storing information about object in both ways at the same time:
1. Metabase should know if an object is covered by a tombstone (that is
not expired yet);
2. It should be possible to physically delete objects covered by a
tombstone immediately (mark with GC) but keep tombstone knowledge.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Morph "NewEpoch" event handling was registered in a closure over
`addNewEpochNotificationHandler` func. That may lead to the data race:
if a shard was initialized before the event registration, everything works
as planned, but if registration was made earlier, it was not able to
include GC handlers since a shard has not called `eventChanInit` yet and,
therefore, it has not registered handler yet.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Add worker pool to the listener to prevent blocking. It is used only for
notary notifications and new block events handling since it uses RPC
calls. That may lead to the deadlock state: neo-go cannot send RPC until
notification channel is read but notification channel cannot be read since
neo-go client cannot send RPC.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Updated client now supports subscription to chain notifications and RPC
switch between provided RPC endpoints.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Provide current heights as an argument to ticker.
Zero height disables any checks, thus corresponding to the old
behaviour. If non-zero height is used, ignore the tick if the height
is less than the timer tick state.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
After `Listen` has raw receiver and calls `listen` which has pointer
receiver, leading to ineffectual assign to `started` field.
Always use `listener` by pointer, to avoid similar bugs.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
When IR node configured without main chain, both
`morphListener` and `mainnetListener` are pointing
into single listener component. We should not call
`Stop()` twice, because it may trigger channel
closing in neo-go or other components and it
can throw panic.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Prevent attemts to connect to the failing endpoint multiple times.
This speeds up all invocations a bit and removes excessive error logs.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Also, remove optimization comments:
1. Having to maintain an execute the same logic for headers as for
objects is quite inefficient, as it increases memory footprint.
2. Unmarshaling object is a cheap operation if data slice is in memory.
3. For unmarshaling header-only, I think we need SDK support.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
`Degraded` mode is set automatically after error counter is over the
threshold. `ReadOnly` mode can still be set by an administrator.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Add `WithEncryption` option that passes ECDSA key to the persistent session
storage. It uses 32 bytes from marshalled ECDSA key in ASN.1 DER from in
AES-256 algorithm encryption in Galois/Counter Mode.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Move in-memory session storage to the separate directory of `storage`. It is
done for future support of different kind of session storages.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
`Batch` can execute the function multiple times leading to multiple
increases of a size approximation.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Alphabet nodes in notary enabled environment cannot call `UpdateState`
method to remove unwanted storage nodes from the network map,
because this method checks witness of the storage node.
To force storage node state update, alphabet nodes should invoke
new method `UpdateStateIR` which is similar to `AddPeerIR`.
State update initiated by the storage node itself is processed
the same way as before -- alphabet nods resign such transaction.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
`Register` was renamed to `AddPeerIR` for consistency with
`UpdateState` changes in
https://github.com/nspcc-dev/neofs-contract/pull/227
This is protocol breaking change for notary enabled environment.
Luckily, there is no notary enabled environments anywhere except
of neofs-dev-env, so we can do such thing. We should avoid such
changes in the future, though.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
`apistatus` package provides types which implement build-in `error`
interface. Add `error of type` pattern when documenting these errors in
order to clarify how these errors should be handled (e.g. `errors.Is` is
not good).
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Return `apistatus.ObjectAccessDenied` error on access violation from ACL
service. Write reason in format of the errors from the previous
implementation. These errors are returned by storage node's server as
NeoFS API statuses.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Replace `core/container.ErrNotFound` error returned by `Source.Get`
interface method with `apistatus.ContainerNotFound` status error. This
error is returned by storage node's server as NeoFS API statuses.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Replace `ErrNotFound`/`ErrAlreadyRemoved` error from
`pkg/core/object` package with `ObjectNotFound`/`ObjectAlreadyRemoved`
one from `apistatus` package. These errors are returned by storage
node's server as NeoFS API statuses.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Return `SessionTokenExpired`/`SessionTokenNotFound` error from
`apistatus` package if private session token is expired/missing. These
errors are returned by storage node's server as NeoFS API statuses.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make all epoch independent in reputation process. Do not reset any timers
related to reputation. Make it possible to finish iteration after the
unexpected `NewEpoch` event.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
There is a need to process expired `LOCK` objects similar to `TOMBSTONE`
ones: we collect them on `Shard`, notify all other shards about
expiration so they could unlock the objects, and only after that mark
lockers as garbage.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation `SignService` converted all `error` values to
`INTERNAL` server failure status. That was done for simplification only.
There is a need to transmit status errors as corresponding status
messages.
Make `SignService` to unwrap errors and convert them to status message
during writing to the response. Non-status errors are converted to
`INTERNAL` server failures. Status errors can also be wrapped in the
depths of the executable code, so `SignService` tries to unwrap them.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make `FormatValidator.ValidateContent` to verify payload of `LOCK`
objects. Pass locked objects to `Locker` interface. Require from
`Locker.Lock` to return `apistatus.IrregularObjectLock` error on a
corresponding condition.
Also add error return to `DeleteHandler.DeleteObjects` method. Require
from method to return `apistatus.ObjectLocked` error on a corresponding
condition. Adopt implementations.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
`Inhume` operation can potentially mark lockers as garbage. There is a
need to update locker list in locked bucket.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make `StorageEngine.Delete` to forward first encountered
`apistatus.ObjectLocked` error during shard processing.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make `StorageEngine.Inhume` to forward first encountered
`apistatus.ObjectLocked` error during shard processing.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement `StorageEngine.Lock` method which works similar to `Inhume`
but calls `Lock` on the processing shards.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make `DB.Lock` to return `apistatus.IrregularObjectLock` if at least one
of the locked objects is irregular (not of type REGULAR).
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make `DB.Inhume` to return `apistatus.ObjectLocked` if at least one of
the inhumed objects is locked.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make `DB.IterateCoveredByTombstones` to not pass locked objects to the
handler. The method is used by GC, therefore it will not consider locked
objects as candidates for deletion even if their tombstone is expired.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make `DB.IterateExpired` to not pass locked objects to the handler. The
method is used by GC, therefore it will not consider them as candidates
for deletion.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
After introduction of LOCK objects (of type `TypeLock`) complicated
extended its behavior:
* create `lockers` container bucket (LCB) during PUT;
* remove object from LCB during DELETE;
* look up object in LCB during EXISTS;
* get object from LCB during GET;
* list objects from LCB during LIST with cursor;
* select objects from LCB during SELECT with '*'.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement `DB.Lock` method which marks list of the objects as locked by
another object. Only regular objects can be locked.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Create class of container buckets with `LOCKED` suffix. Put identifiers
of the objects of type `LOCK` to these buckets during `DB.Put`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
After `putsvc.Service` started to support additional container broadcast
of the saved objects there is no more need to perform broadcast of
tombstone object in `deletesvc.Service`.
Make `putsvc.Service` to perform additional broadcast of `TOMBSTONE`
objects. Remove `broadcastTombstone` stage from `deletesvc.execCtx`,
from now it is encapsulated in `saveTombstone` stage. Remove no longer
needed `putsvc.PutInitPrm.WithTraverseOption` method.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
There are several cases when we need to spread the object around the
container after its primary placement (e.g. objects of type TOMBSTONE).
It'd be convenient to support this feature in `putsvc.Service`.
Add additional stage of container broadcast after the object is stored.
This stage is carried out no more than once and does not affect the
outcome of the main PUT operation.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make all operations that related to `neofs-api-go` library be placed in `v2`
packages. They parse all v2-versioned structs info `neofs-sdk-go`
abstractions and pass them to the corresponding `acl`/`eacl` packages. `v2`
packages are the only packages that do import `neofs-api-go` library. `eacl`
and `acl` provide public functions that only accepts `sdk` structures.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Remove `Object` and `RawObject` types from `pkg/core/object` package.
Use `Object` type from NeoFS SDK Go library everywhere. Avoid using the
deprecated elements.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Similarly to `Get`. Also fix a bug where `ErrNotFound` is returned
instead of `ErrRangeOutOfBounds`.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Metabase is expected to contain actual information about objects stored
in shard. If the object is present in metabase but is missing from
blobstor, peform an additional attempt to fetch it directly without
consulting metabase. Such a situation is unexpected, so error counter
is increased for the shard which has the object in the metabase. We
don't increase error counter for the shard which has the object in
blobstor, because some garbage can be expected there. In this
implementation there is no overhead for objects which are really
missing, i.e. are not present in any metabase.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
There are certain errors which are not expected during usual node
operation and which tell us that something is wrong with the shard.
To prevent possible data corruption, move shard in read-only mode after
amount of errors exceeded some threshold. By default no actions are performed.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
In some places we panic, in some we return error, in some (audit) just return a client.
However in all of the places static client is created immediately before
the sugared-client creation.
This commit makes all constructors to just return a client for the sake
of code simplification and unification.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
We don't use custom names and the only place where custom method option
is used it provides the default name and can be omitted.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
If pre-existing blobovnicza is initialized, it's size should be updated
even if all buckets are in place.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
It was added back in 2fb379b7 when we had many shard modes. Now we have
only two and comments for constants are rather descriptive.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
In previous implementation IR incorrectly verified `SetEACL` event of
`Container` contract. The incorrect behavior could be reproduced in two
ways:
1. Create container using session, and perform `SetEACL` operation
with a key that is different from the session one.
2. Create container using session, and perform `SetEACL` w/o a
session, but sign it using session key from the `Put` operation.
The problem was in the `checkSetEACL` validation method of IR container
processor. It always used session token used for container creation
during session ownership check.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
We could also ignore errors during evacuate, but this requires
unmarshaling objects first which slowers the process considerably.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
In read-only mode modifying operations are immediately returned with
error and all background operations are suspended.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Dump contains magic and a list of objects prefixed by object size in bytes.
We can't use proto-marshaled list because this requires having all dump
in memory. Using TAR induces 512 byte overhead for each object which can
be a problem in some cases.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
- Update `neofs-sdk-go`:
v0.0.0-20211230072947-1fe37df88f80 => v0.0.0-20220113123743-7f3162110659
- Add client interface that duplicates SDK's client behaviour and new
`MultiAddressClient` interface that has method that iterates over wrapped
clients.
- Also start using simple client mode that does not require parsing statuses
outside the SDK library.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
For some data compression makes little sense, as it is already compressed.
This commit allows to leave such data unchanged based on `Content-Type`
attribute. Currently exact, prefix and suffix matching are supported.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Reverse payload overtake triggers direct payload overtake that
sets status and error. We should not override that.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
It is much more convenient to skip source creation.
Also fix some bugs:
1. `cryptoSource.Int63()` now returns number in [0, 1<<63) as required
by `rand.Source` interface.
2. Replace `cryptoSource.Uint63()` with `cryptoSource.Uint64` to allow
generate uint64 numbers directly (see rand.Source64 docs).
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Indent sections corresponding to different shells for faster navigation
and remove `#` at the line beginning.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
If mainchain is disabled in IR config then the node should read inner
ring list via role management contract.
Use `NeoFSAlphabetList` method of morph client as IR lister if
`withoutMainNet` flag is set in IR app.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation `Search` method of transport splitter skipped
responses with empty ID list.
Replace while-loop with do-while one in `TransportSplitter.Search`
method implementation in order to send responses with empty result too.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Provide shard mode information via `DumpInfo()`. Delete atomic field from
Shard structure since it duplicates new field.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Shard's mode was not used in the Node, so added only two modes whose roles
are clear. More modes will be added in the future.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Without sanity check, container service provides successful response,
even though such request will never be approved by Alphabet nodes.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
When application is being terminated, replicator routine
might be on the object picking phase. Storage is terminated
asynchronously, thus `Select()` may return corresponding
error. If we don't process `context.Done()` in this case,
then application freezes on shutdown.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Make `flushBigObjects` routine to mark objects which are written to
`BlobStor`. This prevents already flushed objects from being written on
the next iterator tick.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
For fullness estimation of `Blobovnicza` we use number of object stored
in each size bucket. In previous implementation we multiplied the number
by the difference in bucket boundaries. This expression rather
estimated the minimum volume (and for the smallest bucket, the maximum)
of objects in the bucket.
Multiply number of objects by mean bucket size.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make `syncFullnessCounter` to accept `bbolt.Tx` argument of Bolt
transaction within which counter should be synchronized. Pass
corresponding transaction during `Init`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In notary disabled environment, approval of container creation with nice
name attribute takes much more additional GAS than other operations
(due to NNS invocation).
Morph library changes:
* add the ability to specify per-op fees using `StaticClient` options;
* add the ability to customize fee for `Put` operation with named
container in container morph client.
Inner Ring changes:
* add `fee.named_container_register` config value which specifies
additional GAS fee for the approvals of the named container
registrations;
* pass the config value to `WithCustomFeeForNamedPut` option of
container morph client.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
According to BoltDB documentation bucket `value is only valid for the
life of the transaction`.
Make `DB.IsSmall` copy value slice in order to prevent potential memory
corruptions (e.g. `runtime.stringtobyteslice` cast).
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
After https://github.com/nspcc-dev/neofs-contract/issues/154 alphabet
nodes should call `Register` method for approval of the notary
notifications spawned by `AddPeer` method.
Call `register` method for peer approval in Netmap processor.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Notary event name equals to the name of the method which throws the
event.
Define name const of notary subnet creation.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Use persistent flags on parent command in order to inherit flags in
sub-commands. Turn on notary mode of morph client in `subnet` command of
admin utility for notary environments.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make `Client.EnableNotarySupport` method to call `NNSContractAddress`
for proxy contract if it is not specified in corresponding option.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add `subnet` command which contains all subnet-related commands. Add
sub-commands:
* `create` for creation;
* `remove` for removal;
* `get` for reading;
* `admin` for admin management;
* `client` for client management.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Define notification events, implement parsers. Add morph client of
Subnet contract. Listen, verify and approve events in Inner Ring app.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Continues replication executed in separate pool of goroutines,
so there is no need in worker to handle replication tasks
asynchronously.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Upgrade NeoFS API Go library to version with status returns. Make all API
clients to pull out and return errors from failed statuses. Make signature
service to respond with status if client version supports it.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
ListWithCursor allows listing physically stored objects
from metabase with small chunks. Cursor tracks last
processed object, therefore new chunks are returned
on each request.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Add hash of the TX that generated notification
to neofsid event structures. Adapt all
neofsid wrapper calls to new structures.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Add hash of the TX that generated notification
to neofs/netmap event structures. Adapt all
neofs/netmap wrapper calls to new structures.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Add optional parameters to the client call
signature. Group parameters of a client call
into struct to improve future codebase
support.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Add optional parameters to the client call
signature. Group parameters of a client call
into struct to improve future codebase
support.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
