CLI `storagegroup put` cmd collects information about SG members via NeoFS
API ObjectService.Head RPC in order to compose SG structure. Bearer token
attached to the call was not used in communication, which could lead to data
access problems. These changes fix the described problem.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add `TOMBSTONE_LIFETIME` configuration value of the node which is measured
in NeoFS epoch and is set to 5 by default.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Consider single word of search filter expression as path to file with
protobuf JSON filters. Decode filters from file and add them to the rest.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make `--filters` flag to be repeated. Define new filter expression format:
* `<key> <binary_op> <value>` for binary filters. Supported binary ops: `EQ` (`STRING_EQUAL`), `NE` (`STRING_NOT_EQUAL`).
* `<key> <unary_op>` for unary filters. Supported unary ops: `NOPRESENT` (`NOT_PRESENT`).
Any other string expressions are considered invalid.
Additionally support shorthand flag `-f`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Split the attributes into those that must be explicitly set in the
configuration, and those that, if absent, will be assigned a default value.
Support this logic in `addWellKnownAttributes` function. If no explicit
attribute is set, the application will panic.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add `--bearer` flag to storagegroup command similar to object command.
Attach parsed bearer token to all API requests of storagegroup command.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add locode section to util cmd. Add generate command to locode section for
generating the NeoFS location databases. Add info command to locode section
to work with the entries from NeoFS location database.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
At startup, update only node info status. Leave all other
attributes from node configuration, so user can update them.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Construct used space Controller on node's app-side. Call Controller.Start on
StartEstimation event from sidechain. Call Controller.Stop on
StopEstimation event from sidechain.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement processing of AnnounceUsedSpace RPC from NeoFS API v2 method on
the app-side according to the following algorithm (except common steps for
each RPC):
* check if the request sender is presented in container;
* check route of the request (compose from signatures);
* pass the value to Writer returned by WriterProvider.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Define component on app-side that implements RemoteWriterProvider with next
logic:
* if ServerInfo is nil, call internal dead end provider;
* if address in ServerInfo equals to local node's address, do nothing;
* otherwise create API client instance via client cache and return the
wrapper over it.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement wrapper over StorageEngine on app-side that iterates over the
values returned by ListContainers+ContainerSize method pair. Component will
be used as the source of local metrics.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement wrapper over netmap and container storages on app-side that builds
placement vectors by next steps:
* read container from storage;
* read network map from storage by epoch number;
* build container nodes according to placement policy;
* build placement vectors with epoch-dependent pivot.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement wrapper over API client on app-side that passes used space values
to it. Values are buffered on Put call and sent on Close call.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement wrapper over Container contract's client on app-side that passes
used space values to it. Uses a fixed public key, whose value will be the
key of the local storage node.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
There are two notifications:
- start estimation notification produced at the beginning of the
epoch,
- stop estimation notifications should be produced before
basic audit settlement starts.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Classifier looks at list of inner ring nodes and container
nodes from current and previous epoch to classify request.
Sometimes these checks might return error.
Consider there is a request from unknown key and container's
placement policy valid for current epoch and invalid for past
epoch. Classifier tries to find if key belongs to container
node from current epoch -- it is not. Then it tries to find if
key belongs to container node from past epoch and it throws
error, because placement policy is invalid for past epoch.
This is a legit case and classifier should ignore such errors
to provide best effort in matching. The only error classifier
should return is an error when request does not contain
public key to classify it.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Implement NodeState interface required by Netmap service. Make a single
point of updating the state of the node (for both Netmap and Control
services). Protect node info structure from data race.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Call Netmap contract on SetNetmapStatus rpc of Control service (AddPeer
method if new status is ONLINE, UpdatePeerState method otherwise).
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Switch health status to:
* STARTING after the initialization of app resources and before bootup;
* READY after bootup and until shutting down;
* SHUTTING_DOWN before exiting the application.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Update node network status via update status by getting the latest network
map and finding the node in it. This step is performed at the start of the
application and upon notification of a new epoch.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement methods required for signatures. Receive network map from netmap
storage, convert it to Control service message and return in response.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement HealthChecker on node app structure. Set health status to ONLINE
after node boot. Set health status to OFFLINE on shutdown.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Application should support several NEO endpoints so it can
switch between different RPC nodes when they fail. Application
iterates over endpoints in random order so the default list of
endpoints distribute workload kinda uniformly.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
As in #72 storage application should behave the same way at remote
RPC node failures. The simplest way is to restart application. Later
we can reinitialize it without downtime.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Support processing of NetmapEpoch and NetmapLookupDepth X-headers when
processing object read operations. Placement for operations
Get/Head/GetRange/GetRangeHash/Search is built for the epoch specified in
NetmapEpoch X-header (by default latest). Also the specified operations are
processed until success is achieved for network maps from the past up to
NetmapLookupDepth value. Behavior for default values (zero or missing) left
unchanged.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Transfer the state of the node to offline when the application is shut down.
Updating the state is done by calling UpdateState method of Netmap contract.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
With the update of the local storage engine, the headers of virtual objects
are directly given. In this regard, the step with obtaining the the right
child header is removed.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
