Replace usage of `cache.ClientCache` type with interface with similar
signature. This will further allow overloading clients without affecting the
logic of dependent packages.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement in-memory `Storage` which is going to be used to submit the
results of interactions with network members. `Storage` also provides an
iterator interface, so the component can be used in `Controller`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement `Controller` designed to analyze the values collected by the local
node (the source is hidden behind Writer/WriterProvider interfaces) and
transfer them to the destination (hidden behind Iterator / IteratorProvider
interface).
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
At object.Put operation node does not transfer any
data. It is done in the stream inside, so we should
register duration from `object.Put` invocation until
`CloseAndRecv` on the stream.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
We can't rely on object size in the header because it
might be omitted on initial put or it can be 0xFF.. on
streaming data.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Because options are not used when client is already in cache
providing them to shared cache is misleading at best.
In the worst case `dial_timeout` is set randomly (because of race
condition) which can lead to one service having `dial_timeout` of
another. Thus we set default client creation options when cache is
created.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Replace `eacl.Storage` interface implementation from eACL lib to neofs-node
app package. Remove `eacl.WithMorphClient` option in order to abstract eACL
validator from eACL storage implementation.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Right now we pass redundant copy to callback outside the for-loop through
the helpful boolean variable instead of calling it deeply nested.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Detect redundant local copy of the object in Object Policer. Add redundant
copy callback (`WithRedundantCopyCallback` option). Pass address of the
redundant copy to callback.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Re-compile protobuf definition of Control service. Implement required
messages on DropObjects RPC request and response messages. Implement
`DropObjects` method on Control service server of the node. Use
`StorageEngine.Delete` method as a deleted object handler on server.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
There is a need to define storage node's interface to remove objects from
the local storage.
Define `DropObjects` RPC in `ControlService`. Define `DropObjectsRequest` /
`DropObjectsResponse` structure of the request / response messages.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement `NetworkInfo` calls on full stack of Netmap services. Current
epoch is read from node local state, magic number is read via `MagicNumber`
call of morph client.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make object delete service to use network information to calculate and set
expiration of the created tombstone.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
The lifetime of an object can be limited by specifying a correspondin
well-known attribute. Node should refuse to save expired objects.
Checking objects in FormatValidator is extended with an expiration attribute
parsing step.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add SignWithRFC6979 option to signature verification function since eACL
table are signed by users with this option.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Since the contract started returning the table signature, it became
necessary to check its correctness.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement route Builder interface on wrapper over the container placement
builder, Component implies exactly one transfer to each of the most weight
nodes of the container (according to some weighing algorithm).
Implementation is planned for use when transferring local estimates of
storage nodes.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement a component for transmitting the value of the used container space
along a route defined in the system. Implement WriterProvider interface on
it. By implementation, it is the link between the route planner and the
point-to-point transmitter, and abstracts from the implementation of both of
them. In the future, this implementation will be used as a transmitter of
local estimates of storage nodes among themselves.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement functions to wrap Writer or Iterator. The resulting wrapper
provides WriterProvider or IteratorProvider interface respectively.
Such a wrapper can be used as a single storage instance provider
regardless of context.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement a component that stores the values of the used space of
containers. The storage allows you to write several values for a fixed
container and epoch number, and read the averaged estimates of all
accumulated values. All values are stored in memory. This component is
planned to be used as an accumulator of opinions from various network
participants about the fullness of the container.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement a component that connects the value stores of the used space of
containers. Implement the Start/Stop operations on it, which will later
become the application handlers of the corresponding events from the
sidechain. The main task of the controller is to temporarily synchronize the
stages of calculating the global estimate of the used space in the
container. The details of the score calculation (the way of collecting /
transmitting local scores, the final score formula and writing to the
contract) are encapsulated in the dependency components, the controller is
abstracted from them.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Classifier looks at list of inner ring nodes and container
nodes from current and previous epoch to classify request.
Sometimes these checks might return error.
Consider there is a request from unknown key and container's
placement policy valid for current epoch and invalid for past
epoch. Classifier tries to find if key belongs to container
node from current epoch -- it is not. Then it tries to find if
key belongs to container node from past epoch and it throws
error, because placement policy is invalid for past epoch.
This is a legit case and classifier should ignore such errors
to provide best effort in matching. The only error classifier
should return is an error when request does not contain
public key to classify it.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Replace static NodeInfo structure with NodeState interface that provides
method to read node information in runtime.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add HealthStatus method to HealthChecker interface that should return
current health status of the node application.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement methods required for signatures. Receive network map from netmap
storage, convert it to Control service message and return in response.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add NetmapSnapshot rpc to ControlService protobuf definition. Recompile
proto files. Add required method to server structure.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Support processing of NetmapEpoch and NetmapLookupDepth X-headers when
processing object read operations. Placement for operations
Get/Head/GetRange/GetRangeHash/Search is built for the epoch specified in
NetmapEpoch X-header (by default latest). Also the specified operations are
processed until success is achieved for network maps from the past up to
NetmapLookupDepth value. Behavior for default values (zero or missing) left
unchanged.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add binary salt field to RangeHashPrm struct. Implement field setter. Set
salt from the request in v2 service.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation of eACL service v2 the response X-headers were
validated at the stage of re-checking eACL. This provoked a mismatch of
records in the eACL table with requests. Fix this behavior by checking the
headers from the request, not the response.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Forward request X-headers to client calls during internal processing of
Object operations on the node.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Since PoR audit check uses object.Head with large TTL values to
make real proof of retrievability, we may use random remote
nodes.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
PDP audit check is not quite working with very small objects, so
we try to build coverage with bigger objects.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
This function converts 2-dimension array of container nodes
into single dimension array. Useful when we need to iterate
over whole container.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Add numeric return from TaskManager.Reset method that shows the number of
canceled tasks. This values will be used for assessment of the progress of
the audit.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Define interface of the container communicator which methods are going to be
used in audit checks. Make innerring Server to implement this interface.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement Reset method on audit task manager that cleans task queue.
Extended TaskManager interface with Reset method on IR side. Call Reset
method in audit processor before new audit start.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation parent object header finalized twice in size
limiter + formatter. On the one hand, this added redundant action, on the
other hand, it could provoke a difference in the headers of the linking and
the last part. Change formatter to finalize parent header if it does not
container the signature. Change size limiter to reuse parent header after
last child finalization in linking child.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation eACL validator didn't take into account container
and object ID fields of request bodies.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Every unknown error must not decrease shortage counter and must not
exclude faulty node from the node list, because this list will be used
later for replication.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
With the update of the local storage engine, the headers of virtual objects
are directly given. In this regard, the step with obtaining the the right
child header is removed.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation linking object inherited only ID of the parent
object. From now it inherits full header like last child.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation child objects inherited parent attributes after
split-transformation, which was redundant. From now attributes are not
inherited.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation placement traverser processed incorrectly with
local placement build. Also entity incorrectly traversed the placement
vectors for fixed number read operations until success. The erroneous
behavior was due to the use of a vector number of successes instead of
a scalar number in these scenarios.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
With updated specification of object related operation
we don't have this search attribute any more and we
should not use functions related to this attribute.
This commit breaks object service logic, however it will
be fixed later.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
ErrEACLNotFound error was defined in implementation package. EACL validator
checked this error after the call of eACL storage interface method. Replace
ErrEACLNotFound to core container library. in order to: on the one hand not
use an implementation error, on the other hand, to be able to reuse a
generic type error (404).
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Now morph library returns error if there is not eACL in
sidechain storage. However in this case eACL check should
be passed since it is the same as having empty eACL table.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Sticky bit checks if object owner and request owner are the
same. Container owner should not used in this check.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
`owner` field may be misused as request sender owner, however
it is a owner of a container for that request. New naming
should be clear.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Define ErrRelationNotFound error in searchsvc package. Return
ErrRelationNotFound from RelationSearcher.SearchRelation method if search
result is empty.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Refactored pkg/netmap package provides JSON converters for
NodeInfo and PlacementPolicy structures, that has been used
by client applications.
It also updates Node structure itself so it is a part of
grpc <-> v2 <-> pkg conversion chain.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Replace meta Bucket with meta.DB instance in local storage implementation.
Adopt all dependent components to new local storage.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Call CheckFormat function in container.Put handler for conducting initial
checks of the structure that are not performed by the smart contract.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make Policer to call AddTask method of Replicator when an insufficient
number of copies of an object is detected in the container.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement Replicator service that performs background work to replicate
local object to remote nodes in the container. Replicator is going to be
used by Policer.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
There is a bit to allow or deny bearer token check for
each object service method. If this bit is not set then
ignore bearer token and use extended ACL table from
sidechain.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
If bearer token is presented in the request then check
if it is a valid one and then use it to process extended
ACL checks.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Malicious user can stole public session key and use
it by sending request from it's own scope. To prevent
this each session token is signed and signature private
key must be corresponded with owner id in token. Therefore
malicious node cannot impersonate request without private
key to sign token.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Implement Policer service that performs background work to check compliance
with the placement policy for local objects in the container. In the initial
implementation, the selection of the working queue of objects is
simplified, and there is no transfer of the result to the replicator.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Define ErrNotFound error in headsvc package. Return ErrNotFound from Head
method if the header was not found in the container.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Wrong boolean operation order made matcher return false
on `non-root` search query with non-regular objects. Instead
it should return true for `non-root` query and false for `root`
query.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
If object service produces new request, the should contain
session token. This is the only way for node to grant access
for a private container.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Classifier fetches public key of the request owner
and owner itself. Extended ACL check should rely on
this public key, because it might be extracted from
session token.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
With one tombstone for split objects we can't simply
place it in container. We should inform all nodes that
store split objects of removed original object.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Owner of the request is stored in session token most of the times.
Put request contains session token in the object body, so we have
to fetch it from there.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
GRPC has default message limit of 4MiB. Since every transmitted
neofs message has to be signed, then original message should
be split into transfer fit structures before signature service.
This commit introduce transport payload splitter for object
service pipeline. This splitter works with stream response
for methods:
- object.Get
- object.Range
- object.Search
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
In previous implementation InitRelations call in payload size limiter was
called in write chunk method. This provoked clearing the split header in
children starting from the second.
Replace InitRelations call to the 1st child allocating stage.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Refactor query to match object and its parents in a single call. Support
KeyRoot and KeyLeaf filters.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation object.Search services allowed to search only
physically stored objects. This limitation did not allow getting the ID of
the split object.
Extend search execution logic with parent object filtering. Parent objects
that passed filters are now included in the result
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Use key storage in object services in order to sign requests with private
session key within user session.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add session token argument to object formatter constructor which is written
to the object. Pass session token from trusted object Put.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Basic ACL checker gets request field via getters that are
NPE-free, therefore we don't need to worry about function
invocations on nil structures.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
ACL has to classify request senders by roles:
- owner of the container,
- request from container or inner ring node,
- any other request.
According to this roles ACL checker use different
bits of basic ACL to grant or deny access.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
In previous implementation payload size limiter panicked in case of payload
emptiness. It was caused by the component waiting for at least one write of
a part of the payload.
Fix NPE occurrence with internal initialization after the WriteHeader call.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add SetSeekRange method to RangeTraverser that switches traverser to work
with provided object payload range.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation one excess element could be added to the chain.
Add previous sibling presence check to prevent this.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add FullRange option to get range operation parameters that allows to get
payload range [0:object_size] w/o the actual knowledge of the object size.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Head service receives right child of the processing object in some cases.
Add right child to Head result in order to use it as needed.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementations object size limiter left checksums of parent
object payload unset. This was caused by the closure that written calculated
checksums to the child object instead of parent one.
Construct payload checksum writers in separate function in order to prevent
closure bug.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation session Storage.Get method took a v2 owner
identifier as an argument.
Change method signature to as an argument owner identifier from NeoFS SDK.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation traverser worked like all counts of all selectors
are equal to counts of corresponding replicas. Make traverser to take into
account select count of all replicas.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Define object writer interface. Implement payload size limiter that
restricts payload size of the object. Implement object format moulder that
fill verification fields and finalizes object structure.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
