Commit graph

800 commits

Author SHA1 Message Date
Evgenii Stratonikov
0e60b1d6c9 [#1086] services/control: implement DumpShard RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
aa53418119 [#1086] services/control: fix comments
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Leonard Lyubich
87a45983a1 [#1084] accounting: Keep balance precision from contract
Do not convert balance response of Balance contract to Fixed8.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-26 14:10:02 +03:00
Pavel Karpy
fee2f5a330 [#1100] *: Adopt new SDK's owner.ID API
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 17:58:30 +03:00
Leonard Lyubich
4b5404047c [#1098] object/search: Fix error loss in SignService.Search
Error loss led to incorrect response formation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-21 14:54:22 +03:00
Pavel Karpy
202dbfb95d [#1096] eacl: Rename eacl import aliases
Since `Validator` has been moved to SDK it is more readable to see `eaclSDK`
than `acl` import.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 10:47:33 +03:00
Pavel Karpy
ed156cd738 [#1096] eacl: Use validator from SDK
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 10:47:33 +03:00
Pavel Karpy
7c02a2e251 [#1087] *: Adopt SDK changes
- Update `neofs-sdk-go`:
v0.0.0-20211230072947-1fe37df88f80 => v0.0.0-20220113123743-7f3162110659

- Add client interface that duplicates SDK's client behaviour and new
`MultiAddressClient` interface that has method that iterates over wrapped
clients.

- Also start using simple client mode that does not require parsing statuses
outside the SDK library.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-14 17:29:03 +03:00
Alex Vanin
0f1eb743af [#1077] service/object: Don't override status of payload overtake
Reverse payload overtake triggers direct payload overtake that
sets status and error. We should not override that.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-01-13 10:30:18 +03:00
Evgenii Stratonikov
5828f43e52 [#851] util/rand: use single random source
It is much more convenient to skip source creation.
Also fix some bugs:
1. `cryptoSource.Int63()` now returns number in [0, 1<<63) as required
   by `rand.Source` interface.
2. Replace `cryptoSource.Uint63()` with `cryptoSource.Uint64` to allow
   generate uint64 numbers directly (see rand.Source64 docs).

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-12 19:29:27 +03:00
Leonard Lyubich
0b0c61f4a5 [#1063] node/object: Do not skip empty search responses in splitter
In previous implementation `Search` method of transport splitter skipped
responses with empty ID list.

Replace while-loop with do-while one in `TransportSplitter.Search`
method implementation in order to send responses with empty result too.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-11 12:02:54 +03:00
Pavel Karpy
97fd330fae [#1059] control: Fix ListShards wrapper name
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
77808c7b41 [#1059] control: Provide shard's real mode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
88bb8e2df9 [#1059] control: Add SetShardMode implementation to ctrl svc
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
fa78a37787 [#1059] control: Add unit test for SetShardMode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
29c69f37eb [#1059] control: Add SetShardMode RPC wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
25665cb0fd [#1059] control: Add marshalling and setters to SetShardMode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
4533784e11 [#1059] control: Recompile proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
41dd385b7b [#1059] control: Add SetShardMode rpc to proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
876b0c53de [#1048] control: Rename Blobstore to Blobstor
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
f5a9735e1c [#1048] cli: Add shard list operation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
0e5410603e [#1048] control: Add ListShards implementation to ctrl svc
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
4e989e7133 [#1048] control: Add unit test for ListShards
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
f88a12eaa7 [#1048] control: Add ListShards RPC wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
a6b3e16975 [#1048] control: Add marshalling and setters to ListShards
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
a42b3d37f6 [#1048] control: Recompile proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
20f11c88ca [#1048] control: Add ListShards rpc to proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Alex Vanin
5d46035ae8 [#1052] Tidy INFO logs
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-27 14:28:01 +03:00
Alex Vanin
e610d1ea5f [#1045] Provide sanity check of session token in container service
Without sanity check, container service provides successful response,
even though such request will never be approved by Alphabet nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-20 14:51:07 +03:00
Alex Vanin
bca7cf9470 [#1047] policer: Check context before job selection
When application is being terminated, replicator routine
might be on the object picking phase. Storage is terminated
asynchronously, thus `Select()` may return corresponding
error. If we don't process `context.Done()` in this case,
then application freezes on shutdown.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-20 14:42:57 +03:00
Pavel Karpy
df0d76dc19 [#1032] node/container/router: Do not check nil route points
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 16:02:29 +03:00
Pavel Karpy
fd9514f90e [#1032] node/container/router: Fix interface description
After #872 `NextStage` method description becomes outdated.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 16:02:29 +03:00
Leonard Lyubich
6f41cd86ed [#979] Upgrade SDK NeoFS Go dependency
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
e8f8e58e90 [#979] adm: Add subnet-related commands
Add `subnet` command which contains all subnet-related commands. Add
sub-commands:
  * `create` for creation;
  * `remove` for removal;
  * `get` for reading;
  * `admin` for admin management;
  * `client` for client management.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Pavel Karpy
acf7331034 [#990] object: Fix linter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 17:18:08 +03:00
Alex Vanin
011d0f605b [#965] replicator: Make HandleTask function public
Continues replication executed in separate pool of goroutines,
so there is no need in worker to handle replication tasks
asynchronously.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-26 15:39:38 +03:00
Alex Vanin
a74a402a7d [#965] policer: Implement continuous replication
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-26 15:39:38 +03:00
Leonard Lyubich
7f5fb130c0 [#961] *: Support NeoFS API status returns
Upgrade NeoFS API Go library to version with status returns. Make all API
clients to pull out and return errors from failed statuses. Make signature
service to respond with status if client version supports it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-24 09:26:40 +03:00
Leonard Lyubich
44d0d453ef [#961] client: Clarify error returns in docs
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-24 09:26:40 +03:00
Evgenii Stratonikov
95893927aa *: replace neofs-api-go with neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-12 17:29:09 +03:00
Leonard Lyubich
8f8b638b95 [#922] control: Add MAINTENANCE netmap status
Add `MAINTENANCE` value to `NetmapStatus` enum in Control API. The status is
going to be used to toggle maintenance mode of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
0ec8f529ab [#842] object/delete: Set tombstone local node as tombstone owner
All objects in NeoFS must have owner ID. In previous implementation Object
Delete service handler set owner ID from request session token. If removal
was executed w/o a session, object with tombstone was prepared incorrectly.
In order to fix this node should set its own ID and become an owner of the
tombstone object.

Extend `NetworkInfo` interface required by Object.Delete handler with
`LocalNodeID` method which returns `owner.ID` of the local node. Implement
the method on `networkState` component of storage node application which is
updated on each node state change in NeoFS network map. Set owner returned
by `LocalNodeID` call as tombstone object's owner in Delete handler.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 16:18:02 +03:00
Leonard Lyubich
d421022547 [#842] object/acl: Check owner and key only if sticky bit is set
In previous implementation node returns "access denied" on Object.Put with
object with unset owner. Although object owner must be set, its absence
should not be considered as access error. The same applies to sender key.

Check owner ID and public key emptiness only if sticky bit is set.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 16:18:02 +03:00
Leonard Lyubich
b3708fc530 [#957] services/object: Refactor usage of NeoFS API client
The client needs of the Object service are limited and change not often.
Interface changes of the client library should not affect the operation of
various service packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the Object service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-03 18:30:09 +03:00
Alex Vanin
fbed86da2c [#943] service/object: Remove KeyStore from CommonPrm
There is no point to pass key storage in parameters because
it can be defined on the service level of application.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
c30aa20b04 [#943] service/object: Refactor private key fetching during execution
`CommonPrm` structure has private key for remote operations.
It obtained in the beginning of request processing. However,
not every operation triggers remote calls. Therefore, key
might not be used. It is important to avoid early key fetching
because `TokenStore` now returns error if session token does not
exist. This is valid case when container nodes receive request with
session token (for ACL pass) and they should process request locally.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
2fbdcbdee1 [#943] service/object: Check session token expiration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
508a28fdc0 [#943] service/object: Do not sign put service message with session token
This is invalid operation for storage nodes that receive part of split
object. While object is signed by session token, the message itself
should be signed by the node key.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
46fa07b7cc [#943] service/object: Return error if requested session token is not available
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Evgenii Stratonikov
0057eeb0f7 [#901] *: release worker pools where possible
Some of the pools are initialized during config initialization,
so it isn't possible currently to release them in one place.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Evgenii Stratonikov
0beaed2ef4 [#901] util: implement Release for PseudWorkerPool
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Evgenii Stratonikov
7cb3d0cb4a [#885] policer: remove objects for removed container
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-18 12:14:14 +03:00
Leonard Lyubich
283ccc04b4 [#833] services/netmap: Support new fields of netmap.NetworkInfo
Make the implementation of network info source (Netmap V2 service
dependency) to read MillisecondsPerBlock sidechain parameter and NeoFS
network parameters depending on the client version.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Leonard Lyubich
40a4a7faa2 [#674] object/put: Use pseudo worker pool for local operations
After storage engine started to limit number of PUT operations there is no
need to limited worker pool in Object Put service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
07130855aa [#674] util: Rename SyncWorkerPool with PseudoWorkerPool
`SyncWorkerPool` name is more appropriate for worker pool of size 1.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Alex Vanin
d8f7fed10a [#881] acl: Use session token from request at object.Put
Session token can be present in both object header and
request meta header. They are the same during initial object
placement.

At the object replication, storage node puts object without
any session tokens attached to the request. If container's eACL
denies object.Put for USER role (use bearer to upload), then
replication might fail on objects with session tokens of the
signed by container owner. It is incorrect, so use session
token directly from request meta header.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-08 17:18:32 +03:00
Evgenii Stratonikov
b8ba677c85 [#882] policer: add CID to the error message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 08:21:01 +03:00
Alex Vanin
e41e74b5fc [#872] services/container: Ignore passed routes in load router
Higher level solution for original issue with lost announcements
from single node container.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-05 19:29:27 +03:00
Leonard Lyubich
4661f65975 [#645] client/cache: Check response public key in all client operations
There is a need to check if public key in the RPC response matches the
public key of the related storage node declared in network map.

Define `ErrWrongPublicKey` error. Implement RPC response handler's
constructor `AssertKeyResponseCallback` which checks public key. Construct
handler and pass it to client's option `WithResponseInfoHandler`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
3a497a3c53 [#645] object/put: Pass NodeInfo instead of AddressGroup to relay func
Replace `network.AddressGroup` parameter type of relay function with
`client.NodeInfo`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
e473f3ac91 [#645] *: Use helper functions to build client.NodeInfo structures
Helper functions from core/client package allow to set public keys of
storage nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
2d441a4cc6 [#645] placement/traverser: Rename Key method of Node type to PublicKey
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
7b228b7603 [#645] *: Construct clients from client.NodeInfo in API client cache
There is a need to have the ability to expand the data needed for client
construction.

Replace `network.AddressGroup` parameter of client cache interfaces with
`client.NodeInfo`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
be05bed0b3 [#860] Re-compile protobuf files after make protoc target update
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-29 17:51:33 +03:00
Leonard Lyubich
ee20200c2e [#845] object/put: Separate pools for local and remote operations
In previous implementation Object PUT used single pool of workers for local
and remote ops, but these ops are heterogeneous.

Use remote/local pool for remote/local operations in PUT service. At first
the pools are configured with the same size.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 16:20:19 +03:00
Leonard Lyubich
3b2b6007c6 [#845] object/put: Make distributed target to define is node is local
In previous implementation `distributedTarget` didn't check if next node is
local. This check was performed by the handlers (target initializer and
relay func).

Make `distributedTarget` to calculate node's locality. Pass locality flag to
the handlers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 16:20:19 +03:00
Leonard Lyubich
73fb1a886c [#849] policer: Write message about redundant local object copy
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Leonard Lyubich
d613a856ce [#849] policer: Log object address in processNodes method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Alex Vanin
b97cdbea9e [#839] service/object: Add high level functions to control request forwarding
Applicable for object/get execution context.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-27 11:10:50 +03:00
Alex Vanin
b3464e8140 [#839] service/object: Do not use request forwarding when assembling object
Forwarding mechanism resends original request. During split object chain traversal,
storage node performs multiple `object.Head` requests on each child. If request
forwarding happens, then `object.Head` returns object ID of the original request.
This produces infinite assembly loop.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-27 11:10:50 +03:00
Alex Vanin
01df4ffa61 [#811] service/container: Hide cache invalidation logic in Writer interface
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Alex Vanin
4a1f0de8f4 [#811] service/container: Invalidate cache records on update requests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Leonard Lyubich
61b4baf736 [#778] services/object: Wrap last client's error into errIncompletePut
Make `errIncompletePut` to be a structure which wraps single client error.
Wrap error of the last client into `errIncompletePut` during placement
execution.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 16:49:37 +03:00
Leonard Lyubich
02f2a98bcc [#778] services/object: Return the composite error of PUT operation
In previous implementation Object service's handler returned const error in
case of failure (full or partial) of PUT operation. This did not even allow
us to roughly guess what the reason is. Not as a complete solution, but to
alleviate some cases where all nodes in a container return the same error,
it is suggested to return the error of the last server that responded.

Return latest server error from placement loop of `iteratePlacement` method
of `distributedTarget` type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 16:49:37 +03:00
Evgenii Stratonikov
5f86d54721 [#816] object/acl: fix eACL target processing
Ignore role if public keys are present.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-09-10 15:15:36 +03:00
Leonard Lyubich
7a13053fab [#818] object/acl: Fit sticky bit specification
In previous implementation sticky bit could disrupt access of container
nodes to replication. According to NeoFS specification sticky bit should not
affect the requests sent by nodes from SYSTEM group.

Add role check to `stickyBitCheck`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 15:14:59 +03:00
Pavel Karpy
4a81781c0c [#770] logger: Delete spaces in log message keys
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Leonard Lyubich
358e3ed8c4 [#645] *: Change the locality condition of the node from the placement
Some software components regulate the way of working with placement arrays
when a local node enters it. In the previous implementation, the locality
criterion was the correspondence between the announced network address
(group) and the address with which the node was configured. However, by
design, network addresses are not unique identifiers of storage nodes in the
system.

Change comparisons by network addresses to comparisons by keys in all
packages with the logic described above. Implement `netmap.AnnouncedKeys`
interface on `cfg` type in the storage node application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich
22926e8f28 [#645] placement/traverser: Extend Node type with public key
There is a need to process public keys of the placement result.

Implement `Node.PublicKey` method which returns storage node's key announced
in netmap.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich
fe90456dcc [#645] placement: Overload result of Traverser.Next method
In previous implementation `placement.Traverser.Next` method returned slice
of `network.AddressGroup` elements. There is a need to process keys of
storage nodes besides network addresses for intra-container communication.

Wrap `network.AddressGroup` in a new type `placement.Node` that summarizes
the storage node information required for communication. Return slice of
`Node` instances from `Traverser.Next` method. Fix compilation breaks in
dependent packages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich
e738699fcc [#676] services/container: Cache the results of read operations
In previous implementation Container service handlers didn't cache the
results of `Get` / `GetEACL` / `List` operations. As a consequence of this,
high load on the service caused neo-go client's connection errors. To avoid
this there is a need to use cache. Object service already uses `Get` and
`GetEACL` caches.

Implement cache of `List` results. Share already implemented cache of Object
service with the Container one. Provide new instance of read-only container
storage (defined as an interface)to morph executor's constructor on which
container service is based. Write operations remained unchanged.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-08-30 18:01:26 +03:00
Alex Vanin
d6439369ab [#768] object/acl: Support all eACL filter fields from the spec
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 19:01:03 +03:00
Leonard Lyubich
abfcc7498c [#715] services/policer: Select pseudo-random list of objects to check
In previous implementation of Policer's job queue the same list of objects for
processing was selected at each iteration. This was caused by consistent
return of `engine.List` function.

Use `rand.Shuffle` function to compose pseudo-random list of all objects in
order to approximately evenly distribute objects to work.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-08-25 14:40:12 +03:00
Pavel Karpy
e8665f6cef [#730] node/morph: Use disable_cache config param
Do not init caches for eACL, containers and netmap
if `disable_cache` config options is `true`, use
direct RPC calls instead.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 09:40:10 +03:00
Pavel Karpy
c423aa432a [#730] node/config/morph: Add disable_cache
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 09:40:10 +03:00
ZhangTao1596
01806db612 [#723] container/announcement/storage: Map announcements by epoch and container id
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2021-07-30 09:58:30 +03:00
Leonard Lyubich
7c1432aa57 [#660] services/container: Don't check container format in Put
Storage node should only reflect container Put requests to sidechain w/o any
validation. All verification steps are performed by IR.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-07-05 11:05:44 +03:00
Leonard Lyubich
8192933313 [#660] services/netmap: Don't call IsSupportedVersion in LocalNodeInfo
Clients can have version later than the server. The server should not deny
these clients.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-07-05 11:05:44 +03:00
Leonard Lyubich
8eadf18e01 [#607] services/control: Fix incorrect allocation in NetmapSnapshot
Fix make with length + subsequent append.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
f8ee6b1164 [#607] services/netmap: Support backward compatibility
Make Netmap service to return single network address on `LocalNodeInfo` RPC
in order to older clients.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
43eff09944 [#607] *: Do not use deprecated elements of code
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
1e52e86bbc [#607] node/control: Make group address in NodeInfo message
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
cede2b4ed7 [#607] reputation,container: Support address groups in ServerInfo
There is a need to support multiple server endpoints for reputation and
container transmission.

Replace `ServerInfo.Address` getter with `ServerInfo.IterateAddresses`
iterator.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
d6bb697726 [#607] container/announce: Use public keys as keys to server cache
In future server info will contain multiple endpoints whose string
representation will be worse suited to the cache key.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
44a2c81f8e [#607] reputation/router: Use public keys as keys to server cache
In future server info will contain multiple endpoints whose string
representation will be worse suited to the cache key.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
9033cf5aa1 [#607] placement: Work with AddressGroup
Make `PlacementBuilder` implementations to work `network.AddressGroup` type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
b3dd9a3254 [#607] placement: Make traverser to return list of address groups
Make placement `Traverser.Next` method to return ``[]network.AddressGroup`
in order to support multiple addresses of the storeage nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
8ac3c62518 [#607] object/head: Make client constructor to work with group address
Make Object Head service to work with `AddressGroup` instead of `Address`
in order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
d0e48c949b [#607] object/search: Make client constructor to work with group address
Make Object Search service to work with `AddressGroup` instead of `Address`
in order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
ad14df07f6 [#607] object/get: Make client constructor to work with group address
Make Object Get service to work with `AddressGroup` instead of `Address` in
order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
8972f84672 [#607] object/put: Make client constructor to work with group address
Make Object Put service to work with `AddressGroup` instead of `Address` in
order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
6e5d7f84af [#607] network: Generalize LocalAddressSource to address group
Make `LocalAddressSource.LocalAddress` method to return `AddressGroup`. Make
`IsLocalAddress` function to accept parameter of type `AddressGroup`. Adopt
the application code with temporary `GroupFromAddress` helper.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
3805b0f638 [#607] client: Overload Client interface
There is a need to generalize single-address client to group-address client.
To do this, we can re-implement `Client` interface from NeoFS API Go library
and still use it in the application code. There is a problem with method
`Raw` which must return single-address raw client. So as not to make changes
to API library we need to overload Client interface in order to support
`Raw` method in group-address client implementation.

Define `Client` interface in new `pkg/core/client` package. Completely
inherit API `Client` interface. Add `RawForAddress` method to build raw
client for the single node address. Adopt the application code that used Raw
method to work with new `Client`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Pavel Karpy
48827f42d3 [#643] pkg: Sync method names and commentaries to them
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-24 16:10:44 +03:00
Leonard Lyubich
adbbad0beb [#607] network: Do not work with Address pointers
`network.Address` structure in most cases created once and used read-only.

Replace `AddressFromString` function with `Address.FromString` method with
the same purpose and implementation. Make all libraries to work with value.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich
e5504c7130 [#607] network: Do not use Address.String for address comparison
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich
47fe8911a3 [#607] network: Rename Address.HostAddrString method to HostAddr
Return tyype is clear from the method's signature and docs, there is no
point in reflecting it in the name.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich
35f81729e4 [#607] network: Do not return error from Address.HostAddrString method
Panic if internal `manet.DialArgs` call returns error since this is
unexpected according to `AddressFromString` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich
95ccbbc2f9 [#607] network: Accept value instead of pointer in IsLocalAddress
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Pavel Karpy
6b176e8769 [#613] pkg/reputation: Move manager building to pkg
Move `managers` package to `pkg` since
it can be reused in other packages.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-17 18:30:06 +03:00
Leonard Lyubich
455fd952dd [#414] ir: Serve ControlService
Serve `ControlService` instance on configured endpoint (do not serve if not
specified). Read allowed keys from config.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-15 15:21:52 +03:00
Leonard Lyubich
dcfe9a6504 [#414] ir/control: Implement service server
Implement `ControlServiceServer` on `Server` type. The `Server` requires all
requests to be signed with keys from the so-called whitelist. To obtain
health status, it uses the abstraction in the form of `HealthChecker`
interface.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-15 15:21:52 +03:00
Leonard Lyubich
93803b1a90 [#414] ir: Define Control service
Define `ControlService` for IR similar to the one from storage node. Add
`HealthStatus` RPC which returns health status of the IR application.
Implement getters, setters and methods to sign/verify the messages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-15 15:21:52 +03:00
Evgenii Stratonikov
9142c778c7 [#562] pkg: remove nspcc-dev/neofs-crypto dependency
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-15 14:49:59 +03:00
Leonard Lyubich
096acb2a44 [#580] v2/object/put: Check payload size of the relayed objects
NeoFS network dictates the limitation of the maximum size of the "finished"
objects. The payload size of an object is recorded in its header. When
writing finished objects, the nodes must check:

  * satisfying the constraint;
  * matching the value in the header to the number of payload bytes.

Provide value returned by `MaxSizeSource` component to `validatingTarget`.
Check max size value during the stream of the "finished" objects. Check
header value during the streaming and on-close. Check payload size in v2
relay scenario.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-09 12:08:37 +03:00
Evgenii Stratonikov
b8a7c11e57 [#501] object/put: reduce TTL of the relayed request
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-02 10:31:14 +03:00
Evgenii Stratonikov
3468491224 [#501] object/put: relay requests for signed objects
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-02 10:31:14 +03:00
Evgenii Stratonikov
a422f42ca9 [#501] object/put: refactor distributed target
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-02 10:31:14 +03:00
Leonard Lyubich
3e1463cc76 [#570] *: Use generator of test container IDs from API Go lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 20:45:15 +03:00
Leonard Lyubich
3dd10b6795 [#570] *: Remove usage of deprecated elements from API Go library
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 20:45:15 +03:00
Leonard Lyubich
70a7354e9d [#570] *: Use new Equal method of owner.ID type
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 20:45:15 +03:00
Pavel Karpy
33bef46f31 [#549] network/cache: Change Get signature
Make network cache's `Get` method accept
`network.Address` argument instead of
string.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-27 17:07:04 +03:00
Leonard Lyubich
a1bfcdc74a [#525] v2/container: Write session token from header to removal witness
If container is removed via session, then session token should be included
in removal witness.

Write session token from request meta header to `container.RemovalWitness`
structure which is passed to `wrapper.Delete` function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich
9a0964efa4 [#525] morph/container: Accept RemovalWitness in Delete function
Make `wrapper.Delete` function to accept `container.RemovalWitness` struct
instead of its separated elements. `Signature` type is replaced by binary
signature since public key is unused.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich
b09f212990 [#525] v2/container: Write session token from header to eACL table
If eACL table is set via session, then session token should be written to
it.

Write session token from request meta header to `eacl.Table` structure which
is passed to `wrapper.PutEACL` function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich
5c2b8de87d [#525] v2/container: Make context with token reusable
Rename `PutContext` to `ContextWithToken` and implement its constructor as a
separate function in order to reuse it in other RPCs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich
98cc685a9b [#525] v2/container: Write session token from header to container
If container is created via session, then session token should be written to
it.

Write session token from request meta header to `Container` structure which
is passed to `wrapper.Put` function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich
1deb3f3d01 [#525] morph/container: Do not accept signature in PutEACL function
In previous implementation wrapper over the Container contract's client
accepted the signature of the eACL table in addition to itself. After recent
changes in API Go lib table carries its signature. Thus, it is redundant
to pass the eACL table signature separately.

Make `wrapper.PutEACL` method to accept `eacl.Table` only.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich
32828d2b90 [#525] morph/container: Do not accept signature in Put function
In previous implementation wrapper over the Container contract's client
accepted the signature of the container in addition to itself. After recent
changes in API Go lib container carries its signature. Thus, it is redundant
to pass the container signature separately.

Make `wrapper.Put` method to accept `Container` only.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich
df197dc38b [#525] morph/container: Do not return signature from GetEACL method
In previous implementation wrapper over the Container contract's client
returned the signature of the eACL table in addition to itself. After recent
changes in API Go lib table carries its signature. Thus, it is redundant to
return the table signature separately.

Make `Wrapper.GetEACL` method to return only `eacl.Table` with error.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich
615813d6de [#525] v2/container: Return session token in GetExtendedACL
In recent API changes `GetExtendedACLResponseBody` carries session token.
In recent API Go lib changes `eacl.Table` structure carries related session
token.

Write session token of eACL table from sidechain to GetExtendedACL response
body in node's `ContainerService` server.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich
a7f71bf167 [#525] v2/container: Return session token and signature in Get
In recent API changes `GetResponseBody` carries session token and signature.
In recent API Go lib changes `Container` structure carries its session
token and signature.

Write session token and signature of container from sidechain to Get
response body in node's `ContainerService` server.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Alex Vanin
975ab0dff7 [#561] acl: Add tests for request meta header traverser
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-27 11:21:14 +03:00
Alex Vanin
a025e6250e [#561] acl: Fetch session token from original request meta header
As it explained in previous commit, session token also should
be presented in original meta header but can be omitted in higher
layers.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-27 11:21:14 +03:00
Alex Vanin
d368afffe5 [#561] acl: Fetch bearer token from original request meta header
Request meta headers are organized in a layers, where
upper layers re-sign down layers. Bearer token should be
a part of original meta header and it can be omitted in
upper layers. Therefore we need to traverse over linked list
of meta header to the original meta header to get bearer token.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-27 11:21:14 +03:00
Leonard Lyubich
83c27f6e8a [#505] morph/container: Change get container API
Make `Get` method of the wrapper over Container contract's client to
accept binary container ID. Create `Get` function similar to the previous
`Get` variation. Use this function in Container service server in the place
where `Get` method was used.

Additionally implement `AsContainerSource` function which allows
to simply compose container Source interface from the wrapper.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-25 16:35:52 +03:00
Leonard Lyubich
e3b4c9eda0 [#505] morph/container: Change delete container API
Make `Delete` method of the wrapper over Container contract's client to accept
two binary parameters: container ID and signature. Create `Delete` function
similar to the previous `Delete` variation, but accepting `Signature`
structure instead of binary signature. Use this function in Container
service server in the place where `Delete` method was used.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-25 16:35:52 +03:00
Leonard Lyubich
565ad51b42 [#505] morph/container: Change put container API
Make `Put` method of the wrapper over Container contract's client to accept
three binary parameters: container, key and signature. Create `Put` function
similar to the previous `Put` variation, but accepting `Signature`
structure instead of binary key and signature. Use this function in
Container service server in the place where `Put` method was used.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-25 16:35:52 +03:00
Leonard Lyubich
395fd187ac [#505] morph/container: Add key argument to client wrapper's SetEACL
In recent changes argument list of set eACL call of Container contract
client was extended with binary public key. In the future there will be a
need to pass the table in binary format.

Replace `PutEACL` method with `PutEACLBinary` one which accepts three binary
parameters: eACL table, key and signature. Create `PutEACL` function similar
to the removed method, but accepting `Signature` structure instead of just
a signature. Use this function in Container service server in the place
where `PutEACL` was used.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-25 16:35:52 +03:00
Evgenii Stratonikov
2b2b2c2c45 [#496] Use single contract wrapper constructor
There is no need in a separate `New()` or `WrapClient()`

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-05-25 13:41:39 +03:00
Leonard Lyubich
b587b23e79 [#543] object/search: Fix potential NPE on request forwarder
Request forwarding callback should be called only if set since it is an
optional parameter.

Call `forwarder` function only if it is non-nil.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-20 11:33:24 +03:00
Leonard Lyubich
b2002ffc6d [#543] object/get: Fix potential NPE on request forwarder
Request forwarding callback should be called only if set since it is an
optional parameter. In GetRangeHash forwarder is never set.

Call `forwarder` function only if it is non-nil. Remove no longer needed
`hashOnly` option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-20 11:33:24 +03:00
Alex Vanin
89b147ebe7 [#521] Recompile protobuf files after transition from pkg/errors
Autogenerated files were accidentally modified. This commit returns
them back to the original.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-19 19:09:35 +03:00
Alex Vanin
b5256ccf4c [#521] Fix issues with transition from pkg/errors pkg
Wrap functions at `pkg/errors` return nil if error argument
was nil. fmt.Errorf always returns error so we need to add
missing error checks to the code.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-19 19:09:35 +03:00
Evgenii Stratonikov
71b87155ef [#521] *: use stdlib errors package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-05-19 16:42:54 +03:00
Leonard Lyubich
72e2bc8fb6 [#532] object/hash: Fix NPE during request forwarding
In current implementation `Object.GetRangeHash` RPC handler forwards range
requests for payload data. Missing request forwarder of the original request
caused NPE during execution.

Do not call request forwarder if payload range hash is requested.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-18 18:05:18 +03:00
Alex Vanin
16f13bc0a5 [#522] Use HostAddrString as RPC endpoint instead of IPAddrString
To enable TLS support we can't operate with IP addresses directly.
Certificates are issued with host names so it is required to
pass them into RPC client. DNS resolving should be done by transport
layer and not be a part of node. Therefore `IPAddrString` usage is
removed from code.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-18 12:07:00 +03:00
Pavel Karpy
9fbc6f5efe [#527] reputation: Fix EigenTrust algorithm
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-18 10:29:25 +03:00
Leonard Lyubich
80ef4492c1 [#493] node: Get rid of outdated object GC worker
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Alex Vanin
d49bd4b94a Update neofs-api-go to v1.26.0
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-10 09:45:11 +03:00
Pavel Karpy
20b7295087 [#488] reputation/router: Add commentaries and TODO
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
d1db54acf8 [#488] reputation: Change Writer interface
Includes:
- Delete first `ctx` argument in `Write` method.
- Move intermediate Initial trust struct and method
to `calculator` file.
- Change Alpha to 0.1.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
eb74a9cafc [#488] reputation: Add commentaries
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
2c8c9f69c8 [#488] storage/calculator: Make alpha dynamic
Delete reading `alpha` from env var. Cover
retrieving `alpha` behind interface in
intermediate calculator. Add TODO to decide
if it is necessary to receive that param from
global config or not.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
ea781664cf [#488] reputation/eigentrust/calculator: Implement calc wrapper
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
d3c1fc7dda [#488] reputation/local/storage: Add trusting peer
Fix iteration logic after adding trusting peer
in `Trust` struct.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
e69917b27a [#488] reputation/common: Move ServerInfo to common pkg
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
49d477f466 [#488] reputation/managers: Implement route builders
Add implementation of Builder interface for
intermediate trusts. Move all code associated
with managers to `common` directory in `cmd`
and `pkg/services/reputation`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
e2a1b0e0ee [#488] reputation/eigentrust/storages: Fix args
Change anonymous func arg for `Iterate`
methods of Storages to `PeerTrustsHandler`
type for implementing corresponding
interface.
Implement missing `Iterate` method for
daughter Storage.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
f6783f4f81 [#488] cmd/reputation: Add DaughterStorage
Add `DaughterStorage` init in main pkg
and start write all received daughters'
trusts to it.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Leonard Lyubich
969af596b4 [#431] object/search: Remove no longer used unexported methods
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-04 13:56:55 +03:00
Leonard Lyubich
7366d62e41 [#431] object/range: Fix payload slice allocation in re-sign callback
Allocate capacity instead of length of the slice to write the object payload
range since each chunk is written through `append`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-04 13:56:55 +03:00
Leonard Lyubich
36088949fc [#431] services/object: Re-sign original read requests during forwarding
In previous implementation node's Object Get/Head/GetRange V2 service
handlers created a new request for each RPC. Now original requests are
re-signed according to API specification. Logical handler abstracts from
this version-dependent logic through `RequestForwarder` callback.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-04 13:56:55 +03:00
Leonard Lyubich
e6b30aed36 [#431] object/search: Re-sign original requests during forwarding
In previous implementation node's Object Search V2 service handler created a
new request for each RPC. Now original requests are re-signed according to
API specification. Logical handler abstracts from this version-dependent
logic through `RequestForwarder` callback.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-04 13:56:55 +03:00
ZhangTao1596
2877a343c3 [#498] audit: skip unnecessary statement
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2021-05-04 12:55:01 +03:00
Pavel Karpy
477682adb7 [#482] reputation/router: Move router to ./common
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-19 11:47:04 +03:00
Pavel Karpy
ac8441b718 [#482] reputation/router: Make route pkg independent
Make route package independent from
controller package. Add common interfaces
to `./common` directory.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-19 11:47:04 +03:00
Pavel Karpy
0a16aaacb1 [#482] reputation/router: Use trusting peer
In route building use `trusting` peer
field of `reputation.Trust` struct
instead of `reputation.Peer`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-19 11:47:04 +03:00
Pavel Karpy
1c92dc2414 [#482] reputation: Fix typos in comments
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-19 11:47:04 +03:00
Leonard Lyubich
a97e08cfd7 [#473] Implement EigenTrust calculations
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-04-12 13:25:28 +03:00
Pavel Karpy
60cc3b3e16 [#460] reputation: Add Router to reputation server
Add `Router` to the reputation server. `Router` is
called on every incoming request and inits `Writer`
that sends `Trust`s to the next route point or
handle(logs in that implementation) them if current
node is the end point of the route.

Rename `onlyKeyRemoteServerInfo` struct for container
to separate it from the same implementation of the
same `ServerInfo` interface for reputation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-08 17:29:08 +03:00
Pavel Karpy
5970a5e6ae [#460] reputation/local: Declare Managers
Declare interfaces for building list of
managers of certain peer in certain epoch.
Implement `Builder` struct that aggregates
fields that are necessary for building
list of managers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-08 17:29:08 +03:00
Pavel Karpy
91825a0162 [#460] reputation/local: Implement local trust router
Implement reputation `Router` and its constructor,
designed to define where to send local trusts.
Router is based on dependencies that are hidden
behind interfaces, that are declared in the router's
package.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-08 17:29:08 +03:00
Pavel Karpy
e6c9fb283c [#460] Fix typos in commentaries
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-08 17:29:08 +03:00
Alex Vanin
7cf48d4d91 [#452] Update neo-go to latest master
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-06 10:35:20 +03:00
Leonard Lyubich
3f9c8f2f59 [#444] reputation: Implement sign and response servers
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-04-01 14:54:22 +03:00
Leonard Lyubich
df97e35f30 [#444] reputation: Define RPC Server interface
Define `Server` interface of NeoFS API V2 Reputation service server.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-04-01 14:54:22 +03:00
Leonard Lyubich
106884fc40 [#428] client: Hide client cache behind interface in dependent packages
Replace usage of `cache.ClientCache` type with interface with similar
signature. This will further allow overloading clients without affecting the
logic of dependent packages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Leonard Lyubich
f25253738a [#428] reputation: Implement local trust storage
Implement in-memory `Storage` which is going to be used to submit the
results of interactions with network members. `Storage` also provides an
iterator interface, so the component can be used in `Controller`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Leonard Lyubich
29e974df93 [#428] reputation: Implement local trust controller
Implement `Controller` designed to analyze the values collected by the local
node (the source is hidden behind Writer/WriterProvider interfaces) and
transfer them to the destination (hidden behind Iterator / IteratorProvider
interface).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Leonard Lyubich
eadb3204f0 [#428] reputation: Define basic types
Define PeerID, TrustValue and Trust types and basic methods on them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Leonard Lyubich
383d2494eb [#425] Adapt the Client's refactoring to the interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Leonard Lyubich
4640099a67 [#425] control: Implement service RPCs
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Leonard Lyubich
718a2fad26 [#425] services: Define service interfaces that was removed from API lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Alex Vanin
892b4f4f12 [#426] service/object: Fix put duration
At object.Put operation node does not transfer any
data. It is done in the stream inside, so we should
register duration from `object.Put` invocation until
`CloseAndRecv` on the stream.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
b8e10571c6 [#426] Put prometheus behind pkg/metrics
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
bc6e7a823b [#426] service/object: Add object size metrics
We can't rely on object size in the header because it
might be omitted on initial put or it can be 0xFF.. on
streaming data.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
0b93e8a029 [#426] service/object: Add request duration metrics
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
f8ab7a9c60 [#426] service/object: Add object service metrics collector
Collect number of request per method in object service.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Evgenii Stratonikov
6679d59e89 [#422] pkg/services: Provide client options on cache creation
Because options are not used when client is already in cache
providing them to shared cache is misleading at best.
In the worst case `dial_timeout` is set randomly (because of race
condition) which can lead to one service having `dial_timeout` of
another. Thus we set default client creation options when cache is
created.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-03-15 14:07:53 +03:00
Evgenii Stratonikov
cc7287d6f7 [#422] pkg/services: Cache clients by address only
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-03-15 14:07:53 +03:00
Leonard Lyubich
ed808c3f1b [#419] eacl: Replace eACL storage implementation to app-side
Replace `eacl.Storage` interface implementation from eACL lib to neofs-node
app package. Remove `eacl.WithMorphClient` option in order to abstract eACL
validator from eACL storage implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-04 16:33:29 +03:00
Leonard Lyubich
5900975d58 [#217] object/policer: Leave readability instead of performance comment
Right now we pass redundant copy to callback outside the for-loop through
the helpful boolean variable instead of calling it deeply nested.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
277e3ca20a [#217] policer: Handler redundant local copy of the object
Detect redundant local copy of the object in Object Policer. Add redundant
copy callback (`WithRedundantCopyCallback` option). Pass address of the
redundant copy to callback.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
aa24702ebe [#401] control: Serve DropObjects RPC
Re-compile protobuf definition of Control service. Implement required
messages on DropObjects RPC request and response messages. Implement
`DropObjects` method on Control service server of the node. Use
`StorageEngine.Delete` method as a deleted object handler on server.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 13:07:53 +03:00
Leonard Lyubich
f2337baedc [#401] control: Add RPC to mark objects to be removed from the node
There is a need to define storage node's interface to remove objects from
the local storage.

Define `DropObjects` RPC in `ControlService`. Define `DropObjectsRequest` /
`DropObjectsResponse` structure of the request / response messages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 13:07:53 +03:00
Leonard Lyubich
9073e198b9 [#398] cmd/node: Serve NetmapService.NetworkInfo RPC
Implement `NetworkInfo` calls on full stack of Netmap services. Current
epoch is read from node local state, magic number is read via `MagicNumber`
call of morph client.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 19:16:40 +03:00
Leonard Lyubich
b8d1144839 [#378] object/delete: Set expiration epoch of the created tombstones
Make object delete service to use network information to calculate and set
expiration of the created tombstone.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
38727c2930 [#368] object: Reject expired objects
The lifetime of an object can be limited by specifying a correspondin
well-known attribute. Node should refuse to save expired objects.

Checking objects in FormatValidator is extended with an expiration attribute
parsing step.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-15 19:04:33 +03:00
Leonard Lyubich
f85e5f0238 [#202] object/eacl: Fix algorithm for checking eACL table
Add SignWithRFC6979 option to signature verification function since eACL
table are signed by users with this option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-03 15:14:59 +03:00
Leonard Lyubich
ad348afcd6 [#202] object/eacl: Verify signature of eACL table
Since the contract started returning the table signature, it became
necessary to check its correctness.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-03 15:14:59 +03:00
Leonard Lyubich
bd5c70131b [#328] cmd/node: Serve Container.AnnounceUsedSpace RPC
Register recently implemented handler of AnnounceUsedSpace RPC in node app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
21133aee2f [#328] container/load: Implement route builder based on placement
Implement route Builder interface on wrapper over the container placement
builder, Component implies exactly one transfer to each of the most weight
nodes of the container (according to some weighing algorithm).
Implementation is planned for use when transferring local estimates of
storage nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
d48fb81193 [#328] container/load: Implement route controller
Implement a component for transmitting the value of the used container space
along a route defined in the system. Implement WriterProvider interface on
it. By implementation, it is the link between the route planner and the
point-to-point transmitter, and abstracts from the implementation of both of
them. In the future, this implementation will be used as a transmitter of
local estimates of storage nodes among themselves.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
4415f8dc5b [#328] container/load: Implement simple Iterator and Writer providers
Implement functions to wrap Writer or Iterator. The resulting wrapper
provides WriterProvider or IteratorProvider interface respectively.
Such a wrapper can be used as a single storage instance provider
regardless of context.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
2622e11ae3 [#328] container/load: Implement local storage of used space announcements
Implement a component that stores the values of the used space of
containers. The storage allows you to write several values for a fixed
container and epoch number, and read the averaged estimates of all
accumulated values. All values are stored in memory. This component is
planned to be used as an accumulator of opinions from various network
participants about the fullness of the container.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
cc88320d6b [#328] container/load: Implement a metrics exchange controller
Implement a component that connects the value stores of the used space of
containers. Implement the Start/Stop operations on it, which will later
become the application handlers of the corresponding events from the
sidechain. The main task of the controller is to temporarily synchronize the
stages of calculating the global estimate of the used space in the
container. The details of the score calculation (the way of collecting /
transmitting local scores, the final score formula and writing to the
contract) are encapsulated in the dependency components, the controller is
abstracted from them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Alex Vanin
e252f93dbf [#350] object/acl: Make best effort to classify request
Classifier looks at list of inner ring nodes and container
nodes from current and previous epoch to classify request.
Sometimes these checks might return error.

Consider there is a request from unknown key and container's
placement policy valid for current epoch and invalid for past
epoch. Classifier tries to find if key belongs to container
node from current epoch -- it is not. Then it tries to find if
key belongs to container node from past epoch and it throws
error, because placement policy is invalid for past epoch.

This is a legit case and classifier should ignore such errors
to provide best effort in matching. The only error classifier
should return is an error when request does not contain
public key to classify it.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-27 20:18:33 +03:00
Leonard Lyubich
c77d346016 [#339] services/netmap: Use dynamic node information
Replace static NodeInfo structure with NodeState interface that provides
method to read node information in runtime.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-25 17:13:46 +03:00
Leonard Lyubich
619f8826e1 [#315] control: Implement SetNetmapStatus on Server
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 17:46:12 +03:00
Leonard Lyubich
f39d08bda7 [#315] control: Add SetNetmapStatus rpc
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 17:46:12 +03:00
Leonard Lyubich
e5dc4ede57 [#312] control: Add newline at the end of proto file
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
f327024ee7 [#312] control/server: Extended HealthChecker interface
Add HealthStatus method to HealthChecker interface that should return
current health status of the node application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
6ae51e41f9 [#312] control: Add health status field to health check response body
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
0abb8acef3 [#312] control: Define HealthStatus enum
Define enumeration of the statuses of storage node application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
83d4420a30 [#313] control: Rename HealthStatus enum to NetmapStatus
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Alex Vanin
a89567a88d [#317] morph/client: Return complete eACL signature from contract
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-15 16:25:49 +03:00
Leonard Lyubich
c75a828adf [#311] control: Fix loss of parent attribute keys in netmap converter
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
6c9dab481c [#311] control: Cover StableMarshal methods with unit tests
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
5f22e7fac7 Add newline at end of Control service proto file
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
36d8e2d3b7 [#311] control: Implement JSON marshaler on Netmap message
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
dde24bc9e5 [#311] control: Rewrite methods for binary encoding and signatures
Rewrite marshalers and sizers using proto library from API repo.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
e9d4b2d25b [#311] Recompile protobuf files
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
3e882d963f [#311] services/control: Implement NetmapSnapshot method
Implement methods required for signatures. Receive network map from netmap
storage, convert it to Control service message and return in response.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
28777911fb [#311] services/control: Define NetmapSnapshot rpc
Add NetmapSnapshot rpc to ControlService protobuf definition. Recompile
proto files. Add required method to server structure.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
88023f3655 [#311] services/control: Define Netmap structure
Define NodeInfo protobuf type. Define Netmap protobuf type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
f311be3ab6 [#234] Fix ineffassign linter remark
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
0a0ee89665 [#306] control: Rename WithAllowedKeys function to WithAuthorizedKeys
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
abd9952e46 [#306] Rename Private service to Control service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
44a0fb5a69 [#306] private: Define and use HealthChecker interface to get the status
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
c1b8a4815f [#306] private: Implement server of gRPC private node service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
e75ddb0549 [#306] private: Implement setters on generated proto messages
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
85ec633938 [#306] Define and compile proto files for private node service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
1e170c3812 [#234] services/object: Support netmap epoch and lookup dead in read ops
Support processing of NetmapEpoch and NetmapLookupDepth X-headers when
processing object read operations. Placement for operations
Get/Head/GetRange/GetRangeHash/Search is built for the epoch specified in
NetmapEpoch X-header (by default latest). Also the specified operations are
processed until success is achieved for network maps from the past up to
NetmapLookupDepth value. Behavior for default values (zero or missing) left
unchanged.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-12 19:19:11 +03:00
Leonard Lyubich
2f4d90025f [#234] core/netmap: Extend Source interface
Add GetNetMapByEpoch method. Add Epoch method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-12 19:19:11 +03:00
Alex Vanin
e88c1f750d [#302] Fix golint else/return linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
74a68af5c2 [#302] Fix golint underscore linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
04c0c1b8f5 [#302] Fix errcheck linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
5e231c515a [#302] Fix whitespace linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Leonard Lyubich
3a7f6701d8 [#290] object/rangehash: Apply salt to data before hashing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:40:59 +03:00
Leonard Lyubich
0e2c761481 [#290] object/rangehash: Add salt to RangeHashPrm structure
Add binary salt field to RangeHashPrm struct. Implement field setter. Set
salt from the request in v2 service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:40:59 +03:00
Leonard Lyubich
9a86fff7e0 [#291] Remove some unused code from repository
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
Leonard Lyubich
a51211eda7 [#291] Remove unused code from pkg/services/object/head
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
Leonard Lyubich
3260e9263e [#291] Remove unused pkg/services/object/rangehash packages
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
Leonard Lyubich
d299d94049 [#291] Remove unused pkg/services/object/search/query packages
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
Leonard Lyubich
1dd1762163 [#285] object/put: Fix the loss of X-headers when forwarding objects
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-29 12:22:05 +03:00
Leonard Lyubich
2897e83fb2 [#285] object/eacl: Validate X-headers from the requests, not the responses
In previous implementation of eACL service v2 the response X-headers were
validated at the stage of re-checking eACL. This provoked a mismatch of
records in the eACL table with requests. Fix this behavior by checking the
headers from the request, not the response.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-29 12:22:05 +03:00
Leonard Lyubich
c69f867af1 [#285] services/object: Add X-Headers to client call options
Forward request X-headers to client calls during internal processing of
Object operations on the node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-29 12:22:05 +03:00
Alex Vanin
7174abcc7c [#281] service/audit: Fix typos
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
e5108cf135 [#281] service/audit: Run each SG check in separate routing at PoR
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
0d7832f5e9 [#281] service/audit: Add separate pool for SG checks in PoR
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
8e72abaab7 [#281] service/audit: Make report structure threadsafe
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
44d9f95137 [#276] service/audit: Count Head requests and retries in PoR
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
9273a89f61 [#276] service/audit: Use random nodes for object.Head in PoR
Since PoR audit check uses object.Head with large TTL values to
make real proof of retrievability, we may use random remote
nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
10b548275a [#276] innerring: Use Head with TTL in PoR
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
3ec342b2c3 [#259] services/audit: Implement PDP check
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
6977adec66 [#273] service/audit: Ignore coverage of very small objects
PDP audit check is not quite working with very small objects, so
we try to build coverage with bigger objects.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
afb83c610c [#273] service/audit: Add cache of head responses in PoR
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
9212864f42 [#258] services/audit: Implement PoP check
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
a5320408a5 [#271] service/audit: Implement PoR
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
5f65ec0265 [#271] audit/report: Add PoR related setters
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
85773c419b [#271] service/object: Fill all fields of new short header structure
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
af7ae0ec4c [#271] acl: Add permission to get objects for inner ring
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
dd5e67ff0c [#271] service/audit: Add netmap structure to audit task
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
64b022a5ed [#271] placement: Add FlattenNodes function
This function converts 2-dimension array of container nodes
into single dimension array. Useful when we need to iterate
over whole container.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
58033a3adc [#271] placement: Add BuildObjectPlacement function
This function allows to shuffle prepared container nodes
according to HRW weights from network map.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
2ebcbe70fb [#255] ir/audit: Make task manager to return number of skipped tasks
Add numeric return from TaskManager.Reset method that shows the number of
canceled tasks. This values will be used for assessment of the progress of
the audit.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
03e3afb0e8 [#255] services/audit: Define interface of container communicator
Define interface of the container communicator which methods are going to be
used in audit checks. Make innerring Server to implement this interface.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
76d4e53ea0 [#255] services/audit: Skip all tasks from previous epoch in audit processor
Implement Reset method on audit task manager that cleans task queue.
Extended TaskManager interface with Reset method on IR side. Call Reset
method in audit processor before new audit start.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
33804e024d [#255] services/audit: Complete audit report after the last check
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
580c9c974a [#255] services/audit: Implement task manager
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
0f0be2377b [#255] services/audit: Implement audit executor without checks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
babfbc18f2 [#255] services/audit: Implement Task structure
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
076f201807 [#255] services/audit: Define Report structure and Reporter interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
43ee94a572 [#80] storageroup: Implement function to collect structure from container
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
44466f4a29 [#80] storagegroup: Implement search query generator
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
6245e58d1f [#80] services/object: Implement iterators over split-tree leaves
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
e53bf574b5 [#279] container: Use new methods to work with container format
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-24 13:38:20 +03:00
Leonard Lyubich
bbe8f34ffd [#253] object/eacl: Process only CID/OID object headers in RANGEHASH/DELETE
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-18 18:31:20 +03:00
Leonard Lyubich
9e27263df3 [#253] object/put: Fix loss of session token
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-18 18:31:20 +03:00
Leonard Lyubich
8e661d5c37 [#264] object/get: Prevent receiving payloads through GETRANGE when GET
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-18 16:42:17 +03:00
Leonard Lyubich
66f9532857 [#264] object/get: Check parent address in linking/last child
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-18 16:42:17 +03:00
Leonard Lyubich
ec21040542 [#264] transformer: Finalize parent header once
In previous implementation parent object header finalized twice in size
limiter + formatter. On the one hand, this added redundant action, on the
other hand, it could provoke a difference in the headers of the linking and
the last part. Change formatter to finalize parent header if it does not
container the signature. Change size limiter to reuse parent header after
last child finalization in linking child.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-18 16:42:17 +03:00
Leonard Lyubich
49131f1bc7 [#247] object/eacl: Use address from session token in request validation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-15 13:21:13 +03:00
Leonard Lyubich
dba88c79b4 [#247] object/range: Process only object_id and container_id headers
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-15 13:21:13 +03:00
Leonard Lyubich
8654458b19 [#247] object/eacl: Use object ID from session token context
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-15 13:21:13 +03:00
Leonard Lyubich
168dcbdccd [#247] object/eacl: Process object address from request
In previous implementation eACL validator didn't take into account container
and object ID fields of request bodies.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-15 13:21:13 +03:00
Alex Vanin
bbe700fa37 [#250] service/policer: Don't shrink node list at unknown error
Every unknown error must not decrease shortage counter and must not
exclude faulty node from the node list, because this list will be used
later for replication.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-14 21:49:50 +03:00
Leonard Lyubich
fb50362dcc [#243] services/object: Share common parameters across services
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
a01262d8bd [#243] service/object: Fix object chain assembly for tombstone body
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
9265e31e65 [#243] services/object: Fix lost tokens when generating requests
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
fe3906c295 [#243] object/delete: Implement new service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
510e9ff2ec [#243] core/object: Replace test content of tombstone with an API structure
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
627012e4fe [#241] object/search: Set container ID parameter of storage engine call
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
611a29f682 [#241] object/search: Refactor service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
3bfb18386b object/get: Replace code with function call
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
9fb7190358 [#242] service/object: Support ContainerID field in local search
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
0e1f05ff45 [#239] object/head: Implement new service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
173d34a8a2 [#235] object/get: Set common operation parameters
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
7e56427534 [#235] object/getrangehash: Implement new service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
0fc5ea674c [#235] object/getrange: Fix incorrect processing of range from last child
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
397d912e19 [#235] object/head: Support raw flag in service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1d23483828 [#235] services/object: Implement new GetRange algorithm
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
869d9e571c [#233] object/head: Remove getting the right child header
With the update of the local storage engine, the headers of virtual objects
are directly given. In this regard, the step with obtaining the the right
child header is removed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
f24daa10ff [#233] services/object: Implement new Get algorithm
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
e0350efe00 [#231] services/policer: Use engine.List method
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
351e4b4592 [#222] Support Inhume and Delete in object service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
953387a1e5 [#186] object/acl: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
046206f670 [#186] object/head: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2be8f154a0 [#186] object/put: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
14442a0801 [#186] object/range: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
9b32f131c0 [#186] object/rangehash: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1bce2a4e55 [#186] object/search: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
aa9eb2eaf2 [#186] policer: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1b867e53d7 [#186] replicator: Use new storage engine for work
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
6cddc91cbf [#227] transformer: Inherit parent header in linking object
In previous implementation linking object inherited only ID of the parent
object. From now it inherits full header like last child.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-02 14:53:04 +03:00
Leonard Lyubich
b8e8e1e80d [#227] transformer: Do not inherit attribute in generated objects
In previous implementation child objects inherited parent attributes after
split-transformation, which was redundant. From now attributes are not
inherited.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-02 14:53:04 +03:00
Leonard Lyubich
5470d94416 [#223] placement: Fix local and single-success placement traversal
In previous implementation placement traverser processed incorrectly with
local placement build. Also entity incorrectly traversed the placement
vectors for fixed number read operations until success. The erroneous
behavior was due to the use of a vector number of successes instead of
a scalar number in these scenarios.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-01 19:36:46 +03:00
Leonard Lyubich
d94a0eb25e [#221] eacl: Get rid of deprecated methods usage
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-01 11:08:47 +03:00
Alex Vanin
f96630581a [#208] Set unique splitID in children
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-30 10:44:15 +03:00
Alex Vanin
6280d075b9 [#208] Remove childfree search attribute
With updated specification of object related operation
we don't have this search attribute any more and we
should not use functions related to this attribute.

This commit breaks object service logic, however it will
be fixed later.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-30 10:44:15 +03:00
Leonard Lyubich
706bdf736e [#209] eacl: Support object ID filter in eACL mechanism
Add object ID header to the list of processing object headers in eACL
validation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-25 11:59:15 +03:00
Leonard Lyubich
fddc50fd85 [#203] Replace ErrEACLNotFound to core library
ErrEACLNotFound error was defined in implementation package. EACL validator
checked this error after the call of eACL storage interface method. Replace
ErrEACLNotFound to core container library. in order to: on the one hand not
use an implementation error, on the other hand, to be able to reuse a
generic type error (404).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 20:19:20 +03:00
Leonard Lyubich
9148980bd0 [#193] services/object: Support client options in all Object services
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 15:34:30 +03:00
Alex Vanin
cf1ea983e5 [#203] Do not fail eACL check on EACLNotFound error
Now morph library returns error if there is not eACL in
sidechain storage. However in this case eACL check should
be passed since it is the same as having empty eACL table.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-24 15:32:26 +03:00
Leonard Lyubich
46dab77705 [#195] services/object: Write debug log messages on worker pool errors
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
fa6e4a3ca4 [#195] services/object: Write debug log messages on internal service errors
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
6a5c37d592 [#195] object/search: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
cb46e4b154 [#195] object/rangehash: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
af6484e3b1 [#195] object/range: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
cf2dc37a42 [#195] object/put: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
9fbfc0b5e4 [#195] object/head: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
e1e5a590e9 [#195] object/delete: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Alex Vanin
dbf6c9efef [#190] Use request sender owner in sticky bit check
Sticky bit checks if object owner and request owner are the
same. Container owner should not used in this check.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 19:07:16 +03:00
Alex Vanin
2148e282ec [#190] Rename owner to cnrOwner in object request info
`owner` field may be misused as request sender owner, however
it is a owner of a container for that request. New naming
should be clear.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 19:07:16 +03:00
Alex Vanin
f0537b35c1 [#190] Add isOwnerFromKey helper function in ACL
This function takes public key and returns true if
owner id was produced by this key.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 19:07:16 +03:00
Alex Vanin
a14bb6292b [#182] Reuse search filter in policer
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 17:59:46 +03:00
Alex Vanin
2e605b2435 [#182] Limit policer object filter to physical stored objects
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 17:59:46 +03:00
Alex Vanin
cf85fa9fab [#180] Return isInnerRing flag in request classifier
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 15:17:18 +03:00
Alex Vanin
6f841e319d [#180] Make separate basic ACL check for inner ring requests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-19 15:17:18 +03:00
Alex Vanin
e8fe07edd0 [#184] Use SDK client cache in object.Rangehash
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
f85e88c4f8 [#184] Use SDK client cache in object.Range
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
d485a5967d [#184] Use SDK client cache in object.Search
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
7ba95dd5fc [#184] Use SDK client cache in object.Put
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
e9a6365333 [#184] Use SDK client cache in object.Head
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Leonard Lyubich
1caf15463e [#174] Update to neofs-api-go v1.20.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Leonard Lyubich
58fcb35fb0 [#174] Use Marshal(JSON)/Unmarshal(JSON) methods for encoding/decoding
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Leonard Lyubich
3de8febe57 [#174] Update to latest neofs-api-go changes
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Alex Vanin
32219bb294 [#160] Remove query match function
This function duplicates query processing that
is done in meta-storage now.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Leonard Lyubich
3c42f5b452 [#161] object/head: Inherit common parameters in HeadRelation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 17:19:34 +03:00