Commit Graph

773 Commits (2b755ddb12f9fcd9b51a38ca79b5fa88cdab27ab)

Author SHA1 Message Date
Evgenii Stratonikov 0f376a5d83 [#1388] services/control: Extend tests for `ShardInfo` marshaling
It is nice to have different paths for different components and also
check that the information returned is different for different shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-17 13:48:07 +03:00
Leonard Lyubich f15e6e888f [#1377] oid, cid: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-16 15:33:22 +03:00
Leonard Lyubich 3a188bb2e5 [#1371] bearer: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 15:16:17 +03:00
Leonard Lyubich ae92074272 [#1369] checksum: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Leonard Lyubich 088df0e2a9 [#1369] version: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Leonard Lyubich 7a57d7b076 [#1369] audit: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Elizaveta Chichindaeva cc7a723d77 [#1320] English Check
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-05-11 10:40:02 +03:00
Evgenii Stratonikov a4769d8624 *: use `require.ErrorIs` where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:47:57 +03:00
Evgenii Stratonikov 1219ff89d4 [#1319] services/object_manager: Fix error message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov 057d53459b [#1319] services/object: Remove `Write` method from `local` and `remote` targets
They work with prepared objects only.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov 4ea03c01b5 [#1319] services/object: Store payload directly in `distributedTarget`
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Pavel Karpy 6f2723b47e [#1318] node: Fix linter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy 24ca8ca5c7 [#1318] object: Change default ttl
This is need to reuse object service without passing any API v2 structs.
Also, fix comment.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy e4cfeec449 [#1318] service: Add tombstone checker service
The service fetches tombstones from the network via object service, every
request is handled in the following order:
1. checks local LRU cache;
2. checks local storage engine;
3. tries to find object in the placement nodes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Evgenii Stratonikov 5736b834c3 [#1286] services/object: Validate object when created with sesssion token
All fields set by a user should be verified.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:53:22 +03:00
Evgenii Stratonikov c6a9c5cd8c [#1283] services/object: Disallow creating objects without a session token
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:53:22 +03:00
Evgenii Stratonikov ff1912aa2a services/acl: check session token expiration epoch
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:37:29 +03:00
Evgenii Stratonikov 6472a170eb [#1143] shard: Introduce explicit `Degraded` mode
`Degraded` mode is set automatically after error counter is over the
threshold. `ReadOnly` mode can still be set by an administrator.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Alex Vanin 7ed84d1755 [#1278] acl: Return netmap.Source interface
Application can provide cached netmap source in this case.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-30 14:22:12 +03:00
Evgenii Stratonikov cf119e4ca9 [#1163] services/audit: Randomize the order of PDP checks
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-30 10:56:33 +03:00
Evgenii Stratonikov 2ad8016d75 [#1239] innerring: Use pointer-less slices for object IDs
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-29 16:02:05 +03:00
Pavel Karpy 6ec104d686 [#1255] node/session: Rename constant
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy 90a8c52bdb [#1255] object: Add persistent storage usage
Use persistent storage usage in the node if it was configured so.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy 016eaa25f3 [#1255] node/session: Add encryption tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy 01ed366e99 [#1255] node/session: Add encryption
Add `WithEncryption` option that passes ECDSA key to the persistent session
storage. It uses 32 bytes from marshalled ECDSA key in ASN.1 DER from in
AES-256 algorithm encryption in Galois/Counter Mode.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy a884ad56d9 [#1255] node/session: Add persistent tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy 455b9fb325 [#1255] node/session: Add persistent session storage
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy 929c9851a6 [#1255] node/session: Create separate dir for in-memory storage
Move in-memory session storage to the separate directory of `storage`. It is
done for future support of different kind of session storages.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Evgenii Stratonikov 414ba6e0a2 [#1244] nats: Split client creation into 2 stages
Create and connect to an endpoint using separate functions.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-24 11:51:49 +03:00
Evgenii Stratonikov 2848001dfb [#1246] object/acl: Return more concise description for eACL errors
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-21 19:20:01 +03:00
Leonard Lyubich 459bdcf04b [#1247] object/acl: Return `ObjectAccessDenied` status error
Return `apistatus.ObjectAccessDenied` error on access violation from ACL
service. Write reason in format of the errors from the previous
implementation. These errors are returned by storage node's server as
NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich 967650f2ed [#1247] container: Return `ContainerNotFound` status error
Replace `core/container.ErrNotFound` error returned by `Source.Get`
interface method with `apistatus.ContainerNotFound` status error. This
error is returned by storage node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich 70ffdf3478 [#1247] object: Return `NOT_FOUND` and `ALREADY_REMOVED` statuses
Replace `ErrNotFound`/`ErrAlreadyRemoved` error from
`pkg/core/object` package with `ObjectNotFound`/`ObjectAlreadyRemoved`
one from `apistatus` package. These errors are returned by storage
node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich f32c9670ad [#1247] object: Return status errors from `util.KeyStorage`
Return `SessionTokenExpired`/`SessionTokenNotFound` error from
`apistatus` package if private session token is expired/missing. These
errors are returned by storage node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich a4ee59977e [#1247] session: Remove unused `storage.ErrNotFound` error
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Pavel Karpy 16543a1904 [#1243] node/object: Hide engine behind interface
Do not use engine as explicit dependency in the put service. This allows
wrapping storage engine.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 14:27:52 +03:00
Pavel Karpy d02720c910 [#1243] nats: Fix comments
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 14:27:52 +03:00
Pavel Karpy c3db12d71b [#1210] reputation: Resolve race condition
Make all epoch independent in reputation process. Do not reset any timers
related to reputation. Make it possible to finish iteration after the
unexpected `NewEpoch` event.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-16 17:18:30 +03:00
Pavel Karpy 13af4e6046 [#1210] reputation: Improve debug logs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-16 17:18:30 +03:00
Alex Vanin 362cda53d2 [#1238] Call `String()` explicitly instead of using stringer pointer
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Alex Vanin 8f476f3c4d [#1238] Adopt neofs-node for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Alex Vanin 9fad29dfe0 [#1238] Adopt neofs-ir for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Alex Vanin 7f3195b197 [#1238] Adopt neofs-cli for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Leonard Lyubich ebd84f6dd4 [#1175] object/put: Perform additional container broadcast of LOCKs
`LOCK` objects should be broadcast across container like `TOMBSTONE`
ones.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich 323dea95c6 [#1181] Do not use deprecated elements
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich 8a2a096680 [#1175] services/sign: Convert `error` to status return
In previous implementation `SignService` converted all `error` values to
`INTERNAL` server failure status. That was done for simplification only.
There is a need to transmit status errors as corresponding status
messages.

Make `SignService` to unwrap errors and convert them to status message
during writing to the response. Non-status errors are converted to
`INTERNAL` server failures. Status errors can also be wrapped in the
depths of the executable code, so `SignService` tries to unwrap them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Alex Vanin b6720d5f97 [#1231] Update new SDK Client interface
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-14 13:18:48 +03:00
Alex Vanin e38d7dda6b [#1230] replicator: Improve error log
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-10 19:19:18 +03:00
Leonard Lyubich 14ea7b2c47 [#1219] object/delete: Change approach of the tombstone broadcast
After `putsvc.Service` started to support additional container broadcast
of the saved objects there is no more need to perform broadcast of
tombstone object in `deletesvc.Service`.

Make `putsvc.Service` to perform additional broadcast of `TOMBSTONE`
objects. Remove `broadcastTombstone` stage from `deletesvc.execCtx`,
from now it is encapsulated in `saveTombstone` stage. Remove no longer
needed `putsvc.PutInitPrm.WithTraverseOption` method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-05 16:23:21 +03:00
Leonard Lyubich a55af18ad1 [#1219] object/put: Support additional container broadcast
There are several cases when we need to spread the object around the
container after its primary placement (e.g. objects of type TOMBSTONE).
It'd be convenient to support this feature in `putsvc.Service`.

Add additional stage of container broadcast after the object is stored.
This stage is carried out no more than once and does not affect the
outcome of the main PUT operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-05 16:23:21 +03:00
Evgenii Stratonikov 0bf59522f7 [#1191] object/acl: check session token verb
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-05 16:03:16 +03:00
Pavel Karpy 79c6f52a27 [#1183] node: Add NATS client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 15:57:24 +03:00
Pavel Karpy 03b601b594 [#1183] node: Init notificator on startup
It uses logger as a notification writer. NATS messaging will be implemented
later.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 11:41:18 +03:00
Pavel Karpy 90ff066940 [#1183] node/notificator: Implement notificator
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 11:41:18 +03:00
Pavel Karpy cf17686321 [#1111] control: Rename aliaces
`objectSDKAddress` -> `addressSDK`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 10:06:14 +03:00
Pavel Karpy 99b31e3235 [#1111] object/acl: Refactor service
Make all operations that related to `neofs-api-go` library be placed in `v2`
packages. They parse all v2-versioned structs info `neofs-sdk-go`
abstractions and pass them to the corresponding `acl`/`eacl` packages. `v2`
packages are the only packages that do import `neofs-api-go` library. `eacl`
and `acl` provide public functions that only accepts `sdk` structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 10:06:14 +03:00
Leonard Lyubich 7ccd1625af [#1214] *: Use single `Object` type in whole project
Remove `Object` and `RawObject` types from `pkg/core/object` package.
Use `Object` type from NeoFS SDK Go library everywhere. Avoid using the
deprecated elements.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 17:45:00 +03:00
Alex Vanin cbc816d57d [#1211] node: Fix object get init error check
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Leonard Lyubich b5bdcfc076 [#1194] client: Support request X-headers
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-03 15:38:38 +03:00
Leonard Lyubich e0dce1043a [#1195] Adopt recent changes in NeoFS SDK
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-02-28 10:43:03 +03:00
Evgenii Stratonikov 47cd10a4d7 [#767] services/object: replace duplicate link in TODO
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:59:27 +03:00
Evgenii Stratonikov 050a4bb2b0 [#1115] *: link TODOs to corresponding issues
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov 692790a899 [#1115] services/object: reuse errors from core service
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov 0695ec4125 [#1133] services/session: remove expired tokens every epoch
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 14:58:32 +03:00
Pavel Karpy 1667ec9e6d [#1131] *: Adopt SDK changes
`object.Address` has been moved to `object/address`
`object.ID` has been moved to `object/id`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-02-08 09:45:38 +03:00
Evgenii Stratonikov 6f50fefbea [#625] client/balance: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov ed50cf6207 [#1118] services/control: return error counter in `ListShards`
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Evgenii Stratonikov 7fb15fa1d0 [#1118] services/control: allow to reset error counter in `SetShardMode`
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Evgenii Stratonikov a82ac0edcd [#1023] services/object: remove TTL from `Get` parameters
It was unused an should be provided as a `CallOption` instead.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 13:01:58 +03:00
Evgenii Stratonikov fbb95cff14 [#1094] services/object: remove some FIXME comments
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov 1bdbd6ed22 [#1094] eacl/v2: preallocate slice in `headersFromObject`
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov 18cfd8b042 [#1086] services/control: implement `RestoreShard` RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov 2ec4a3c897 [#1086] neofs-cli: add command for shard dump
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov 0e60b1d6c9 [#1086] services/control: implement `DumpShard` RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov aa53418119 [#1086] services/control: fix comments
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Leonard Lyubich 87a45983a1 [#1084] accounting: Keep balance precision from contract
Do not convert balance response of Balance contract to Fixed8.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-26 14:10:02 +03:00
Pavel Karpy fee2f5a330 [#1100] *: Adopt new SDK's `owner.ID` API
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 17:58:30 +03:00
Leonard Lyubich 4b5404047c [#1098] object/search: Fix error loss in `SignService.Search`
Error loss led to incorrect response formation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-21 14:54:22 +03:00
Pavel Karpy 202dbfb95d [#1096] eacl: Rename `eacl` import aliases
Since `Validator` has been moved to SDK it is more readable to see `eaclSDK`
than `acl` import.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 10:47:33 +03:00
Pavel Karpy ed156cd738 [#1096] eacl: Use validator from SDK
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 10:47:33 +03:00
Pavel Karpy 7c02a2e251 [#1087] *: Adopt SDK changes
- Update `neofs-sdk-go`:
v0.0.0-20211230072947-1fe37df88f80 => v0.0.0-20220113123743-7f3162110659

- Add client interface that duplicates SDK's client behaviour and new
`MultiAddressClient` interface that has method that iterates over wrapped
clients.

- Also start using simple client mode that does not require parsing statuses
outside the SDK library.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-14 17:29:03 +03:00
Alex Vanin 0f1eb743af [#1077] service/object: Don't override status of payload overtake
Reverse payload overtake triggers direct payload overtake that
sets status and error. We should not override that.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-01-13 10:30:18 +03:00
Evgenii Stratonikov 5828f43e52 [#851] util/rand: use single random source
It is much more convenient to skip source creation.
Also fix some bugs:
1. `cryptoSource.Int63()` now returns number in [0, 1<<63) as required
   by `rand.Source` interface.
2. Replace `cryptoSource.Uint63()` with `cryptoSource.Uint64` to allow
   generate uint64 numbers directly (see rand.Source64 docs).

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-12 19:29:27 +03:00
Leonard Lyubich 0b0c61f4a5 [#1063] node/object: Do not skip empty search responses in splitter
In previous implementation `Search` method of transport splitter skipped
responses with empty ID list.

Replace while-loop with do-while one in `TransportSplitter.Search`
method implementation in order to send responses with empty result too.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-11 12:02:54 +03:00
Pavel Karpy 97fd330fae [#1059] control: Fix `ListShards` wrapper name
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy 77808c7b41 [#1059] control: Provide shard's real mode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy 88bb8e2df9 [#1059] control: Add `SetShardMode` implementation to ctrl svc
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy fa78a37787 [#1059] control: Add unit test for `SetShardMode`
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy 29c69f37eb [#1059] control: Add `SetShardMode` RPC wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy 25665cb0fd [#1059] control: Add marshalling and setters to `SetShardMode`
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy 4533784e11 [#1059] control: Recompile proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy 41dd385b7b [#1059] control: Add `SetShardMode` rpc to proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy 876b0c53de [#1048] control: Rename `Blobstore` to `Blobstor`
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy f5a9735e1c [#1048] cli: Add `shard list` operation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy 0e5410603e [#1048] control: Add `ListShards` implementation to ctrl svc
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy 4e989e7133 [#1048] control: Add unit test for `ListShards`
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy f88a12eaa7 [#1048] control: Add `ListShards` RPC wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy a6b3e16975 [#1048] control: Add marshalling and setters to `ListShards`
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy a42b3d37f6 [#1048] control: Recompile proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy 20f11c88ca [#1048] control: Add `ListShards` rpc to proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Alex Vanin 5d46035ae8 [#1052] Tidy INFO logs
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-27 14:28:01 +03:00
Alex Vanin e610d1ea5f [#1045] Provide sanity check of session token in container service
Without sanity check, container service provides successful response,
even though such request will never be approved by Alphabet nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-20 14:51:07 +03:00
Alex Vanin bca7cf9470 [#1047] policer: Check context before job selection
When application is being terminated, replicator routine
might be on the object picking phase. Storage is terminated
asynchronously, thus `Select()` may return corresponding
error. If we don't process `context.Done()` in this case,
then application freezes on shutdown.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-20 14:42:57 +03:00
Pavel Karpy df0d76dc19 [#1032] node/container/router: Do not check nil route points
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 16:02:29 +03:00
Pavel Karpy fd9514f90e [#1032] node/container/router: Fix interface description
After #872 `NextStage` method description becomes outdated.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 16:02:29 +03:00
Leonard Lyubich 6f41cd86ed [#979] Upgrade SDK NeoFS Go dependency
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich e8f8e58e90 [#979] adm: Add subnet-related commands
Add `subnet` command which contains all subnet-related commands. Add
sub-commands:
  * `create` for creation;
  * `remove` for removal;
  * `get` for reading;
  * `admin` for admin management;
  * `client` for client management.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Pavel Karpy acf7331034 [#990] object: Fix linter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 17:18:08 +03:00
Alex Vanin 011d0f605b [#965] replicator: Make HandleTask function public
Continues replication executed in separate pool of goroutines,
so there is no need in worker to handle replication tasks
asynchronously.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-26 15:39:38 +03:00
Alex Vanin a74a402a7d [#965] policer: Implement continuous replication
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-26 15:39:38 +03:00
Leonard Lyubich 7f5fb130c0 [#961] *: Support NeoFS API status returns
Upgrade NeoFS API Go library to version with status returns. Make all API
clients to pull out and return errors from failed statuses. Make signature
service to respond with status if client version supports it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-24 09:26:40 +03:00
Leonard Lyubich 44d0d453ef [#961] client: Clarify error returns in docs
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-24 09:26:40 +03:00
Evgenii Stratonikov 95893927aa *: replace neofs-api-go with neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-12 17:29:09 +03:00
Leonard Lyubich 8f8b638b95 [#922] control: Add MAINTENANCE netmap status
Add `MAINTENANCE` value to `NetmapStatus` enum in Control API. The status is
going to be used to toggle maintenance mode of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich 0ec8f529ab [#842] object/delete: Set tombstone local node as tombstone owner
All objects in NeoFS must have owner ID. In previous implementation Object
Delete service handler set owner ID from request session token. If removal
was executed w/o a session, object with tombstone was prepared incorrectly.
In order to fix this node should set its own ID and become an owner of the
tombstone object.

Extend `NetworkInfo` interface required by Object.Delete handler with
`LocalNodeID` method which returns `owner.ID` of the local node. Implement
the method on `networkState` component of storage node application which is
updated on each node state change in NeoFS network map. Set owner returned
by `LocalNodeID` call as tombstone object's owner in Delete handler.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 16:18:02 +03:00
Leonard Lyubich d421022547 [#842] object/acl: Check owner and key only if sticky bit is set
In previous implementation node returns "access denied" on Object.Put with
object with unset owner. Although object owner must be set, its absence
should not be considered as access error. The same applies to sender key.

Check owner ID and public key emptiness only if sticky bit is set.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 16:18:02 +03:00
Leonard Lyubich b3708fc530 [#957] services/object: Refactor usage of NeoFS API client
The client needs of the Object service are limited and change not often.
Interface changes of the client library should not affect the operation of
various service packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the Object service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-03 18:30:09 +03:00
Alex Vanin fbed86da2c [#943] service/object: Remove KeyStore from CommonPrm
There is no point to pass key storage in parameters because
it can be defined on the service level of application.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin c30aa20b04 [#943] service/object: Refactor private key fetching during execution
`CommonPrm` structure has private key for remote operations.
It obtained in the beginning of request processing. However,
not every operation triggers remote calls. Therefore, key
might not be used. It is important to avoid early key fetching
because `TokenStore` now returns error if session token does not
exist. This is valid case when container nodes receive request with
session token (for ACL pass) and they should process request locally.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin 2fbdcbdee1 [#943] service/object: Check session token expiration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin 508a28fdc0 [#943] service/object: Do not sign put service message with session token
This is invalid operation for storage nodes that receive part of split
object. While object is signed by session token, the message itself
should be signed by the node key.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin 46fa07b7cc [#943] service/object: Return error if requested session token is not available
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Evgenii Stratonikov 0057eeb0f7 [#901] *: release worker pools where possible
Some of the pools are initialized during config initialization,
so it isn't possible currently to release them in one place.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Evgenii Stratonikov 0beaed2ef4 [#901] util: implement `Release` for `PseudWorkerPool`
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Evgenii Stratonikov 7cb3d0cb4a [#885] policer: remove objects for removed container
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-18 12:14:14 +03:00
Leonard Lyubich 283ccc04b4 [#833] services/netmap: Support new fields of netmap.NetworkInfo
Make the implementation of network info source (Netmap V2 service
dependency) to read MillisecondsPerBlock sidechain parameter and NeoFS
network parameters depending on the client version.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Leonard Lyubich 40a4a7faa2 [#674] object/put: Use pseudo worker pool for local operations
After storage engine started to limit number of PUT operations there is no
need to limited worker pool in Object Put service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich 07130855aa [#674] util: Rename SyncWorkerPool with PseudoWorkerPool
`SyncWorkerPool` name is more appropriate for worker pool of size 1.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Alex Vanin d8f7fed10a [#881] acl: Use session token from request at object.Put
Session token can be present in both object header and
request meta header. They are the same during initial object
placement.

At the object replication, storage node puts object without
any session tokens attached to the request. If container's eACL
denies object.Put for USER role (use bearer to upload), then
replication might fail on objects with session tokens of the
signed by container owner. It is incorrect, so use session
token directly from request meta header.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-08 17:18:32 +03:00
Evgenii Stratonikov b8ba677c85 [#882] policer: add CID to the error message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 08:21:01 +03:00
Alex Vanin e41e74b5fc [#872] services/container: Ignore passed routes in load router
Higher level solution for original issue with lost announcements
from single node container.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-05 19:29:27 +03:00
Leonard Lyubich 4661f65975 [#645] client/cache: Check response public key in all client operations
There is a need to check if public key in the RPC response matches the
public key of the related storage node declared in network map.

Define `ErrWrongPublicKey` error. Implement RPC response handler's
constructor `AssertKeyResponseCallback` which checks public key. Construct
handler and pass it to client's option `WithResponseInfoHandler`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich 3a497a3c53 [#645] object/put: Pass NodeInfo instead of AddressGroup to relay func
Replace `network.AddressGroup` parameter type of relay function with
`client.NodeInfo`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich e473f3ac91 [#645] *: Use helper functions to build client.NodeInfo structures
Helper functions from core/client package allow to set public keys of
storage nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich 2d441a4cc6 [#645] placement/traverser: Rename Key method of Node type to PublicKey
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich 7b228b7603 [#645] *: Construct clients from client.NodeInfo in API client cache
There is a need to have the ability to expand the data needed for client
construction.

Replace `network.AddressGroup` parameter of client cache interfaces with
`client.NodeInfo`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich be05bed0b3 [#860] Re-compile protobuf files after `make protoc` target update
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-29 17:51:33 +03:00
Leonard Lyubich ee20200c2e [#845] object/put: Separate pools for local and remote operations
In previous implementation Object PUT used single pool of workers for local
and remote ops, but these ops are heterogeneous.

Use remote/local pool for remote/local operations in PUT service. At first
the pools are configured with the same size.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 16:20:19 +03:00
Leonard Lyubich 3b2b6007c6 [#845] object/put: Make distributed target to define is node is local
In previous implementation `distributedTarget` didn't check if next node is
local. This check was performed by the handlers (target initializer and
relay func).

Make `distributedTarget` to calculate node's locality. Pass locality flag to
the handlers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 16:20:19 +03:00
Leonard Lyubich 73fb1a886c [#849] policer: Write message about redundant local object copy
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Leonard Lyubich d613a856ce [#849] policer: Log object address in processNodes method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Alex Vanin b97cdbea9e [#839] service/object: Add high level functions to control request forwarding
Applicable for object/get execution context.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-27 11:10:50 +03:00
Alex Vanin b3464e8140 [#839] service/object: Do not use request forwarding when assembling object
Forwarding mechanism resends original request. During split object chain traversal,
storage node performs multiple `object.Head` requests on each child. If request
forwarding happens, then `object.Head` returns object ID of the original request.
This produces infinite assembly loop.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-27 11:10:50 +03:00
Alex Vanin 01df4ffa61 [#811] service/container: Hide cache invalidation logic in Writer interface
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Alex Vanin 4a1f0de8f4 [#811] service/container: Invalidate cache records on update requests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Leonard Lyubich 61b4baf736 [#778] services/object: Wrap last client's error into errIncompletePut
Make `errIncompletePut` to be a structure which wraps single client error.
Wrap error of the last client into `errIncompletePut` during placement
execution.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 16:49:37 +03:00
Leonard Lyubich 02f2a98bcc [#778] services/object: Return the composite error of PUT operation
In previous implementation Object service's handler returned const error in
case of failure (full or partial) of PUT operation. This did not even allow
us to roughly guess what the reason is. Not as a complete solution, but to
alleviate some cases where all nodes in a container return the same error,
it is suggested to return the error of the last server that responded.

Return latest server error from placement loop of `iteratePlacement` method
of `distributedTarget` type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 16:49:37 +03:00
Evgenii Stratonikov 5f86d54721 [#816] object/acl: fix eACL target processing
Ignore role if public keys are present.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-09-10 15:15:36 +03:00
Leonard Lyubich 7a13053fab [#818] object/acl: Fit sticky bit specification
In previous implementation sticky bit could disrupt access of container
nodes to replication. According to NeoFS specification sticky bit should not
affect the requests sent by nodes from SYSTEM group.

Add role check to `stickyBitCheck`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 15:14:59 +03:00
Pavel Karpy 4a81781c0c [#770] logger: Delete spaces in log message keys
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Leonard Lyubich 358e3ed8c4 [#645] *: Change the locality condition of the node from the placement
Some software components regulate the way of working with placement arrays
when a local node enters it. In the previous implementation, the locality
criterion was the correspondence between the announced network address
(group) and the address with which the node was configured. However, by
design, network addresses are not unique identifiers of storage nodes in the
system.

Change comparisons by network addresses to comparisons by keys in all
packages with the logic described above. Implement `netmap.AnnouncedKeys`
interface on `cfg` type in the storage node application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich 22926e8f28 [#645] placement/traverser: Extend Node type with public key
There is a need to process public keys of the placement result.

Implement `Node.PublicKey` method which returns storage node's key announced
in netmap.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich fe90456dcc [#645] placement: Overload result of Traverser.Next method
In previous implementation `placement.Traverser.Next` method returned slice
of `network.AddressGroup` elements. There is a need to process keys of
storage nodes besides network addresses for intra-container communication.

Wrap `network.AddressGroup` in a new type `placement.Node` that summarizes
the storage node information required for communication. Return slice of
`Node` instances from `Traverser.Next` method. Fix compilation breaks in
dependent packages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich e738699fcc [#676] services/container: Cache the results of read operations
In previous implementation Container service handlers didn't cache the
results of `Get` / `GetEACL` / `List` operations. As a consequence of this,
high load on the service caused neo-go client's connection errors. To avoid
this there is a need to use cache. Object service already uses `Get` and
`GetEACL` caches.

Implement cache of `List` results. Share already implemented cache of Object
service with the Container one. Provide new instance of read-only container
storage (defined as an interface)to morph executor's constructor on which
container service is based. Write operations remained unchanged.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-08-30 18:01:26 +03:00
Alex Vanin d6439369ab [#768] object/acl: Support all eACL filter fields from the spec
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 19:01:03 +03:00
Leonard Lyubich abfcc7498c [#715] services/policer: Select pseudo-random list of objects to check
In previous implementation of Policer's job queue the same list of objects for
processing was selected at each iteration. This was caused by consistent
return of `engine.List` function.

Use `rand.Shuffle` function to compose pseudo-random list of all objects in
order to approximately evenly distribute objects to work.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-08-25 14:40:12 +03:00
Pavel Karpy e8665f6cef [#730] node/morph: Use `disable_cache` config param
Do not init caches for eACL, containers and netmap
if `disable_cache` config options is `true`, use
direct RPC calls instead.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 09:40:10 +03:00
Pavel Karpy c423aa432a [#730] node/config/morph: Add `disable_cache`
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 09:40:10 +03:00
ZhangTao1596 01806db612 [#723] container/announcement/storage: Map announcements by epoch and container id
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2021-07-30 09:58:30 +03:00
Leonard Lyubich 7c1432aa57 [#660] services/container: Don't check container format in Put
Storage node should only reflect container Put requests to sidechain w/o any
validation. All verification steps are performed by IR.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-07-05 11:05:44 +03:00
Leonard Lyubich 8192933313 [#660] services/netmap: Don't call IsSupportedVersion in LocalNodeInfo
Clients can have version later than the server. The server should not deny
these clients.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-07-05 11:05:44 +03:00
Leonard Lyubich 8eadf18e01 [#607] services/control: Fix incorrect allocation in NetmapSnapshot
Fix make with length + subsequent append.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich f8ee6b1164 [#607] services/netmap: Support backward compatibility
Make Netmap service to return single network address on `LocalNodeInfo` RPC
in order to older clients.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich 43eff09944 [#607] *: Do not use deprecated elements of code
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich 1e52e86bbc [#607] node/control: Make group address in NodeInfo message
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich cede2b4ed7 [#607] reputation,container: Support address groups in ServerInfo
There is a need to support multiple server endpoints for reputation and
container transmission.

Replace `ServerInfo.Address` getter with `ServerInfo.IterateAddresses`
iterator.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich d6bb697726 [#607] container/announce: Use public keys as keys to server cache
In future server info will contain multiple endpoints whose string
representation will be worse suited to the cache key.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich 44a2c81f8e [#607] reputation/router: Use public keys as keys to server cache
In future server info will contain multiple endpoints whose string
representation will be worse suited to the cache key.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich 9033cf5aa1 [#607] placement: Work with AddressGroup
Make `PlacementBuilder` implementations to work `network.AddressGroup` type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich b3dd9a3254 [#607] placement: Make traverser to return list of address groups
Make placement `Traverser.Next` method to return ``[]network.AddressGroup`
in order to support multiple addresses of the storeage nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich 8ac3c62518 [#607] object/head: Make client constructor to work with group address
Make Object Head service to work with `AddressGroup` instead of `Address`
in order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich d0e48c949b [#607] object/search: Make client constructor to work with group address
Make Object Search service to work with `AddressGroup` instead of `Address`
in order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich ad14df07f6 [#607] object/get: Make client constructor to work with group address
Make Object Get service to work with `AddressGroup` instead of `Address` in
order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich 8972f84672 [#607] object/put: Make client constructor to work with group address
Make Object Put service to work with `AddressGroup` instead of `Address` in
order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich 6e5d7f84af [#607] network: Generalize LocalAddressSource to address group
Make `LocalAddressSource.LocalAddress` method to return `AddressGroup`. Make
`IsLocalAddress` function to accept parameter of type `AddressGroup`. Adopt
the application code with temporary `GroupFromAddress` helper.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich 3805b0f638 [#607] client: Overload Client interface
There is a need to generalize single-address client to group-address client.
To do this, we can re-implement `Client` interface from NeoFS API Go library
and still use it in the application code. There is a problem with method
`Raw` which must return single-address raw client. So as not to make changes
to API library we need to overload Client interface in order to support
`Raw` method in group-address client implementation.

Define `Client` interface in new `pkg/core/client` package. Completely
inherit API `Client` interface. Add `RawForAddress` method to build raw
client for the single node address. Adopt the application code that used Raw
method to work with new `Client`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Pavel Karpy 48827f42d3 [#643] pkg: Sync method names and commentaries to them
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-24 16:10:44 +03:00
Leonard Lyubich adbbad0beb [#607] network: Do not work with Address pointers
`network.Address` structure in most cases created once and used read-only.

Replace `AddressFromString` function with `Address.FromString` method with
the same purpose and implementation. Make all libraries to work with value.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich e5504c7130 [#607] network: Do not use Address.String for address comparison
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich 47fe8911a3 [#607] network: Rename Address.HostAddrString method to HostAddr
Return tyype is clear from the method's signature and docs, there is no
point in reflecting it in the name.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich 35f81729e4 [#607] network: Do not return error from `Address.HostAddrString` method
Panic if internal `manet.DialArgs` call returns error since this is
unexpected according to `AddressFromString` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich 95ccbbc2f9 [#607] network: Accept value instead of pointer in IsLocalAddress
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Pavel Karpy 6b176e8769 [#613] pkg/reputation: Move manager building to `pkg`
Move `managers` package to `pkg` since
it can be reused in other packages.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-17 18:30:06 +03:00
Leonard Lyubich 455fd952dd [#414] ir: Serve ControlService
Serve `ControlService` instance on configured endpoint (do not serve if not
specified). Read allowed keys from config.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-15 15:21:52 +03:00
Leonard Lyubich dcfe9a6504 [#414] ir/control: Implement service server
Implement `ControlServiceServer` on `Server` type. The `Server` requires all
requests to be signed with keys from the so-called whitelist. To obtain
health status, it uses the abstraction in the form of `HealthChecker`
interface.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-15 15:21:52 +03:00
Leonard Lyubich 93803b1a90 [#414] ir: Define Control service
Define `ControlService` for IR similar to the one from storage node. Add
`HealthStatus` RPC which returns health status of the IR application.
Implement getters, setters and methods to sign/verify the messages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-15 15:21:52 +03:00
Evgenii Stratonikov 9142c778c7 [#562] pkg: remove `nspcc-dev/neofs-crypto` dependency
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-15 14:49:59 +03:00
Leonard Lyubich 096acb2a44 [#580] v2/object/put: Check payload size of the relayed objects
NeoFS network dictates the limitation of the maximum size of the "finished"
objects. The payload size of an object is recorded in its header. When
writing finished objects, the nodes must check:

  * satisfying the constraint;
  * matching the value in the header to the number of payload bytes.

Provide value returned by `MaxSizeSource` component to `validatingTarget`.
Check max size value during the stream of the "finished" objects. Check
header value during the streaming and on-close. Check payload size in v2
relay scenario.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-09 12:08:37 +03:00
Evgenii Stratonikov b8a7c11e57 [#501] object/put: reduce TTL of the relayed request
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-02 10:31:14 +03:00
Evgenii Stratonikov 3468491224 [#501] object/put: relay requests for signed objects
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-02 10:31:14 +03:00
Evgenii Stratonikov a422f42ca9 [#501] object/put: refactor distributed target
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-02 10:31:14 +03:00
Leonard Lyubich 3e1463cc76 [#570] *: Use generator of test container IDs from API Go lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 20:45:15 +03:00
Leonard Lyubich 3dd10b6795 [#570] *: Remove usage of deprecated elements from API Go library
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 20:45:15 +03:00
Leonard Lyubich 70a7354e9d [#570] *: Use new Equal method of owner.ID type
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 20:45:15 +03:00
Pavel Karpy 33bef46f31 [#549] network/cache: Change `Get` signature
Make network cache's `Get` method accept
`network.Address` argument instead of
string.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-27 17:07:04 +03:00
Leonard Lyubich a1bfcdc74a [#525] v2/container: Write session token from header to removal witness
If container is removed via session, then session token should be included
in removal witness.

Write session token from request meta header to `container.RemovalWitness`
structure which is passed to `wrapper.Delete` function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich 9a0964efa4 [#525] morph/container: Accept RemovalWitness in Delete function
Make `wrapper.Delete` function to accept `container.RemovalWitness` struct
instead of its separated elements. `Signature` type is replaced by binary
signature since public key is unused.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich b09f212990 [#525] v2/container: Write session token from header to eACL table
If eACL table is set via session, then session token should be written to
it.

Write session token from request meta header to `eacl.Table` structure which
is passed to `wrapper.PutEACL` function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich 5c2b8de87d [#525] v2/container: Make context with token reusable
Rename `PutContext` to `ContextWithToken` and implement its constructor as a
separate function in order to reuse it in other RPCs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich 98cc685a9b [#525] v2/container: Write session token from header to container
If container is created via session, then session token should be written to
it.

Write session token from request meta header to `Container` structure which
is passed to `wrapper.Put` function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich 1deb3f3d01 [#525] morph/container: Do not accept signature in PutEACL function
In previous implementation wrapper over the Container contract's client
accepted the signature of the eACL table in addition to itself. After recent
changes in API Go lib table carries its signature. Thus, it is redundant
to pass the eACL table signature separately.

Make `wrapper.PutEACL` method to accept `eacl.Table` only.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich 32828d2b90 [#525] morph/container: Do not accept signature in Put function
In previous implementation wrapper over the Container contract's client
accepted the signature of the container in addition to itself. After recent
changes in API Go lib container carries its signature. Thus, it is redundant
to pass the container signature separately.

Make `wrapper.Put` method to accept `Container` only.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich df197dc38b [#525] morph/container: Do not return signature from GetEACL method
In previous implementation wrapper over the Container contract's client
returned the signature of the eACL table in addition to itself. After recent
changes in API Go lib table carries its signature. Thus, it is redundant to
return the table signature separately.

Make `Wrapper.GetEACL` method to return only `eacl.Table` with error.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich 615813d6de [#525] v2/container: Return session token in GetExtendedACL
In recent API changes `GetExtendedACLResponseBody` carries session token.
In recent API Go lib changes `eacl.Table` structure carries related session
token.

Write session token of eACL table from sidechain to GetExtendedACL response
body in node's `ContainerService` server.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich a7f71bf167 [#525] v2/container: Return session token and signature in Get
In recent API changes `GetResponseBody` carries session token and signature.
In recent API Go lib changes `Container` structure carries its session
token and signature.

Write session token and signature of container from sidechain to Get
response body in node's `ContainerService` server.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Alex Vanin 975ab0dff7 [#561] acl: Add tests for request meta header traverser
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-27 11:21:14 +03:00
Alex Vanin a025e6250e [#561] acl: Fetch session token from original request meta header
As it explained in previous commit, session token also should
be presented in original meta header but can be omitted in higher
layers.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-27 11:21:14 +03:00
Alex Vanin d368afffe5 [#561] acl: Fetch bearer token from original request meta header
Request meta headers are organized in a layers, where
upper layers re-sign down layers. Bearer token should be
a part of original meta header and it can be omitted in
upper layers. Therefore we need to traverse over linked list
of meta header to the original meta header to get bearer token.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-27 11:21:14 +03:00
Leonard Lyubich 83c27f6e8a [#505] morph/container: Change get container API
Make `Get` method of the wrapper over Container contract's client to
accept binary container ID. Create `Get` function similar to the previous
`Get` variation. Use this function in Container service server in the place
where `Get` method was used.

Additionally implement `AsContainerSource` function which allows
to simply compose container Source interface from the wrapper.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-25 16:35:52 +03:00
Leonard Lyubich e3b4c9eda0 [#505] morph/container: Change delete container API
Make `Delete` method of the wrapper over Container contract's client to accept
two binary parameters: container ID and signature. Create `Delete` function
similar to the previous `Delete` variation, but accepting `Signature`
structure instead of binary signature. Use this function in Container
service server in the place where `Delete` method was used.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-25 16:35:52 +03:00
Leonard Lyubich 565ad51b42 [#505] morph/container: Change put container API
Make `Put` method of the wrapper over Container contract's client to accept
three binary parameters: container, key and signature. Create `Put` function
similar to the previous `Put` variation, but accepting `Signature`
structure instead of binary key and signature. Use this function in
Container service server in the place where `Put` method was used.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-25 16:35:52 +03:00
Leonard Lyubich 395fd187ac [#505] morph/container: Add key argument to client wrapper's SetEACL
In recent changes argument list of set eACL call of Container contract
client was extended with binary public key. In the future there will be a
need to pass the table in binary format.

Replace `PutEACL` method with `PutEACLBinary` one which accepts three binary
parameters: eACL table, key and signature. Create `PutEACL` function similar
to the removed method, but accepting `Signature` structure instead of just
a signature. Use this function in Container service server in the place
where `PutEACL` was used.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-25 16:35:52 +03:00
Evgenii Stratonikov 2b2b2c2c45 [#496] Use single contract wrapper constructor
There is no need in a separate `New()` or `WrapClient()`

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-05-25 13:41:39 +03:00
Leonard Lyubich b587b23e79 [#543] object/search: Fix potential NPE on request forwarder
Request forwarding callback should be called only if set since it is an
optional parameter.

Call `forwarder` function only if it is non-nil.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-20 11:33:24 +03:00
Leonard Lyubich b2002ffc6d [#543] object/get: Fix potential NPE on request forwarder
Request forwarding callback should be called only if set since it is an
optional parameter. In GetRangeHash forwarder is never set.

Call `forwarder` function only if it is non-nil. Remove no longer needed
`hashOnly` option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-20 11:33:24 +03:00
Alex Vanin 89b147ebe7 [#521] Recompile protobuf files after transition from `pkg/errors`
Autogenerated files were accidentally modified. This commit returns
them back to the original.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-19 19:09:35 +03:00
Alex Vanin b5256ccf4c [#521] Fix issues with transition from `pkg/errors` pkg
Wrap functions at `pkg/errors` return nil if error argument
was nil. fmt.Errorf always returns error so we need to add
missing error checks to the code.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-19 19:09:35 +03:00
Evgenii Stratonikov 71b87155ef [#521] *: use stdlib `errors` package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-05-19 16:42:54 +03:00
Leonard Lyubich 72e2bc8fb6 [#532] object/hash: Fix NPE during request forwarding
In current implementation `Object.GetRangeHash` RPC handler forwards range
requests for payload data. Missing request forwarder of the original request
caused NPE during execution.

Do not call request forwarder if payload range hash is requested.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-18 18:05:18 +03:00
Alex Vanin 16f13bc0a5 [#522] Use `HostAddrString` as RPC endpoint instead of `IPAddrString`
To enable TLS support we can't operate with IP addresses directly.
Certificates are issued with host names so it is required to
pass them into RPC client. DNS resolving should be done by transport
layer and not be a part of node. Therefore `IPAddrString` usage is
removed from code.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-18 12:07:00 +03:00
Pavel Karpy 9fbc6f5efe [#527] reputation: Fix EigenTrust algorithm
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-18 10:29:25 +03:00
Leonard Lyubich 80ef4492c1 [#493] node: Get rid of outdated object GC worker
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Alex Vanin d49bd4b94a Update neofs-api-go to v1.26.0
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-10 09:45:11 +03:00
Pavel Karpy 20b7295087 [#488] reputation/router: Add commentaries and TODO
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy d1db54acf8 [#488] reputation: Change `Writer` interface
Includes:
- Delete first `ctx` argument in `Write` method.
- Move intermediate Initial trust struct and method
to `calculator` file.
- Change Alpha to 0.1.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy eb74a9cafc [#488] reputation: Add commentaries
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy 2c8c9f69c8 [#488] storage/calculator: Make `alpha` dynamic
Delete reading `alpha` from env var. Cover
retrieving `alpha` behind interface in
intermediate calculator. Add TODO to decide
if it is necessary to receive that param from
global config or not.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy ea781664cf [#488] reputation/eigentrust/calculator: Implement calc wrapper
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy d3c1fc7dda [#488] reputation/local/storage: Add trusting peer
Fix iteration logic after adding trusting peer
in `Trust` struct.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy e69917b27a [#488] reputation/common: Move `ServerInfo` to `common` pkg
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy 49d477f466 [#488] reputation/managers: Implement route builders
Add implementation of Builder interface for
intermediate trusts. Move all code associated
with managers to `common` directory in `cmd`
and `pkg/services/reputation`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy e2a1b0e0ee [#488] reputation/eigentrust/storages: Fix args
Change anonymous func arg for `Iterate`
methods of Storages to `PeerTrustsHandler`
type for implementing corresponding
interface.
Implement missing `Iterate` method for
daughter Storage.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy f6783f4f81 [#488] cmd/reputation: Add `DaughterStorage`
Add `DaughterStorage` init in main pkg
and start write all received daughters'
trusts to it.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Leonard Lyubich 969af596b4 [#431] object/search: Remove no longer used unexported methods
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-04 13:56:55 +03:00
Leonard Lyubich 7366d62e41 [#431] object/range: Fix payload slice allocation in re-sign callback
Allocate capacity instead of length of the slice to write the object payload
range since each chunk is written through `append`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-04 13:56:55 +03:00
Leonard Lyubich 36088949fc [#431] services/object: Re-sign original read requests during forwarding
In previous implementation node's Object Get/Head/GetRange V2 service
handlers created a new request for each RPC. Now original requests are
re-signed according to API specification. Logical handler abstracts from
this version-dependent logic through `RequestForwarder` callback.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-04 13:56:55 +03:00
Leonard Lyubich e6b30aed36 [#431] object/search: Re-sign original requests during forwarding
In previous implementation node's Object Search V2 service handler created a
new request for each RPC. Now original requests are re-signed according to
API specification. Logical handler abstracts from this version-dependent
logic through `RequestForwarder` callback.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-04 13:56:55 +03:00
ZhangTao1596 2877a343c3 [#498] audit: skip unnecessary statement
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2021-05-04 12:55:01 +03:00
Pavel Karpy 477682adb7 [#482] reputation/router: Move router to `./common`
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-19 11:47:04 +03:00
Pavel Karpy ac8441b718 [#482] reputation/router: Make route pkg independent
Make route package independent from
controller package. Add common interfaces
to `./common` directory.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-19 11:47:04 +03:00
Pavel Karpy 0a16aaacb1 [#482] reputation/router: Use trusting peer
In route building use `trusting` peer
field of `reputation.Trust` struct
instead of `reputation.Peer`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-19 11:47:04 +03:00
Pavel Karpy 1c92dc2414 [#482] reputation: Fix typos in comments
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-19 11:47:04 +03:00
Leonard Lyubich a97e08cfd7 [#473] Implement EigenTrust calculations
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-04-12 13:25:28 +03:00
Pavel Karpy 60cc3b3e16 [#460] reputation: Add `Router` to reputation server
Add `Router` to the reputation server. `Router` is
called on every incoming request and inits `Writer`
that sends `Trust`s to the next route point or
handle(logs in that implementation) them if current
node is the end point of the route.

Rename `onlyKeyRemoteServerInfo` struct for container
to separate it from the same implementation of the
same `ServerInfo` interface for reputation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-08 17:29:08 +03:00
Pavel Karpy 5970a5e6ae [#460] reputation/local: Declare Managers
Declare interfaces for building list of
managers of certain peer in certain epoch.
Implement `Builder` struct that aggregates
fields that are necessary for building
list of managers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-08 17:29:08 +03:00
Pavel Karpy 91825a0162 [#460] reputation/local: Implement local trust router
Implement reputation `Router` and its constructor,
designed to define where to send local trusts.
Router is based on dependencies that are hidden
behind interfaces, that are declared in the router's
package.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-08 17:29:08 +03:00
Pavel Karpy e6c9fb283c [#460] Fix typos in commentaries
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-08 17:29:08 +03:00
Alex Vanin 7cf48d4d91 [#452] Update neo-go to latest master
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-06 10:35:20 +03:00
Leonard Lyubich 3f9c8f2f59 [#444] reputation: Implement sign and response servers
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-04-01 14:54:22 +03:00
Leonard Lyubich df97e35f30 [#444] reputation: Define RPC Server interface
Define `Server` interface of NeoFS API V2 Reputation service server.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-04-01 14:54:22 +03:00