Commit graph

1753 commits

Author SHA1 Message Date
Pavel Karpy
2aec5736e0 [#1318] node/gc: Fix tombstones handling
Handle only expired tombstones.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
24ca8ca5c7 [#1318] object: Change default ttl
This is need to reuse object service without passing any API v2 structs.
Also, fix comment.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
7799f8e4cf [#1318] engine: Change tombstone clear process
- Delete objects physically on tombstone's arrival;
- Store information about tombstones in the Graveyard;
- Clear Graveyard every epoch based on the information about TS in the
network.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
e4cfeec449 [#1318] service: Add tombstone checker service
The service fetches tombstones from the network via object service, every
request is handled in the following order:
1. checks local LRU cache;
2. checks local storage engine;
3. tries to find object in the placement nodes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
2583f608e8 [#1318] metabase: Add DropGraves method
Method clean up Graveyard and should be called by the routine that handles
tombstone expiration.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
daab30c391 [#1318] metabase: Add iteration offset
Add offset element to the iterations over deleted objects (both the
Graveyard and the Garbage buckets).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
fe8076e60a [#1318] metabase: Add immediate object deletion
Mark objects with GC immediately after a Tombstone is received.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
8107c8d1a9 [#1318] metabase: Separate buckets with TS and GC marks
It allows storing information about object in both ways at the same time:
1. Metabase should know if an object is covered by a tombstone (that is
not expired yet);
2. It should be possible to physically delete objects covered by a
tombstone immediately (mark with GC) but keep tombstone knowledge.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Evgenii Stratonikov
5736b834c3 [#1286] services/object: Validate object when created with sesssion token
All fields set by a user should be verified.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:53:22 +03:00
Evgenii Stratonikov
c6a9c5cd8c [#1283] services/object: Disallow creating objects without a session token
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:53:22 +03:00
Pavel Karpy
bd27837364 [#1321] node: Register GC event channel before shard init
Morph "NewEpoch" event handling was registered in a closure over
`addNewEpochNotificationHandler` func. That may lead to the data race:
if a shard was initialized before the event registration, everything works
as planned, but if registration was made earlier, it was not able to
include GC handlers since a shard has not called `eventChanInit` yet and,
therefore, it has not registered handler yet.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-25 10:39:37 +03:00
Evgenii Stratonikov
6c08ec9ca2 [#1312] morph/netmap: Fix panic during parameter parsing
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-12 17:37:12 +03:00
Pavel Karpy
b6e5aac604 [#1170] pkg/morph: Delete unused function
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
01e69f2f7a [#1170] pkg/morph: Add worker pool
Add worker pool to the listener to prevent blocking. It is used only for
notary notifications and new block events handling since it uses RPC
calls. That may lead to the deadlock state: neo-go cannot send RPC until
notification channel is read but notification channel cannot be read since
neo-go client cannot send RPC.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
4ed5b1ceef [#1170] pkg/morph: Invalidate cache on RPC switch
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
6508136204 [#1170] ir, node: Stop the app on WS connection lost
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
2427593da6 [#1170] pkg/morph: Add inactive mode callback
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
424cc6d495 [#1170] go.mod: Update neo-go
github.com/nspcc-dev/neo-go v0.98.0 => v0.98.2

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
fdf1338d65 [#1170] config: Delete notification endpoints
Also, change `http` to `ws` in examples.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
3e45b4a085 [#1170] pkg/morph/subscriber: Adopt new WS client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
402f488bec [#1170] pkg/morph: Change HTTP for WS client
Updated client now supports subscription to chain notifications and RPC
switch between provided RPC endpoints.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Evgenii Stratonikov
622ea4818f [#1250] *: Remove io/ioutil imports
It is deprecated starting from go1.16.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 17:23:28 +03:00
Evgenii Stratonikov
e8bf18c0b4 [#1208] innerring: Fix race condition between handling new epoch and block
Before resetting the timer, ensure the block tick is processed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 11:52:24 +03:00
Evgenii Stratonikov
4770cb8bf6 [#1208] innerring: Disallow to tick timer twice on the same height
Provide current heights as an argument to ticker.
Zero height disables any checks, thus corresponding to the old
behaviour. If non-zero height is used, ignore the tick if the height
is less than the timer tick state.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 11:52:24 +03:00
Evgenii Stratonikov
c0e65dadaf [#1292] morph/event: Make listener methods use pointer receiver
After `Listen` has raw receiver and calls `listen` which has pointer
receiver, leading to ineffectual assign to `started` field.
Always use `listener` by pointer, to avoid similar bugs.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 11:48:57 +03:00
Leonard Lyubich
629a4f79cd [#1293] metabase: Fix out-of-range panic in freePotentialLocks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-05 15:38:51 +03:00
Alex Vanin
e62e02815b [#1291] listener: Stop only once
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-05 14:46:40 +03:00
Alex Vanin
77f4a5844b [#1291] innerring: Don't stop the same listener twice
When IR node configured without main chain, both
`morphListener` and `mainnetListener` are pointing
into single listener component. We should not call
`Stop()` twice, because it may trigger channel
closing in neo-go or other components and it
can throw panic.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-05 14:46:40 +03:00
Evgenii Stratonikov
923db59149 [#1213] morph/client: Remember last succesfully used client
Prevent attemts to connect to the failing endpoint multiple times.
This speeds up all invocations a bit and removes excessive error logs.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov
9bbb136e4a [#1213] morph/client: Protect cached values with a mutex
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov
a8d2001b35 [#1213] morph/client: Use a separate cache for every client
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov
61f60b8461 [#1213] morph/client: Unlock mutex to perform network requests
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov
48eb87d32c [#1149] writecache: Prevent corruption in Head
Also, remove optimization comments:
1. Having to maintain an execute the same logic for headers as for
   objects is quite inefficient, as it increases memory footprint.
2. Unmarshaling object is a cheap operation if data slice is in memory.
3. For unmarshaling header-only, I think we need SDK support.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:40:25 +03:00
Evgenii Stratonikov
ff1912aa2a services/acl: check session token expiration epoch
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:37:29 +03:00
Evgenii Stratonikov
68903c9fd9 [#1143] shard: Support degraded mode in Get and GetRange
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
f058cead8f [#1143] shard: Handle some errors in degraded mode
If metabase is corrupted for some reason, failback to
checking blobstor directly.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
08e7914729 [#1143] blobstor: Implement existsSmall check
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
aa0cc1f824 [#1143] blobovnicza: Copy object data in Get
Data returned from `*bbolt.Bucket.Get()` is only valid for the lifetime
of the transaction.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
6472a170eb [#1143] shard: Introduce explicit Degraded mode
`Degraded` mode is set automatically after error counter is over the
threshold. `ReadOnly` mode can still be set by an administrator.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Alex Vanin
7ed84d1755 [#1278] acl: Return netmap.Source interface
Application can provide cached netmap source in this case.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-30 14:22:12 +03:00
Evgenii Stratonikov
cf119e4ca9 [#1163] services/audit: Randomize the order of PDP checks
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-30 10:56:33 +03:00
Evgenii Stratonikov
882236a03b [#1239] morph/client: Remove intermediate conversion in morph client
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-29 16:02:05 +03:00
Evgenii Stratonikov
6936195afa [#1239] util/attributes: Remove excessive slice copy during parsing
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-29 16:02:05 +03:00
Evgenii Stratonikov
2ad8016d75 [#1239] innerring: Use pointer-less slices for object IDs
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-29 16:02:05 +03:00
Pavel Karpy
6ec104d686 [#1255] node/session: Rename constant
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
90a8c52bdb [#1255] object: Add persistent storage usage
Use persistent storage usage in the node if it was configured so.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
016eaa25f3 [#1255] node/session: Add encryption tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
01ed366e99 [#1255] node/session: Add encryption
Add `WithEncryption` option that passes ECDSA key to the persistent session
storage. It uses 32 bytes from marshalled ECDSA key in ASN.1 DER from in
AES-256 algorithm encryption in Galois/Counter Mode.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
a884ad56d9 [#1255] node/session: Add persistent tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
455b9fb325 [#1255] node/session: Add persistent session storage
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
929c9851a6 [#1255] node/session: Create separate dir for in-memory storage
Move in-memory session storage to the separate directory of `storage`. It is
done for future support of different kind of session storages.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Evgenii Stratonikov
2a69aaf976 [#1157] network/cache: Optimize client fetch from multiClient
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 18:11:20 +03:00
Evgenii Stratonikov
a4261243fc [#1157] network/cache: Cache multiclients based on public key only
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 18:11:20 +03:00
Evgenii Stratonikov
de5a2f6574 [#1262] metabase: Remove list index in place Delete
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
43867a3093 [#1262] metabase: Do not allocate intermediate slices for indices
```
name              old alloc/op   new alloc/op   delta
Put/parallel-8       123kB ± 4%     119kB ± 3%  -2.72%  (p=0.006 n=10+9)
Put/sequential-8     170kB ± 1%     168kB ± 1%  -1.42%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op  delta
Put/parallel-8         473 ± 1%       469 ± 0%  -0.87%  (p=0.000 n=10+10)
Put/sequential-8       792 ± 0%       787 ± 0%  -0.58%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
0e9b6be3fd [#1262] metabase: Remove intermediate allocations in decodeList
```
name              old time/op    new time/op    delta
Put/parallel-8      1.57ms ±11%    1.51ms ± 3%   -4.06%  (p=0.043 n=9+10)
Put/sequential-8    5.16ms ± 2%    5.16ms ± 3%     ~     (p=1.000 n=9+10)

name              old alloc/op   new alloc/op   delta
Put/parallel-8       126kB ± 4%     123kB ± 4%   -2.54%  (p=0.016 n=8+10)
Put/sequential-8     171kB ± 1%     170kB ± 1%     ~     (p=0.182 n=9+10)

name              old allocs/op  new allocs/op  delta
Put/parallel-8         565 ± 2%       473 ± 1%  -16.18%  (p=0.000 n=9+10)
Put/sequential-8       819 ± 1%       792 ± 0%   -3.34%  (p=0.000 n=9+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
d45df614fb [#1262] metabase: Optimize decodeList
Prevent additional allocation during `append` in `Put`.

```
name              old alloc/op   new alloc/op   delta
Put/parallel-8       131kB ± 1%     126kB ± 4%  -3.87%  (p=0.005 n=8+8)
Put/sequential-8     172kB ± 1%     171kB ± 1%  -0.73%  (p=0.028 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
456e1584d6 [#1262] metabase: Add benchmarks for Put
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
f1223b46df [#1262] blobovnicza: Make helper in Put function idempotent
`Batch` can execute the function multiple times leading to multiple
increases of a size approximation.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
ad92493b86 [#1268] blobstor: Cleanup zstd encoders/decoders
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-25 14:05:06 +03:00
Alex Vanin
7e06d0aa69 [#1253] ir: Call UpdateStateIR method to remove dead storage node
Alphabet nodes in notary enabled environment cannot call `UpdateState`
method to remove unwanted storage nodes from the network map,
because this method checks witness of the storage node.

To force storage node state update, alphabet nodes should invoke
new method `UpdateStateIR` which is similar to `AddPeerIR`.

State update initiated by the storage node itself is processed
the same way as before -- alphabet nods resign such transaction.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-24 11:52:19 +03:00
Alex Vanin
e81081e0e0 [#1253] ir: Update AddPeer method name for notary enabled env
`Register` was renamed to `AddPeerIR` for consistency with
`UpdateState` changes in
https://github.com/nspcc-dev/neofs-contract/pull/227

This is protocol breaking change for notary enabled environment.
Luckily, there is no notary enabled environments anywhere except
of neofs-dev-env, so we can do such thing. We should avoid such
changes in the future, though.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-24 11:52:19 +03:00
Evgenii Stratonikov
414ba6e0a2 [#1244] nats: Split client creation into 2 stages
Create and connect to an endpoint using separate functions.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-24 11:51:49 +03:00
Evgenii Stratonikov
e2062013cf network: remove unused constants
Fix linter complaints. These constants are unused after
nspcc-dev/neofs-node#1232.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-22 11:55:19 +03:00
Evgenii Stratonikov
2848001dfb [#1246] object/acl: Return more concise description for eACL errors
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-21 19:20:01 +03:00
Leonard Lyubich
318639e5bf [#1247] *: Clarify docs about returned errors from apistatus package
`apistatus` package provides types which implement build-in `error`
interface. Add `error of type` pattern when documenting these errors in
order to clarify how these errors should be handled (e.g. `errors.Is` is
not good).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
459bdcf04b [#1247] object/acl: Return ObjectAccessDenied status error
Return `apistatus.ObjectAccessDenied` error on access violation from ACL
service. Write reason in format of the errors from the previous
implementation. These errors are returned by storage node's server as
NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
967650f2ed [#1247] container: Return ContainerNotFound status error
Replace `core/container.ErrNotFound` error returned by `Source.Get`
interface method with `apistatus.ContainerNotFound` status error. This
error is returned by storage node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
70ffdf3478 [#1247] object: Return NOT_FOUND and ALREADY_REMOVED statuses
Replace `ErrNotFound`/`ErrAlreadyRemoved` error from
`pkg/core/object` package with `ObjectNotFound`/`ObjectAlreadyRemoved`
one from `apistatus` package. These errors are returned by storage
node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
f32c9670ad [#1247] object: Return status errors from util.KeyStorage
Return `SessionTokenExpired`/`SessionTokenNotFound` error from
`apistatus` package if private session token is expired/missing. These
errors are returned by storage node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
a4ee59977e [#1247] session: Remove unused storage.ErrNotFound error
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Evgenii Stratonikov
737d40f2d9 [#1152] network: Reuse URI parsing function from api-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-17 16:33:08 +03:00
Pavel Karpy
16543a1904 [#1243] node/object: Hide engine behind interface
Do not use engine as explicit dependency in the put service. This allows
wrapping storage engine.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 14:27:52 +03:00
Pavel Karpy
d02720c910 [#1243] nats: Fix comments
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 14:27:52 +03:00
Pavel Karpy
c3db12d71b [#1210] reputation: Resolve race condition
Make all epoch independent in reputation process. Do not reset any timers
related to reputation. Make it possible to finish iteration after the
unexpected `NewEpoch` event.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-16 17:18:30 +03:00
Pavel Karpy
13af4e6046 [#1210] reputation: Improve debug logs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-16 17:18:30 +03:00
Alex Vanin
0adb29c035 [#1214] Fix metabase put logging
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 17:17:18 +03:00
Alex Vanin
362cda53d2 [#1238] Call String() explicitly instead of using stringer pointer
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Alex Vanin
8f476f3c4d [#1238] Adopt neofs-node for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Alex Vanin
9fad29dfe0 [#1238] Adopt neofs-ir for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Alex Vanin
7f3195b197 [#1238] Adopt neofs-cli for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Leonard Lyubich
98c84670e3 [#1175] core/object: Check emptiness of LOCK members
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
9feb5f9405 [#1175] engine: Fix AddShard implementation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
9dff07200c [#1181] local storage: Process expired locks similar to tombstones
There is a need to process expired `LOCK` objects similar to `TOMBSTONE`
ones: we collect them on `Shard`, notify all other shards about
expiration so they could unlock the objects, and only after that mark
lockers as garbage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
ebd84f6dd4 [#1175] object/put: Perform additional container broadcast of LOCKs
`LOCK` objects should be broadcast across container like `TOMBSTONE`
ones.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
323dea95c6 [#1181] Do not use deprecated elements
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
e3f4e48d87 [#1175] Fix typo in error return docs
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
8a2a096680 [#1175] services/sign: Convert error to status return
In previous implementation `SignService` converted all `error` values to
`INTERNAL` server failure status. That was done for simplification only.
There is a need to transmit status errors as corresponding status
messages.

Make `SignService` to unwrap errors and convert them to status message
during writing to the response. Non-status errors are converted to
`INTERNAL` server failures. Status errors can also be wrapped in the
depths of the executable code, so `SignService` tries to unwrap them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
ced854bc2e [#1175] object/fmt: Handle LOCK objects
Make `FormatValidator.ValidateContent` to verify payload of `LOCK`
objects. Pass locked objects to `Locker` interface. Require from
`Locker.Lock` to return `apistatus.IrregularObjectLock` error on a
corresponding condition.

Also add error return to `DeleteHandler.DeleteObjects` method. Require
from method to return `apistatus.ObjectLocked` error on a corresponding
condition. Adopt implementations.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
d163008b63 [#1175] storage engine: Fix Lock implementation
`Lock` must return `apistatus.IrregularObjectLock` error instead of
`apistatus.ObjectLocked` one.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
07107c8bfd [#1175] metabase: Update locker list during Inhume
`Inhume` operation can potentially mark lockers as garbage. There is a
need to update locker list in locked bucket.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
26b8e33ed7 [#1175] metabase: Fix encoding of updated locker list
Even single locker must be encoded using `encodeList`, or subsequent
`decodeList` call will fail.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
88cefa7c5f [#1175] storage engine: Return status error on Delete of locked object
Make `StorageEngine.Delete` to forward first encountered
`apistatus.ObjectLocked` error during shard processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
97b6ec8150 [#1175] storage engine: Return status error on Inhume of locked object
Make `StorageEngine.Inhume` to forward first encountered
`apistatus.ObjectLocked` error during shard processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
545861d1bc [#1175] storage engine: Implement Lock operation
Implement `StorageEngine.Lock` method which works similar to `Inhume`
but calls `Lock` on the processing shards.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
6ed85ff1e1 [#1175] shard: Implement Lock operation
Implement `Shard.Lock` method which required rw mode and calls `Lock` on
underlying metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
e21eedee06 [#1175] shard: Return status error on Inhume of locked object
Make `Shard.Inhume` to forward `apistatus.ObjectLocked` returned by
underlying metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
b585791d6e [#1175] metabase: Return status error on Lock of irregular object
Make `DB.Lock` to return `apistatus.IrregularObjectLock` if at least one
of the locked objects is irregular (not of type REGULAR).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
23fcacd3f2 [#1175] metabase: Return status error on Inhume of locked object
Make `DB.Inhume` to return `apistatus.ObjectLocked` if at least one of
the inhumed objects is locked.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
303eb2e078 [#1176] metabase: Ignore locked objs in IterateCoveredByTombstones
Make `DB.IterateCoveredByTombstones` to not pass locked objects to the
handler. The method is used by GC, therefore it will not consider locked
objects as candidates for deletion even if their tombstone is expired.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
dba3e58dc5 [#1176] metabase: Ignore locked objects in IterateExpired
Make `DB.IterateExpired` to not pass locked objects to the handler. The
method is used by GC, therefore it will not consider them as candidates
for deletion.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
9508633a7e [#1175] metabase: Work with LOCK objects
After introduction of LOCK objects (of type `TypeLock`) complicated
extended its behavior:
  * create `lockers` container bucket (LCB) during PUT;
  * remove object from LCB during DELETE;
  * look up object in LCB during EXISTS;
  * get object from LCB during GET;
  * list objects from LCB during LIST with cursor;
  * select objects from LCB during SELECT with '*'.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
9f13674a10 [#1175] metabase: Implement LOCK operation
Implement `DB.Lock` method which marks list of the objects as locked by
another object. Only regular objects can be locked.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
14d27455f3 [#1175] metabase: Add container bucket for LOCKs
Create class of container buckets with `LOCKED` suffix. Put identifiers
of the objects of type `LOCK` to these buckets during `DB.Put`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
e90ac9c6f9 [#1181] metabase: Remove unused CleanUp method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Alex Vanin
b6720d5f97 [#1231] Update new SDK Client interface
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-14 13:18:48 +03:00
Alex Vanin
e38d7dda6b [#1230] replicator: Improve error log
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-10 19:19:18 +03:00
Leonard Lyubich
14ea7b2c47 [#1219] object/delete: Change approach of the tombstone broadcast
After `putsvc.Service` started to support additional container broadcast
of the saved objects there is no more need to perform broadcast of
tombstone object in `deletesvc.Service`.

Make `putsvc.Service` to perform additional broadcast of `TOMBSTONE`
objects. Remove `broadcastTombstone` stage from `deletesvc.execCtx`,
from now it is encapsulated in `saveTombstone` stage. Remove no longer
needed `putsvc.PutInitPrm.WithTraverseOption` method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-05 16:23:21 +03:00
Leonard Lyubich
a55af18ad1 [#1219] object/put: Support additional container broadcast
There are several cases when we need to spread the object around the
container after its primary placement (e.g. objects of type TOMBSTONE).
It'd be convenient to support this feature in `putsvc.Service`.

Add additional stage of container broadcast after the object is stored.
This stage is carried out no more than once and does not affect the
outcome of the main PUT operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-05 16:23:21 +03:00
Evgenii Stratonikov
0bf59522f7 [#1191] object/acl: check session token verb
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-05 16:03:16 +03:00
Pavel Karpy
79c6f52a27 [#1183] node: Add NATS client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 15:57:24 +03:00
Pavel Karpy
03b601b594 [#1183] node: Init notificator on startup
It uses logger as a notification writer. NATS messaging will be implemented
later.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 11:41:18 +03:00
Pavel Karpy
90ff066940 [#1183] node/notificator: Implement notificator
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 11:41:18 +03:00
Pavel Karpy
cf17686321 [#1111] control: Rename aliaces
`objectSDKAddress` -> `addressSDK`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 10:06:14 +03:00
Pavel Karpy
99b31e3235 [#1111] object/acl: Refactor service
Make all operations that related to `neofs-api-go` library be placed in `v2`
packages. They parse all v2-versioned structs info `neofs-sdk-go`
abstractions and pass them to the corresponding `acl`/`eacl` packages. `v2`
packages are the only packages that do import `neofs-api-go` library. `eacl`
and `acl` provide public functions that only accepts `sdk` structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 10:06:14 +03:00
Leonard Lyubich
7ccd1625af [#1214] *: Use single Object type in whole project
Remove `Object` and `RawObject` types from `pkg/core/object` package.
Use `Object` type from NeoFS SDK Go library everywhere. Avoid using the
deprecated elements.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 17:45:00 +03:00
Evgenii Stratonikov
19ad349b27 [#1204] shard: Save ID in the metabase
`AddShard` must return shard id, so we temporarily open metabase
there.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 17:13:13 +03:00
Evgenii Stratonikov
1fe9cd4d36 [#1186] engine: Allow to skip metabase in GetRange
Similarly to `Get`. Also fix a bug where `ErrNotFound` is returned
instead of `ErrRangeOutOfBounds`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
2b5550ccf6 [#1186] blobstor: Fix comment
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
b33fb0f739 [#1186] blobstor: Unify errors for Get and GetRangeBig
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
69e1e6ca20 [#1186] engine: Read object directly from blobstor in case of conflicts
Metabase is expected to contain actual information about objects stored
in shard. If the object is present in metabase but is missing from
blobstor, peform an additional attempt to fetch it directly without
consulting metabase. Such a situation is unexpected, so error counter
is increased for the shard which has the object in the metabase. We
don't increase error counter for the shard which has the object in
blobstor, because some garbage can be expected there. In this
implementation there is no overhead for objects which are really
missing, i.e. are not present in any metabase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
6e6f3648d2 [#1188] metabase: log PUT / DELETE operations
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 15:21:35 +03:00
Evgenii Stratonikov
ce76325840 [#1146] engine: remove TODO and add a comment
Parallel check seems to be slower, so remove TODO.
```
name \ time/op      old          new           pool
Exists/2_shards-40  52.3µs ±23%   91.6µs ± 6%   84.2µs ± 1%
Exists/4_shards-40  72.7µs ±11%  121.8µs ± 8%  116.9µs ± 5%

name \ alloc/op     old          new           pool
Exists/2_shards-40  5.00kB ± 0%   5.03kB ± 0%   5.16kB ± 0%
Exists/4_shards-40  9.89kB ± 0%   9.93kB ± 1%  10.19kB ± 0%

name \ allocs/op    old          new           pool
Exists/2_shards-40     112 ± 1%      115 ± 1%      117 ± 1%
Exists/4_shards-40     207 ± 1%      211 ± 1%      216 ± 1%
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 12:31:41 +03:00
Evgenii Stratonikov
dab6bcabe4 [#1146] engine: Add benchmarks for exists
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 12:31:41 +03:00
Pavel Karpy
0e0a675f35 [#1212] morph: Fix NPE in multi client
Share multi cache between single clients.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-04 12:16:11 +03:00
Alex Vanin
cbc816d57d [#1211] node: Fix object get init error check
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Alex Vanin
bc45a79d1f [#1211] innerring: Fix object get init error check
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Alex Vanin
ca17eb0eb4 [#1211] innerring: Fix objectIDs list reader
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Leonard Lyubich
b5bdcfc076 [#1194] client: Support request X-headers
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-03 15:38:38 +03:00
Evgenii Stratonikov
e10b8f53d6 [#1151] morph/client: Cache notary transaction heights
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-01 10:35:54 +03:00
Leonard Lyubich
e0dce1043a [#1195] Adopt recent changes in NeoFS SDK
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-02-28 10:43:03 +03:00
ZhangTao1596
dd0e10d306 [#1180] ir/internal: Fix audit range hash type
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2022-02-14 12:36:02 +03:00
Evgenii Stratonikov
47cd10a4d7 [#767] services/object: replace duplicate link in TODO
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:59:27 +03:00
Evgenii Stratonikov
e21d054fe2 [#749] morph/client: set group signer scope
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:26:27 +03:00
Evgenii Stratonikov
fa947b85a6 [#749] morph/client: reuse TestInvokePrm in InvokePrm
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:26:27 +03:00
Evgenii Stratonikov
9e2f7ac371 [#749] morph/client: allow to fetch contract group key
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:26:27 +03:00
Evgenii Stratonikov
4838d3bb80 [#749] morph/client: cache NNS contract hash
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:26:27 +03:00
Evgenii Stratonikov
050a4bb2b0 [#1115] *: link TODOs to corresponding issues
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
692790a899 [#1115] services/object: reuse errors from core service
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
4caa330ddc [#1115] morph/client: reuse errors from container contract
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
f9d3111825 [#1115] blobovnicza: remove GetRange method
It is unused and incorrect as described in FIXME.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
c1530dec5e [#1115] shard/gc: provide parsed addresses in HandleExpiredTombstones
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
0695ec4125 [#1133] services/session: remove expired tokens every epoch
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 14:58:32 +03:00
Pavel Karpy
1667ec9e6d [#1131] *: Adopt SDK changes
`object.Address` has been moved to `object/address`
`object.ID` has been moved to `object/id`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-02-08 09:45:38 +03:00
Evgenii Stratonikov
1c821d6c36 [#625] morph/client: remove unused interface
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
c34cfa1f35 [#625] client/container: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
819d80a7a9 [#625] client/audit: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
767ee5c0cd [#625] client/reputation: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
97d18bc515 [#625] client/netmap: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
8c5c3ac9e8 [#625] client/neofsid: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00