Commit graph

140 commits

Author SHA1 Message Date
Evgenii Stratonikov
9426fd5046 WIP: pilorama: add custom batches
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-20 12:13:14 +03:00
Evgenii Stratonikov
609dbe83db [#1559] engine: Do not count logical errors as storage ones
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-13 10:09:30 +03:00
Evgenii Stratonikov
f9eb15254e engine: remove default error threshold
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 13:47:39 +03:00
Evgenii Stratonikov
85aa30e89c local_object_storage: ignore pilorama errors
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 13:47:39 +03:00
Evgenii Stratonikov
7b5b735fb2 [#1550] engine: Split errors on write- and meta- errors
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 13:45:57 +03:00
Evgenii Stratonikov
dafc21b052 [#1550] engine: Set default error threshold
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 13:45:57 +03:00
Leonard Lyubich
596d877a44 [#1549] engine: Disable shard on blobovnicza init failure
There is a need to support working w/o shard if it has problems with
blobovnicza tree.

Make `BlobStor.Init` to return new `ErrInitBlobovniczas` error. Remove
shard from storage engine's shard set if it returned this error from
`Init` call. So if some of the shards (but not all) return this error,
the node will be able to continue working without them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-08 13:45:57 +03:00
Evgenii Stratonikov
e0e4f1f7ee engine: initialize shards in parallel
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 13:44:05 +03:00
Evgenii Stratonikov
1fed255c5b [#1505] pilorama: Allow to customize database parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 13:27:59 +03:00
Evgenii Stratonikov
982cb987a3 [#1333] engine: Increase error counter for pilorama errors
1. Modifying operations are not expected to fail, unless the shard is
   read-only.
2. `Get*` operations should increase error counter too, unless the
   error is `ErrTreeNotFound`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 13:27:38 +03:00
Evgenii Stratonikov
62b2769a66 [#1333] local_object_storage: Support ReadOnly mode in pilorama
The tricky part here is the engine itself: we stop iteration on
`ErrReadOnly` because it is better to synchronize the shard later than
to have partial trees stored in 2 shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 13:26:53 +03:00
Evgenii Stratonikov
06f2681178 [#1442] pilorama: Generate timestamp based on node position in the container
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 13:19:50 +03:00
Evgenii Stratonikov
241d4d6810 [#1431] engine: Add benchmark for Select vs TreeGetByPath
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 13:18:15 +03:00
Evgenii Stratonikov
9cbd4271f1 [#1329] services/tree: Implement GetOpLog RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 13:17:22 +03:00
Evgenii Stratonikov
730f14e4eb [#1406] pilorama: Return parent from TreeGetMeta
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 13:15:52 +03:00
Evgenii Stratonikov
59bd5ac973 [#1344] engine: Log errors in Tree* operations
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 13:13:32 +03:00
Evgenii Stratonikov
dd7c4385c6 [#1326] services/tree: Implement GetSubTree RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 12:50:13 +03:00
Evgenii Stratonikov
4a65eb7e5f [#1324] engine: Implement Forest interface for storage engine
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-08 12:47:40 +03:00
Evgenii Stratonikov
f602d05b0a [#1494] *: Fix linter warnings
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 12:26:10 +03:00
Pavel Karpy
08bf8a68f1 [#1460] engine: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
14366bbd89 [#1418] engine: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
5f57db6bf8 [#1418] shard: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
babd382ba5 [#1418] engine: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
6e752f36dc [#1418] shard: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Leonard Lyubich
1c30414a6c [#1454] Upgrade NeoFS SDK Go module with new IDs
Core changes:
 * avoid package-colliding variable naming
 * avoid using pointers to IDs where unnecessary
 * avoid using `idSDK` import alias pattern
 * use `EncodeToString` for protocol string calculation and `String` for
  printing

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Leonard Lyubich
883f2f1ac6 [#1423] local storage: Activate skipped unit tests after fix
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Leonard Lyubich
86552cf3ae engine: make corruptSubDir to ignore blobovnicza dir
There is a dir for Blobovnicza tree in BlobStor, so we need to ignore
it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:33:14 +03:00
Leonard Lyubich
bb25ecbd15 [#1400] owner: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:33:14 +03:00
Leonard Lyubich
f15e6e888f [#1377] oid, cid: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-16 15:33:22 +03:00
Leonard Lyubich
ae92074272 [#1369] checksum: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Leonard Lyubich
088df0e2a9 [#1369] version: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Elizaveta Chichindaeva
cc7a723d77 [#1320] English Check
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-05-11 10:40:02 +03:00
Evgenii Stratonikov
a4769d8624 *: use require.ErrorIs where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:47:57 +03:00
Pavel Karpy
7799f8e4cf [#1318] engine: Change tombstone clear process
- Delete objects physically on tombstone's arrival;
- Store information about tombstones in the Graveyard;
- Clear Graveyard every epoch based on the information about TS in the
network.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
bd27837364 [#1321] node: Register GC event channel before shard init
Morph "NewEpoch" event handling was registered in a closure over
`addNewEpochNotificationHandler` func. That may lead to the data race:
if a shard was initialized before the event registration, everything works
as planned, but if registration was made earlier, it was not able to
include GC handlers since a shard has not called `eventChanInit` yet and,
therefore, it has not registered handler yet.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-25 10:39:37 +03:00
Evgenii Stratonikov
6472a170eb [#1143] shard: Introduce explicit Degraded mode
`Degraded` mode is set automatically after error counter is over the
threshold. `ReadOnly` mode can still be set by an administrator.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Leonard Lyubich
318639e5bf [#1247] *: Clarify docs about returned errors from apistatus package
`apistatus` package provides types which implement build-in `error`
interface. Add `error of type` pattern when documenting these errors in
order to clarify how these errors should be handled (e.g. `errors.Is` is
not good).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
70ffdf3478 [#1247] object: Return NOT_FOUND and ALREADY_REMOVED statuses
Replace `ErrNotFound`/`ErrAlreadyRemoved` error from
`pkg/core/object` package with `ObjectNotFound`/`ObjectAlreadyRemoved`
one from `apistatus` package. These errors are returned by storage
node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Alex Vanin
8f476f3c4d [#1238] Adopt neofs-node for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Leonard Lyubich
9feb5f9405 [#1175] engine: Fix AddShard implementation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
9dff07200c [#1181] local storage: Process expired locks similar to tombstones
There is a need to process expired `LOCK` objects similar to `TOMBSTONE`
ones: we collect them on `Shard`, notify all other shards about
expiration so they could unlock the objects, and only after that mark
lockers as garbage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
323dea95c6 [#1181] Do not use deprecated elements
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
e3f4e48d87 [#1175] Fix typo in error return docs
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
ced854bc2e [#1175] object/fmt: Handle LOCK objects
Make `FormatValidator.ValidateContent` to verify payload of `LOCK`
objects. Pass locked objects to `Locker` interface. Require from
`Locker.Lock` to return `apistatus.IrregularObjectLock` error on a
corresponding condition.

Also add error return to `DeleteHandler.DeleteObjects` method. Require
from method to return `apistatus.ObjectLocked` error on a corresponding
condition. Adopt implementations.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
d163008b63 [#1175] storage engine: Fix Lock implementation
`Lock` must return `apistatus.IrregularObjectLock` error instead of
`apistatus.ObjectLocked` one.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
88cefa7c5f [#1175] storage engine: Return status error on Delete of locked object
Make `StorageEngine.Delete` to forward first encountered
`apistatus.ObjectLocked` error during shard processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
97b6ec8150 [#1175] storage engine: Return status error on Inhume of locked object
Make `StorageEngine.Inhume` to forward first encountered
`apistatus.ObjectLocked` error during shard processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
545861d1bc [#1175] storage engine: Implement Lock operation
Implement `StorageEngine.Lock` method which works similar to `Inhume`
but calls `Lock` on the processing shards.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
7ccd1625af [#1214] *: Use single Object type in whole project
Remove `Object` and `RawObject` types from `pkg/core/object` package.
Use `Object` type from NeoFS SDK Go library everywhere. Avoid using the
deprecated elements.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 17:45:00 +03:00
Evgenii Stratonikov
19ad349b27 [#1204] shard: Save ID in the metabase
`AddShard` must return shard id, so we temporarily open metabase
there.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 17:13:13 +03:00