fyrchik/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
2982 commits 20 branches 26 tags 30 MiB
Commit graph

573 commits

Author SHA1 Message Date
Evgenii Stratonikov
8cf71b7f1c [#1324] local_object_storage: Implement tree service backend 
In this commit we implement algorithm for CRDT trees from
https://martin.klepmann.com/papers/move-op.pdf

Each tree is identified by the ID of a container it belongs to
and the tree name itself. Essentially, it is a sequence of operations
which should be applied in chronological order to get a usual tree
representation.

There are 2 backends for now: bbolt database and in-memory.
In-memory backend is here for debugging and will eventually act
as a memory-cache for the on-disk database.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
11c9df6f00 [#1611] shard: Print shard ID in component logs 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-19 15:57:53 +03:00
Pavel Karpy
9f7a22e2aa [#1561] object: Return OUT_OF_RANGE status 
Replace `ErrRangeOutOfBounds` error from `pkg/core/object` package with
`ObjectOutOfRange` from `apistatus` package. That error is returned by
storage node's server as NeoFS API statuses.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-19 13:29:45 +03:00
Pavel Karpy
cb0bb7207c [#1461] shard: Add a separate ErrLockObjectRemoval 
Do not return `meta.ErrLockObjectRemoval` from shard's methods, add shard's
own error for that.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
558cc1193a [#1461] engine: Clarify force removal 
Document force removal behaviour in all the Storage engine parts.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
51afcc1182 [#1461] engine, policer: Force remove objects w/o container 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
18ec5d7c8e [#1461] meta: Return error on lock object removal 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
b5c56d459a [#1461] engine: Add force lock removal tests 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
63c00e785d [#1461] shard: Fix option naming 
`WitDeletedLockCallback` => `WithDeletedLockCallback`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
5122be34e7 [#1461] shard: Add lock tests 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
eaf96bccf7 [#1461] meta: Add lock tests 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
c6cf8e5c0b [#1461] engine: Fix LOCK test 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
9c5ef3bab8 [#1461] node: Allow force LOCK removal 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
fed9e6679d [#1461] node: Unlock locked object on its lock removal 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Evgenii Stratonikov
72c044e2eb [#1599] engine: Parallelize shard initialization 
Shard is intended to be used as a separate failure domain,
which usually resides on a separate disk. Thus, sequential
initialization is bound by IO and this change speeds up thing a bit.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-15 13:44:54 +03:00
Evgenii Stratonikov
633b4e7d2d [#1483] metabase: Add VERSION.md 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
6f243a2a76 [#1483] metabase: Store version 
The main problem is to distinguish the case of initial initialization
and update from version 0. We can't do this at `Open`, because of
`resync_metabase` flag. Thus, the following approach was taken:
1. During `Open` check whether the metabase was initialized.
2. Check for the version in `Init` or write the new one if the metabase
   is new.
3. Update version in `Reset`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
7df50297cd [#1520] shard: Ignore errors on metabase refill 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-21 17:25:28 +03:00
Evgenii Stratonikov
78ea450c25 [#1502] shard: Process locks on metabase refill 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-21 11:07:26 +03:00
Evgenii Stratonikov
972ca83e23 [#1524] writecache: Add some bolt parameters to the configuration 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
07e06249d5 [#1524] metabase: Add some bolt parameters to the configuration 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
81f925d5a0 [#1516] metabase: Optimize ListWithCursor for long listings 
Cache buckets outside of the main loop and allocate memory for the
resulting offset only once.

```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       6.45µs ±14%    5.79µs ±11%  -10.24%  (p=0.002 n=10+10)
ListWithCursor/10_items-8     20.9µs ±17%    17.3µs ± 9%  -17.27%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     153µs ±12%     131µs ± 9%  -14.63%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       2.31kB ± 0%    1.91kB ± 0%  -17.46%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     6.94kB ± 0%    5.50kB ± 0%  -20.78%  (p=0.000 n=8+8)
ListWithCursor/100_items-8    53.3kB ± 0%    41.5kB ± 0%  -22.18%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         40.0 ± 0%      34.0 ± 0%  -15.00%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        121 ± 0%       100 ± 0%  -17.36%  (p=0.000 n=10+10)
ListWithCursor/100_items-8       930 ± 0%       758 ± 0%  -18.49%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
a93373fe71 [#1516] metabase: Cache graveyard buckets in ListWithCursor 
```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       6.40µs ±13%    6.45µs ±14%     ~     (p=0.739 n=10+10)
ListWithCursor/10_items-8     30.9µs ±21%    20.9µs ±17%  -32.49%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     274µs ±27%     153µs ±12%  -44.09%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       2.26kB ± 0%    2.31kB ± 0%   +2.46%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     10.8kB ± 0%     6.9kB ± 0%  -36.07%  (p=0.000 n=8+8)
ListWithCursor/100_items-8    96.8kB ± 0%    53.3kB ± 0%  -44.98%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         39.0 ± 0%      40.0 ± 0%   +2.56%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        192 ± 0%       121 ± 0%  -36.98%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     1.72k ± 0%     0.93k ± 0%  -45.93%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>

name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       5.23µs ±19%    5.26µs ±15%     ~     (p=0.853 n=10+10)
ListWithCursor/10_items-8     27.2µs ±15%    18.0µs ±19%  -33.80%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     250µs ±13%     139µs ±15%  -44.27%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       1.99kB ± 0%    2.04kB ± 0%   +2.82%  (p=0.000 n=8+8)
ListWithCursor/10_items-8     10.3kB ± 0%     6.4kB ± 0%  -37.83%  (p=0.000 n=8+10)
ListWithCursor/100_items-8    93.9kB ± 0%    50.4kB ± 0%  -46.37%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         35.0 ± 0%      36.0 ± 0%   +2.86%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        184 ± 0%       113 ± 0%  -38.59%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     1.67k ± 0%     0.88k ± 0%  -47.29%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
af4db8a73b [#1516] metabase: Cache address key and do not decode address twice 
```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       10.6µs ± 1%     6.4µs ±13%  -39.62%  (p=0.000 n=7+10)
ListWithCursor/10_items-8     75.3µs ± 2%    30.9µs ±21%  -58.97%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     726µs ± 2%     274µs ±27%  -62.28%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       3.19kB ± 0%    2.26kB ± 0%  -29.21%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     20.7kB ± 0%    10.8kB ± 0%  -47.68%  (p=0.000 n=10+8)
ListWithCursor/100_items-8     196kB ± 0%      97kB ± 0%  -50.65%  (p=0.000 n=7+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         55.0 ± 0%      39.0 ± 0%  -29.09%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        346 ± 0%       192 ± 0%  -44.51%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     3.25k ± 0%     1.72k ± 0%  -47.13%  (p=0.000 n=9+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
504f45e9ee [#1516] metabase: Add benchmark for ListWithCursor 
```
name                        time/op
ListWithCursor/1_item-8     10.6µs ± 1%
ListWithCursor/10_items-8   75.3µs ± 2%
ListWithCursor/100_items-8   726µs ± 2%

name                        alloc/op
ListWithCursor/1_item-8     3.19kB ± 0%
ListWithCursor/10_items-8   20.7kB ± 0%
ListWithCursor/100_items-8   196kB ± 0%

name                        allocs/op
ListWithCursor/1_item-8       55.0 ± 0%
ListWithCursor/10_items-8      346 ± 0%
ListWithCursor/100_items-8   3.25k ± 0%
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
f602d05b0a [#1494] *: Fix linter warnings 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
38772e1a2e writecache: provide timeout to bbolt.Open 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-09 14:19:47 +03:00
Pavel Karpy
010253a97a [#1460] blobovnicza: Do not use pointers as the results 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
0e4a1beecf [#1460] blobstor: Do not use pointers as the results 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
08bf8a68f1 [#1460] engine: Do not use pointers as the results 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
7b6363f4c6 [#1460] shard: Do not use pointers as the results 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
9b2932609b [#1460] meta: Do not use pointers as the results 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
a580429996 [#1460] meta: Add a benchmark on Get operation 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
b0c7b7851a [#1418] blobstor: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
14366bbd89 [#1418] engine: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
5f57db6bf8 [#1418] shard: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
6d4d920bad [#1418] meta: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Evgenii Stratonikov
feef9a98f7 [#1462] writecache: remove useless logs 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
2ae7c94cd6 [#1462] *: Remove log.With invocations 
`log.With` is suitable during initialization, but in other places it induces
some overhead, even when branches with logging are not taken.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
43e776dfb1 [#1462] writecache: Persist objects in batches 
`Update` becomes a botleneck under a heavy load.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
6ad87e7959 [#1462] writecache: Fill flush marks during startup 
Some of the objects are already flushed, don't do it twice.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
f2a7503964 [#1462] fstree: Allow to fetch file content lazily 
If we should process address based on some condition, there is no need
to read file content in memory.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
54d4503701 [#1462] writecache: Rename flushWorkersCount 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
0148209168 [#1462] writecache: Use Timer instead of Ticker in flush 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
5073a37930 [#1433] metabase: Optimize Select 
For some filters we can scan only a subset of keys instead of checking
each key.

```
name                       old time/op  new time/op  delta
Select/string_equal-8      49.3µs ± 4%  11.0µs ± 4%  -77.68%  (p=0.000 n=10+10)
Select/string_not_equal-8  7.01ms ± 5%  7.06ms ±10%     ~     (p=0.971 n=10+10)
Select/common_prefix-8      118µs ± 6%    79µs ± 5%  -33.04%  (p=0.000 n=10+9)
Select/unknown-8           21.3µs ± 4%   3.2µs ± 4%  -84.88%  (p=0.000 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
5c48588c64 [#1433] metabase: Add more tests for Select 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
ea76c989ca [#1433] metabase: Add a benchmark for Select 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
b602fff01f [#1463] blobovnicza: Remove pointers from struct members 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 13:42:54 +03:00
Evgenii Stratonikov
d9f0ac8909 [#1463] blobovnicza: Determine db size via os.Stat 
Currently we use `(*bbolt.Bucket).Stats().KeyN` for estimating database
size. However, it iterates over all pages in bucket and thus heavily
depends on the bucket size. This commit replaces initial size estimation
with a single `os.Stat` call.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 13:42:54 +03:00
Pavel Karpy
021aa97965 [#1418] blobovnicza: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Pavel Karpy
281befec67 [#1418] blobstor: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Pavel Karpy
babd382ba5 [#1418] engine: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Pavel Karpy
6e752f36dc [#1418] shard: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Pavel Karpy
e265ce2d52 [#1418] meta: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Pavel Karpy
1c100fb4b0 [#1418] writecache: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Leonard Lyubich
1c30414a6c [#1454] Upgrade NeoFS SDK Go module with new IDs 
Core changes:
 * avoid package-colliding variable naming
 * avoid using pointers to IDs where unnecessary
 * avoid using `idSDK` import alias pattern
 * use `EncodeToString` for protocol string calculation and `String` for
  printing

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-01 17:41:45 +03:00
Leonard Lyubich
883f2f1ac6 [#1423] local storage: Activate skipped unit tests after fix 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-27 15:45:33 +03:00
Leonard Lyubich
86552cf3ae engine: make corruptSubDir to ignore blobovnicza dir 
There is a dir for Blobovnicza tree in BlobStor, so we need to ignore
it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-23 15:33:14 +03:00
Leonard Lyubich
bb25ecbd15 [#1400] owner: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-23 15:33:14 +03:00
Leonard Lyubich
f15e6e888f [#1377] oid, cid: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-16 15:33:22 +03:00
Leonard Lyubich
ae92074272 [#1369] checksum: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-12 09:48:31 +03:00
Leonard Lyubich
088df0e2a9 [#1369] version: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-12 09:48:31 +03:00
Elizaveta Chichindaeva
cc7a723d77 [#1320] English Check 
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
 2022-05-11 10:40:02 +03:00
Evgenii Stratonikov
a4769d8624 *: use require.ErrorIs where possible 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-04-29 16:47:57 +03:00
Evgenii Stratonikov
3c39e6df11 [#1319] blobstor: Compress big objects in a streaming fashion 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-04-29 16:46:40 +03:00
Pavel Karpy
9da7df4d42 [#1318] metabase: Init metabase in test 
`Init()` creates buckets. It prevents NPE in tests.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
486ec8a2e5 [#1318] metabase: Reuse comparing func from require 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
1caeb0b648 [#1318] metabase: Add missing error check in tests 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
2aec5736e0 [#1318] node/gc: Fix tombstones handling 
Handle only expired tombstones.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
7799f8e4cf [#1318] engine: Change tombstone clear process 
- Delete objects physically on tombstone's arrival;
- Store information about tombstones in the Graveyard;
- Clear Graveyard every epoch based on the information about TS in the
network.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
2583f608e8 [#1318] metabase: Add DropGraves method 
Method clean up Graveyard and should be called by the routine that handles
tombstone expiration.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
daab30c391 [#1318] metabase: Add iteration offset 
Add offset element to the iterations over deleted objects (both the
Graveyard and the Garbage buckets).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
fe8076e60a [#1318] metabase: Add immediate object deletion 
Mark objects with GC immediately after a Tombstone is received.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
8107c8d1a9 [#1318] metabase: Separate buckets with TS and GC marks 
It allows storing information about object in both ways at the same time:
1. Metabase should know if an object is covered by a tombstone (that is
not expired yet);
2. It should be possible to physically delete objects covered by a
tombstone immediately (mark with GC) but keep tombstone knowledge.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
bd27837364 [#1321] node: Register GC event channel before shard init 
Morph "NewEpoch" event handling was registered in a closure over
`addNewEpochNotificationHandler` func. That may lead to the data race:
if a shard was initialized before the event registration, everything works
as planned, but if registration was made earlier, it was not able to
include GC handlers since a shard has not called `eventChanInit` yet and,
therefore, it has not registered handler yet.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-25 10:39:37 +03:00
Evgenii Stratonikov
622ea4818f [#1250] *: Remove io/ioutil imports 
It is deprecated starting from go1.16.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-04-07 17:23:28 +03:00
Leonard Lyubich
629a4f79cd [#1293] metabase: Fix out-of-range panic in freePotentialLocks 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-04-05 15:38:51 +03:00
Evgenii Stratonikov
48eb87d32c [#1149] writecache: Prevent corruption in Head 
Also, remove optimization comments:
1. Having to maintain an execute the same logic for headers as for
   objects is quite inefficient, as it increases memory footprint.
2. Unmarshaling object is a cheap operation if data slice is in memory.
3. For unmarshaling header-only, I think we need SDK support.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-31 15:40:25 +03:00
Evgenii Stratonikov
68903c9fd9 [#1143] shard: Support degraded mode in Get and GetRange 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
f058cead8f [#1143] shard: Handle some errors in degraded mode 
If metabase is corrupted for some reason, failback to
checking blobstor directly.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
08e7914729 [#1143] blobstor: Implement existsSmall check 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
aa0cc1f824 [#1143] blobovnicza: Copy object data in Get 
Data returned from `*bbolt.Bucket.Get()` is only valid for the lifetime
of the transaction.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
6472a170eb [#1143] shard: Introduce explicit Degraded mode 
`Degraded` mode is set automatically after error counter is over the
threshold. `ReadOnly` mode can still be set by an administrator.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
de5a2f6574 [#1262] metabase: Remove list index in place Delete 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
43867a3093 [#1262] metabase: Do not allocate intermediate slices for indices 
```
name              old alloc/op   new alloc/op   delta
Put/parallel-8       123kB ± 4%     119kB ± 3%  -2.72%  (p=0.006 n=10+9)
Put/sequential-8     170kB ± 1%     168kB ± 1%  -1.42%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op  delta
Put/parallel-8         473 ± 1%       469 ± 0%  -0.87%  (p=0.000 n=10+10)
Put/sequential-8       792 ± 0%       787 ± 0%  -0.58%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
0e9b6be3fd [#1262] metabase: Remove intermediate allocations in decodeList 
```
name              old time/op    new time/op    delta
Put/parallel-8      1.57ms ±11%    1.51ms ± 3%   -4.06%  (p=0.043 n=9+10)
Put/sequential-8    5.16ms ± 2%    5.16ms ± 3%     ~     (p=1.000 n=9+10)

name              old alloc/op   new alloc/op   delta
Put/parallel-8       126kB ± 4%     123kB ± 4%   -2.54%  (p=0.016 n=8+10)
Put/sequential-8     171kB ± 1%     170kB ± 1%     ~     (p=0.182 n=9+10)

name              old allocs/op  new allocs/op  delta
Put/parallel-8         565 ± 2%       473 ± 1%  -16.18%  (p=0.000 n=9+10)
Put/sequential-8       819 ± 1%       792 ± 0%   -3.34%  (p=0.000 n=9+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
d45df614fb [#1262] metabase: Optimize decodeList 
Prevent additional allocation during `append` in `Put`.

```
name              old alloc/op   new alloc/op   delta
Put/parallel-8       131kB ± 1%     126kB ± 4%  -3.87%  (p=0.005 n=8+8)
Put/sequential-8     172kB ± 1%     171kB ± 1%  -0.73%  (p=0.028 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
456e1584d6 [#1262] metabase: Add benchmarks for Put 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
f1223b46df [#1262] blobovnicza: Make helper in Put function idempotent 
`Batch` can execute the function multiple times leading to multiple
increases of a size approximation.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
ad92493b86 [#1268] blobstor: Cleanup zstd encoders/decoders 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-25 14:05:06 +03:00
Leonard Lyubich
318639e5bf [#1247] *: Clarify docs about returned errors from apistatus package 
`apistatus` package provides types which implement build-in `error`
interface. Add `error of type` pattern when documenting these errors in
order to clarify how these errors should be handled (e.g. `errors.Is` is
not good).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-17 16:34:00 +03:00
Leonard Lyubich
70ffdf3478 [#1247] object: Return NOT_FOUND and ALREADY_REMOVED statuses 
Replace `ErrNotFound`/`ErrAlreadyRemoved` error from
`pkg/core/object` package with `ObjectNotFound`/`ObjectAlreadyRemoved`
one from `apistatus` package. These errors are returned by storage
node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-17 16:34:00 +03:00
Alex Vanin
0adb29c035 [#1214] Fix metabase put logging 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-03-16 17:17:18 +03:00
Alex Vanin
8f476f3c4d [#1238] Adopt neofs-node for non pointer slices in SDK 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-03-16 15:29:31 +03:00
Alex Vanin
7f3195b197 [#1238] Adopt neofs-cli for non pointer slices in SDK 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-03-16 15:29:31 +03:00
Leonard Lyubich
9feb5f9405 [#1175] engine: Fix AddShard implementation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
9dff07200c [#1181] local storage: Process expired locks similar to tombstones 
There is a need to process expired `LOCK` objects similar to `TOMBSTONE`
ones: we collect them on `Shard`, notify all other shards about
expiration so they could unlock the objects, and only after that mark
lockers as garbage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
323dea95c6 [#1181] Do not use deprecated elements 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
e3f4e48d87 [#1175] Fix typo in error return docs 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
ced854bc2e [#1175] object/fmt: Handle LOCK objects 
Make `FormatValidator.ValidateContent` to verify payload of `LOCK`
objects. Pass locked objects to `Locker` interface. Require from
`Locker.Lock` to return `apistatus.IrregularObjectLock` error on a
corresponding condition.

Also add error return to `DeleteHandler.DeleteObjects` method. Require
from method to return `apistatus.ObjectLocked` error on a corresponding
condition. Adopt implementations.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
d163008b63 [#1175] storage engine: Fix Lock implementation 
`Lock` must return `apistatus.IrregularObjectLock` error instead of
`apistatus.ObjectLocked` one.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
07107c8bfd [#1175] metabase: Update locker list during Inhume 
`Inhume` operation can potentially mark lockers as garbage. There is a
need to update locker list in locked bucket.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
26b8e33ed7 [#1175] metabase: Fix encoding of updated locker list 
Even single locker must be encoded using `encodeList`, or subsequent
`decodeList` call will fail.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
88cefa7c5f [#1175] storage engine: Return status error on Delete of locked object 
Make `StorageEngine.Delete` to forward first encountered
`apistatus.ObjectLocked` error during shard processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
97b6ec8150 [#1175] storage engine: Return status error on Inhume of locked object 
Make `StorageEngine.Inhume` to forward first encountered
`apistatus.ObjectLocked` error during shard processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
545861d1bc [#1175] storage engine: Implement Lock operation 
Implement `StorageEngine.Lock` method which works similar to `Inhume`
but calls `Lock` on the processing shards.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
6ed85ff1e1 [#1175] shard: Implement Lock operation 
Implement `Shard.Lock` method which required rw mode and calls `Lock` on
underlying metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
e21eedee06 [#1175] shard: Return status error on Inhume of locked object 
Make `Shard.Inhume` to forward `apistatus.ObjectLocked` returned by
underlying metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
b585791d6e [#1175] metabase: Return status error on Lock of irregular object 
Make `DB.Lock` to return `apistatus.IrregularObjectLock` if at least one
of the locked objects is irregular (not of type REGULAR).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
23fcacd3f2 [#1175] metabase: Return status error on Inhume of locked object 
Make `DB.Inhume` to return `apistatus.ObjectLocked` if at least one of
the inhumed objects is locked.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
303eb2e078 [#1176] metabase: Ignore locked objs in IterateCoveredByTombstones 
Make `DB.IterateCoveredByTombstones` to not pass locked objects to the
handler. The method is used by GC, therefore it will not consider locked
objects as candidates for deletion even if their tombstone is expired.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
dba3e58dc5 [#1176] metabase: Ignore locked objects in IterateExpired 
Make `DB.IterateExpired` to not pass locked objects to the handler. The
method is used by GC, therefore it will not consider them as candidates
for deletion.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
9508633a7e [#1175] metabase: Work with LOCK objects 
After introduction of LOCK objects (of type `TypeLock`) complicated
extended its behavior:
  * create `lockers` container bucket (LCB) during PUT;
  * remove object from LCB during DELETE;
  * look up object in LCB during EXISTS;
  * get object from LCB during GET;
  * list objects from LCB during LIST with cursor;
  * select objects from LCB during SELECT with '*'.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
9f13674a10 [#1175] metabase: Implement LOCK operation 
Implement `DB.Lock` method which marks list of the objects as locked by
another object. Only regular objects can be locked.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
14d27455f3 [#1175] metabase: Add container bucket for LOCKs 
Create class of container buckets with `LOCKED` suffix. Put identifiers
of the objects of type `LOCK` to these buckets during `DB.Put`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
e90ac9c6f9 [#1181] metabase: Remove unused CleanUp method 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-15 13:03:23 +03:00
Leonard Lyubich
7ccd1625af [#1214] *: Use single Object type in whole project 
Remove `Object` and `RawObject` types from `pkg/core/object` package.
Use `Object` type from NeoFS SDK Go library everywhere. Avoid using the
deprecated elements.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 17:45:00 +03:00
Evgenii Stratonikov
19ad349b27 [#1204] shard: Save ID in the metabase 
`AddShard` must return shard id, so we temporarily open metabase
there.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-04 17:13:13 +03:00
Evgenii Stratonikov
1fe9cd4d36 [#1186] engine: Allow to skip metabase in GetRange 
Similarly to `Get`. Also fix a bug where `ErrNotFound` is returned
instead of `ErrRangeOutOfBounds`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
2b5550ccf6 [#1186] blobstor: Fix comment 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
b33fb0f739 [#1186] blobstor: Unify errors for Get and GetRangeBig 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
69e1e6ca20 [#1186] engine: Read object directly from blobstor in case of conflicts 
Metabase is expected to contain actual information about objects stored
in shard. If the object is present in metabase but is missing from
blobstor, peform an additional attempt to fetch it directly without
consulting metabase. Such a situation is unexpected, so error counter
is increased for the shard which has the object in the metabase. We
don't increase error counter for the shard which has the object in
blobstor, because some garbage can be expected there. In this
implementation there is no overhead for objects which are really
missing, i.e. are not present in any metabase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
6e6f3648d2 [#1188] metabase: log PUT / DELETE operations 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-04 15:21:35 +03:00
Evgenii Stratonikov
ce76325840 [#1146] engine: remove TODO and add a comment 
Parallel check seems to be slower, so remove TODO.
```
name \ time/op      old          new           pool
Exists/2_shards-40  52.3µs ±23%   91.6µs ± 6%   84.2µs ± 1%
Exists/4_shards-40  72.7µs ±11%  121.8µs ± 8%  116.9µs ± 5%

name \ alloc/op     old          new           pool
Exists/2_shards-40  5.00kB ± 0%   5.03kB ± 0%   5.16kB ± 0%
Exists/4_shards-40  9.89kB ± 0%   9.93kB ± 1%  10.19kB ± 0%

name \ allocs/op    old          new           pool
Exists/2_shards-40     112 ± 1%      115 ± 1%      117 ± 1%
Exists/4_shards-40     207 ± 1%      211 ± 1%      216 ± 1%
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-04 12:31:41 +03:00
Evgenii Stratonikov
dab6bcabe4 [#1146] engine: Add benchmarks for exists 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-03-04 12:31:41 +03:00
Evgenii Stratonikov
050a4bb2b0 [#1115] *: link TODOs to corresponding issues 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
f9d3111825 [#1115] blobovnicza: remove GetRange method 
It is unused and incorrect as described in FIXME.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
c1530dec5e [#1115] shard/gc: provide parsed addresses in HandleExpiredTombstones 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-02-11 12:58:59 +03:00
Pavel Karpy
1667ec9e6d [#1131] *: Adopt SDK changes 
`object.Address` has been moved to `object/address`
`object.ID` has been moved to `object/id`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-02-08 09:45:38 +03:00
Evgenii Stratonikov
ed50cf6207 [#1118] services/control: return error counter in ListShards 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-02-03 15:14:27 +03:00
Evgenii Stratonikov
7fb15fa1d0 [#1118] services/control: allow to reset error counter in SetShardMode 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-02-03 15:14:27 +03:00
Evgenii Stratonikov
6ad2624552 [#1118] engine: allow to set error threshold 
There are certain errors which are not expected during usual node
operation and which tell us that something is wrong with the shard.
To prevent possible data corruption, move shard in read-only mode after
amount of errors exceeded some threshold. By default no actions are performed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-02-03 15:14:27 +03:00
Evgenii Stratonikov
674f520da7 [#1132] *: Use path/filepath package when working with files 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-02-03 10:12:07 +03:00
Evgenii Stratonikov
5a03a14940 [#1094] metabase: move Select filters sanity check outside of transaction 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
651adf46c6 [#1094] blobovnicza: calculate size in Init properly 
If pre-existing blobovnicza is initialized, it's size should be updated
even if all buckets are in place.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
418b553920 [#1094] shard: remove TODO for shard mode description 
It was added back in 2fb379b7 when we had many shard modes. Now we have
only two and comments for constants are rather descriptive.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
ed7c732676 [#1094] shard: unify collection of expired objects 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
18cfd8b042 [#1086] services/control: implement RestoreShard RPC 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
0e60b1d6c9 [#1086] services/control: implement DumpShard RPC 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
adb80bebb2 [#1085] shard: rename Evacuate to Dump 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
d06425c852 [#1085] shard: allow to ignore errors in Evacuate 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
36eebb5932 [#1085] writecache: allow to ignore errors during iteration 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
cd75638ce3 [#1085] blobstor: allow to ignore errors during iteration 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
c954f0e71b [#1085] fstree: allow to ignore errors during iteration 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
e53ad2f468 [#1085] blobovnicza: allow to ignore errors during iteration 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
9c60ab893c [#1085] shard: allow to ignore errors during restore 
We could also ignore errors during evacuate, but this requires
unmarshaling objects first which slowers the process considerably.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
0ef3d5ab03 [#1085] writecache: persist in-memory objects before shutdown 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
3215210d0c [#1085] shard: Close components in reverse order 
Write-cache should be closed before blobstor and metabase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
b432ec3a03 [#1085] shard/test: add object integrity check for Evacuate 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
c0f6c988f0 [#1085] shard: dump data from write-cache 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-01-26 15:28:13 +03:00