Commit graph

409 commits

Author SHA1 Message Date
Leonard Lyubich
14d27455f3 [#1175] metabase: Add container bucket for LOCKs
Create class of container buckets with `LOCKED` suffix. Put identifiers
of the objects of type `LOCK` to these buckets during `DB.Put`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
e90ac9c6f9 [#1181] metabase: Remove unused CleanUp method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
7ccd1625af [#1214] *: Use single Object type in whole project
Remove `Object` and `RawObject` types from `pkg/core/object` package.
Use `Object` type from NeoFS SDK Go library everywhere. Avoid using the
deprecated elements.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 17:45:00 +03:00
Evgenii Stratonikov
19ad349b27 [#1204] shard: Save ID in the metabase
`AddShard` must return shard id, so we temporarily open metabase
there.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 17:13:13 +03:00
Evgenii Stratonikov
1fe9cd4d36 [#1186] engine: Allow to skip metabase in GetRange
Similarly to `Get`. Also fix a bug where `ErrNotFound` is returned
instead of `ErrRangeOutOfBounds`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
2b5550ccf6 [#1186] blobstor: Fix comment
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
b33fb0f739 [#1186] blobstor: Unify errors for Get and GetRangeBig
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
69e1e6ca20 [#1186] engine: Read object directly from blobstor in case of conflicts
Metabase is expected to contain actual information about objects stored
in shard. If the object is present in metabase but is missing from
blobstor, peform an additional attempt to fetch it directly without
consulting metabase. Such a situation is unexpected, so error counter
is increased for the shard which has the object in the metabase. We
don't increase error counter for the shard which has the object in
blobstor, because some garbage can be expected there. In this
implementation there is no overhead for objects which are really
missing, i.e. are not present in any metabase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
6e6f3648d2 [#1188] metabase: log PUT / DELETE operations
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 15:21:35 +03:00
Evgenii Stratonikov
ce76325840 [#1146] engine: remove TODO and add a comment
Parallel check seems to be slower, so remove TODO.
```
name \ time/op      old          new           pool
Exists/2_shards-40  52.3µs ±23%   91.6µs ± 6%   84.2µs ± 1%
Exists/4_shards-40  72.7µs ±11%  121.8µs ± 8%  116.9µs ± 5%

name \ alloc/op     old          new           pool
Exists/2_shards-40  5.00kB ± 0%   5.03kB ± 0%   5.16kB ± 0%
Exists/4_shards-40  9.89kB ± 0%   9.93kB ± 1%  10.19kB ± 0%

name \ allocs/op    old          new           pool
Exists/2_shards-40     112 ± 1%      115 ± 1%      117 ± 1%
Exists/4_shards-40     207 ± 1%      211 ± 1%      216 ± 1%
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 12:31:41 +03:00
Evgenii Stratonikov
dab6bcabe4 [#1146] engine: Add benchmarks for exists
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 12:31:41 +03:00
Evgenii Stratonikov
050a4bb2b0 [#1115] *: link TODOs to corresponding issues
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
f9d3111825 [#1115] blobovnicza: remove GetRange method
It is unused and incorrect as described in FIXME.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
c1530dec5e [#1115] shard/gc: provide parsed addresses in HandleExpiredTombstones
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Pavel Karpy
1667ec9e6d [#1131] *: Adopt SDK changes
`object.Address` has been moved to `object/address`
`object.ID` has been moved to `object/id`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-02-08 09:45:38 +03:00
Evgenii Stratonikov
ed50cf6207 [#1118] services/control: return error counter in ListShards
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Evgenii Stratonikov
7fb15fa1d0 [#1118] services/control: allow to reset error counter in SetShardMode
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Evgenii Stratonikov
6ad2624552 [#1118] engine: allow to set error threshold
There are certain errors which are not expected during usual node
operation and which tell us that something is wrong with the shard.
To prevent possible data corruption, move shard in read-only mode after
amount of errors exceeded some threshold. By default no actions are performed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Evgenii Stratonikov
674f520da7 [#1132] *: Use path/filepath package when working with files
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 10:12:07 +03:00
Evgenii Stratonikov
5a03a14940 [#1094] metabase: move Select filters sanity check outside of transaction
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
651adf46c6 [#1094] blobovnicza: calculate size in Init properly
If pre-existing blobovnicza is initialized, it's size should be updated
even if all buckets are in place.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
418b553920 [#1094] shard: remove TODO for shard mode description
It was added back in 2fb379b7 when we had many shard modes. Now we have
only two and comments for constants are rather descriptive.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
ed7c732676 [#1094] shard: unify collection of expired objects
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
18cfd8b042 [#1086] services/control: implement RestoreShard RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
0e60b1d6c9 [#1086] services/control: implement DumpShard RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
adb80bebb2 [#1085] shard: rename Evacuate to Dump
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
d06425c852 [#1085] shard: allow to ignore errors in Evacuate
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
36eebb5932 [#1085] writecache: allow to ignore errors during iteration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
cd75638ce3 [#1085] blobstor: allow to ignore errors during iteration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
c954f0e71b [#1085] fstree: allow to ignore errors during iteration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
e53ad2f468 [#1085] blobovnicza: allow to ignore errors during iteration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
9c60ab893c [#1085] shard: allow to ignore errors during restore
We could also ignore errors during evacuate, but this requires
unmarshaling objects first which slowers the process considerably.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
0ef3d5ab03 [#1085] writecache: persist in-memory objects before shutdown
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
3215210d0c [#1085] shard: Close components in reverse order
Write-cache should be closed before blobstor and metabase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
b432ec3a03 [#1085] shard/test: add object integrity check for Evacuate
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
c0f6c988f0 [#1085] shard: dump data from write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
ad01aaf8bf [#1085] writecache: add read-only mode
In read-only mode modifying operations are immediately returned with
error and all background operations are suspended.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
9f963e001b [#1085] writecache: set metaOnly flag for receiving from priority channel
This can lead to objects being flushed twice.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
7eedb23eb7 [#1085] shard: allow to dump/restore objects
Dump contains magic and a list of objects prefixed by object size in bytes.
We can't use proto-marshaled list because this requires having all dump
in memory. Using TAR induces 512 byte overhead for each object which can
be a problem in some cases.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Pavel Karpy
fee2f5a330 [#1100] *: Adopt new SDK's owner.ID API
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 17:58:30 +03:00
ZhangTao1596
ac82899e85 [#1088] writecache: Fix small flushed when persist
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2022-01-19 12:55:33 +03:00
Evgenii Stratonikov
486d5c2e86 [#1060] writecache: compress big object if needed
Small objects use `blobstor.Put`, so no changes are required.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-13 14:20:36 +03:00
Evgenii Stratonikov
0d969d7a06 [#1060] blobstor: allow to disable compression based on content-type
For some data compression makes little sense, as it is already compressed.
This commit allows to leave such data unchanged based on `Content-Type`
attribute. Currently exact, prefix and suffix matching are supported.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-13 14:20:36 +03:00
Pavel Karpy
375394dc99 [#1059] shard: Add shard mode to shard Info
Provide shard mode information via `DumpInfo()`. Delete atomic field from
Shard structure since it duplicates new field.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
93bd6be743 [#1057] shard: Do not allow memory change operations in "read-only"
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-29 18:06:15 +03:00
Pavel Karpy
4f756bf121 [#1057] shard: Make shard to have only two mode
Shard's mode was not used in the Node, so added only two modes whose roles
are clear. More modes will be added in the future.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-29 18:06:15 +03:00
Pavel Karpy
284188f8f9 [#1048] node: Fill shard's info with its components' infos
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
ZhangTao1596
1be0c42726 [#1022] writecache: Fix db counter decrement and flushed big object
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2021-12-15 14:23:49 +03:00
Leonard Lyubich
fc42844fbf [#1028] writecache: Evict flushed big objects
Make `flushBigObjects` routine to call `evictObjects` which was
successfully written to `BlobStor`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-14 13:49:47 +03:00
Leonard Lyubich
d246e5193b [#1028] writecache: Mark big objects as flushed after write
Make `flushBigObjects` routine to mark objects which are written to
`BlobStor`. This prevents already flushed objects from being written on
the next iterator tick.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-14 13:49:47 +03:00
Leonard Lyubich
90fd883e32 [#915] blobovnicza: Use mean bucket size multiplier instead of min
For fullness estimation of `Blobovnicza` we use number of object stored
in each size bucket. In previous implementation we multiplied the number
by the difference in bucket boundaries. This expression rather
estimated the minimum volume (and for the smallest bucket, the maximum)
of objects in the bucket.

Multiply number of objects by mean bucket size.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-08 14:24:49 +03:00
Leonard Lyubich
2920c5203b [#915] blobovnicza: Re-use Bolt transaction in syncFullnessCounter
Make `syncFullnessCounter` to accept `bbolt.Tx` argument of Bolt
transaction within which counter should be synchronized. Pass
corresponding transaction during `Init`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-08 14:24:49 +03:00
Leonard Lyubich
f9bcb6f5e3 [#1007] metabase: Copy BoltDB value returned by IsSmall
According to BoltDB documentation bucket `value is only valid for the
life of the transaction`.

Make `DB.IsSmall` copy value slice in order to prevent potential memory
corruptions (e.g. `runtime.stringtobyteslice` cast).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-07 18:59:42 +03:00
Leonard Lyubich
6f41cd86ed [#979] Upgrade SDK NeoFS Go dependency
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Alex Vanin
1f911830a5 [#948] engine: Define EndOfListing error in component packages
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
c80c83b0b8 [#948] metabase: Simplify cursor structure
Use bbolt bucket keys to seek offset in root bucket
and container buckets.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
468caa83d9 [#948] Adopt sdk-go client
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
08bdd0d561 [#948] engine: Fix comments of object listing with cursor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
aa9ce8a853 [#948] engine: Define cursor for object listing as a type
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
164cd10af8 [#948] engine: Hide shard IDs in ListWithCursor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
0f6d8f6eea [#948] metabase: Use seek for faster cursor listing
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
8d471c7e36 [#948] core/object: Move listing objects error to core
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
c02c7bee5b [#948] engine: Add ListWithCursor method
Interface for shard.ListWithCursor invocations.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
5b6be7bc1c [#948] engine: Fix test file path
Before: `<t.Name()>.<id>.blobstor`
After:  `<t.Name()>/<id>.blobstor`
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
1950724a04 [#948] shard: Add ListWithCursor method
Interface for metabase.ListWithCursor method invocation.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
1a829a521f [#948] metabase: Add ListWithCursor method
ListWithCursor allows listing physically stored objects
from metabase with small chunks. Cursor tracks last
processed object, therefore new chunks are returned
on each request.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
cb36f8b50e [#868] blobstor: Speed up compression test
Compression test does not require wide blobovnicza tree.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-16 13:35:34 +03:00
Alex Vanin
e4087893a1 [#966] engine: Fix test after neofs-api-go replacement
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-15 13:06:10 +03:00
Evgenii Stratonikov
95893927aa *: replace neofs-api-go with neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-12 17:29:09 +03:00
Leonard Lyubich
6c0b29e3e3 [#922] storage engine: Prevent any operations after first Close call
Make `BlockExecution` / `ResumeExecution` to not release per-shard worker
pools. Make `StorageEngine.Close` to block these methods and any
data-related operations. It is still releases the pools.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
d6457ee485 [#922] shard: Allow multiple gc.stop() calls
Use `sync.Once` to prevent locks of stopping GC. It will also allow to
safely call `Shard.Close` multiple  times.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
10f0bd91d6 [#922] engine: Change interface of container operations
Add `error` to return. Improve docs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
a537334f33 [#922] engine: Fix typos and improve naming related to exec blocks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
ec04e787aa [#922] storage engine: Support operation blocking
There is a need to disable execution of local data operation on storage
engine in runtime. If storage engine ops are blocked, node will act like
always but all local object operations will be denied.

Implement `BlockExecution` / `ResumeExecution` methods on `StorageEngine`
which blocks / resumes the execution of data ops. Wait for the completion of
all operations executed at the time of the call. Return error passed to
`BlockExecution` from all data-related methods until `ResumeExecution` call.
Make `Close` to block operations as well.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
cea1de3a27 [#922] storage engine: Remove unused Exists method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Evgenii Stratonikov
1462824ab8 [#947] writecache: refactor object persisting
a1696a8 introduced some logic which in some situations prevented big objects
to be persisted in FSTree. In this commit a refactoring is done with the
goal of simplifying the code and also checking #866 issue.

1. Split a monstrous function into multiple simple ones: memory objects
   can only be small and for writing through the cache we can do a dispatch
   in `Put` itself.
2. Determine objects to be put in database before the actual update
   as setting up a transaction has non-zero overhead.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-01 15:52:56 +03:00
Alex Vanin
2271944064 [#945] metabase: Use cleanup instead of defer in tests
Cleanup is more idiomatic way to clean test data

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 16:10:01 +03:00
Alex Vanin
e41aba610d [#945] metabase: Fix containers listing
Container listing should not ignore tombstone and
storage group objects which are not stored in
primary buckets.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 16:10:01 +03:00
Evgenii Stratonikov
0057eeb0f7 [#901] *: release worker pools where possible
Some of the pools are initialized during config initialization,
so it isn't possible currently to release them in one place.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Evgenii Stratonikov
10afd26354 [#901] engine: release shard pools in Close
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Pavel Karpy
616013cb8a [#933] write-cache: Use real number of objects in counters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 18:49:10 +03:00
Pavel Karpy
29955a3036 [#933] FSTree: Add NumberOfObjects method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 18:49:10 +03:00
Evgenii Stratonikov
1a1435be3d [#791] neofs-lens: inspect objects by address
```
> neofs-lens list --path ./blob/blobovnicza/1/1/0
6ay4GfhR9RgN28d5ufg63toPetkYHGcpcW7G3b7QWSek/9ibXu6v4uTwLEcME5vyHev6Zi8LpxqiWTe1dahKpAbb6
H3VBttoLQoknzMDgnVNyLZ8EpkDnQjnaxDr9fnAWeEHA/Hw1titdGh7BrTe2yLotiYbVh9FQaRRNhoNzXTyetpFgt

> neofs-lens inspect --path ./blob/blobovnicza/1/1/0 \
    --address 6ay4GfhR9RgN28d5ufg63toPetkYHGcpcW7G3b7QWSek/9ibXu6v4uTwLEcME5vyHev6Zi8LpxqiWTe1dahKpAbb6 \
    --header --out payload
Version: v2.1
Type: REGULAR
CID: 6ay4GfhR9RgN28d5ufg63toPetkYHGcpcW7G3b7QWSek
ID: 9ibXu6v4uTwLEcME5vyHev6Zi8LpxqiWTe1dahKpAbb6
Owner: 2dokPzmmcLnnR21jQB3qPppTQRgwMNMKEWD
CreatedAt: 0
PayloadSize: 32
Attributes:
  foo: bar

> hexdump -C payload
00000000  ff 6c d4 71 c4 83 f1 5f  b9 0b ad b3 7c 58 21 b6  |.l.q..._....|X!.|
00000010  d9 55 26 a4 1a 95 04 68  0b 4e 7c 8b 76 3a 1b 1d  |.U&....h.N|.v:..|
00000020
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-18 11:55:02 +03:00
Evgenii Stratonikov
123328a2f4 blobstor: close all opened blobovniczas
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-18 10:44:50 +03:00
Leonard Lyubich
1b3637aad7 [#884] shard: Mark expired tombstones as garbage
Mark expired tombstones as garbage in `Shard.HandleExpiredTombstones`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 16:09:44 +03:00
Leonard Lyubich
3e4f0c1eb9 [#674] object/put: Handle job submission errors
Close wait channel on shard pool error in order to prevent deadlock.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
43ccab3294 [#674] storage engine: Fix unit test failed on shard pool NPE
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
5b1975d52a [#674] storage engine: Use per-shard worker pools for PUT operation
Make `StorageEngine` to use non-blocking worker pools with the same
(configurable) size for PUT operation. This allows you to switch to using
more free shards when overloading others, thereby more evenly distributing
the write load.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Evgenii Stratonikov
dd678cd976 [#868] blobstor: allow to decompress objects on-the-fly
We should be able to read whatever we have written earlier.
Compression setting applies only to the new objects.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 16:01:45 +03:00
Evgenii Stratonikov
cc377b34d2 [#868] blobstor: initialize (de-)compressors in Init
Do not log in options constructors. Also failure to
initialize compression module (possibly due to invalid options) is
certainly an error deserving proper treatment.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 16:01:45 +03:00
Leonard Lyubich
8a1593fdcc [#791] writecache: Export OpenDB function
The function will be useful for the tool which works with write-cache parts.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
ca4c9d4673 [#791] writecache: Implement iterator over small objects
There is a need to list addresses of the small objects stored in WriteCache
database.

Implement `IterateDB` function which accepts BoltDB instance and iterate
over all saved objects and passes their addresses to the hander.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
f73c5c2259 [#791] blobovnicza: Support read-only mode
There is a need to open Blobovnicza instances in read-only mode in some
cases.

Add `ReadOnly` option. Do not create dir path in RO. Open underlying BoltDB
instance with ReadOnly flag. Document thal all writing operations should not
be called in ro (otherwise BoltDB txs fail).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
cc5d3288a1 [#791] blobovnicza: Don't iterate over range buckets in Iterate
`Blobovnicza` can be initialized with any number of range buckets, and
reconstructed with different size limit. In previous implementation
`Iterate` could miss some stored objects if we construct `Blobovnicza` with
smaller number of ranges.

Make `Iterate` to traverse all buckets regardless of current instance
bounds.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
5e2ca0d04b [#791] blobovnicza: Implement method to iterate over object addresses
In previous implementation `Blobovnicza.Iterate` op decoded object data only
and passed it to the handler. There is a need to iterate over all addresses
of the stored objects.

Add `DecodeAddresses` and `WithoutData` methods of `IteratePrm` type. Add
`Address` method to `IterationElement` type. Make `Iterate` to decode object
addresses if `DecodeAddress` was called and not read the data if
`WithoutData` was called. Implement `IterateAddresses` helper function to
simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
c96c455125 [#840] shard: Fix metabase Inhume parameters for tombstones
Replace `WithAddresses` with `WithTombstoneAddress` on `InhumePrm`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Leonard Lyubich
14329ab565 [#840] metabase: Distinguish objects with tombstones and GC marks
Each object from graveyard has tombstone or GC mark. If object has
tombstone, metabase should return `ErrAlreadyRemoved` on object requests.
This is the case when user clearly removed the object from container. GC
marks are used for physical removal which can appear even if object is still
presented in container (Control service, Policer job, etc.). In this case
metabase should return 404 error on object requests.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Leonard Lyubich
02e6df683a [#840] shard: Return only physically stored objects from List
`List` method of `Shard` must return only physically stored objects.

Use `AddPhyFilter` to select only phy objects.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Leonard Lyubich
504abdda06 [#789] shard: Exclude object.ErrAlreadyRemoved errors in refillMetabase
Tombstone and "alive" objects can be both stored in BlobStor. They can
appear during iterating in different order. Metabase returns
`ErrAlreadyRemoved` error if object is inhumed.

Ignore `object.ErrAlreadyRemoved` errors of `metabase.Put`in Shard's
`refillMetabase` operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
6bf7a00cfe [#789] shard: Add option to refill metabase on initialization
Add `WithRefillMetabase` option constructor which allows to set flag to
refill metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00