Commit graph

370 commits

Author SHA1 Message Date
Evgenii Stratonikov
cc377b34d2 [#868] blobstor: initialize (de-)compressors in Init
Do not log in options constructors. Also failure to
initialize compression module (possibly due to invalid options) is
certainly an error deserving proper treatment.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 16:01:45 +03:00
Leonard Lyubich
8a1593fdcc [#791] writecache: Export OpenDB function
The function will be useful for the tool which works with write-cache parts.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
ca4c9d4673 [#791] writecache: Implement iterator over small objects
There is a need to list addresses of the small objects stored in WriteCache
database.

Implement `IterateDB` function which accepts BoltDB instance and iterate
over all saved objects and passes their addresses to the hander.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
f73c5c2259 [#791] blobovnicza: Support read-only mode
There is a need to open Blobovnicza instances in read-only mode in some
cases.

Add `ReadOnly` option. Do not create dir path in RO. Open underlying BoltDB
instance with ReadOnly flag. Document thal all writing operations should not
be called in ro (otherwise BoltDB txs fail).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
cc5d3288a1 [#791] blobovnicza: Don't iterate over range buckets in Iterate
`Blobovnicza` can be initialized with any number of range buckets, and
reconstructed with different size limit. In previous implementation
`Iterate` could miss some stored objects if we construct `Blobovnicza` with
smaller number of ranges.

Make `Iterate` to traverse all buckets regardless of current instance
bounds.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
5e2ca0d04b [#791] blobovnicza: Implement method to iterate over object addresses
In previous implementation `Blobovnicza.Iterate` op decoded object data only
and passed it to the handler. There is a need to iterate over all addresses
of the stored objects.

Add `DecodeAddresses` and `WithoutData` methods of `IteratePrm` type. Add
`Address` method to `IterationElement` type. Make `Iterate` to decode object
addresses if `DecodeAddress` was called and not read the data if
`WithoutData` was called. Implement `IterateAddresses` helper function to
simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-05 10:51:10 +03:00
Leonard Lyubich
c96c455125 [#840] shard: Fix metabase Inhume parameters for tombstones
Replace `WithAddresses` with `WithTombstoneAddress` on `InhumePrm`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Leonard Lyubich
14329ab565 [#840] metabase: Distinguish objects with tombstones and GC marks
Each object from graveyard has tombstone or GC mark. If object has
tombstone, metabase should return `ErrAlreadyRemoved` on object requests.
This is the case when user clearly removed the object from container. GC
marks are used for physical removal which can appear even if object is still
presented in container (Control service, Policer job, etc.). In this case
metabase should return 404 error on object requests.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Leonard Lyubich
02e6df683a [#840] shard: Return only physically stored objects from List
`List` method of `Shard` must return only physically stored objects.

Use `AddPhyFilter` to select only phy objects.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 11:27:41 +03:00
Leonard Lyubich
504abdda06 [#789] shard: Exclude object.ErrAlreadyRemoved errors in refillMetabase
Tombstone and "alive" objects can be both stored in BlobStor. They can
appear during iterating in different order. Metabase returns
`ErrAlreadyRemoved` error if object is inhumed.

Ignore `object.ErrAlreadyRemoved` errors of `metabase.Put`in Shard's
`refillMetabase` operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
6bf7a00cfe [#789] shard: Add option to refill metabase on initialization
Add `WithRefillMetabase` option constructor which allows to set flag to
refill metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
b9c22e21b1 [#789] shard: Add method to refill the metabase
There is a need to refill Metabase data with the objects from BlobStor.

Implement `refillMetabase` method which iterates over all objects from
BlobStor and saves them in Metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
8d016d2529 [#789] blobstor: Implement iterator
There is a need to be able to process all objects saved in `BlobStor`.

Implement `BlobStor.Iterate` method which iterates over all objects.
Implement `IterateBinaryObjects` and `IterateObjects` helper functions to
simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
b618a44d69 [#789] blobovnicza: Implement iterator
There is a need to be able to process all stored objects saved in
`Blobovnicza`.

Implement `Blobovnicza.Iterate` method which iterates over all objects.
Implement `IterateObjects` helper function to simplify the code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
55c94a0152 [#789] metabase: Implement Reset method
In the previous implementation of the metabase, there was no possibility of
reinitializing the metabase: clearing information about existing objects and
bringing it back to its initial state. This operation can be useful in
cases when the stored metadata about objects has lost (or possibly lost)
relevance, and you need to generate data from scratch. Also at the
initialization stage, static resources of the base were not created -
container-independent buckets.

Make `Metabase.Init` method to allocate graveyard, container-size and
to-move-it buckets in underlying BoltDB instance. Implement `Metabase.Reset`
method: it works like `Init` but clean up all static buckets and removes
other ones. Due to the logical similarity, the methods share a single piece
of code.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
a1696a81b6 [#776] writecache: Limit size of used disk space
There is a need to limit disk space used by write-cache. It is almost
impossible to calculate the value exactly. It is proposed to estimate the
size of the cache by the number of objects stored in it.

Track amounts of objects saved in DB and FSTree separately. To do this,
`ObjectCounters` interface is defined. It is generalized to a store of
numbers that can be made persistent (new option `WithObjectCounters`). By
default DB number is calculated as key number in default bucket, and FS
number is set same to DB since it is currently hard to read the actual value
from `FSTree` instance. Each PUT/DELETE operation to DB or FS
increases/decreases corresponding counter. Before each PUT op an overflow
check is performed with the following formula for evaluating the occupied
space: `NumDB * MaxDBSize + NumFS * MaxFSSize`. If next PUT can cause
write-cache overflow, object is written to the main storage.

By default maximum write-cache size is set to 1GB.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:07:36 +03:00
Leonard Lyubich
3258d9c616 [#790] writecache: Log each writing operation
Call `storagelog.Write` in all places after the successful writing op.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Leonard Lyubich
4f73c00776 [#790] blobstor: Log each writing operation
Call `storagelog.Write` in all places after the successful writing op.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Leonard Lyubich
3cfb58aabd [#790] storage engine: Add common template of log messages
There is a need to keep track of each local storage change. Log messages are
the most convenient way to do it.

Implement function which writes log message about the completed writing
operation in storage engine.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Leonard Lyubich
3f293cb55e [#762] storage/shard: Try to read headers from write-cache
Shard should try to read object headers from write-cache if it is enabled.

Extend `writecache.Cache` interface with `Head` method. Call the method in
`Shard.Head` if `Shard.hasWriteCache` returns true.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:33:17 +03:00
Leonard Lyubich
aa2151fbeb [#762] write-cache: Implement Head operation
Write cache should be able to execute HEAD operations according to spec.

Add simple implementation of `Head` method through the `Get` one. Leave
notes for future optimization.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:33:17 +03:00
Pavel Karpy
c54f524df9 [#773] writecache: Delete unused dbSize param
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-27 15:05:53 +03:00
Pavel Karpy
4ff98a7e2b [#760] metabase: Support COMMON_PREFIX matchtype
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-27 15:03:15 +03:00
Alex Vanin
53f031e98c [#766] Fix misspell linter error
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 17:57:15 +03:00
Alex Vanin
30738549ef [#766] Fix makezero linter error
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 17:57:15 +03:00
Pavel Karpy
3d981320c6 [#745] gc: Stop GC work on Shard's Close
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-05 14:44:05 +03:00
Pavel Karpy
7a10d902be [#727] Use util.MkdirAllX instead of os.MkdirAll
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-30 16:42:22 +03:00
Leonard Lyubich
56d4410913 [#638] Update to Go 1.16
Changes:

  * replace `iotuil` elements with the ones from `os` package;
  * replace `os.Filemode` with `fs.FileMode`;
  * use `signal.NotifyContext` instead of `NewGracefulContext` (removed).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-29 13:44:59 +03:00
Pavel Karpy
48827f42d3 [#643] pkg: Sync method names and commentaries to them
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-24 16:10:44 +03:00
ZhangTao1596
9b87e6267d [#568] shard/writecache: Maintain memory size and db size
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2021-06-21 14:26:30 +03:00
Leonard Lyubich
cc81723b01 [#615] blobovnicza: Simplify byte size stringer
Make `stringifyByteSize` to return number of bytes as string.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-16 12:14:04 +03:00
Leonard Lyubich
d34de558f0 [#570] *: Use generator of test owner IDs from API Go lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 20:45:15 +03:00
Leonard Lyubich
3e1463cc76 [#570] *: Use generator of test container IDs from API Go lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 20:45:15 +03:00
Leonard Lyubich
3dd10b6795 [#570] *: Remove usage of deprecated elements from API Go library
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 20:45:15 +03:00
Evgenii Stratonikov
213bbcbf2b [#541] blobstor/fstree: fix a bug in Iterate()
Be able to recover address from the path. Also add tests.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-05-20 19:24:42 +03:00
Leonard Lyubich
1b51dcf8a4 [#546] engine/inhume: Fix incorrect Inhume behavior for root objects
If object to be inhumed is root we need to continue first traverse over the
shards. In case when several children are stored in different shards,
inhuming object in a single shard leads to appearance of inhumed object in
subsequent selections. Also, any object can be already inhumed, and this
case is equivalent to successful inhume.

Do not fail on `object.ErrAlreadyRemoved` error. Continue first iterating
over shards if we detected root object (`SplitInfoError`).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-20 17:39:38 +03:00
Leonard Lyubich
c5dae76c7d [#546] engine/inhume: Write unit tests
Write unit tests of `StorageEngine.Inhume` which assert that inhumed objects
don't appear in `Select` result.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-20 17:39:38 +03:00
Leonard Lyubich
d1d846cf4d [#546] engine/inhume: Fix incorrect reaction on successful traverse
In previous implementation storage engine false-negatively reacted to a
successful Inhume operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-20 17:39:38 +03:00
Evgenii Stratonikov
71b87155ef [#521] *: use stdlib errors package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-05-19 16:42:54 +03:00
Evgenii Stratonikov
59de521fd1 [#472] blobstor: implement write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-04-29 17:36:36 +03:00
Evgenii Stratonikov
96a8ee7c83 [#472] blobstor: allow to put raw objects
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-04-29 17:36:36 +03:00
Evgenii Stratonikov
4a1ca4ecc1 [#443] metabase: speedup encodelist
GOB appears to be almost twice as slow as this implementation.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-04-29 17:36:36 +03:00
Evgenii Stratonikov
62b8958177 [#472] blobstor: check for blobovnicza without mutex first
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-04-29 17:36:36 +03:00
Evgenii Stratonikov
828c8695c1 [#472] blobstor: use DB.Batch() for metabase and blobstor
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-04-29 17:36:36 +03:00
Evgenii Stratonikov
934e394e28 [#472] blobstor: move fsTree to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-04-29 17:36:36 +03:00
Evgenii Stratonikov
de74fcc38f [#481] blobstor: use simplelru.LRU cache
Evicting from cache requires closing blobovnicza which
in turn needs to lock `activeMtx`. This lock is not needed on
every addition, but our LRU library doesn't return evicted keys.
In future we may consider switching to other implementation.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-04-21 14:41:34 +03:00
Alex Vanin
9a961e21b1 [#477] metabase: Move MergeSplitInfo to storage/util pkg
This function already reused in different storage engine parts
so it makes sense to keep it in separate package.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-15 11:18:39 +03:00
Alex Vanin
bc09e29bfd [#477] engine: Reuse SplitInfoError fix for engine.Range
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-15 11:18:39 +03:00
Alex Vanin
7e435db292 [#477] engine: Reuse SplitInfoError fix for engine.Get
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-15 11:18:39 +03:00
Alex Vanin
7547592ce3 [#477] engine: Iterate over all shards to return complete SplitInfoError
Different SplitInfo parts may be stored in different shards. Storage
engine must not stop at first SplitInfoError and should make
best effort to complete SplitInfo structure if needed.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-15 11:18:39 +03:00
Alex Vanin
e020fe5597 [#477] engine: Add test for missing link issue
There were no unit tests of storage engine. This commit
adds first test to reproduce missing link ID in split info
at `engine.Head(raw)` request.

Engine tests uses some constructors from metabase tests,
so it is better to locate such functions in common
package at local_object_storage.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-15 11:18:39 +03:00
Alex Vanin
2716000b08 [#477] engine: Make generateShardID function independent
This function does not use any engine internal states.
This way it can be reused in tests.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-15 11:18:39 +03:00
Leonard Lyubich
df909984bd [#461] metabase: Delete graves regardless of the presence of objects
`Inhume` operation can be performed on already deleted objects, and in this
case the entry will be added to the graveyard. `Delete` operation finishes
with error if object is not presented in metabase. However, the entry in the
cemetery must be deleted regardless of the presence of the object.

Additionally, now `Delete` does not return an error in the absence of an
object.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-04-06 17:34:01 +03:00
Leonard Lyubich
8d5c17facd [#438] metabase: Fix saving of object payload in Put
Metabase should not store payloads of objects. Make Put operation to cut
object payload before saving binary object in metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-22 21:40:27 +03:00
Leonard Lyubich
e3f8470d7a [#438] metabase: Write unit tests for checking the absence of payloads
Metabase should not store payloads of objects. Set payload in generated test
object. Ascertain that objects returned by Get method have no payload.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-22 21:40:27 +03:00
Leonard Lyubich
b62a2a0f54 [#425] blobstor: Remove hard-code used for Blobovnicza test
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Alex Vanin
b8e10571c6 [#426] Put prometheus behind pkg/metrics
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
980b774af2 [#426] engine: Support duration metrics
With `enable metrics` option, engine will collect
durations for all public methods.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Leonard Lyubich
35073fb61b [#217] storage engine: Add GC marking to Inhume operation
Add `InhumePrm.MarkAsGarbage` method which marks passed objects to be
removed from local storage. Update `InhumePrm.WithTarget` doc to prevent
conflicting use with the new method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
7a4e3efa95 [#217] storage engine: Transform Inhume to batch operation
Make `StorageEngine.Inhume` to process list of objects per single operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
3ed0065455 [#395] metabase: Do not add tombstone-on-tombstone records to graveyard
Creating tombstones for tombstones is prohibited in NeoFS system. Metabase
graveyard contains records of the form {address: address}: key is an address
of inhumed object, value is an address of the tombstone. To prevent creation
tombstones for tombstones metabase must control incoming Inhume calls:

 * if Inhume target is a tombstone, then "grave" should not be added;
 * if {a1:a2} "grave" was created earlier and {a2: a3} "grave" came later,
   then first "grave" must be removed as tomb-on-tomb.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 16:19:26 +03:00
Leonard Lyubich
bc5e04f502 [#378] metabase: Calculate expiration once for each fake bucket
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
a6d5556dc1 [#378] shard/gc: Reuse addressFromKey func in all appropriate places
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
72f7df3339 [#378] shard/gc: Limit the accumulated batch of objects to be deleted
Change Shard's garbage remover to interrupt iterating over the metabase
graveyard when the buffer is full to the max size (`WithRemoverBatchSize`
Shard's option).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
f5f416268f [#378] metabase: Add the ability to break the loop in IterateOverGraveyard
Make `DB.IterateOverGraveyard` to immediately return nil if GraveHandler
returns ErrInterruptIterator.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
ff442e9559 [#378] shard/gc: Avoid usage of the range-loop vars in go-routines
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
717f2beb47 [#378] shard: Collect expired tombstones in GC every epoch
Add new epoch event handler to GC that finds all expired tombstones and
marks them and underlying objects to be removed. Shard uses callbacks
provided by the storage engine to mark underlying objects.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
3d5169c4c9 [#378] metabase: Implement iterator over the objects under tombstones
Implement `DB.IterateCoveredByTombstones` method that iterates over graves
and handles all objects under one of the tombstones.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
dab45050b9 [#378] shard: Collect expired non-tombstone objects in GC every epoch
Add new epoch event handler to GC that finds all expired non-tombstone
objects and marks them to be removed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
89a22450e5 [#378] shard: Define new epoch event
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
182df23859 [#378] metabase: Implement iterator over expired objects
Implement `DB.IterateExpired` method that iterates over the objects in
metabase that are expired at particular epoch.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
a9a1acc880 [#378] shard: Control the completion of all handlers of the previous event
Group handlers of the particular event to a WaitGroup and wait for it before
the next event handling. This will ensure that all handlers complete and
prevent potential conflicts between past and present jobs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
4da41613c3 [#378] shard: Initialize GC processes on Init
`Shard.Init` method creates a new GC instance from shard configuration and
starts GC's workers through `init` call. In initial implementation GC
routines are indefinite and can be killed only with by application shutdown.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
13087dc3dd [#378] shard: Implement skeleton of internal GC
Shard's GC component consists of:
 * asynchronous remover that periodically wake up and removes all garbage
   objects from the shard, and goes to sleep for particular time interval;
 * external event listener that distributes jobs between workers;
 * group of workers that can handle a single job related to particular
   external event.

Remover and event listener represents go-routines which are started by
`init` method (calls from `Shard.Init`). In initial version all event
handlers are interrupted: this means that next event of the same type will
interrupt previous handling and start the new one.

GC is fully encapsulated in Shard. All GC configurations are reflected in
Shard's configuration.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
d72d37a322 [#378] metabase: Implement iterator over metabase graveyard
Implement `DB.IterateOverGraveyard` method that iterates over all graves and
passes passes their descriptors (new type `Grave`) to handler (new type
`GraveHandler`). `Grave` currently have buried object address and garbage
flag.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
3d7d074af4 [#377] shard: Support batch Inhume operation
Replace single target address in `InhumePrm` with the list of addresses.
Change corresponding parameter in `WithTarget` and `MarkAsGarbage` methods
to variadic.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
0d6d195d0d [#377] metabase: Support batch Inhume operation
Replace single target address in `InhumePrm` with the list of addresses.
Rename `WithAddress` method to `WithAddresses` and change parameter to
variadic.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
6ec7433e14 [#377] storage engine: Change the implementation of Delete operation
Make `StorageEngine.Delete` to execute `Inhume` operation with
`MarkAsGarbage` parameter on the `Shard` that holds the object. Searching of
the particular shard is performed through iterating over HRW-sorted shards.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
d3a0079d1d [#377] shard: Add Inhume parameter to mark the object as garbage
Implement `InhumePrm.MarkAsGarbage` method that leads to marking object as
garbage in metabase. Update `InhumePrm.WithTarget` doc indicating a conflict
with the new method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
30ac234c20 [#377] metabase: Add Inhume parameter to mark the object as garbage
Implement `InhumePrm.WithGCMark` method that marks the object as garbage in
graveyard. Update `InhumePrm.WithTombstoneAddress` doc indicating a conflict
with the new method. Update `Inhume` function doc about tombstone address
parameter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
7c3f828893 [#219] metabase: Count parent references in Delete operation
Delete operation of Metabase is performed on group of objects. The set being
removed can contain descendants of a common parent. In the case when all
descendants of a parent object are deleted, it must also be deleted from
the metabase. In the previous implementation, this was not done due to the
chosen approach to counting references to the parent.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-18 09:54:19 +03:00
Leonard Lyubich
a2c2241356 [#379] storage engine: Inhume object in one shard
In previous implementation StorageEngine.Inhume operation forced Shard
.Inhume call on all internal shards. There is a need to inhume object in a
single shard. To achieve this, Inhume operation is performed in next steps:

 1. iterate over sorted shards, check object presence through Exists call;
 2. if object exists at any shard in step 1 => inhume it and return on
    success;
 3. if no shards contain the object => iterate over sorted shards again and
    try to inhume the object at first possible shard;
 4. if all Inhume calls are failed => return an error.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-15 18:58:18 +03:00
Leonard Lyubich
ffbf6b922f [#361] metabase: Support new match types
Support STRING_NOT_EQUAL and NOT_PRESENT match types.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-03 10:32:15 +03:00
Alex Vanin
de1f601765 [#337] engine: Add container listing for both engine and shard
Container listing already supported in the metabase for `engine.List`
operation. To get container statistics engine should provide both the
option to get container volume estimation and list of all containers.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:04:30 +03:00
Alex Vanin
3926e76c32 [#337] engine: Get all container sizes estimation from shards
Objects of one container can be split among shards, so engine
should iterate over all available shards to sum all size
estimations.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:04:30 +03:00
Alex Vanin
dc5a481f17 [#337] shard: Get container size estimation from metabase
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:04:30 +03:00
Alex Vanin
41578001e4 [#337] metabase: Keep container size estimation
Storage nodes keep container size estimation so they
can announce this info and hope for some basic income
settlements. This is also useful for monitoring.

Container size does not include non regular or inhumed
object sizes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:04:30 +03:00
Alex Vanin
e03d906cb7 [#334] engine: Make tests more predictable
There is a codecov issue because objects are not placed
in the engine the same way every unit test. Therefore
sometimes there are more coverage, sometimes there are
less. Seeded RNG should solve this issue for engine tests.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-21 11:00:23 +03:00
Leonard Lyubich
a9c1f88b8a [#323] storage/blobovnicza: Add Timeout option of BoltDB instances
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
06085ba8be [#323] storage/shard: Fix message of the error returned by Open method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
1ab630eb24 [#323] storage/shard: Fix message of the error returned by Init method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
b5a1c03fb9 [#323] storage/engine: Fix message of the error returned by Open method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Alex Vanin
1d56e60589 [#302] Remove unused FSBucket component
FSBucket became obsolete when storage object engine has
been implemented.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
e88c1f750d [#302] Fix golint else/return linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
5e231c515a [#302] Fix whitespace linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Leonard Lyubich
9a86fff7e0 [#291] Remove some unused code from repository
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
Leonard Lyubich
edef9463d7 [#253] metabase: Add prefix to Graveyard and ToMoveIt bucket names
In previous implementation DB.Containers method could return an error about
invalid container ID string format. This could happen if some of top-level
buckets had name w/o "_" substring.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-17 12:03:18 +03:00
Alex Vanin
e6f2d84736 [#246] blobovnivza: Fix deadlock on concurrent evict and open new blobovnicza
Deadlock occurs when `getActivate` function opens new blobovnicza and that
invokes evict in LRU cache of open blobovniczas. `getActivate` makes
`activeMtx.Lock()` and then cache evict makes `activeMtx.RLock()` and deadlock
happens.

Fix contains two steps:
- add separate mutex to open blobovniczas (1),
- split single Lock outside of `updateAndGet` (2).

As for the (1) `bbolt.Open()` locks when it tries to open the same file from
two threads. So separate mutex will prevent that.

As for the (2) `updateAndGet` function contains from two parts. At first it
checks if required blobovnicza is ready and it returns it. In this case we can
use the simple RLock. But then there is an option when we should open new
blobovnicza and update map of active blobovniczas.

In this case we call `openBlobovnicza` without activeMtx lock. Cache evict
happens there and it won't cause deadlock.

Then we lock activeMtx to update the map of active blobovniczas. Concurrency can
happen there. However `openBlobovnicza` will not open the same blobovnicza twice,
so we can make one more check if opened blobovnicza was activated while thread was
locked in activeMtx. If so, then return active blobovnicza, else finish activation.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-14 13:37:45 +03:00
Leonard Lyubich
91bea44a1a [#245] blobovnicza: Fix initializing dimensional buckets
In previous implementation Blobovnicza could incorrectly initialize
dimensional buckets: if SmallSizeLimit = 2 ^ X + Y && Y < 2 ^ X, then
largest dimensional bucket was [2 ^ (X - 1) : 2 ^ X]. This was caused by an
incorrect condition for stopping the iterator along the dimensional
boundaries.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-14 12:48:59 +03:00
Leonard Lyubich
4f5d5c7e45 Write system path to BoltDB on blobovnicza closing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
63bc3aab82 [#242] engine: Do not iterate over shards if cid is not set
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
511b5f782a [#242] engine: Support ContainerID argument in shard selects
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
f585772746 [#242] shard: Support ContainerID argument in metabase selects
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
b97f818323 [#242] metabase: Make ContainerID a mandatory parameter for Select
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
9dd83bdf0d [#235] blobstor: Return object.ErrRangeOutOfBounds from shallow dir
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
41b9fa5b45 [#235] engine: Return OutOfBounds error
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1d23483828 [#235] services/object: Implement new GetRange algorithm
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
91d8e0a4de [#238] engine: Support raw flag in Head method
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
da42ee3a26 [#238] engine: Stop shard iteration at split info error in Head
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
69dee6ec6d [#238] shard: Support raw flag in Head method
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
bf7a390304 [#238] metabase: Support raw flag in Get method
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
c267a08886 [#236] engine: Stop shard iteration at split info error in GetRange
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
590745204c [#237] metabase: Structure parameters and results of all operations
All parameters and resulting values of all metabase operations are
structured in new types. The most popular scenarios for using operations are
moved to auxiliary functions.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
a875d80491 Remove outdated code of metabase and localstore
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
26f03c6301 [#232] Fix SplitInfo update in existed virtual object
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
ee1e3931a5 [#231] engine: Add List method
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
a7087faa1a [#231] shard: Add List method to return all available objects
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
a1cb48f800 [#231] metabase: Add container lister
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
3a37eda410 [#230] engine: Stop on first SplitInfoError in Get method
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
a488d202e1 [#230] engine: Rename WithAddress to WithAddresses for delete operations
Delete works with multiple addresses at once so it is convenient to
show it in parameter setter function.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
7c97bd1360 [#230] shard: Add test for getting SplitInfo
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
bf9e938a3b [#230] metabase: Return SplitInfoError in Exist method
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
e2de95e3f6 [#230] metabase: Index split info structures
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
eaae5a5dd7 [#229] blobovnicza: Store objects in a binary format
In previous implementation Blobovnicza's stored objects in protocol format
which did not allow working with externally compressed objects. To achieve
this goal, operations Get and Put no longer work with the structure of the
object, but only with abstract binary data. Operation GetRange has become
incorrect in its original purpose to receive the payload range. In this
regard, BlobStor receives the payload range of the object through Get
operation. In the future either Blobovnicza will learn to compress objects
by itself, or the GetRange operation will be eliminated.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
3d77fdb347 [#222] Set write cache specific config on application level
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
af2719689d [#222] Do not compress data in blobovniczas
Blobovnicza returns object, so we can't put compressed
data there. Compressed data won't be deserialized correctly.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
6bec50963b [#222] Add tests for shards
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
ce92df3543 [#222] Return nil in metabase.IsSmall if object is big
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
862f9a3f1a [#222] Put small objects in blobovnicza in blobStor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
21708d5408 [#222] Add Exists method in storage engine
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
f470022594 [#222] Update Select method in storage engine
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
25486194b9 [#222] Update Head method in storage engine
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
67d4b0db12 [#222] Move ErrAlreadyRemoved to core package
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
6a3e3e5642 [#222] Update Range method in storage engine
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
c9b1c6603e [#222] Update Get method in storage engine
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
f3e96f8c19 [#222] Update Delete method in storage engine
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
aa8dd18b07 [#222] Update Put method in storage engine
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
f45b7a048d [#222] Rename WithTreeRoot.. to WithRoot.. in blobstor opts
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
d4e48b28b1 [#222] Add Inhume method to storage engine
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
92c95150cd [#222] Use write cache options in storage app
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
69ba295077 [#222] Add ToMoveIt method in shard
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
ccd8760fbc [#222] Update Exists method in shard
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
97a2e26b71 [#222] Add Head method in shard
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
244e244c0e [#222] Add GetRange method in shard
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
73505fe582 [#222] Update Get method in shard
Now `Get`, `Head` and `GetRange` will be
separate methods on shard.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
31371771e8 [#222] Update Delete method in shard
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
5fc54386b0 [#222] Update Put method in shard
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
72f8ca1afd [#222] Add Inhume method for shard
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
1b76458684 [#222] Add writeCache instance to shard
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
cbdcfb9159 [#220] shard: Enable write cache through option
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00