Evgenii Stratonikov
3d882e9f47
[ #1806 ] engine: Allow to flush write-cache
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
0b4c867ef1
[ #1806 ] writecache: Allow to start flush manually
...
Allow user to initiate flushing objects from a writecache.
We need this in 2 cases:
1. During writecache storage schema update, it should be flushed with
the old version of node and started clean with a new one.
2. During SSD replacement, to avoid data loss.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
55148404ee
[ #1806 ] writecache: Use interfaces for blobstor and metabase
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
6f45cc81fc
[ #1764 ] neofs-node: Use constants for storage types
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-27 08:57:20 +03:00
Evgenii Stratonikov
a49137349b
[ #1731 ] engine: Allow to use user handler for evacuated objects
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-24 13:47:48 +03:00
Evgenii Stratonikov
7cc68cf4d4
[ #1810 ] blobovniczatree: Add a test for blobovnicva usage
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-23 10:19:26 +04:00
Evgenii Stratonikov
3df98ce7ba
[ #1731 ] engine: Return the amount of actually moved objects in Evacuate
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
a51b76056e
[ #1731 ] engine: Add Evacuate command
...
Make it possible to move all data from 1 shard to other shards.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
7377979e12
[ #1731 ] engine: Move single shard PUT to a separate function
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
5321f8ef9c
[ #1786 ] engine: Unify parameter setters
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov
b064fb24d8
[ #1616 ] engine: Do not use batches in delete
...
Use a simple loop at the callsite. This way we remove as much as we can.
Also, `Delete` metrics is more meaningful now.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov
ae1dab29bc
[ #1482 ] metabase: Encode database keys in binary
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:26 +03:00
Pavel Karpy
532c41ca05
[ #1658 ] meta: Do not check object expiration in counters
...
Do not perform operations that produce unused results in `syncCounter`.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
431e331373
[ #1658 ] shard: Update metric counters
...
Use meta's operation results to change the metrics. Support typed object
counters.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
ad47e2a985
[ #1658 ] meta: Add logic counter
...
- Meta now supports (and requires) inc/dec labeled counters
- The new logic counter is incremented on `Put` operations and is
decremented on `Inhume` and `Delete` operations that are performed on
_stored_ objects only
- Allow force counters sync. "Force" mode should be used on metabase resync
and should not be used on a regular meta start
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
d872862710
[ #1658 ] meta: Force counters resync process
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
beb7f2a048
[ #1634 ] node: Change default epoch in tests
...
Do not treat objects with expiration as expired by default.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
edef26a4fd
[ #1658 ] engine: Update metrics interfaces
...
It now supports typed object counter metrics.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
4afb928ab6
[ #1674 ] *: Expire entities after the expiration epoch
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-12 19:56:59 +04:00
Evgenii Stratonikov
60b2930417
[ #1778 ] neofs-node: Fix blobovnicza config type
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 17:06:00 +03:00
Evgenii Stratonikov
87bd49563e
[ #1761 ] pilorama: Use batch size of 1 for tests
...
Improve tests speed by a lot and use more iterations in `ApplyRandom` test.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Evgenii Stratonikov
a2bb3a2a96
[ #1630 ] pilorama: Support dropping trees
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Leonard Lyubich
8e8265d5ea
[ #1707 ] blobovnicza/get: Iterate over all buckets regardless of config
...
Re-configuration of Blobovnicza's object size limit must not affect
already stored objects. In previous implementation `Blobovnicza` didn't
see objects stored in buckets which became too big after size limit
re-configuration. For example, lets consider 1st configuration with 64KB
size limit for stored objects. Lets assume that we stored object of 64KB
size, and re-configured `Blobovnicza` with 32KB limit. After reboot
object should be still available, but actually it isn't. This is caused
by `Get` operation algorithm which iterates over configured size ranges
only, and doesn't process any other existing size bucket. By the way,
increasing of the object size limit didn't lead to the problem even in
previous implementation.
Make `Blobovnicza.Get` method to iterate over all size buckets
regardless of current configuration. This covers the described scenario.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-08 12:24:28 +04:00
Evgenii Stratonikov
e6cf0b3d42
[ #1745 ] writecache: Set flush parameters based on max size
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
c04126f35e
[ #1745 ] writecache: Remove IsErrNotFound
...
We specify the error in the doc-comment, and it is the same for all our
components.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
9e41e85295
[ #1745 ] writecache: Actualize docs
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
e9c6ee2623
[ #1745 ] writecache: Rename constants for default values
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
177e8e01b1
[ #1745 ] neofs-node: Remove memcache_capacity
from the configuration
...
It is unused since ddaed283e9
.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
bda084f331
[ #1745 ] writecache: Simplify object counters
...
Remove unused option and additional pointers to db/fstree.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
20abdaeed4
[ #1745 ] writecache: Set flush mark only on success
...
Set flush mark in the inside the flush worker because writing to the blobstor
can fail. Because each evicted object must be deleted, it is reasonable
to do this in the evict callback.
The evict callback is protected by LRU mutex and thus potentially interferes
with `Get` and `Iterate` methods. This problem will be addressed in the
future.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
d8a00c365a
[ #1691 ] blobovniczatree: Fix active blobovnicza caching
...
Maintain an invariant that any blobovnicza is present either
in `opened` or in `active` map. Otherwise, the logic becomes too
complicate because it is not obvious when we should close the blobovnicza.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-31 19:11:48 +03:00
Evgenii Stratonikov
dfac4e1c0b
[ #1691 ] blobovniczatree: Do not fill cache during initialization
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-31 19:11:48 +03:00
Evgenii Stratonikov
ae52d53609
[ #1698 ] pilorama: Add a test for the empty FileName
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 16:52:51 +03:00
Evgenii Stratonikov
78bf17c0b0
[ #1686 ] blobstor/*: Remove unneeded tests
...
The blobovniczatree test also checks the amount of objects we can put,
so leave it here.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
b9a2055e1c
[ #1686 ] blobstor/*: Return ErrReadOnly for modifying operations
...
This check should occur on the shard level, but because
blobstor components expose `Open(readOnly bool)` interface,
it is reasonable to expect an error here.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
6c2d3b020f
[ #1686 ] fstree: Return proper error for concurrent Delete
...
If the file doesn't exist, return `apistatus.ObjectNotFound`.
First check is still there as a shortcut.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
0b95a21701
[ #1686 ] blobstor: Add generic tests
...
This tests check that each blobstor component behaves similarly when
same methods are being used. It is intended to serve as a specification
for all future components.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
b2d4cc556e
[ #1686 ] blobovnicza: Remove IterateObjects
...
It is used only once, makes sense to inline.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
5f95498364
[ #1686 ] blobstor: Add generic tests
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
9792e6a4ef
[ #1686 ] metabase: Add generic tests
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
482a7e7f2f
[ #1686 ] pilorama: Add generic tests
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
b8b9d25f9d
[ #1686 ] local_object_storage: Add generic tests
...
Use them for writecache as a simple example.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
bc20851cd1
[ #1686 ] shard: Open all components in case metabase has failed
...
All components must be opened first, before any other operation is
performed.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Pavel Karpy
c7c1c257e1
[ #1658 ] node: Init object counter on meta's Init
...
Includes:
1. Renaming counter key to distinguish logical and physical objects
2. Version update dropping since changes could be done in a compatible way
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
9c7b3ce799
[ #1658 ] node: Read metrics from meta on startup
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
745f72fff0
[ #1658 ] node: Support object counter metrics
...
Increment shard's object counter on successful `Put` calls and decrement on
`Delete`.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
626766db08
[ #1658 ] meta: Add object counter
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
30341f2192
[ #1687 ] *: Perform go fmt
using go v1.19
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Evgenii Stratonikov
e8c6dce466
[ #1523 ] blobstor: Rename compression.CConfig
to Config
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
0042c1d4a6
[ #1584 ] blobovniczatree: move public methods to different files
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
26b4a258e0
[ #1523 ] neofs-node: Refactor configuration
...
1. Move compression parameters to the `shard` section.
2. Allow to use multiple sub-storage components in the blobstor.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
13cdbde2e2
[ #1523 ] blobstor: Remove unused types
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
0c9e4e6a35
[ #1523 ] blobstor: Unify request dispatch logic
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
266458fe5c
[ #1523 ] blobstor: Unify fstree and blobovnicza interfaces
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
5aa3defc67
[ #1523 ] writecache: Remove cloneBytes
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
ddaed283e9
[ #1523 ] writecache: Simplify logic
...
1. Remove in-memory cache. It doesn't persist objects and if we want
more speed, `NoSync` option can be used for the bolt DB.
2. Put to the metabase in a synchronous fashion. This considerably
simplifies overall logic and plays nicely with the metabase bolt DB
batch settings.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
4176b2a1bc
[ #1523 ] local_object_storage: Unify parameters for the Iterate
operation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
73f8bb3e5f
[ #1523 ] shard: Store generic storage ID in metabase
...
Allow to extend blobstor with more storage sub-systems. Currently
objects stored in the FSTree have empty byte slice descriptor and object
from blobovnicza tree have the same id as earlier. Each such change in
the identifier formation should be accompanied with metabase version
increase.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
9eb018672c
[ #1523 ] blobstor: Unify parameters for Put
operation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
7d6df543d7
[ #1523 ] blobstor: Export single Get
, GetRange
and Delete
methods
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
ca15083a50
[ #1523 ] local_object_storage: Unify parameters for the Exists
operation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
d75d030a90
[ #1523 ] local_object_storage: Unify parameters for the Delete
operation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
36c88f0dc8
[ #1523 ] local_object_storage: Unify parameters for the GetRange
operation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
d9a2f280c9
[ #1523 ] local_object_storage: Unify parameters for the Get
operation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
b621f5983a
[ #1523 ] local_object_storage: Move blobovnicza tree to a separate package
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
4e361e2ab4
[ #1647 ] writecache: Open DB in sync mode
...
Close #1647 .
Initially, `Sync: false` was provided because we can already lose
objects cached in memory. However, future changes in writecache will
remove inmemory cache and speed up it via other means.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-16 15:27:18 +04:00
Pavel Karpy
da2975a2f9
[ #1664 ] write-cache: Fix panic on Delete
operation
...
If an object is found in the Write-cache and is placed at the end of
the in-memory cache, the memory counter update operation tries to
dereference the index that is out of the sliced array. Moreover, even if
panic does not appear, the counter is updated with the wrong value.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-09 20:18:26 +03:00
Pavel Karpy
a5da665e69
[ #1656 ] meta: Fix MatchCommonPrefix
operation
...
Return all the objects on the empty common prefix search without search
optimizations that breaks boltDB logic.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:32:00 +03:00
Pavel Karpy
156ba85326
[ #1634 ] node: Do not return expired objects
...
If an object has not been marked for removal by the GC in the current epoch
yet but has already expired, respond with `ErrObjectNotFound` api status.
Also, optimize shard iteration: a node must stop any iteration if the object
is found but gonna be removed soon.
All the checks are performed by the Metabase.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
Pavel Karpy
9aba0ba512
[ #1634 ] meta: Add epoch state
...
It allows performing expiration checks on the stored objects.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
Evgenii Stratonikov
8ffc2fdf5e
[ #1642 ] engine: Do not increase error counter if the pilorama is disabled
...
After a4adb79db
new logical error could be returned. Do not increase
error counter in this case.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-30 18:39:22 +03:00
Evgenii Stratonikov
8d0884e74f
[ #1644 ] storagelog: Fix doc comment
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-30 18:39:08 +03:00
Evgenii Stratonikov
4abe5a7245
shard: add more checks for GetRange
parameters
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 11:22:04 +03:00
Evgenii Stratonikov
72586f17d4
shard: fix GetRange
for objects stored in the write-cache
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 11:22:04 +03:00
Evgenii Stratonikov
1691364653
[ #1559 ] local_object_storage: Fix tests and some data races
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
50e28f22f9
[ #1559 ] shard: Change Degraded
mode string representation
...
It is a flag, but is a `degraded-read-write` mode for a user.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Leonard Lyubich
fabe717d32
[ #1549 ] shard: Turn to read-only mode on metabase failure
...
If metabase can't be opened in the default mode, try opening shard
first in `ReadOnly` mode and then in `DegradedReadOnly`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
4944490ffb
[ #1559 ] local_object_storage: Move shard to the DegradedReadOnly
mode
...
`Degraded` mode can be set by the administrator if needed.
Modifying operations in this mode can lead node into an inconsistent state
because metabase checks such as lock checking are not performed.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
3df62769c0
[ #1559 ] local_object_storage: Allow to set mode for all components
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
1e786233bf
[ #1559 ] local_object_storage: Provide readOnly flag to Open
...
We should be able to reopen storage in readonly in runtime.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Leonard Lyubich
e38b0aa4ba
[ #1549 ] engine: Disable shard on blobovnicza init failure
...
There is a need to support working w/o shard if it has problems with
blobovnicza tree.
Make `BlobStor.Init` to return new `ErrInitBlobovniczas` error. Remove
shard from storage engine's shard set if it returned this error from
`Init` call. So if some of the shards (but not all) return this error,
the node will be able to continue working without them.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
40a56c6b42
[ #1559 ] engine: Do not count logical errors as storage ones
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
c8911d72d0
[ #1559 ] shard: Do not consult metabase in a degraded mode
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
339864b720
[ #1559 ] local_object_storage: Move shard.Mode
to a separate package
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
d8ba954aff
[ #1559 ] shard: Use Set
prefix for parameter setting
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
7b882b26d8
[ #1559 ] shard: Remove public functions
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
34d319fed2
[ #1559 ] metabase: Use Set
prefix for parameter setting
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
f58234aa2f
[ #1559 ] metabase: Remove public functions
...
Reduce public interface of this package. Later each result will contain
an additional status, so it makes more sense to use the same functions
and result processing everywhere.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
a4adb79db7
[ #1607 ] pilorama: Enable tree service explicitly
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
d62723f038
[ #1505 ] pilorama: Provide timeout to bbolt.Open
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
26041f18bf
[ #1505 ] pilorama: Allow to customize database parameters
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
c7b10598f9
[ #1333 ] engine: Increase error counter for pilorama errors
...
1. Modifying operations are not expected to fail, unless the shard is
read-only.
2. `Get*` operations should increase error counter too, unless the
error is `ErrTreeNotFound`.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
5e843a73f9
[ #1333 ] services/control: Return pilorama info in ListShards
RPC
...
Do not return backend type from the service for now, because memory
backend is expected to vanish.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8f4ee1aded
[ #1333 ] local_object_storage: Support ReadOnly mode in pilorama
...
The tricky part here is the engine itself: we stop iteration on
`ErrReadOnly` because it is better to synchronize the shard later than
to have partial trees stored in 2 shards.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
735931c842
[ #1481 ] pilorama: Fix TreeApply
...
Current implementation prevents invalid operations to become valid at
some later point (consider adding a child to the non-existent parent and
then adding the parent). This seems to diverge from the paper algorithm
and complicates implementation. Make it simpler.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad3038d16d
[ #1444 ] pilorama: Fix TreeMove
in bbolt backend
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8027b7bb6b
[ #1444 ] pilorama: Optimize internal encoding/decoding
...
```
name old time/op new time/op delta
ApplySequential/bbolt-8 55.5µs ± 4% 55.5µs ± 3% ~ (p=1.000 n=10+7)
ApplyReorderLast/bbolt-8 108µs ± 6% 112µs ± 8% ~ (p=0.077 n=9+9)
name old alloc/op new alloc/op delta
ApplySequential/bbolt-8 28.8kB ± 3% 27.7kB ± 6% -3.79% (p=0.005 n=10+10)
ApplyReorderLast/bbolt-8 41.4kB ± 5% 38.9kB ± 5% -6.19% (p=0.001 n=10+9)
name old allocs/op new allocs/op delta
ApplySequential/bbolt-8 262 ± 2% 235 ±10% -10.41% (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8 684 ± 6% 616 ± 7% -10.04% (p=0.000 n=10+9)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
4437cd7113
[ #1442 ] pilorama: Generate timestamp based on node position in the container
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3312924b82
[ #1431 ] pilorama: Use Batch
for write transactions
...
Helps a lot in case of concurrent request flow.
```
name old time/op new time/op delta
ApplySequential/bbolt-8 78.0µs ± 9% 59.8µs ± 4% -23.39% (p=0.000 n=10+9)
ApplyReorderLast/bbolt-8 143µs ± 5% 113µs ±15% -21.06% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ApplySequential/bbolt-8 56.9kB ± 8% 28.9kB ± 3% -49.22% (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8 87.3kB ± 3% 40.9kB ±10% -53.16% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ApplySequential/bbolt-8 224 ±11% 262 ± 5% +16.93% (p=0.000 n=9+10)
ApplyReorderLast/bbolt-8 518 ± 4% 674 ±11% +30.09% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f0a67f948d
[ #1431 ] pilorama: Cache attributes in the index
...
Currently to find a node by path we iterate over all the children on
each level. This is far from optimal and scales badly with the number of
nodes on a single level. Thus we introduce "indexed attributes" for
which an additional information is stored and which can be use in
`*ByPath` operations. Currently this set only includes `FileName`
attribute but this may change in future.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f5d35571d0
[ #1431 ] engine: Add benchmark for Select
vs TreeGetByPath
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
835170e452
[ #1329 ] pilorama: Allow to benchmark all tree backends
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
536857ea5a
[ #1329 ] services/tree: Implement GetOpLog
RPC
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
7703dd5d7f
[ #1419 ] pilorama: Create new nodes in path if needed
...
Consider a node `{FileName: "dir", Attribute: "xxx"}`. In case we add
a new node by path `["dir", "file.txt"]`, create a new intermediate node
with a single attribute.
`GetByPath` now also considers only nodes with a single attribute while building a path.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad48918a97
[ #1406 ] pilorama: Return parent from TreeGetMeta
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8844d9b2db
[ #1344 ] pilorama: Document errors for Get* methods
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
d8ad68d613
[ #1344 ] engine: Log errors in Tree* operations
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
910db42748
[ #1344 ] pilorama: Use require.ErrorIs
in tests
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
aea855e8f3
[ #1326 ] services/tree: Implement GetSubTree RPC
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
46f4ce2773
[ #1324 ] engine: Implement Forest interface for storage engine
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8cf71b7f1c
[ #1324 ] local_object_storage: Implement tree service backend
...
In this commit we implement algorithm for CRDT trees from
https://martin.klepmann.com/papers/move-op.pdf
Each tree is identified by the ID of a container it belongs to
and the tree name itself. Essentially, it is a sequence of operations
which should be applied in chronological order to get a usual tree
representation.
There are 2 backends for now: bbolt database and in-memory.
In-memory backend is here for debugging and will eventually act
as a memory-cache for the on-disk database.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
11c9df6f00
[ #1611 ] shard: Print shard ID in component logs
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 15:57:53 +03:00
Pavel Karpy
9f7a22e2aa
[ #1561 ] object: Return OUT_OF_RANGE
status
...
Replace `ErrRangeOutOfBounds` error from `pkg/core/object` package with
`ObjectOutOfRange` from `apistatus` package. That error is returned by
storage node's server as NeoFS API statuses.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-19 13:29:45 +03:00
Pavel Karpy
cb0bb7207c
[ #1461 ] shard: Add a separate ErrLockObjectRemoval
...
Do not return `meta.ErrLockObjectRemoval` from shard's methods, add shard's
own error for that.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
558cc1193a
[ #1461 ] engine: Clarify force removal
...
Document force removal behaviour in all the Storage engine parts.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
51afcc1182
[ #1461 ] engine, policer: Force remove objects w/o container
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
18ec5d7c8e
[ #1461 ] meta: Return error on lock object removal
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
b5c56d459a
[ #1461 ] engine: Add force lock removal tests
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
63c00e785d
[ #1461 ] shard: Fix option naming
...
`WitDeletedLockCallback` => `WithDeletedLockCallback`.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
5122be34e7
[ #1461 ] shard: Add lock tests
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
eaf96bccf7
[ #1461 ] meta: Add lock tests
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
c6cf8e5c0b
[ #1461 ] engine: Fix LOCK test
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
9c5ef3bab8
[ #1461 ] node: Allow force LOCK removal
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
fed9e6679d
[ #1461 ] node: Unlock locked object on its lock removal
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Evgenii Stratonikov
72c044e2eb
[ #1599 ] engine: Parallelize shard initialization
...
Shard is intended to be used as a separate failure domain,
which usually resides on a separate disk. Thus, sequential
initialization is bound by IO and this change speeds up thing a bit.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-15 13:44:54 +03:00
Evgenii Stratonikov
633b4e7d2d
[ #1483 ] metabase: Add VERSION.md
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
6f243a2a76
[ #1483 ] metabase: Store version
...
The main problem is to distinguish the case of initial initialization
and update from version 0. We can't do this at `Open`, because of
`resync_metabase` flag. Thus, the following approach was taken:
1. During `Open` check whether the metabase was initialized.
2. Check for the version in `Init` or write the new one if the metabase
is new.
3. Update version in `Reset`.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
7df50297cd
[ #1520 ] shard: Ignore errors on metabase refill
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:25:28 +03:00
Evgenii Stratonikov
78ea450c25
[ #1502 ] shard: Process locks on metabase refill
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 11:07:26 +03:00
Evgenii Stratonikov
972ca83e23
[ #1524 ] writecache: Add some bolt parameters to the configuration
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
07e06249d5
[ #1524 ] metabase: Add some bolt parameters to the configuration
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
81f925d5a0
[ #1516 ] metabase: Optimize ListWithCursor
for long listings
...
Cache buckets outside of the main loop and allocate memory for the
resulting offset only once.
```
name old time/op new time/op delta
ListWithCursor/1_item-8 6.45µs ±14% 5.79µs ±11% -10.24% (p=0.002 n=10+10)
ListWithCursor/10_items-8 20.9µs ±17% 17.3µs ± 9% -17.27% (p=0.000 n=10+10)
ListWithCursor/100_items-8 153µs ±12% 131µs ± 9% -14.63% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 2.31kB ± 0% 1.91kB ± 0% -17.46% (p=0.000 n=10+10)
ListWithCursor/10_items-8 6.94kB ± 0% 5.50kB ± 0% -20.78% (p=0.000 n=8+8)
ListWithCursor/100_items-8 53.3kB ± 0% 41.5kB ± 0% -22.18% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 40.0 ± 0% 34.0 ± 0% -15.00% (p=0.000 n=10+10)
ListWithCursor/10_items-8 121 ± 0% 100 ± 0% -17.36% (p=0.000 n=10+10)
ListWithCursor/100_items-8 930 ± 0% 758 ± 0% -18.49% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
a93373fe71
[ #1516 ] metabase: Cache graveyard buckets in ListWithCursor
...
```
name old time/op new time/op delta
ListWithCursor/1_item-8 6.40µs ±13% 6.45µs ±14% ~ (p=0.739 n=10+10)
ListWithCursor/10_items-8 30.9µs ±21% 20.9µs ±17% -32.49% (p=0.000 n=10+10)
ListWithCursor/100_items-8 274µs ±27% 153µs ±12% -44.09% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 2.26kB ± 0% 2.31kB ± 0% +2.46% (p=0.000 n=10+10)
ListWithCursor/10_items-8 10.8kB ± 0% 6.9kB ± 0% -36.07% (p=0.000 n=8+8)
ListWithCursor/100_items-8 96.8kB ± 0% 53.3kB ± 0% -44.98% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 39.0 ± 0% 40.0 ± 0% +2.56% (p=0.000 n=10+10)
ListWithCursor/10_items-8 192 ± 0% 121 ± 0% -36.98% (p=0.000 n=10+10)
ListWithCursor/100_items-8 1.72k ± 0% 0.93k ± 0% -45.93% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
name old time/op new time/op delta
ListWithCursor/1_item-8 5.23µs ±19% 5.26µs ±15% ~ (p=0.853 n=10+10)
ListWithCursor/10_items-8 27.2µs ±15% 18.0µs ±19% -33.80% (p=0.000 n=10+10)
ListWithCursor/100_items-8 250µs ±13% 139µs ±15% -44.27% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 1.99kB ± 0% 2.04kB ± 0% +2.82% (p=0.000 n=8+8)
ListWithCursor/10_items-8 10.3kB ± 0% 6.4kB ± 0% -37.83% (p=0.000 n=8+10)
ListWithCursor/100_items-8 93.9kB ± 0% 50.4kB ± 0% -46.37% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 35.0 ± 0% 36.0 ± 0% +2.86% (p=0.000 n=10+10)
ListWithCursor/10_items-8 184 ± 0% 113 ± 0% -38.59% (p=0.000 n=10+10)
ListWithCursor/100_items-8 1.67k ± 0% 0.88k ± 0% -47.29% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
af4db8a73b
[ #1516 ] metabase: Cache address key and do not decode address twice
...
```
name old time/op new time/op delta
ListWithCursor/1_item-8 10.6µs ± 1% 6.4µs ±13% -39.62% (p=0.000 n=7+10)
ListWithCursor/10_items-8 75.3µs ± 2% 30.9µs ±21% -58.97% (p=0.000 n=10+10)
ListWithCursor/100_items-8 726µs ± 2% 274µs ±27% -62.28% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 3.19kB ± 0% 2.26kB ± 0% -29.21% (p=0.000 n=10+10)
ListWithCursor/10_items-8 20.7kB ± 0% 10.8kB ± 0% -47.68% (p=0.000 n=10+8)
ListWithCursor/100_items-8 196kB ± 0% 97kB ± 0% -50.65% (p=0.000 n=7+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 55.0 ± 0% 39.0 ± 0% -29.09% (p=0.000 n=10+10)
ListWithCursor/10_items-8 346 ± 0% 192 ± 0% -44.51% (p=0.000 n=10+10)
ListWithCursor/100_items-8 3.25k ± 0% 1.72k ± 0% -47.13% (p=0.000 n=9+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
504f45e9ee
[ #1516 ] metabase: Add benchmark for ListWithCursor
...
```
name time/op
ListWithCursor/1_item-8 10.6µs ± 1%
ListWithCursor/10_items-8 75.3µs ± 2%
ListWithCursor/100_items-8 726µs ± 2%
name alloc/op
ListWithCursor/1_item-8 3.19kB ± 0%
ListWithCursor/10_items-8 20.7kB ± 0%
ListWithCursor/100_items-8 196kB ± 0%
name allocs/op
ListWithCursor/1_item-8 55.0 ± 0%
ListWithCursor/10_items-8 346 ± 0%
ListWithCursor/100_items-8 3.25k ± 0%
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
f602d05b0a
[ #1494 ] *: Fix linter warnings
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
38772e1a2e
writecache: provide timeout to bbolt.Open
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-09 14:19:47 +03:00
Pavel Karpy
010253a97a
[ #1460 ] blobovnicza: Do not use pointers as the results
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
0e4a1beecf
[ #1460 ] blobstor: Do not use pointers as the results
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
08bf8a68f1
[ #1460 ] engine: Do not use pointers as the results
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
7b6363f4c6
[ #1460 ] shard: Do not use pointers as the results
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
9b2932609b
[ #1460 ] meta: Do not use pointers as the results
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
a580429996
[ #1460 ] meta: Add a benchmark on Get
operation
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
b0c7b7851a
[ #1418 ] blobstor: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
14366bbd89
[ #1418 ] engine: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
5f57db6bf8
[ #1418 ] shard: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
6d4d920bad
[ #1418 ] meta: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Evgenii Stratonikov
feef9a98f7
[ #1462 ] writecache: remove useless logs
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
2ae7c94cd6
[ #1462 ] *: Remove log.With
invocations
...
`log.With` is suitable during initialization, but in other places it induces
some overhead, even when branches with logging are not taken.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
43e776dfb1
[ #1462 ] writecache: Persist objects in batches
...
`Update` becomes a botleneck under a heavy load.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
6ad87e7959
[ #1462 ] writecache: Fill flush marks during startup
...
Some of the objects are already flushed, don't do it twice.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
f2a7503964
[ #1462 ] fstree: Allow to fetch file content lazily
...
If we should process address based on some condition, there is no need
to read file content in memory.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
54d4503701
[ #1462 ] writecache: Rename flushWorkersCount
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
0148209168
[ #1462 ] writecache: Use Timer
instead of Ticker
in flush
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
5073a37930
[ #1433 ] metabase: Optimize Select
...
For some filters we can scan only a subset of keys instead of checking
each key.
```
name old time/op new time/op delta
Select/string_equal-8 49.3µs ± 4% 11.0µs ± 4% -77.68% (p=0.000 n=10+10)
Select/string_not_equal-8 7.01ms ± 5% 7.06ms ±10% ~ (p=0.971 n=10+10)
Select/common_prefix-8 118µs ± 6% 79µs ± 5% -33.04% (p=0.000 n=10+9)
Select/unknown-8 21.3µs ± 4% 3.2µs ± 4% -84.88% (p=0.000 n=10+9)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
5c48588c64
[ #1433 ] metabase: Add more tests for Select
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
ea76c989ca
[ #1433 ] metabase: Add a benchmark for Select
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
b602fff01f
[ #1463 ] blobovnicza: Remove pointers from struct members
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 13:42:54 +03:00
Evgenii Stratonikov
d9f0ac8909
[ #1463 ] blobovnicza: Determine db size via os.Stat
...
Currently we use `(*bbolt.Bucket).Stats().KeyN` for estimating database
size. However, it iterates over all pages in bucket and thus heavily
depends on the bucket size. This commit replaces initial size estimation
with a single `os.Stat` call.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 13:42:54 +03:00
Pavel Karpy
021aa97965
[ #1418 ] blobovnicza: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
281befec67
[ #1418 ] blobstor: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
babd382ba5
[ #1418 ] engine: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
6e752f36dc
[ #1418 ] shard: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
e265ce2d52
[ #1418 ] meta: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
1c100fb4b0
[ #1418 ] writecache: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Leonard Lyubich
1c30414a6c
[ #1454 ] Upgrade NeoFS SDK Go module with new IDs
...
Core changes:
* avoid package-colliding variable naming
* avoid using pointers to IDs where unnecessary
* avoid using `idSDK` import alias pattern
* use `EncodeToString` for protocol string calculation and `String` for
printing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Leonard Lyubich
883f2f1ac6
[ #1423 ] local storage: Activate skipped unit tests after fix
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Leonard Lyubich
86552cf3ae
engine: make corruptSubDir
to ignore blobovnicza
dir
...
There is a dir for Blobovnicza tree in BlobStor, so we need to ignore
it.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:33:14 +03:00
Leonard Lyubich
bb25ecbd15
[ #1400 ] owner: Upgrade SDK package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:33:14 +03:00
Leonard Lyubich
f15e6e888f
[ #1377 ] oid, cid: Upgrade SDK package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-16 15:33:22 +03:00
Leonard Lyubich
ae92074272
[ #1369 ] checksum: Upgrade SDK package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Leonard Lyubich
088df0e2a9
[ #1369 ] version: Upgrade SDK package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Elizaveta Chichindaeva
cc7a723d77
[ #1320 ] English Check
...
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-05-11 10:40:02 +03:00
Evgenii Stratonikov
a4769d8624
*: use require.ErrorIs
where possible
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:47:57 +03:00
Evgenii Stratonikov
3c39e6df11
[ #1319 ] blobstor: Compress big objects in a streaming fashion
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Pavel Karpy
9da7df4d42
[ #1318 ] metabase: Init metabase in test
...
`Init()` creates buckets. It prevents NPE in tests.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
486ec8a2e5
[ #1318 ] metabase: Reuse comparing func from require
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
1caeb0b648
[ #1318 ] metabase: Add missing error check in tests
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
2aec5736e0
[ #1318 ] node/gc: Fix tombstones handling
...
Handle only expired tombstones.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
7799f8e4cf
[ #1318 ] engine: Change tombstone clear process
...
- Delete objects physically on tombstone's arrival;
- Store information about tombstones in the Graveyard;
- Clear Graveyard every epoch based on the information about TS in the
network.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
2583f608e8
[ #1318 ] metabase: Add DropGraves
method
...
Method clean up Graveyard and should be called by the routine that handles
tombstone expiration.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
daab30c391
[ #1318 ] metabase: Add iteration offset
...
Add offset element to the iterations over deleted objects (both the
Graveyard and the Garbage buckets).
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
fe8076e60a
[ #1318 ] metabase: Add immediate object deletion
...
Mark objects with GC immediately after a Tombstone is received.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
8107c8d1a9
[ #1318 ] metabase: Separate buckets with TS and GC marks
...
It allows storing information about object in both ways at the same time:
1. Metabase should know if an object is covered by a tombstone (that is
not expired yet);
2. It should be possible to physically delete objects covered by a
tombstone immediately (mark with GC) but keep tombstone knowledge.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
bd27837364
[ #1321 ] node: Register GC event channel before shard init
...
Morph "NewEpoch" event handling was registered in a closure over
`addNewEpochNotificationHandler` func. That may lead to the data race:
if a shard was initialized before the event registration, everything works
as planned, but if registration was made earlier, it was not able to
include GC handlers since a shard has not called `eventChanInit` yet and,
therefore, it has not registered handler yet.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-25 10:39:37 +03:00
Evgenii Stratonikov
622ea4818f
[ #1250 ] *: Remove io/ioutil imports
...
It is deprecated starting from go1.16.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 17:23:28 +03:00
Leonard Lyubich
629a4f79cd
[ #1293 ] metabase: Fix out-of-range panic in freePotentialLocks
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-05 15:38:51 +03:00
Evgenii Stratonikov
48eb87d32c
[ #1149 ] writecache: Prevent corruption in Head
...
Also, remove optimization comments:
1. Having to maintain an execute the same logic for headers as for
objects is quite inefficient, as it increases memory footprint.
2. Unmarshaling object is a cheap operation if data slice is in memory.
3. For unmarshaling header-only, I think we need SDK support.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:40:25 +03:00
Evgenii Stratonikov
68903c9fd9
[ #1143 ] shard: Support degraded mode in Get
and GetRange
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
f058cead8f
[ #1143 ] shard: Handle some errors in degraded mode
...
If metabase is corrupted for some reason, failback to
checking blobstor directly.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
08e7914729
[ #1143 ] blobstor: Implement existsSmall
check
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
aa0cc1f824
[ #1143 ] blobovnicza: Copy object data in Get
...
Data returned from `*bbolt.Bucket.Get()` is only valid for the lifetime
of the transaction.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
6472a170eb
[ #1143 ] shard: Introduce explicit Degraded
mode
...
`Degraded` mode is set automatically after error counter is over the
threshold. `ReadOnly` mode can still be set by an administrator.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
de5a2f6574
[ #1262 ] metabase: Remove list index in place Delete
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
43867a3093
[ #1262 ] metabase: Do not allocate intermediate slices for indices
...
```
name old alloc/op new alloc/op delta
Put/parallel-8 123kB ± 4% 119kB ± 3% -2.72% (p=0.006 n=10+9)
Put/sequential-8 170kB ± 1% 168kB ± 1% -1.42% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
Put/parallel-8 473 ± 1% 469 ± 0% -0.87% (p=0.000 n=10+10)
Put/sequential-8 792 ± 0% 787 ± 0% -0.58% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
0e9b6be3fd
[ #1262 ] metabase: Remove intermediate allocations in decodeList
...
```
name old time/op new time/op delta
Put/parallel-8 1.57ms ±11% 1.51ms ± 3% -4.06% (p=0.043 n=9+10)
Put/sequential-8 5.16ms ± 2% 5.16ms ± 3% ~ (p=1.000 n=9+10)
name old alloc/op new alloc/op delta
Put/parallel-8 126kB ± 4% 123kB ± 4% -2.54% (p=0.016 n=8+10)
Put/sequential-8 171kB ± 1% 170kB ± 1% ~ (p=0.182 n=9+10)
name old allocs/op new allocs/op delta
Put/parallel-8 565 ± 2% 473 ± 1% -16.18% (p=0.000 n=9+10)
Put/sequential-8 819 ± 1% 792 ± 0% -3.34% (p=0.000 n=9+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
d45df614fb
[ #1262 ] metabase: Optimize decodeList
...
Prevent additional allocation during `append` in `Put`.
```
name old alloc/op new alloc/op delta
Put/parallel-8 131kB ± 1% 126kB ± 4% -3.87% (p=0.005 n=8+8)
Put/sequential-8 172kB ± 1% 171kB ± 1% -0.73% (p=0.028 n=10+9)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
456e1584d6
[ #1262 ] metabase: Add benchmarks for Put
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
f1223b46df
[ #1262 ] blobovnicza: Make helper in Put
function idempotent
...
`Batch` can execute the function multiple times leading to multiple
increases of a size approximation.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
ad92493b86
[ #1268 ] blobstor: Cleanup zstd encoders/decoders
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-25 14:05:06 +03:00