7d6df543d7
[ #1523 ] blobstor: Export single `Get`, `GetRange` and `Delete` methods
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
ca15083a50
[ #1523 ] local_object_storage: Unify parameters for the `Exists` operation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
d75d030a90
[ #1523 ] local_object_storage: Unify parameters for the `Delete` operation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
36c88f0dc8
[ #1523 ] local_object_storage: Unify parameters for the `GetRange` operation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
d9a2f280c9
[ #1523 ] local_object_storage: Unify parameters for the `Get` operation
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
b621f5983a
[ #1523 ] local_object_storage: Move blobovnicza tree to a separate package
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
4e361e2ab4
[ #1647 ] writecache: Open DB in sync mode
...
Close #1647 .
Initially, `Sync: false` was provided because we can already lose
objects cached in memory. However, future changes in writecache will
remove inmemory cache and speed up it via other means.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-16 15:27:18 +04:00
da2975a2f9
[ #1664 ] write-cache: Fix panic on `Delete` operation
...
If an object is found in the Write-cache and is placed at the end of
the in-memory cache, the memory counter update operation tries to
dereference the index that is out of the sliced array. Moreover, even if
panic does not appear, the counter is updated with the wrong value.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-09 20:18:26 +03:00
a5da665e69
[ #1656 ] meta: Fix `MatchCommonPrefix` operation
...
Return all the objects on the empty common prefix search without search
optimizations that breaks boltDB logic.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:32:00 +03:00
156ba85326
[ #1634 ] node: Do not return expired objects
...
If an object has not been marked for removal by the GC in the current epoch
yet but has already expired, respond with `ErrObjectNotFound` api status.
Also, optimize shard iteration: a node must stop any iteration if the object
is found but gonna be removed soon.
All the checks are performed by the Metabase.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
9aba0ba512
[ #1634 ] meta: Add epoch state
...
It allows performing expiration checks on the stored objects.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
8ffc2fdf5e
[ #1642 ] engine: Do not increase error counter if the pilorama is disabled
...
After a4adb79db
2022-07-30 18:39:22 +03:00
8d0884e74f
[ #1644 ] storagelog: Fix doc comment
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-30 18:39:08 +03:00
4abe5a7245
shard: add more checks for `GetRange` parameters
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 11:22:04 +03:00
72586f17d4
shard: fix `GetRange` for objects stored in the write-cache
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 11:22:04 +03:00
1691364653
[ #1559 ] local_object_storage: Fix tests and some data races
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
50e28f22f9
[ #1559 ] shard: Change `Degraded` mode string representation
...
It is a flag, but is a `degraded-read-write` mode for a user.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
fabe717d32
[ #1549 ] shard: Turn to read-only mode on metabase failure
...
If metabase can't be opened in the default mode, try opening shard
first in `ReadOnly` mode and then in `DegradedReadOnly`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
4944490ffb
[ #1559 ] local_object_storage: Move shard to the `DegradedReadOnly` mode
...
`Degraded` mode can be set by the administrator if needed.
Modifying operations in this mode can lead node into an inconsistent state
because metabase checks such as lock checking are not performed.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
3df62769c0
[ #1559 ] local_object_storage: Allow to set mode for all components
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
1e786233bf
[ #1559 ] local_object_storage: Provide readOnly flag to `Open`
...
We should be able to reopen storage in readonly in runtime.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
e38b0aa4ba
[ #1549 ] engine: Disable shard on blobovnicza init failure
...
There is a need to support working w/o shard if it has problems with
blobovnicza tree.
Make `BlobStor.Init` to return new `ErrInitBlobovniczas` error. Remove
shard from storage engine's shard set if it returned this error from
`Init` call. So if some of the shards (but not all) return this error,
the node will be able to continue working without them.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
40a56c6b42
[ #1559 ] engine: Do not count logical errors as storage ones
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
c8911d72d0
[ #1559 ] shard: Do not consult metabase in a degraded mode
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
339864b720
[ #1559 ] local_object_storage: Move `shard.Mode` to a separate package
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
d8ba954aff
[ #1559 ] shard: Use `Set` prefix for parameter setting
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
7b882b26d8
[ #1559 ] shard: Remove public functions
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
34d319fed2
[ #1559 ] metabase: Use `Set` prefix for parameter setting
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
f58234aa2f
[ #1559 ] metabase: Remove public functions
...
Reduce public interface of this package. Later each result will contain
an additional status, so it makes more sense to use the same functions
and result processing everywhere.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
a4adb79db7
[ #1607 ] pilorama: Enable tree service explicitly
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
d62723f038
[ #1505 ] pilorama: Provide timeout to `bbolt.Open`
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
26041f18bf
[ #1505 ] pilorama: Allow to customize database parameters
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
c7b10598f9
[ #1333 ] engine: Increase error counter for pilorama errors
...
1. Modifying operations are not expected to fail, unless the shard is
read-only.
2. `Get*` operations should increase error counter too, unless the
error is `ErrTreeNotFound`.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
5e843a73f9
[ #1333 ] services/control: Return pilorama info in `ListShards` RPC
...
Do not return backend type from the service for now, because memory
backend is expected to vanish.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
8f4ee1aded
[ #1333 ] local_object_storage: Support ReadOnly mode in pilorama
...
The tricky part here is the engine itself: we stop iteration on
`ErrReadOnly` because it is better to synchronize the shard later than
to have partial trees stored in 2 shards.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
735931c842
[ #1481 ] pilorama: Fix `TreeApply`
...
Current implementation prevents invalid operations to become valid at
some later point (consider adding a child to the non-existent parent and
then adding the parent). This seems to diverge from the paper algorithm
and complicates implementation. Make it simpler.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
ad3038d16d
[ #1444 ] pilorama: Fix `TreeMove` in bbolt backend
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
8027b7bb6b
[ #1444 ] pilorama: Optimize internal encoding/decoding
...
```
name old time/op new time/op delta
ApplySequential/bbolt-8 55.5µs ± 4% 55.5µs ± 3% ~ (p=1.000 n=10+7)
ApplyReorderLast/bbolt-8 108µs ± 6% 112µs ± 8% ~ (p=0.077 n=9+9)
name old alloc/op new alloc/op delta
ApplySequential/bbolt-8 28.8kB ± 3% 27.7kB ± 6% -3.79% (p=0.005 n=10+10)
ApplyReorderLast/bbolt-8 41.4kB ± 5% 38.9kB ± 5% -6.19% (p=0.001 n=10+9)
name old allocs/op new allocs/op delta
ApplySequential/bbolt-8 262 ± 2% 235 ±10% -10.41% (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8 684 ± 6% 616 ± 7% -10.04% (p=0.000 n=10+9)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
4437cd7113
[ #1442 ] pilorama: Generate timestamp based on node position in the container
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
3312924b82
[ #1431 ] pilorama: Use `Batch` for write transactions
...
Helps a lot in case of concurrent request flow.
```
name old time/op new time/op delta
ApplySequential/bbolt-8 78.0µs ± 9% 59.8µs ± 4% -23.39% (p=0.000 n=10+9)
ApplyReorderLast/bbolt-8 143µs ± 5% 113µs ±15% -21.06% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ApplySequential/bbolt-8 56.9kB ± 8% 28.9kB ± 3% -49.22% (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8 87.3kB ± 3% 40.9kB ±10% -53.16% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ApplySequential/bbolt-8 224 ±11% 262 ± 5% +16.93% (p=0.000 n=9+10)
ApplyReorderLast/bbolt-8 518 ± 4% 674 ±11% +30.09% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
f0a67f948d
[ #1431 ] pilorama: Cache attributes in the index
...
Currently to find a node by path we iterate over all the children on
each level. This is far from optimal and scales badly with the number of
nodes on a single level. Thus we introduce "indexed attributes" for
which an additional information is stored and which can be use in
`*ByPath` operations. Currently this set only includes `FileName`
attribute but this may change in future.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
f5d35571d0
[ #1431 ] engine: Add benchmark for `Select` vs `TreeGetByPath`
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
835170e452
[ #1329 ] pilorama: Allow to benchmark all tree backends
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
536857ea5a
[ #1329 ] services/tree: Implement `GetOpLog` RPC
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
7703dd5d7f
[ #1419 ] pilorama: Create new nodes in path if needed
...
Consider a node `{FileName: "dir", Attribute: "xxx"}`. In case we add
a new node by path `["dir", "file.txt"]`, create a new intermediate node
with a single attribute.
`GetByPath` now also considers only nodes with a single attribute while building a path.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
ad48918a97
[ #1406 ] pilorama: Return parent from `TreeGetMeta`
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
8844d9b2db
[ #1344 ] pilorama: Document errors for Get* methods
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
d8ad68d613
[ #1344 ] engine: Log errors in Tree* operations
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
910db42748
[ #1344 ] pilorama: Use `require.ErrorIs` in tests
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
aea855e8f3
[ #1326 ] services/tree: Implement GetSubTree RPC
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
46f4ce2773
[ #1324 ] engine: Implement Forest interface for storage engine
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
8cf71b7f1c
[ #1324 ] local_object_storage: Implement tree service backend
...
In this commit we implement algorithm for CRDT trees from
https://martin.klepmann.com/papers/move-op.pdf
Each tree is identified by the ID of a container it belongs to
and the tree name itself. Essentially, it is a sequence of operations
which should be applied in chronological order to get a usual tree
representation.
There are 2 backends for now: bbolt database and in-memory.
In-memory backend is here for debugging and will eventually act
as a memory-cache for the on-disk database.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
11c9df6f00
[ #1611 ] shard: Print shard ID in component logs
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 15:57:53 +03:00
9f7a22e2aa
[ #1561 ] object: Return `OUT_OF_RANGE` status
...
Replace `ErrRangeOutOfBounds` error from `pkg/core/object` package with
`ObjectOutOfRange` from `apistatus` package. That error is returned by
storage node's server as NeoFS API statuses.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-19 13:29:45 +03:00
cb0bb7207c
[ #1461 ] shard: Add a separate `ErrLockObjectRemoval`
...
Do not return `meta.ErrLockObjectRemoval` from shard's methods, add shard's
own error for that.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
558cc1193a
[ #1461 ] engine: Clarify force removal
...
Document force removal behaviour in all the Storage engine parts.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
51afcc1182
[ #1461 ] engine, policer: Force remove objects w/o container
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
18ec5d7c8e
[ #1461 ] meta: Return error on lock object removal
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
b5c56d459a
[ #1461 ] engine: Add force lock removal tests
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
63c00e785d
[ #1461 ] shard: Fix option naming
...
`WitDeletedLockCallback` => `WithDeletedLockCallback`.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
5122be34e7
[ #1461 ] shard: Add lock tests
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
eaf96bccf7
[ #1461 ] meta: Add lock tests
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
c6cf8e5c0b
[ #1461 ] engine: Fix LOCK test
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
9c5ef3bab8
[ #1461 ] node: Allow force LOCK removal
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
fed9e6679d
[ #1461 ] node: Unlock locked object on its lock removal
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
72c044e2eb
[ #1599 ] engine: Parallelize shard initialization
...
Shard is intended to be used as a separate failure domain,
which usually resides on a separate disk. Thus, sequential
initialization is bound by IO and this change speeds up thing a bit.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-15 13:44:54 +03:00
633b4e7d2d
[ #1483 ] metabase: Add VERSION.md
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:48:28 +03:00
6f243a2a76
[ #1483 ] metabase: Store version
...
The main problem is to distinguish the case of initial initialization
and update from version 0. We can't do this at `Open`, because of
`resync_metabase` flag. Thus, the following approach was taken:
1. During `Open` check whether the metabase was initialized.
2. Check for the version in `Init` or write the new one if the metabase
is new.
3. Update version in `Reset`.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:48:28 +03:00
7df50297cd
[ #1520 ] shard: Ignore errors on metabase refill
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:25:28 +03:00
78ea450c25
[ #1502 ] shard: Process locks on metabase refill
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 11:07:26 +03:00
972ca83e23
[ #1524 ] writecache: Add some bolt parameters to the configuration
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 17:04:35 +03:00
07e06249d5
[ #1524 ] metabase: Add some bolt parameters to the configuration
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 17:04:35 +03:00
81f925d5a0
[ #1516 ] metabase: Optimize `ListWithCursor` for long listings
...
Cache buckets outside of the main loop and allocate memory for the
resulting offset only once.
```
name old time/op new time/op delta
ListWithCursor/1_item-8 6.45µs ±14% 5.79µs ±11% -10.24% (p=0.002 n=10+10)
ListWithCursor/10_items-8 20.9µs ±17% 17.3µs ± 9% -17.27% (p=0.000 n=10+10)
ListWithCursor/100_items-8 153µs ±12% 131µs ± 9% -14.63% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 2.31kB ± 0% 1.91kB ± 0% -17.46% (p=0.000 n=10+10)
ListWithCursor/10_items-8 6.94kB ± 0% 5.50kB ± 0% -20.78% (p=0.000 n=8+8)
ListWithCursor/100_items-8 53.3kB ± 0% 41.5kB ± 0% -22.18% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 40.0 ± 0% 34.0 ± 0% -15.00% (p=0.000 n=10+10)
ListWithCursor/10_items-8 121 ± 0% 100 ± 0% -17.36% (p=0.000 n=10+10)
ListWithCursor/100_items-8 930 ± 0% 758 ± 0% -18.49% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
a93373fe71
[ #1516 ] metabase: Cache graveyard buckets in `ListWithCursor`
...
```
name old time/op new time/op delta
ListWithCursor/1_item-8 6.40µs ±13% 6.45µs ±14% ~ (p=0.739 n=10+10)
ListWithCursor/10_items-8 30.9µs ±21% 20.9µs ±17% -32.49% (p=0.000 n=10+10)
ListWithCursor/100_items-8 274µs ±27% 153µs ±12% -44.09% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 2.26kB ± 0% 2.31kB ± 0% +2.46% (p=0.000 n=10+10)
ListWithCursor/10_items-8 10.8kB ± 0% 6.9kB ± 0% -36.07% (p=0.000 n=8+8)
ListWithCursor/100_items-8 96.8kB ± 0% 53.3kB ± 0% -44.98% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 39.0 ± 0% 40.0 ± 0% +2.56% (p=0.000 n=10+10)
ListWithCursor/10_items-8 192 ± 0% 121 ± 0% -36.98% (p=0.000 n=10+10)
ListWithCursor/100_items-8 1.72k ± 0% 0.93k ± 0% -45.93% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
name old time/op new time/op delta
ListWithCursor/1_item-8 5.23µs ±19% 5.26µs ±15% ~ (p=0.853 n=10+10)
ListWithCursor/10_items-8 27.2µs ±15% 18.0µs ±19% -33.80% (p=0.000 n=10+10)
ListWithCursor/100_items-8 250µs ±13% 139µs ±15% -44.27% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 1.99kB ± 0% 2.04kB ± 0% +2.82% (p=0.000 n=8+8)
ListWithCursor/10_items-8 10.3kB ± 0% 6.4kB ± 0% -37.83% (p=0.000 n=8+10)
ListWithCursor/100_items-8 93.9kB ± 0% 50.4kB ± 0% -46.37% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 35.0 ± 0% 36.0 ± 0% +2.86% (p=0.000 n=10+10)
ListWithCursor/10_items-8 184 ± 0% 113 ± 0% -38.59% (p=0.000 n=10+10)
ListWithCursor/100_items-8 1.67k ± 0% 0.88k ± 0% -47.29% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
af4db8a73b
[ #1516 ] metabase: Cache address key and do not decode address twice
...
```
name old time/op new time/op delta
ListWithCursor/1_item-8 10.6µs ± 1% 6.4µs ±13% -39.62% (p=0.000 n=7+10)
ListWithCursor/10_items-8 75.3µs ± 2% 30.9µs ±21% -58.97% (p=0.000 n=10+10)
ListWithCursor/100_items-8 726µs ± 2% 274µs ±27% -62.28% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 3.19kB ± 0% 2.26kB ± 0% -29.21% (p=0.000 n=10+10)
ListWithCursor/10_items-8 20.7kB ± 0% 10.8kB ± 0% -47.68% (p=0.000 n=10+8)
ListWithCursor/100_items-8 196kB ± 0% 97kB ± 0% -50.65% (p=0.000 n=7+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 55.0 ± 0% 39.0 ± 0% -29.09% (p=0.000 n=10+10)
ListWithCursor/10_items-8 346 ± 0% 192 ± 0% -44.51% (p=0.000 n=10+10)
ListWithCursor/100_items-8 3.25k ± 0% 1.72k ± 0% -47.13% (p=0.000 n=9+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
504f45e9ee
[ #1516 ] metabase: Add benchmark for `ListWithCursor`
...
```
name time/op
ListWithCursor/1_item-8 10.6µs ± 1%
ListWithCursor/10_items-8 75.3µs ± 2%
ListWithCursor/100_items-8 726µs ± 2%
name alloc/op
ListWithCursor/1_item-8 3.19kB ± 0%
ListWithCursor/10_items-8 20.7kB ± 0%
ListWithCursor/100_items-8 196kB ± 0%
name allocs/op
ListWithCursor/1_item-8 55.0 ± 0%
ListWithCursor/10_items-8 346 ± 0%
ListWithCursor/100_items-8 3.25k ± 0%
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
f602d05b0a
[ #1494 ] *: Fix linter warnings
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 12:26:10 +03:00
38772e1a2e
writecache: provide timeout to `bbolt.Open`
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-09 14:19:47 +03:00
010253a97a
[ #1460 ] blobovnicza: Do not use pointers as the results
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
0e4a1beecf
[ #1460 ] blobstor: Do not use pointers as the results
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
08bf8a68f1
[ #1460 ] engine: Do not use pointers as the results
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
7b6363f4c6
[ #1460 ] shard: Do not use pointers as the results
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
9b2932609b
[ #1460 ] meta: Do not use pointers as the results
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
a580429996
[ #1460 ] meta: Add a benchmark on `Get` operation
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
b0c7b7851a
[ #1418 ] blobstor: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
14366bbd89
[ #1418 ] engine: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
5f57db6bf8
[ #1418 ] shard: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
6d4d920bad
[ #1418 ] meta: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
feef9a98f7
[ #1462 ] writecache: remove useless logs
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
2ae7c94cd6
[ #1462 ] *: Remove `log.With` invocations
...
`log.With` is suitable during initialization, but in other places it induces
some overhead, even when branches with logging are not taken.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
43e776dfb1
[ #1462 ] writecache: Persist objects in batches
...
`Update` becomes a botleneck under a heavy load.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
6ad87e7959
[ #1462 ] writecache: Fill flush marks during startup
...
Some of the objects are already flushed, don't do it twice.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
f2a7503964
[ #1462 ] fstree: Allow to fetch file content lazily
...
If we should process address based on some condition, there is no need
to read file content in memory.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
54d4503701
[ #1462 ] writecache: Rename `flushWorkersCount`
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
0148209168
[ #1462 ] writecache: Use `Timer` instead of `Ticker` in `flush`
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
5073a37930
[ #1433 ] metabase: Optimize `Select`
...
For some filters we can scan only a subset of keys instead of checking
each key.
```
name old time/op new time/op delta
Select/string_equal-8 49.3µs ± 4% 11.0µs ± 4% -77.68% (p=0.000 n=10+10)
Select/string_not_equal-8 7.01ms ± 5% 7.06ms ±10% ~ (p=0.971 n=10+10)
Select/common_prefix-8 118µs ± 6% 79µs ± 5% -33.04% (p=0.000 n=10+9)
Select/unknown-8 21.3µs ± 4% 3.2µs ± 4% -84.88% (p=0.000 n=10+9)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
5c48588c64
[ #1433 ] metabase: Add more tests for `Select`
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
ea76c989ca
[ #1433 ] metabase: Add a benchmark for `Select`
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
b602fff01f
[ #1463 ] blobovnicza: Remove pointers from struct members
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 13:42:54 +03:00
d9f0ac8909
[ #1463 ] blobovnicza: Determine db size via `os.Stat`
...
Currently we use `(*bbolt.Bucket).Stats().KeyN` for estimating database
size. However, it iterates over all pages in bucket and thus heavily
depends on the bucket size. This commit replaces initial size estimation
with a single `os.Stat` call.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 13:42:54 +03:00
021aa97965
[ #1418 ] blobovnicza: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
281befec67
[ #1418 ] blobstor: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
babd382ba5
[ #1418 ] engine: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
6e752f36dc
[ #1418 ] shard: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
e265ce2d52
[ #1418 ] meta: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
1c100fb4b0
[ #1418 ] writecache: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
1c30414a6c
[ #1454 ] Upgrade NeoFS SDK Go module with new IDs
...
Core changes:
* avoid package-colliding variable naming
* avoid using pointers to IDs where unnecessary
* avoid using `idSDK` import alias pattern
* use `EncodeToString` for protocol string calculation and `String` for
printing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
883f2f1ac6
[ #1423 ] local storage: Activate skipped unit tests after fix
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
86552cf3ae
engine: make `corruptSubDir` to ignore `blobovnicza` dir
...
There is a dir for Blobovnicza tree in BlobStor, so we need to ignore
it.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:33:14 +03:00
bb25ecbd15
[ #1400 ] owner: Upgrade SDK package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:33:14 +03:00
f15e6e888f
[ #1377 ] oid, cid: Upgrade SDK package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-16 15:33:22 +03:00
ae92074272
[ #1369 ] checksum: Upgrade SDK package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
088df0e2a9
[ #1369 ] version: Upgrade SDK package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
cc7a723d77
[ #1320 ] English Check
...
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-05-11 10:40:02 +03:00
a4769d8624
*: use `require.ErrorIs` where possible
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:47:57 +03:00
3c39e6df11
[ #1319 ] blobstor: Compress big objects in a streaming fashion
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
9da7df4d42
[ #1318 ] metabase: Init metabase in test
...
`Init()` creates buckets. It prevents NPE in tests.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
486ec8a2e5
[ #1318 ] metabase: Reuse comparing func from `require`
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
1caeb0b648
[ #1318 ] metabase: Add missing error check in tests
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
2aec5736e0
[ #1318 ] node/gc: Fix tombstones handling
...
Handle only expired tombstones.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
7799f8e4cf
[ #1318 ] engine: Change tombstone clear process
...
- Delete objects physically on tombstone's arrival;
- Store information about tombstones in the Graveyard;
- Clear Graveyard every epoch based on the information about TS in the
network.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
2583f608e8
[ #1318 ] metabase: Add `DropGraves` method
...
Method clean up Graveyard and should be called by the routine that handles
tombstone expiration.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
daab30c391
[ #1318 ] metabase: Add iteration offset
...
Add offset element to the iterations over deleted objects (both the
Graveyard and the Garbage buckets).
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
fe8076e60a
[ #1318 ] metabase: Add immediate object deletion
...
Mark objects with GC immediately after a Tombstone is received.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
8107c8d1a9
[ #1318 ] metabase: Separate buckets with TS and GC marks
...
It allows storing information about object in both ways at the same time:
1. Metabase should know if an object is covered by a tombstone (that is
not expired yet);
2. It should be possible to physically delete objects covered by a
tombstone immediately (mark with GC) but keep tombstone knowledge.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
bd27837364
[ #1321 ] node: Register GC event channel before shard init
...
Morph "NewEpoch" event handling was registered in a closure over
`addNewEpochNotificationHandler` func. That may lead to the data race:
if a shard was initialized before the event registration, everything works
as planned, but if registration was made earlier, it was not able to
include GC handlers since a shard has not called `eventChanInit` yet and,
therefore, it has not registered handler yet.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-25 10:39:37 +03:00
622ea4818f
[ #1250 ] *: Remove io/ioutil imports
...
It is deprecated starting from go1.16.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 17:23:28 +03:00
629a4f79cd
[ #1293 ] metabase: Fix out-of-range panic in `freePotentialLocks`
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-05 15:38:51 +03:00
48eb87d32c
[ #1149 ] writecache: Prevent corruption in `Head`
...
Also, remove optimization comments:
1. Having to maintain an execute the same logic for headers as for
objects is quite inefficient, as it increases memory footprint.
2. Unmarshaling object is a cheap operation if data slice is in memory.
3. For unmarshaling header-only, I think we need SDK support.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:40:25 +03:00
68903c9fd9
[ #1143 ] shard: Support degraded mode in `Get` and `GetRange`
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
f058cead8f
[ #1143 ] shard: Handle some errors in degraded mode
...
If metabase is corrupted for some reason, failback to
checking blobstor directly.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
08e7914729
[ #1143 ] blobstor: Implement `existsSmall` check
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
aa0cc1f824
[ #1143 ] blobovnicza: Copy object data in `Get`
...
Data returned from `*bbolt.Bucket.Get()` is only valid for the lifetime
of the transaction.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
6472a170eb
[ #1143 ] shard: Introduce explicit `Degraded` mode
...
`Degraded` mode is set automatically after error counter is over the
threshold. `ReadOnly` mode can still be set by an administrator.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
de5a2f6574
[ #1262 ] metabase: Remove list index in place `Delete`
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
43867a3093
[ #1262 ] metabase: Do not allocate intermediate slices for indices
...
```
name old alloc/op new alloc/op delta
Put/parallel-8 123kB ± 4% 119kB ± 3% -2.72% (p=0.006 n=10+9)
Put/sequential-8 170kB ± 1% 168kB ± 1% -1.42% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
Put/parallel-8 473 ± 1% 469 ± 0% -0.87% (p=0.000 n=10+10)
Put/sequential-8 792 ± 0% 787 ± 0% -0.58% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
0e9b6be3fd
[ #1262 ] metabase: Remove intermediate allocations in `decodeList`
...
```
name old time/op new time/op delta
Put/parallel-8 1.57ms ±11% 1.51ms ± 3% -4.06% (p=0.043 n=9+10)
Put/sequential-8 5.16ms ± 2% 5.16ms ± 3% ~ (p=1.000 n=9+10)
name old alloc/op new alloc/op delta
Put/parallel-8 126kB ± 4% 123kB ± 4% -2.54% (p=0.016 n=8+10)
Put/sequential-8 171kB ± 1% 170kB ± 1% ~ (p=0.182 n=9+10)
name old allocs/op new allocs/op delta
Put/parallel-8 565 ± 2% 473 ± 1% -16.18% (p=0.000 n=9+10)
Put/sequential-8 819 ± 1% 792 ± 0% -3.34% (p=0.000 n=9+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
d45df614fb
[ #1262 ] metabase: Optimize `decodeList`
...
Prevent additional allocation during `append` in `Put`.
```
name old alloc/op new alloc/op delta
Put/parallel-8 131kB ± 1% 126kB ± 4% -3.87% (p=0.005 n=8+8)
Put/sequential-8 172kB ± 1% 171kB ± 1% -0.73% (p=0.028 n=10+9)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
456e1584d6
[ #1262 ] metabase: Add benchmarks for `Put`
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
f1223b46df
[ #1262 ] blobovnicza: Make helper in `Put` function idempotent
...
`Batch` can execute the function multiple times leading to multiple
increases of a size approximation.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
ad92493b86
[ #1268 ] blobstor: Cleanup zstd encoders/decoders
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-25 14:05:06 +03:00
318639e5bf
[ #1247 ] *: Clarify docs about returned errors from `apistatus` package
...
`apistatus` package provides types which implement build-in `error`
interface. Add `error of type` pattern when documenting these errors in
order to clarify how these errors should be handled (e.g. `errors.Is` is
not good).
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
70ffdf3478
[ #1247 ] object: Return `NOT_FOUND` and `ALREADY_REMOVED` statuses
...
Replace `ErrNotFound`/`ErrAlreadyRemoved` error from
`pkg/core/object` package with `ObjectNotFound`/`ObjectAlreadyRemoved`
one from `apistatus` package. These errors are returned by storage
node's server as NeoFS API statuses.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
0adb29c035
[ #1214 ] Fix metabase put logging
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 17:17:18 +03:00
8f476f3c4d
[ #1238 ] Adopt neofs-node for non pointer slices in SDK
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
7f3195b197
[ #1238 ] Adopt neofs-cli for non pointer slices in SDK
...
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
9feb5f9405
[ #1175 ] engine: Fix `AddShard` implementation
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
9dff07200c
[ #1181 ] local storage: Process expired locks similar to tombstones
...
There is a need to process expired `LOCK` objects similar to `TOMBSTONE`
ones: we collect them on `Shard`, notify all other shards about
expiration so they could unlock the objects, and only after that mark
lockers as garbage.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
323dea95c6
[ #1181 ] Do not use deprecated elements
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
e3f4e48d87
[ #1175 ] Fix typo in error return docs
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Evgenii Stratonikov
Evgenii Stratonikov
Pavel Karpy
Leonard Lyubich
Elizaveta Chichindaeva
Alex Vanin