Evgenii Stratonikov
9792e6a4ef
[ #1686 ] metabase: Add generic tests
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Pavel Karpy
c7c1c257e1
[ #1658 ] node: Init object counter on meta's Init
...
Includes:
1. Renaming counter key to distinguish logical and physical objects
2. Version update dropping since changes could be done in a compatible way
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
745f72fff0
[ #1658 ] node: Support object counter metrics
...
Increment shard's object counter on successful `Put` calls and decrement on
`Delete`.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
626766db08
[ #1658 ] meta: Add object counter
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
30341f2192
[ #1687 ] *: Perform go fmt
using go v1.19
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Evgenii Stratonikov
73f8bb3e5f
[ #1523 ] shard: Store generic storage ID in metabase
...
Allow to extend blobstor with more storage sub-systems. Currently
objects stored in the FSTree have empty byte slice descriptor and object
from blobovnicza tree have the same id as earlier. Each such change in
the identifier formation should be accompanied with metabase version
increase.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Pavel Karpy
a5da665e69
[ #1656 ] meta: Fix MatchCommonPrefix
operation
...
Return all the objects on the empty common prefix search without search
optimizations that breaks boltDB logic.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:32:00 +03:00
Pavel Karpy
156ba85326
[ #1634 ] node: Do not return expired objects
...
If an object has not been marked for removal by the GC in the current epoch
yet but has already expired, respond with `ErrObjectNotFound` api status.
Also, optimize shard iteration: a node must stop any iteration if the object
is found but gonna be removed soon.
All the checks are performed by the Metabase.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
Pavel Karpy
9aba0ba512
[ #1634 ] meta: Add epoch state
...
It allows performing expiration checks on the stored objects.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
Evgenii Stratonikov
1691364653
[ #1559 ] local_object_storage: Fix tests and some data races
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Leonard Lyubich
fabe717d32
[ #1549 ] shard: Turn to read-only mode on metabase failure
...
If metabase can't be opened in the default mode, try opening shard
first in `ReadOnly` mode and then in `DegradedReadOnly`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
4944490ffb
[ #1559 ] local_object_storage: Move shard to the DegradedReadOnly
mode
...
`Degraded` mode can be set by the administrator if needed.
Modifying operations in this mode can lead node into an inconsistent state
because metabase checks such as lock checking are not performed.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
3df62769c0
[ #1559 ] local_object_storage: Allow to set mode for all components
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
1e786233bf
[ #1559 ] local_object_storage: Provide readOnly flag to Open
...
We should be able to reopen storage in readonly in runtime.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
40a56c6b42
[ #1559 ] engine: Do not count logical errors as storage ones
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
34d319fed2
[ #1559 ] metabase: Use Set
prefix for parameter setting
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
f58234aa2f
[ #1559 ] metabase: Remove public functions
...
Reduce public interface of this package. Later each result will contain
an additional status, so it makes more sense to use the same functions
and result processing everywhere.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
11c9df6f00
[ #1611 ] shard: Print shard ID in component logs
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 15:57:53 +03:00
Pavel Karpy
558cc1193a
[ #1461 ] engine: Clarify force removal
...
Document force removal behaviour in all the Storage engine parts.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
18ec5d7c8e
[ #1461 ] meta: Return error on lock object removal
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
eaf96bccf7
[ #1461 ] meta: Add lock tests
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
9c5ef3bab8
[ #1461 ] node: Allow force LOCK removal
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
fed9e6679d
[ #1461 ] node: Unlock locked object on its lock removal
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Evgenii Stratonikov
633b4e7d2d
[ #1483 ] metabase: Add VERSION.md
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
6f243a2a76
[ #1483 ] metabase: Store version
...
The main problem is to distinguish the case of initial initialization
and update from version 0. We can't do this at `Open`, because of
`resync_metabase` flag. Thus, the following approach was taken:
1. During `Open` check whether the metabase was initialized.
2. Check for the version in `Init` or write the new one if the metabase
is new.
3. Update version in `Reset`.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
07e06249d5
[ #1524 ] metabase: Add some bolt parameters to the configuration
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
81f925d5a0
[ #1516 ] metabase: Optimize ListWithCursor
for long listings
...
Cache buckets outside of the main loop and allocate memory for the
resulting offset only once.
```
name old time/op new time/op delta
ListWithCursor/1_item-8 6.45µs ±14% 5.79µs ±11% -10.24% (p=0.002 n=10+10)
ListWithCursor/10_items-8 20.9µs ±17% 17.3µs ± 9% -17.27% (p=0.000 n=10+10)
ListWithCursor/100_items-8 153µs ±12% 131µs ± 9% -14.63% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 2.31kB ± 0% 1.91kB ± 0% -17.46% (p=0.000 n=10+10)
ListWithCursor/10_items-8 6.94kB ± 0% 5.50kB ± 0% -20.78% (p=0.000 n=8+8)
ListWithCursor/100_items-8 53.3kB ± 0% 41.5kB ± 0% -22.18% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 40.0 ± 0% 34.0 ± 0% -15.00% (p=0.000 n=10+10)
ListWithCursor/10_items-8 121 ± 0% 100 ± 0% -17.36% (p=0.000 n=10+10)
ListWithCursor/100_items-8 930 ± 0% 758 ± 0% -18.49% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
a93373fe71
[ #1516 ] metabase: Cache graveyard buckets in ListWithCursor
...
```
name old time/op new time/op delta
ListWithCursor/1_item-8 6.40µs ±13% 6.45µs ±14% ~ (p=0.739 n=10+10)
ListWithCursor/10_items-8 30.9µs ±21% 20.9µs ±17% -32.49% (p=0.000 n=10+10)
ListWithCursor/100_items-8 274µs ±27% 153µs ±12% -44.09% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 2.26kB ± 0% 2.31kB ± 0% +2.46% (p=0.000 n=10+10)
ListWithCursor/10_items-8 10.8kB ± 0% 6.9kB ± 0% -36.07% (p=0.000 n=8+8)
ListWithCursor/100_items-8 96.8kB ± 0% 53.3kB ± 0% -44.98% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 39.0 ± 0% 40.0 ± 0% +2.56% (p=0.000 n=10+10)
ListWithCursor/10_items-8 192 ± 0% 121 ± 0% -36.98% (p=0.000 n=10+10)
ListWithCursor/100_items-8 1.72k ± 0% 0.93k ± 0% -45.93% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
name old time/op new time/op delta
ListWithCursor/1_item-8 5.23µs ±19% 5.26µs ±15% ~ (p=0.853 n=10+10)
ListWithCursor/10_items-8 27.2µs ±15% 18.0µs ±19% -33.80% (p=0.000 n=10+10)
ListWithCursor/100_items-8 250µs ±13% 139µs ±15% -44.27% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 1.99kB ± 0% 2.04kB ± 0% +2.82% (p=0.000 n=8+8)
ListWithCursor/10_items-8 10.3kB ± 0% 6.4kB ± 0% -37.83% (p=0.000 n=8+10)
ListWithCursor/100_items-8 93.9kB ± 0% 50.4kB ± 0% -46.37% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 35.0 ± 0% 36.0 ± 0% +2.86% (p=0.000 n=10+10)
ListWithCursor/10_items-8 184 ± 0% 113 ± 0% -38.59% (p=0.000 n=10+10)
ListWithCursor/100_items-8 1.67k ± 0% 0.88k ± 0% -47.29% (p=0.000 n=10+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
af4db8a73b
[ #1516 ] metabase: Cache address key and do not decode address twice
...
```
name old time/op new time/op delta
ListWithCursor/1_item-8 10.6µs ± 1% 6.4µs ±13% -39.62% (p=0.000 n=7+10)
ListWithCursor/10_items-8 75.3µs ± 2% 30.9µs ±21% -58.97% (p=0.000 n=10+10)
ListWithCursor/100_items-8 726µs ± 2% 274µs ±27% -62.28% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithCursor/1_item-8 3.19kB ± 0% 2.26kB ± 0% -29.21% (p=0.000 n=10+10)
ListWithCursor/10_items-8 20.7kB ± 0% 10.8kB ± 0% -47.68% (p=0.000 n=10+8)
ListWithCursor/100_items-8 196kB ± 0% 97kB ± 0% -50.65% (p=0.000 n=7+10)
name old allocs/op new allocs/op delta
ListWithCursor/1_item-8 55.0 ± 0% 39.0 ± 0% -29.09% (p=0.000 n=10+10)
ListWithCursor/10_items-8 346 ± 0% 192 ± 0% -44.51% (p=0.000 n=10+10)
ListWithCursor/100_items-8 3.25k ± 0% 1.72k ± 0% -47.13% (p=0.000 n=9+10)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
504f45e9ee
[ #1516 ] metabase: Add benchmark for ListWithCursor
...
```
name time/op
ListWithCursor/1_item-8 10.6µs ± 1%
ListWithCursor/10_items-8 75.3µs ± 2%
ListWithCursor/100_items-8 726µs ± 2%
name alloc/op
ListWithCursor/1_item-8 3.19kB ± 0%
ListWithCursor/10_items-8 20.7kB ± 0%
ListWithCursor/100_items-8 196kB ± 0%
name allocs/op
ListWithCursor/1_item-8 55.0 ± 0%
ListWithCursor/10_items-8 346 ± 0%
ListWithCursor/100_items-8 3.25k ± 0%
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Pavel Karpy
9b2932609b
[ #1460 ] meta: Do not use pointers as the results
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
a580429996
[ #1460 ] meta: Add a benchmark on Get
operation
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
6d4d920bad
[ #1418 ] meta: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Evgenii Stratonikov
5073a37930
[ #1433 ] metabase: Optimize Select
...
For some filters we can scan only a subset of keys instead of checking
each key.
```
name old time/op new time/op delta
Select/string_equal-8 49.3µs ± 4% 11.0µs ± 4% -77.68% (p=0.000 n=10+10)
Select/string_not_equal-8 7.01ms ± 5% 7.06ms ±10% ~ (p=0.971 n=10+10)
Select/common_prefix-8 118µs ± 6% 79µs ± 5% -33.04% (p=0.000 n=10+9)
Select/unknown-8 21.3µs ± 4% 3.2µs ± 4% -84.88% (p=0.000 n=10+9)
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
5c48588c64
[ #1433 ] metabase: Add more tests for Select
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
ea76c989ca
[ #1433 ] metabase: Add a benchmark for Select
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Pavel Karpy
e265ce2d52
[ #1418 ] meta: Do not use pointers as parameters
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Leonard Lyubich
1c30414a6c
[ #1454 ] Upgrade NeoFS SDK Go module with new IDs
...
Core changes:
* avoid package-colliding variable naming
* avoid using pointers to IDs where unnecessary
* avoid using `idSDK` import alias pattern
* use `EncodeToString` for protocol string calculation and `String` for
printing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Leonard Lyubich
883f2f1ac6
[ #1423 ] local storage: Activate skipped unit tests after fix
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Leonard Lyubich
bb25ecbd15
[ #1400 ] owner: Upgrade SDK package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:33:14 +03:00
Leonard Lyubich
f15e6e888f
[ #1377 ] oid, cid: Upgrade SDK package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-16 15:33:22 +03:00
Leonard Lyubich
ae92074272
[ #1369 ] checksum: Upgrade SDK package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Leonard Lyubich
088df0e2a9
[ #1369 ] version: Upgrade SDK package
...
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Elizaveta Chichindaeva
cc7a723d77
[ #1320 ] English Check
...
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-05-11 10:40:02 +03:00
Pavel Karpy
9da7df4d42
[ #1318 ] metabase: Init metabase in test
...
`Init()` creates buckets. It prevents NPE in tests.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
486ec8a2e5
[ #1318 ] metabase: Reuse comparing func from require
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
1caeb0b648
[ #1318 ] metabase: Add missing error check in tests
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
7799f8e4cf
[ #1318 ] engine: Change tombstone clear process
...
- Delete objects physically on tombstone's arrival;
- Store information about tombstones in the Graveyard;
- Clear Graveyard every epoch based on the information about TS in the
network.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
2583f608e8
[ #1318 ] metabase: Add DropGraves
method
...
Method clean up Graveyard and should be called by the routine that handles
tombstone expiration.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
daab30c391
[ #1318 ] metabase: Add iteration offset
...
Add offset element to the iterations over deleted objects (both the
Graveyard and the Garbage buckets).
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00