s.makhov/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Pull requests Activity
3145 commits 8 branches 70 tags 41 MiB
Commit graph

186 commits

Author SHA1 Message Date
Pavel Karpy
49c38d642d [#1902] engine: Search for the tree IDs in every shard 
Iterate over every shard and search for the container's trees. Final result
is a concatenation of shards' results. It is considered that one fixed tree
is placed on one fixed shard but the different trees of a fixed container
could be placed on different shards.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-20 16:17:57 +03:00
Pavel Karpy
24e9e3f3bf [#1902] engine, shard: Implement TreeList method 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-20 16:17:57 +03:00
Pavel Karpy
19850ef157 [#1902] pilorama: Add TreeList method 
To both `bolt` and `memory` forests; extend `Forest` interface.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-20 16:17:57 +03:00
Evgenii Stratonikov
d772e35aba [#1910] .golangci.yml: Add godot linker 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-18 15:08:26 +03:00
Pavel Karpy
31c623636d [#1863] node: Fix shard id in the object counter metrics 
If shard ID is stored in metabase (it is not the first time boot), read it,
set it, use it (not a generated one) in the metrics writer.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-13 13:06:41 +03:00
Pavel Karpy
f037022a7a [#1770] logger: Refactor Logger component 
Make it store its internal `zap.Logger`'s level. Also, make all the
components to accept internal `logger.Logger` instead of `zap.Logger`; it
will simplify future refactor.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-12 18:11:05 +03:00
Evgenii Stratonikov
19c0a74e94 [#1867] services/control: Allow to provide multiple shard IDs to some commands 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-12 11:20:48 +03:00
Evgenii Stratonikov
2d43892fc9 [#1840] neofs-node: Use blobstor paths to identify shard 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
4b005d3178 [#1840] blobstor: Return info about all components 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
b2aa9947c2 [#1829] engine: Delete split objects properly 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-07 16:35:46 +03:00
Evgenii Stratonikov
6557f5d249 [#1839] engine: Handle Inhume errors properly 
If shard is in read-only or degraded mode, there is no need to increase
error counter.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-07 14:25:52 +03:00
Evgenii Stratonikov
2e3ef817f4 [#1819] engine: Increase error counter for PUT errors 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-04 10:11:52 +03:00
Evgenii Stratonikov
af56574849 [#1819] engine: Fix error counter in Inhume 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-10-04 10:11:52 +03:00
Pavel Karpy
8ebe95747e [#1770] node: Do not lock on shard's Close call 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-04 10:08:55 +03:00
Pavel Karpy
887afeaddb [#1770] engine: Do not lock on shard init 
Init can take a lot of time. Because the mutex is taken, all new operations
are blocked.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-04 10:08:55 +03:00
Pavel Karpy
fbd5bc8c38 [#1770] engine: Support configuration reload 
Currently, it only supports changing the compound of the shards.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-04 10:08:55 +03:00
Pavel Karpy
2d7166f8d0 [#1770] shard: Move NewEpoch event routing on SE level 
It will allow dynamic shard management. Closing a shard does not allow
removing event handlers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-10-04 10:08:55 +03:00
Evgenii Stratonikov
0a411908ee [#1806] writecache: Allow to ignore read errors during flush 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
3d882e9f47 [#1806] engine: Allow to flush write-cache 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
a49137349b [#1731] engine: Allow to use user handler for evacuated objects 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-24 13:47:48 +03:00
Evgenii Stratonikov
3df98ce7ba [#1731] engine: Return the amount of actually moved objects in Evacuate 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
a51b76056e [#1731] engine: Add Evacuate command 
Make it possible to move all data from 1 shard to other shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
7377979e12 [#1731] engine: Move single shard PUT to a separate function 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
5321f8ef9c [#1786] engine: Unify parameter setters 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-15 10:28:48 +03:00
Evgenii Stratonikov
b064fb24d8 [#1616] engine: Do not use batches in delete 
Use a simple loop at the callsite. This way we remove as much as we can.
Also, `Delete` metrics is more meaningful now.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-15 10:28:48 +03:00
Pavel Karpy
beb7f2a048 [#1634] node: Change default epoch in tests 
Do not treat objects with expiration as expired by default.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 21:32:37 +04:00
Pavel Karpy
edef26a4fd [#1658] engine: Update metrics interfaces 
It now supports typed object counter metrics.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-09-13 21:32:37 +04:00
Evgenii Stratonikov
a2bb3a2a96 [#1630] pilorama: Support dropping trees 
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-09-12 09:54:15 +03:00
Pavel Karpy
9c7b3ce799 [#1658] node: Read metrics from meta on startup 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-25 19:20:33 +03:00
Pavel Karpy
745f72fff0 [#1658] node: Support object counter metrics 
Increment shard's object counter on successful `Put` calls and decrement on
`Delete`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-25 19:20:33 +03:00
Pavel Karpy
30341f2192 [#1687] *: Perform go fmt using go v1.19 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-22 18:59:57 +03:00
Evgenii Stratonikov
26b4a258e0 [#1523] neofs-node: Refactor configuration 
1. Move compression parameters to the `shard` section.
2. Allow to use multiple sub-storage components in the blobstor.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-08-22 13:14:19 +03:00
Pavel Karpy
156ba85326 [#1634] node: Do not return expired objects 
If an object has not been marked for removal by the GC in the current epoch
yet but has already expired, respond with `ErrObjectNotFound` api status.
Also, optimize shard iteration: a node must stop any iteration if the object
 is found but gonna be removed soon.
All the checks are performed by the Metabase.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-04 16:31:49 +03:00
Pavel Karpy
9aba0ba512 [#1634] meta: Add epoch state 
It allows performing expiration checks on the stored objects.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-08-04 16:31:49 +03:00
Evgenii Stratonikov
8ffc2fdf5e [#1642] engine: Do not increase error counter if the pilorama is disabled 
After a4adb79db new logical error could be returned. Do not increase
error counter in this case.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
 2022-07-30 18:39:22 +03:00
Evgenii Stratonikov
4944490ffb [#1559] local_object_storage: Move shard to the DegradedReadOnly mode 
`Degraded` mode can be set by the administrator if needed.
Modifying operations in this mode can lead node into an inconsistent state
because metabase checks such as lock checking are not performed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
3df62769c0 [#1559] local_object_storage: Allow to set mode for all components 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Leonard Lyubich
e38b0aa4ba [#1549] engine: Disable shard on blobovnicza init failure 
There is a need to support working w/o shard if it has problems with
blobovnicza tree.

Make `BlobStor.Init` to return new `ErrInitBlobovniczas` error. Remove
shard from storage engine's shard set if it returned this error from
`Init` call. So if some of the shards (but not all) return this error,
the node will be able to continue working without them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
40a56c6b42 [#1559] engine: Do not count logical errors as storage ones 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
c8911d72d0 [#1559] shard: Do not consult metabase in a degraded mode 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
339864b720 [#1559] local_object_storage: Move shard.Mode to a separate package 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
d8ba954aff [#1559] shard: Use Set prefix for parameter setting 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
7b882b26d8 [#1559] shard: Remove public functions 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
26041f18bf [#1505] pilorama: Allow to customize database parameters 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
c7b10598f9 [#1333] engine: Increase error counter for pilorama errors 
1. Modifying operations are not expected to fail, unless the shard is
   read-only.
2. `Get*` operations should increase error counter too, unless the
   error is `ErrTreeNotFound`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8f4ee1aded [#1333] local_object_storage: Support ReadOnly mode in pilorama 
The tricky part here is the engine itself: we stop iteration on
`ErrReadOnly` because it is better to synchronize the shard later than
to have partial trees stored in 2 shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
4437cd7113 [#1442] pilorama: Generate timestamp based on node position in the container 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f5d35571d0 [#1431] engine: Add benchmark for Select vs TreeGetByPath 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
536857ea5a [#1329] services/tree: Implement GetOpLog RPC 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad48918a97 [#1406] pilorama: Return parent from TreeGetMeta 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00