c785e11b20
[ #1869 ] shard: Allow to reload metabase on SIGHUP
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
49c38d642d
[ #1902 ] engine: Search for the tree IDs in every shard
...
Iterate over every shard and search for the container's trees. Final result
is a concatenation of shards' results. It is considered that one fixed tree
is placed on one fixed shard but the different trees of a fixed container
could be placed on different shards.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
24e9e3f3bf
[ #1902 ] engine, shard: Implement `TreeList` method
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
19850ef157
[ #1902 ] pilorama: Add `TreeList` method
...
To both `bolt` and `memory` forests; extend `Forest` interface.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
d772e35aba
[ #1910 ] .golangci.yml: Add `godot` linker
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
31c623636d
[ #1863 ] node: Fix shard id in the object counter metrics
...
If shard ID is stored in metabase (it is not the first time boot), read it,
set it, use it (not a generated one) in the metrics writer.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 13:06:41 +03:00
f037022a7a
[ #1770 ] logger: Refactor `Logger` component
...
Make it store its internal `zap.Logger`'s level. Also, make all the
components to accept internal `logger.Logger` instead of `zap.Logger`; it
will simplify future refactor.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
19c0a74e94
[ #1867 ] services/control: Allow to provide multiple shard IDs to some commands
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
2d43892fc9
[ #1840 ] neofs-node: Use blobstor paths to identify shard
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
4b005d3178
[ #1840 ] blobstor: Return info about all components
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
b2aa9947c2
[ #1829 ] engine: Delete split objects properly
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 16:35:46 +03:00
6557f5d249
[ #1839 ] engine: Handle Inhume errors properly
...
If shard is in read-only or degraded mode, there is no need to increase
error counter.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 14:25:52 +03:00
2e3ef817f4
[ #1819 ] engine: Increase error counter for PUT errors
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
af56574849
[ #1819 ] engine: Fix error counter in Inhume
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
8ebe95747e
[ #1770 ] node: Do not lock on shard's `Close` call
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
887afeaddb
[ #1770 ] engine: Do not lock on shard init
...
Init can take a lot of time. Because the mutex is taken, all new operations
are blocked.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
fbd5bc8c38
[ #1770 ] engine: Support configuration reload
...
Currently, it only supports changing the compound of the shards.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
2d7166f8d0
[ #1770 ] shard: Move NewEpoch event routing on SE level
...
It will allow dynamic shard management. Closing a shard does not allow
removing event handlers.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
0a411908ee
[ #1806 ] writecache: Allow to ignore read errors during flush
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
3d882e9f47
[ #1806 ] engine: Allow to flush write-cache
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
a49137349b
[ #1731 ] engine: Allow to use user handler for evacuated objects
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-24 13:47:48 +03:00
3df98ce7ba
[ #1731 ] engine: Return the amount of actually moved objects in `Evacuate`
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
a51b76056e
[ #1731 ] engine: Add Evacuate command
...
Make it possible to move all data from 1 shard to other shards.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
7377979e12
[ #1731 ] engine: Move single shard PUT to a separate function
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
5321f8ef9c
[ #1786 ] engine: Unify parameter setters
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
b064fb24d8
[ #1616 ] engine: Do not use batches in delete
...
Use a simple loop at the callsite. This way we remove as much as we can.
Also, `Delete` metrics is more meaningful now.
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
beb7f2a048
[ #1634 ] node: Change default epoch in tests
...
Do not treat objects with expiration as expired by default.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
edef26a4fd
[ #1658 ] engine: Update metrics interfaces
...
It now supports typed object counter metrics.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
a2bb3a2a96
[ #1630 ] pilorama: Support dropping trees
...
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
9c7b3ce799
[ #1658 ] node: Read metrics from meta on startup
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
745f72fff0
[ #1658 ] node: Support object counter metrics
...
Increment shard's object counter on successful `Put` calls and decrement on
`Delete`.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
30341f2192
[ #1687 ] *: Perform `go fmt` using go `v1.19`
...
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
26b4a258e0
[ #1523 ] neofs-node: Refactor configuration
...
1. Move compression parameters to the `shard` section.
2. Allow to use multiple sub-storage components in the blobstor.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
156ba85326
[ #1634 ] node: Do not return expired objects
...
If an object has not been marked for removal by the GC in the current epoch
yet but has already expired, respond with `ErrObjectNotFound` api status.
Also, optimize shard iteration: a node must stop any iteration if the object
is found but gonna be removed soon.
All the checks are performed by the Metabase.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
9aba0ba512
[ #1634 ] meta: Add epoch state
...
It allows performing expiration checks on the stored objects.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
8ffc2fdf5e
[ #1642 ] engine: Do not increase error counter if the pilorama is disabled
...
After a4adb79db
2022-07-30 18:39:22 +03:00
4944490ffb
[ #1559 ] local_object_storage: Move shard to the `DegradedReadOnly` mode
...
`Degraded` mode can be set by the administrator if needed.
Modifying operations in this mode can lead node into an inconsistent state
because metabase checks such as lock checking are not performed.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
3df62769c0
[ #1559 ] local_object_storage: Allow to set mode for all components
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
e38b0aa4ba
[ #1549 ] engine: Disable shard on blobovnicza init failure
...
There is a need to support working w/o shard if it has problems with
blobovnicza tree.
Make `BlobStor.Init` to return new `ErrInitBlobovniczas` error. Remove
shard from storage engine's shard set if it returned this error from
`Init` call. So if some of the shards (but not all) return this error,
the node will be able to continue working without them.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
40a56c6b42
[ #1559 ] engine: Do not count logical errors as storage ones
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
c8911d72d0
[ #1559 ] shard: Do not consult metabase in a degraded mode
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
339864b720
[ #1559 ] local_object_storage: Move `shard.Mode` to a separate package
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
d8ba954aff
[ #1559 ] shard: Use `Set` prefix for parameter setting
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
7b882b26d8
[ #1559 ] shard: Remove public functions
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
26041f18bf
[ #1505 ] pilorama: Allow to customize database parameters
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
c7b10598f9
[ #1333 ] engine: Increase error counter for pilorama errors
...
1. Modifying operations are not expected to fail, unless the shard is
read-only.
2. `Get*` operations should increase error counter too, unless the
error is `ErrTreeNotFound`.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
8f4ee1aded
[ #1333 ] local_object_storage: Support ReadOnly mode in pilorama
...
The tricky part here is the engine itself: we stop iteration on
`ErrReadOnly` because it is better to synchronize the shard later than
to have partial trees stored in 2 shards.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
4437cd7113
[ #1442 ] pilorama: Generate timestamp based on node position in the container
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
f5d35571d0
[ #1431 ] engine: Add benchmark for `Select` vs `TreeGetByPath`
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
536857ea5a
[ #1329 ] services/tree: Implement `GetOpLog` RPC
...
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
Pavel Karpy
Evgenii Stratonikov
Leonard Lyubich