Commit graph

635 commits

Author SHA1 Message Date
Evgenii Stratonikov
5cf75404dc [#1818] metabase: Add UpdateStorageID operation
By default writecache puts the whole object to update storage ID.
This logic comes from the times when we needed to put objects
in the metabase by the writecache itself. Now this is done by the
blobstor at unmarshaling objects during flush only to update storage ID
is an overkill.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
b64b14eb54 [#1818] writecache: Reuse FSTree flushing code between flushes
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
a56927e3d4 [#1818] writecache: Remove unused variable
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
98a152256b [#1992] writecache: Allow to open in NOSYNC mode
Applicable only to FSTree as we cannot handle corrupted databases
properly yet.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Evgenii Stratonikov
f564430b90 [#1992] fstree: Allow working in SYNC mode
Make O_SYNC the default and allow to opt-out explicitly.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Evgenii Stratonikov
34501685b7 [#1969] local_object_storage: Move ErrObjectIsExpired to another package
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
56de2f1363 [#1969] local_object_storage: Simplify logic error construction
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
fcdbf5e509 [#1969] local_object_storage: Add a type for logical errors
All logic errors are wrapped in `logicerr.Logical` type and do not
affect shard error counter.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Leonard Lyubich
e8c5f03c30 [#1905] shard: Don't log read-only errors of write-cache
There is no need to log `writecache.ErrReadOnly` errors in `Delete`
method of the `Shard`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:45 +03:00
Leonard Lyubich
b1fa084756 [#1905] shard: Decrease severity level of write-cache failure logs
In previous implementation `Shard.Delete` logged writecache's removal
failures in `error` level. There is a need to decrease severity of these
log records since they aren't critical and don't require individual
review.

Change level of the message to `info`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:45 +03:00
Evgenii Stratonikov
1e6588e761 [#1974] shard: Do not panic in degraded mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00
Evgenii Stratonikov
7395ab8ef7 [#1944] metabase: Assume static buckets are created on Init
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00
Evgenii Stratonikov
e4bc3d0e9d [#1944] metabase: Recreate static buckets instead of resetting
From the `Bucket.ForEach` doc:
```
The provided function must not modify the bucket; this will result in undefined behavior.
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00
Evgenii Stratonikov
3b939d190c [#1957] engine: Move shard to read-only if cannot move to degraded
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 08:20:53 +03:00
Evgenii Stratonikov
713fdab177 [#1907] shard: Return from Close after GC has stopped
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-25 11:54:45 +03:00
Pavel Karpy
41f93dcc1d [#1953] blobstor: Log any object removal operation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:52:18 +03:00
Pavel Karpy
942cc38146 [#1953] engine: Add refactored storage helpers
After the refactor there are new storage characteristics: a type and
a general storage id (that could be stringified).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:52:18 +03:00
Pavel Karpy
0371d15b2f [#1953] shard: Fix debug log messages
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:52:18 +03:00
Pavel Karpy
f8180447a1 [#1938] meta: Make version error messages more descriptive
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:46:18 +03:00
Evgenii Stratonikov
1beafea0b5 [#1869] shard: Add logs for SetMode operations on reload
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
87be4f1629 [#1869] shard: Restore shard mode on failed reloads
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
c785e11b20 [#1869] shard: Allow to reload metabase on SIGHUP
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
f769fc83fc [#1869] shard: Embed gcCfg as raw struct
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Pavel Karpy
49c38d642d [#1902] engine: Search for the tree IDs in every shard
Iterate over every shard and search for the container's trees. Final result
is a concatenation of shards' results. It is considered that one fixed tree
is placed on one fixed shard but the different trees of a fixed container
could be placed on different shards.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
24e9e3f3bf [#1902] engine, shard: Implement TreeList method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
2e199c7ab1 [#1902] shard: Fix pilorama disabled err message
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
19850ef157 [#1902] pilorama: Add TreeList method
To both `bolt` and `memory` forests; extend `Forest` interface.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Evgenii Stratonikov
1cb892c579 [#1910] .golangci.yml: Add misspell linker
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
d772e35aba [#1910] .golangci.yml: Add godot linker
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Pavel Karpy
f76516a883 [#1860] meta: Fix 0,1 -> 2+ version migration
In the 2nd version, there was a database format change: buckets have changed
their keys, so it becomes impossible to check the version in the 1 -> 2+
migrations because of different buckets that store info about the version.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 13:42:05 +03:00
Pavel Karpy
31c623636d [#1863] node: Fix shard id in the object counter metrics
If shard ID is stored in metabase (it is not the first time boot), read it,
set it, use it (not a generated one) in the metrics writer.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 13:06:41 +03:00
Pavel Karpy
f037022a7a [#1770] logger: Refactor Logger component
Make it store its internal `zap.Logger`'s level. Also, make all the
components to accept internal `logger.Logger` instead of `zap.Logger`; it
will simplify future refactor.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Evgenii Stratonikov
19c0a74e94 [#1867] services/control: Allow to provide multiple shard IDs to some commands
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
Evgenii Stratonikov
2d43892fc9 [#1840] neofs-node: Use blobstor paths to identify shard
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
4b005d3178 [#1840] blobstor: Return info about all components
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
6d7ffefec5 [#1840] blobstor/test: Add tests for auxiliary functions
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
328691c94f [#1840] blobstor: Return Path from components
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
9b241e4a17 [#1840] neofs-node: Allow to use mode: disabled in config
Currently, when removing shard special care must be taken with respect
to shard numbering. `mode: disabled` allows to leave shard configuration
in place while also ignoring it during initialization. This makes
disk replacement much more convenient.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
b2aa9947c2 [#1829] engine: Delete split objects properly
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 16:35:46 +03:00
Evgenii Stratonikov
6557f5d249 [#1839] engine: Handle Inhume errors properly
If shard is in read-only or degraded mode, there is no need to increase
error counter.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 14:25:52 +03:00
Pavel Karpy
4eb0ed11f8 [#1809] node: Do not boot up if metabase is outdated
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 12:32:10 +03:00
Evgenii Stratonikov
8b3b16fe62 [#1825] writecache: Flush cache when moving to the DEGRADED mode
Degraded mode allows us to operate without an SSD,
thus writecache should be unavailable in this mode.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 12:13:09 +03:00
Evgenii Stratonikov
2e3ef817f4 [#1819] engine: Increase error counter for PUT errors
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
Evgenii Stratonikov
b89e71fa78 [#1819] common: Add ErrNoSpace
Add a common error for this case because it is not an error
which should increase error counter. Single error simplifies checks on
the call-site.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
Evgenii Stratonikov
af56574849 [#1819] engine: Fix error counter in Inhume
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
Pavel Karpy
8ebe95747e [#1770] node: Do not lock on shard's Close call
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
887afeaddb [#1770] engine: Do not lock on shard init
Init can take a lot of time. Because the mutex is taken, all new operations
are blocked.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
fbd5bc8c38 [#1770] engine: Support configuration reload
Currently, it only supports changing the compound of the shards.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
5c69e19016 [#1770] fstree: Depth parameter int -> uint64
Negative values have no sense. On the other hand it differs from the
blobovnicza's configuration and prevents unification.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
2d7166f8d0 [#1770] shard: Move NewEpoch event routing on SE level
It will allow dynamic shard management. Closing a shard does not allow
removing event handlers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00