Commit Graph

3238 Commits (aadd2ad050fc089f5eb1d254b36ae7c37c027506)

Author SHA1 Message Date
Evgenii Stratonikov 8fc88487db [#1731] neofs-cli: Add `control shards evacuate` command
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov 091d7d30f6 [#1731] services/control: Allow to evacuate data from shard
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov a51b76056e [#1731] engine: Add Evacuate command
Make it possible to move all data from 1 shard to other shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov 7377979e12 [#1731] engine: Move single shard PUT to a separate function
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov 4208f7c0cf [#1741] neofs-adm: Use `ValidUntilBlock` to wait for tx persist
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 14:27:16 +03:00
Evgenii Stratonikov 5321f8ef9c [#1786] engine: Unify parameter setters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov b064fb24d8 [#1616] engine: Do not use batches in delete
Use a simple loop at the callsite. This way we remove as much as we can.
Also, `Delete` metrics is more meaningful now.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov cda8f9df2e [#1786] services/control: Remove `WithDeletedObjectHandler` option
Use storage engine directly instead. It is already provided in the
options.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov ae1dab29bc [#1482] metabase: Encode database keys in binary
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:26 +03:00
Leonard Lyubich d6fef68a62 Release v0.32.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-14 17:52:49 +04:00
Leonard Lyubich d895a272ad docs: Improve release instructions
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-14 17:52:49 +04:00
Leonard Lyubich ae655b74f0 *: Exec `make fmts`
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-14 17:52:49 +04:00
Evgenii Stratonikov 4b8b575ca4 [#1782] go.mod: Update neofs-sdk-go
Add fixes for key parameter in `object.Get` operation.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-14 13:18:19 +04:00
Pavel Karpy 532c41ca05 [#1658] meta: Do not check object expiration in counters
Do not perform operations that produce unused results in `syncCounter`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy 431e331373 [#1658] shard: Update metric counters
Use meta's operation results to change the metrics. Support typed object
counters.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy ad47e2a985 [#1658] meta: Add logic counter
- Meta now supports (and requires) inc/dec labeled counters
- The new logic counter is incremented on `Put` operations and is
decremented on `Inhume` and `Delete` operations that are performed on
_stored_ objects only
- Allow force counters sync. "Force" mode should be used on metabase resync
and should not be used on a regular meta start

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy d872862710 [#1658] meta: Force counters resync process
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy beb7f2a048 [#1634] node: Change default epoch in tests
Do not treat objects with expiration as expired by default.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy edef26a4fd [#1658] engine: Update metrics interfaces
It now supports typed object counter metrics.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy 175e2b9fa0 [#1658] metrics: Add object type label to the counters
It will allow to use labeled object counters, e.g.: "phy" for all
the physically stored objects, "logic" for only available objects. Also, it
will allow to add typed (regular, TS, SG, LOCK) counters if needed.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Leonard Lyubich a789159e5b [#1189] config/example: Mention `persistent_sessions.path` default
In current implementation storage node app uses in-memory session
storage if `persistent_sessions.path` value is empty/missing in the
config.

Clarify default behavior in `config/example/node.yaml`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-13 21:29:04 +04:00
Leonard Lyubich d25b7e177b [#1189] node/config: Fix `PersistentSessionsConfig.Path` method's docs
Method never returns `PersistentStatePathDefault` value, and this is
correct.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-13 21:29:04 +04:00
Pavel Karpy 4083a62679 [#1628] tree: Pre-allocate errors in ACL checks
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy 5adf089c1d [#1628] tree: Log unacceptable bearer attachment
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy 2ffcd02ac3 [#1628] tree: Document ACL checks in tree service
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy 8d0906c6ab [#1628] tree: Skip eACL filters for tree requests
Do not call `CalculateAction` for the eACL checks since it requires object
headers that are meaningless in the tree context.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy 876e014b5d [#1628] tree: Make ACL checks the same way as for object requests
1. Do not require a request to be signed by the container owner if a
bearer token is missing
2. Do not check the system role since public requests are not expected to
be signed by IR or a container node (unlike the object requests)

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy 4f18893d9b [#1628] node: Move common `EACLSource` interface to `core` pkg
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy 4afb928ab6 [#1674] *: Expire entities after the expiration epoch
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-12 19:56:59 +04:00
Evgenii Stratonikov f1572a674b [#1775] go.mod: Update neofs-sdk-go
Make use of fixes for the placement policy parser.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 19:10:49 +04:00
Evgenii Stratonikov d25ec52459 [#1778] neofs-node: Panic if config type is invalid
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 17:06:00 +03:00
Evgenii Stratonikov 60b2930417 [#1778] neofs-node: Fix blobovnicza config type
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 17:06:00 +03:00
Evgenii Stratonikov 9da5d784cb [#1630] neofs-node: Remove trees on container removal event
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Evgenii Stratonikov 87bd49563e [#1761] pilorama: Use batch size of 1 for tests
Improve tests speed by a lot and use more iterations in `ApplyRandom` test.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Evgenii Stratonikov a2bb3a2a96 [#1630] pilorama: Support dropping trees
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Leonard Lyubich 699b534416 [#1763] node/netmap: Write log message about parsed NewEpoch event
There is a need to have the ability to track NeoFS timeline on storage
nodes. Epochs tick on notifications receipt, so the most obvious way to
know about received epochs is logging the events.

Wrap `morphEvent.ParseNewEpoch` event parser into function which writes
log message about new epoch number.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-12 09:53:05 +04:00
Leonard Lyubich 3a206b5d7a [#1763] morph/event: Write log message about new notification event
Sometimes it is useful to track sidechain events processed by the node.
For example, we need some easy way to look up for log messages about new
epoch notifications.

Make `subscriber.Subscriber` to log names of all events received from
the sidechain notification channel.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-12 09:53:05 +04:00
Leonard Lyubich 8e8265d5ea [#1707] blobovnicza/get: Iterate over all buckets regardless of config
Re-configuration of Blobovnicza's object size limit must not affect
already stored objects. In previous implementation `Blobovnicza` didn't
see objects stored in buckets which became too big after size limit
re-configuration. For example, lets consider 1st configuration with 64KB
size limit for stored objects. Lets assume that we stored object of 64KB
size, and re-configured `Blobovnicza` with 32KB limit. After reboot
object should be still available, but actually it isn't. This is caused
by `Get` operation algorithm which iterates over configured size ranges
only, and doesn't process any other existing size bucket. By the way,
increasing of the object size limit didn't lead to the problem even in
previous implementation.

Make `Blobovnicza.Get` method to iterate over all size buckets
regardless of current configuration. This covers the described scenario.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-08 12:24:28 +04:00
Pavel Karpy 8e6e89aca3 [#1711] amd: Cache committee actor in the init context
Also simplify the code using `invoker.Invoker`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Pavel Karpy 60aa53651b [#1711] adm: Append witnesses only if they are missing
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Pavel Karpy 1e57565f6c [#1711] adm: Implement local client's methods
Implement methods that `actor.Actor` requires:
1. `InvokeFunction` -- wrapper over `InvokeScript`
2. `GetVersion` returns default struct
3. `CalculateNetworkFee` copied and simplified from Neo-go server side.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Pavel Karpy 4bd8608b37 [#1711] *: Simplify code using neo-go actors
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Evgenii Stratonikov 4354359aed [#1746] network: Set timeout for streaming operations
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 16:00:51 +04:00
Evgenii Stratonikov 0140ac354b [#1759] services/tree: Make logs more descriptive
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 12:57:33 +04:00
Evgenii Stratonikov 74c861342e [#1753] services/tree: Add parent ID to the `GetNodeByPath` response
Currently, you need to use `GetSubTree` to get parent, which seems an
overkill.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 10:52:37 +03:00
Evgenii Stratonikov 8a47f9ba12 [#1753] services/tree: Remove MaxGetSubTreeCount constant
It was used only in `GetOpLog` and ignored on the server side.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 10:52:37 +03:00
Evgenii Stratonikov 9840936a4f [#1753] services/tree: Do not restrict depth in `GetSubTree`
Previously, the depth was restricted because with BFS the amount of
nodes we have in memory blows up exponentially. With DFS is is linear,
so we can process trees of arbitrary depth.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 10:52:37 +03:00
Alexey Vanin c4c2840b52 [#1632] neofs-adm: Update autogenerated storage node config
Replace `diable_cache` with `cache_ttl` value.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-09-07 10:34:46 +03:00
Alexey Vanin 944665dd5d [#1632] neofs-node: Update docs
Replace `diable_cache` with `cache_ttl` value

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-09-07 10:34:46 +03:00
Evgenii Stratonikov e4dcc4d6a9 [#1749] neofs-adm: Register contract hashes in NNS in 2 formats
NNS proposal describes string N3 address format, however we must also
have hex-string for backwards compatibility.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 09:58:05 +03:00