Commit graph

396 commits

Author SHA1 Message Date
Evgenii Stratonikov
cbf8ca12b0 [#859] config: rename refill_metabase to resync_metabase
This flag enables metabase content synchronization with blobstor on
start.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Leonard Lyubich
fb89d29574 [#969] node: Do not require shard config in relay mode
Relay storage node doesn't exec local object operations, so it doesn't need
shard configuration.

Add `required` bool parameter to `engineconfig.IterateShards`. Make it to
panic if it is `true`, and immediately return otherwise. Pass `false` if
node is configured as relay in app (it also prevents panic).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-15 06:47:25 +03:00
Evgenii Stratonikov
95893927aa *: replace neofs-api-go with neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-12 17:29:09 +03:00
Leonard Lyubich
10f0bd91d6 [#922] engine: Change interface of container operations
Add `error` to return. Improve docs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
6b1ce99c35 [#922] node: Block data ops of storage engine via Control API
There is a need to block execution of local object storage operations if
node is put into maintenance mode (resume if the node is taken out of
maintenance mode).

Call `BlockExecution` method if `ControlService.SetNetmapStatus` was called
with `MAINTENANCE` status. Call `ResumeExecution` if it was called with
another status.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
0ec8f529ab [#842] object/delete: Set tombstone local node as tombstone owner
All objects in NeoFS must have owner ID. In previous implementation Object
Delete service handler set owner ID from request session token. If removal
was executed w/o a session, object with tombstone was prepared incorrectly.
In order to fix this node should set its own ID and become an owner of the
tombstone object.

Extend `NetworkInfo` interface required by Object.Delete handler with
`LocalNodeID` method which returns `owner.ID` of the local node. Implement
the method on `networkState` component of storage node application which is
updated on each node state change in NeoFS network map. Set owner returned
by `LocalNodeID` call as tombstone object's owner in Delete handler.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 16:18:02 +03:00
Leonard Lyubich
e500efb9b3 [#957] services/reputation: Refactor usage of NeoFS API client
The client needs of the Reputation service are limited and change not often.
Interface changes of the client library should not affect the operation of
various service packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the Reputation service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-03 18:30:09 +03:00
Evgenii Stratonikov
49c9dbfba8 [#877] config/engine: use default section for shards
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-28 15:04:14 +03:00
Evgenii Stratonikov
cb9bf00ceb [#877] neofs-node/config: allow to provide default values
In case we have multiple sections with similar structure (e.g. shards)
having defaults in a single place is easier to work with.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-28 15:04:14 +03:00
Alex Vanin
fbed86da2c [#943] service/object: Remove KeyStore from CommonPrm
There is no point to pass key storage in parameters because
it can be defined on the service level of application.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
2fbdcbdee1 [#943] service/object: Check session token expiration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Evgenii Stratonikov
41a9261069 [#935] config/example: quote spaces in node.env
```
> source config/example/node.env
config/example/node.env: строка 14: /dns4/s02.neofs.devenv/tcp/8081: Нет такого файла или каталога
config/example/node.env: строка 16: MSK: команда не найдена
config/example/node.env: строка 35: 028f42cfcb74499d7b15b35d9bff260a1c8d27de4f446a627406a382d8961486d6: команда не найдена
config/example/node.env: строка 48: https://rpc2.morph.fs.neo.org:40341: Нет такого файла или каталога
config/example/node.env: строка 49: wss://rpc2.morph.fs.neo.org:40341/ws: Нет такого файла или каталога
config/example/node.env: строка 53: https://rpc2.n3.nspcc.ru:30341: Нет такого файла или каталога
```

New implementation is still basic, but we don't need to parse any env
file, just have a correct example.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 11:10:01 +03:00
Pavel Karpy
208d150500 [#937] node: Init node state before gRPC server start
Fix panic when node processes RPC requests
before state initialization is finished.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-22 15:05:00 +03:00
Leonard Lyubich
283ccc04b4 [#833] services/netmap: Support new fields of netmap.NetworkInfo
Make the implementation of network info source (Netmap V2 service
dependency) to read MillisecondsPerBlock sidechain parameter and NeoFS
network parameters depending on the client version.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Pavel Karpy
2f2258733b [#873] node: Delete useless notary deposit Amount, Duration envs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
8374c3d2f5 [#873] node: Delete useless notary timer
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
2a3a6cc0ba [#873] node: Use dynamic notary deposit duration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
5e816dc01a [#873] node: Make deposits depend on epoch and balances
SN tries to keep 1:3 proportion of GAS and
notary balances respectively. If that proportion
has been messed(means that notary balance is
lower than required) it sends half of its
GAS balance to the notary service.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Leonard Lyubich
40a4a7faa2 [#674] object/put: Use pseudo worker pool for local operations
After storage engine started to limit number of PUT operations there is no
need to limited worker pool in Object Put service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
2126235f0e [#674] node: Configure size of per-shard worker pools
Add `shard_pool_size` config to `storage` section. Set app default to 20.
Pass the value to `WithShardPoolSize` option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Evgenii Stratonikov
cc377b34d2 [#868] blobstor: initialize (de-)compressors in Init
Do not log in options constructors. Also failure to
initialize compression module (possibly due to invalid options) is
certainly an error deserving proper treatment.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 16:01:45 +03:00
Evgenii Stratonikov
00f14d4dcd [#857] config: use size suffixes where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 11:26:24 +03:00
Evgenii Stratonikov
a0abf10c8e [#857] config: allow to parse sizes with suffix
The code is taken from viper as `cast` package that we use
doesn't have needed converter.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 11:26:24 +03:00
Alex Vanin
e41e74b5fc [#872] services/container: Ignore passed routes in load router
Higher level solution for original issue with lost announcements
from single node container.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-05 19:29:27 +03:00
Alex Vanin
f08636c518 [#872] cmd/neofs-node: Don't lose local container size estimations
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-05 19:29:27 +03:00
Leonard Lyubich
312e356a75 [#645] object: Compare public keys in reputation client constructor
In previous implementation of reputation client constructor compared network
addresses to check the membership of the network map.

Replace network addresses comparison with public key comparison.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
e473f3ac91 [#645] *: Use helper functions to build client.NodeInfo structures
Helper functions from core/client package allow to set public keys of
storage nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
7b228b7603 [#645] *: Construct clients from client.NodeInfo in API client cache
There is a need to have the ability to expand the data needed for client
construction.

Replace `network.AddressGroup` parameter of client cache interfaces with
`client.NodeInfo`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-30 20:57:00 +03:00
Leonard Lyubich
287683c590 [#863] node: Do not open main chain client connection
In current implementation storage node doesn't use main chain client.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-29 12:55:48 +03:00
Leonard Lyubich
50c4c5116d [#863] node: Prevent panic about missing Neo RPC endpoints
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-29 12:55:48 +03:00
Alex Vanin
409b72cb5b [#786] neofs-node: Ignore proxy contract in notary disabled environment
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-27 16:20:31 +03:00
Leonard Lyubich
0e3e8db5c0 [#845] node: Configure Object PUT remote and local pools separately
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 16:20:19 +03:00
Leonard Lyubich
ee20200c2e [#845] object/put: Separate pools for local and remote operations
In previous implementation Object PUT used single pool of workers for local
and remote ops, but these ops are heterogeneous.

Use remote/local pool for remote/local operations in PUT service. At first
the pools are configured with the same size.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-27 16:20:19 +03:00
Leonard Lyubich
d938c7267b [#789] node/config: Add Shard's refill_metabase config to examples
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
b5fff810f4 [#789] cmd/node: Add refill_metabase config to shard section
Implement `RefillMetabase` method to `shardconfig.Config` type which reads
`refill_metabase` config value. Pass the result to `WithRefillMetabase`
option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:22:09 +03:00
Leonard Lyubich
368e280413 [#776] cmd/node: Configure write-cache size limit
Add `size_limit` config value to write-cache section. Add `SizeLimit` method
of `writecache.Config` type. Pass its value to `WithMaxCacheSize` option on
app construction.

Reflect config update in examples. Cover new value in unit test.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-15 18:07:36 +03:00
Alex Vanin
d840627816 [#786] cmd/neofs-node: Use NNS to find contract script hashes missing in config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin
affae68061 [#812] cmd/neofs-node: Support notary notifications in reputation service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
90823a7d05 [#812] cmd/neofs-node: Fix reputation wrapper import typo
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Pavel Karpy
288a8784d0 [#815] node: Make notary deposit before bootstrap
Bootstrap process involves `addPeer` contract
call => it is necessary to make deposit before
booting up.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Pavel Karpy
b303e49408 [#815] morph/event/netmap: Add updateState notary notification support
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Alex Vanin
01df4ffa61 [#811] service/container: Hide cache invalidation logic in Writer interface
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Alex Vanin
49b6b5b49d [#811] cmd/node: Use cache invalidator in container service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Pavel Karpy
19e97e4d7c [#807] node: Fix notary container invocation
Create non-notary container wrapper and
use it in load announcing since it shouldn't
be invoked as notary request.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Alex Vanin
1edf40f4d6 [#798] morph/subscriber: Remove RPC Init timeout
More convenient way is to fail straight away and
expect external restart.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Alex Vanin
4874b4ae92 [#798] cmd/neofs-node: Save latest processed block number
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Alex Vanin
cdb3b71070 [#798] neofs-node/config: Add persistent_state section
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Pavel Karpy
4a81781c0c [#770] logger: Delete spaces in log message keys
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
2f343a15e5 [#770] node/cmd: Add TryNotary to SN container wrapper
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
722b844aa2 [#770] node: Add notary deposit timer
Storage Node needs to have notary deposit
for successful notary request sending.
Add notary deposit on startup(and wait for
its acceptance). Add notary deposit timer,
its config in `morph` section and env vars
for its tuning.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00