Commit graph

423 commits

Author SHA1 Message Date
Pavel Karpy
fee2f5a330 [] *: Adopt new SDK's owner.ID API
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 17:58:30 +03:00
Pavel Karpy
ed156cd738 [] eacl: Use validator from SDK
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 10:47:33 +03:00
Pavel Karpy
7c02a2e251 [] *: Adopt SDK changes
- Update `neofs-sdk-go`:
v0.0.0-20211230072947-1fe37df88f80 => v0.0.0-20220113123743-7f3162110659

- Add client interface that duplicates SDK's client behaviour and new
`MultiAddressClient` interface that has method that iterates over wrapped
clients.

- Also start using simple client mode that does not require parsing statuses
outside the SDK library.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-14 17:29:03 +03:00
Evgenii Stratonikov
0d969d7a06 [] blobstor: allow to disable compression based on content-type
For some data compression makes little sense, as it is already compressed.
This commit allows to leave such data unchanged based on `Content-Type`
attribute. Currently exact, prefix and suffix matching are supported.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-13 14:20:36 +03:00
Evgenii Stratonikov
5828f43e52 [] util/rand: use single random source
It is much more convenient to skip source creation.
Also fix some bugs:
1. `cryptoSource.Int63()` now returns number in [0, 1<<63) as required
   by `rand.Source` interface.
2. Replace `cryptoSource.Uint63()` with `cryptoSource.Uint64` to allow
   generate uint64 numbers directly (see rand.Source64 docs).

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-12 19:29:27 +03:00
Leonard Lyubich
e74ff7c15a [] node/object: Change the sequence of servers
In previous implementation ACL server was the 1st (except metric server
in some cases) server in pipeline of Object service servers. This led to
the fact that errors of this handler could not be reduced to status
responses.

Nest object ACL server into signature and response servers to support
common response format.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-11 11:08:55 +03:00
Pavel Karpy
a8ba573ec8 [] node/config: Use mode from config in shard/s constructor
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-29 18:06:15 +03:00
Pavel Karpy
8b8a815fb3 [] node/config: Add shard mode config param
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-29 18:06:15 +03:00
Evgenii Stratonikov
e96eb3e00b [] neofs-node: add epoch metric
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-28 15:34:20 +03:00
Pavel Karpy
0e5410603e [] control: Add ListShards implementation to ctrl svc
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
c49e53ba9d [] morph: Add maxConnPerHost option
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 18:55:53 +03:00
Pavel Karpy
63e035bd8a [] node: Add maxConnPerHost to config
It allows configuring number of neo-go client opened connections per one
host.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 18:55:53 +03:00
Sergio Nemirowski
4ad2961c51 [] neofs-node: fix usage message
Signed-off-by: Sergio Nemirowski <sergio@nspcc.ru>
2021-12-03 10:37:24 +03:00
Evgenii Stratonikov
d1be5b5f9e [] neofs-node: default to secure TLS settings
Support TLS >=1.2 only and strong cipher suites.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-02 12:05:41 +03:00
Leonard Lyubich
0fb838b169 [] node: Configure subnets
Add `subnet` sub-section to `node` section of storage node config. Add
`entries` value which allows to enumerate subnets for entrance. Add
`exit_zero` value which allows to not enter zero subnet.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-28 16:42:46 +03:00
Alex Vanin
a74a402a7d [] policer: Implement continuous replication
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-26 15:39:38 +03:00
Leonard Lyubich
7f5fb130c0 [] *: Support NeoFS API status returns
Upgrade NeoFS API Go library to version with status returns. Make all API
clients to pull out and return errors from failed statuses. Make signature
service to respond with status if client version supports it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-24 09:26:40 +03:00
Leonard Lyubich
44d0d453ef [] client: Clarify error returns in docs
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-24 09:26:40 +03:00
Pavel Karpy
dbf3a2f2fb [] *: Add optional parameters to container morph client calls
Adapt all container wrapper calls to new
structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
bad739258e [] *: Add notification TX hash to neofs/netmap morph client calls
Add hash of the TX that generated notification
to neofs/netmap event structures. Adapt all
neofs/netmap wrapper calls to new structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Evgenii Stratonikov
aba09bb853 [] config: support floating-point sizes
Rounding is done using the suffix boundary (i.e. floating point value
with `gb` suffix will return size which is an integer number of
megabytes).

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-16 12:43:55 +03:00
Evgenii Stratonikov
e7fd980951 [] config: rename writecache.size_limit to writecache.capacity
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Evgenii Stratonikov
c015b04ed5 [] config: rename shallow_* to width and depth
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Evgenii Stratonikov
684e3e0ae0 [] config: rename mem_size to memcache_capacity
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Evgenii Stratonikov
a180f09523 [] config: rename max_size to max_object_size
`_size` suffix by itself makes me think about restrictions on database
size.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Evgenii Stratonikov
da51c119d5 [] config: rename opened_cache_size to opened_cache_capacity
Capacity is a common term to use with cache, see e.g. leveldb
configuration.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Evgenii Stratonikov
0759d8d0e5 [] config: rename small_size_limit to small_object_size
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Evgenii Stratonikov
cbf8ca12b0 [] config: rename refill_metabase to resync_metabase
This flag enables metabase content synchronization with blobstor on
start.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Leonard Lyubich
fb89d29574 [] node: Do not require shard config in relay mode
Relay storage node doesn't exec local object operations, so it doesn't need
shard configuration.

Add `required` bool parameter to `engineconfig.IterateShards`. Make it to
panic if it is `true`, and immediately return otherwise. Pass `false` if
node is configured as relay in app (it also prevents panic).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-15 06:47:25 +03:00
Evgenii Stratonikov
95893927aa *: replace neofs-api-go with neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-12 17:29:09 +03:00
Leonard Lyubich
10f0bd91d6 [] engine: Change interface of container operations
Add `error` to return. Improve docs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
6b1ce99c35 [] node: Block data ops of storage engine via Control API
There is a need to block execution of local object storage operations if
node is put into maintenance mode (resume if the node is taken out of
maintenance mode).

Call `BlockExecution` method if `ControlService.SetNetmapStatus` was called
with `MAINTENANCE` status. Call `ResumeExecution` if it was called with
another status.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
0ec8f529ab [] object/delete: Set tombstone local node as tombstone owner
All objects in NeoFS must have owner ID. In previous implementation Object
Delete service handler set owner ID from request session token. If removal
was executed w/o a session, object with tombstone was prepared incorrectly.
In order to fix this node should set its own ID and become an owner of the
tombstone object.

Extend `NetworkInfo` interface required by Object.Delete handler with
`LocalNodeID` method which returns `owner.ID` of the local node. Implement
the method on `networkState` component of storage node application which is
updated on each node state change in NeoFS network map. Set owner returned
by `LocalNodeID` call as tombstone object's owner in Delete handler.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 16:18:02 +03:00
Leonard Lyubich
e500efb9b3 [] services/reputation: Refactor usage of NeoFS API client
The client needs of the Reputation service are limited and change not often.
Interface changes of the client library should not affect the operation of
various service packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the Reputation service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-03 18:30:09 +03:00
Evgenii Stratonikov
49c9dbfba8 [] config/engine: use default section for shards
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-28 15:04:14 +03:00
Evgenii Stratonikov
cb9bf00ceb [] neofs-node/config: allow to provide default values
In case we have multiple sections with similar structure (e.g. shards)
having defaults in a single place is easier to work with.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-28 15:04:14 +03:00
Alex Vanin
fbed86da2c [] service/object: Remove KeyStore from CommonPrm
There is no point to pass key storage in parameters because
it can be defined on the service level of application.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
2fbdcbdee1 [] service/object: Check session token expiration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Evgenii Stratonikov
41a9261069 [] config/example: quote spaces in node.env
```
> source config/example/node.env
config/example/node.env: строка 14: /dns4/s02.neofs.devenv/tcp/8081: Нет такого файла или каталога
config/example/node.env: строка 16: MSK: команда не найдена
config/example/node.env: строка 35: 028f42cfcb74499d7b15b35d9bff260a1c8d27de4f446a627406a382d8961486d6: команда не найдена
config/example/node.env: строка 48: https://rpc2.morph.fs.neo.org:40341: Нет такого файла или каталога
config/example/node.env: строка 49: wss://rpc2.morph.fs.neo.org:40341/ws: Нет такого файла или каталога
config/example/node.env: строка 53: https://rpc2.n3.nspcc.ru:30341: Нет такого файла или каталога
```

New implementation is still basic, but we don't need to parse any env
file, just have a correct example.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 11:10:01 +03:00
Pavel Karpy
208d150500 [] node: Init node state before gRPC server start
Fix panic when node processes RPC requests
before state initialization is finished.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-22 15:05:00 +03:00
Leonard Lyubich
283ccc04b4 [] services/netmap: Support new fields of netmap.NetworkInfo
Make the implementation of network info source (Netmap V2 service
dependency) to read MillisecondsPerBlock sidechain parameter and NeoFS
network parameters depending on the client version.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-15 13:56:19 +03:00
Pavel Karpy
2f2258733b [] node: Delete useless notary deposit Amount, Duration envs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
8374c3d2f5 [] node: Delete useless notary timer
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
2a3a6cc0ba [] node: Use dynamic notary deposit duration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Pavel Karpy
5e816dc01a [] node: Make deposits depend on epoch and balances
SN tries to keep 1:3 proportion of GAS and
notary balances respectively. If that proportion
has been messed(means that notary balance is
lower than required) it sends half of its
GAS balance to the notary service.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-15 12:22:25 +03:00
Leonard Lyubich
40a4a7faa2 [] object/put: Use pseudo worker pool for local operations
After storage engine started to limit number of PUT operations there is no
need to limited worker pool in Object Put service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Leonard Lyubich
2126235f0e [] node: Configure size of per-shard worker pools
Add `shard_pool_size` config to `storage` section. Set app default to 20.
Pass the value to `WithShardPoolSize` option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-10-14 10:20:39 +03:00
Evgenii Stratonikov
cc377b34d2 [] blobstor: initialize (de-)compressors in Init
Do not log in options constructors. Also failure to
initialize compression module (possibly due to invalid options) is
certainly an error deserving proper treatment.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 16:01:45 +03:00
Evgenii Stratonikov
00f14d4dcd [] config: use size suffixes where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 11:26:24 +03:00
Evgenii Stratonikov
a0abf10c8e [] config: allow to parse sizes with suffix
The code is taken from viper as `cast` package that we use
doesn't have needed converter.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-08 11:26:24 +03:00