Commit graph

140 commits

Author SHA1 Message Date
Leonard Lyubich
1bed03995b [#328] container/load: Implement Iterator on local storage engine
Implement wrapper over StorageEngine on app-side that iterates over the
values returned by ListContainers+ContainerSize method pair. Component will
be used as the source of local metrics.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
910017b72f [#328] container/load: Implement PlacementBuilder for used space router
Implement wrapper over netmap and container storages on app-side that builds
placement vectors by next steps:

 * read container from storage;
 * read network map from storage by epoch number;
 * build container nodes according to placement policy;
 * build placement vectors with epoch-dependent pivot.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
6f552c8599 [#328] container/load: Implement WriterProvider on API client
Implement wrapper over API client on app-side that passes used space values
to it. Values are buffered on Put call and sent on Close call.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
fa685160f9 [#328] container/load: Implement Writer on Container contract's client
Implement wrapper over Container contract's client on app-side that passes
used space values to it. Uses a fixed public key, whose value will be the
key of the local storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Alex Vanin
e252f93dbf [#350] object/acl: Make best effort to classify request
Classifier looks at list of inner ring nodes and container
nodes from current and previous epoch to classify request.
Sometimes these checks might return error.

Consider there is a request from unknown key and container's
placement policy valid for current epoch and invalid for past
epoch. Classifier tries to find if key belongs to container
node from current epoch -- it is not. Then it tries to find if
key belongs to container node from past epoch and it throws
error, because placement policy is invalid for past epoch.

This is a legit case and classifier should ignore such errors
to provide best effort in matching. The only error classifier
should return is an error when request does not contain
public key to classify it.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-27 20:18:33 +03:00
Leonard Lyubich
198eb06032 [#339] cmd/node: Change the way the local NodeInfo structure is stored
Implement NodeState interface required by Netmap service. Make a single
point of updating the state of the node (for both Netmap and Control
services). Protect node info structure from data race.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-25 17:13:46 +03:00
Leonard Lyubich
3bf1519b1d [#323] cmd/node: Set BoltDB timeout option of Metabase component
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
cf00682065 [#323] cmd/node: Call Close method of the storage engine on app shutdown
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
9f41192bff [#323] cmd/node: Compose functions for closing components in app
Closing callback can be registered in app through onShutdown method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
707434efa9 [#323] cmd/node: Set default value of Blobovnicza's storage size limit
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
53f8f06f2d [#323] cmd/node: Set default value of small size limit cfg of BlobStor
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
fa0ea35f83 [#315] cmd/node: Implement node status updating in application
Call Netmap contract on SetNetmapStatus rpc of Control service (AddPeer
method if new status is ONLINE, UpdatePeerState method otherwise).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 17:46:12 +03:00
Leonard Lyubich
4666a7797b [#312] cmd/node: Switch health status while the application is running
Switch health status to:
  * STARTING after the initialization of app resources and before bootup;
  * READY after bootup and until shutting down;
  * SHUTTING_DOWN before exiting the application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
147399f547 [#312] cmd/node: Implement HealthStatus method on node application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
71fcbd3ed4 [#313] cmd/node: Update netmap status in the right way
Update node network status via update status by getting the latest network
map and finding the node in it. This step is performed at the start of the
application and upon notification of a new epoch.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
83d4420a30 [#313] control: Rename HealthStatus enum to NetmapStatus
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
3e882d963f [#311] services/control: Implement NetmapSnapshot method
Implement methods required for signatures. Receive network map from netmap
storage, convert it to Control service message and return in response.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
73e99fa590 [#306] cmd/node: Rename permitted_keys config value to authorized_keys
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
0a0ee89665 [#306] control: Rename WithAllowedKeys function to WithAuthorizedKeys
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
abd9952e46 [#306] Rename Private service to Control service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
df3746fa68 [#306] cmd/node: Switch health status on boot and shutdown
Implement HealthChecker on node app structure. Set health status to ONLINE
after node boot. Set health status to OFFLINE on shutdown.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
f3cac6cc31 [#306] cmd/node: Serve private node service in storage node app
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Alex Vanin
ca225fa3e8 [#304] cmd/neofs-node: Update morph endpoint config indent
Now it has symmetric indent for RPC and notification endpoints.

```
  morph:
    rpc_endpoint:
      - http://seed1.neo.org:20333
      - http://seed2.neo.org:20333
      - http://seed3.neo.org:20333
    notification_endpoint:
      - ws://seed1.neo.org:20333/ws
      - ws://seed-go.nspcc.ru:30333/ws
    dial_timeout: 10s
```

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Alex Vanin
842fb9248c [#304] cmd/neofs-node: Tidy default configuration and log output
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Alex Vanin
63ebe41991 [#304] cmd/neofs-node: Select random NEO endpoints from the list
Application should support several NEO endpoints so it can
switch between different RPC nodes when they fail. Application
iterates over endpoints in random order so the default list of
endpoints distribute workload kinda uniformly.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Alex Vanin
3774c5d69a [#304] cmd/neofs-node: Catch closing channel of listener endpoint
As in #72 storage application should behave the same way at remote
RPC node failures. The simplest way is to restart application. Later
we can reinitialize it without downtime.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Leonard Lyubich
1e170c3812 [#234] services/object: Support netmap epoch and lookup dead in read ops
Support processing of NetmapEpoch and NetmapLookupDepth X-headers when
processing object read operations. Placement for operations
Get/Head/GetRange/GetRangeHash/Search is built for the epoch specified in
NetmapEpoch X-header (by default latest). Also the specified operations are
processed until success is achieved for network maps from the past up to
NetmapLookupDepth value. Behavior for default values (zero or missing) left
unchanged.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-12 19:19:11 +03:00
Alex Vanin
e88c1f750d [#302] Fix golint else/return linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
74a68af5c2 [#302] Fix golint underscore linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
5e231c515a [#302] Fix whitespace linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Leonard Lyubich
d58e28afee [#298] morph/netmap: Use netmap.NodeState argument in UpdatePeerState
Replace custom NodeState enum with the one from netmap lib. Remove no longer
used NodeState enum.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:42:09 +03:00
Leonard Lyubich
b45360b933 [#298] cmd/node: Go offline on application shutdown
Transfer the state of the node to offline when the application is shut down.
Updating the state is done by calling UpdateState method of Netmap contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:42:09 +03:00
Leonard Lyubich
9a86fff7e0 [#291] Remove some unused code from repository
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
Alex Vanin
feec559e47 [#254] Update neo-go to preview4 compatible version
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
fe3906c295 [#243] object/delete: Implement new service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
611a29f682 [#241] object/search: Refactor service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
0e1f05ff45 [#239] object/head: Implement new service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
7e56427534 [#235] object/getrangehash: Implement new service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1d23483828 [#235] services/object: Implement new GetRange algorithm
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
a875d80491 Remove outdated code of metabase and localstore
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
869d9e571c [#233] object/head: Remove getting the right child header
With the update of the local storage engine, the headers of virtual objects
are directly given. In this regard, the step with obtaining the the right
child header is removed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
f24daa10ff [#233] services/object: Implement new Get algorithm
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
a488d202e1 [#230] engine: Rename WithAddress to WithAddresses for delete operations
Delete works with multiple addresses at once so it is convenient to
show it in parameter setter function.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
3d77fdb347 [#222] Set write cache specific config on application level
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
4114028a63 [#222] Add shard amount check
There should be at least one correctly set up shard in node config.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
351e4b4592 [#222] Support Inhume and Delete in object service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
f45b7a048d [#222] Rename WithTreeRoot.. to WithRoot.. in blobstor opts
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
92c95150cd [#222] Use write cache options in storage app
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2eb83624cf [#220] cmd/neofs-node: Configure Shard's write cache usage
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
cd2228cd99 [#220] cmd/neofs-node: Support new configuration of storage components
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00