Commit graph

230 commits

Author SHA1 Message Date
Alex Vanin
7131e6f339 [#426] cmd/neofs-node: Fix metric option in local storage
`WithMetrics` can't make nil check without reflection so we
have to explicitly check if metrics enabled outside of
engine constructor.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
b8e10571c6 [#426] Put prometheus behind pkg/metrics
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
3c0e47e6fd [#426] cmd/neofs-node: Add metrics collector in node
Disabled by default.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Evgenii Stratonikov
6679d59e89 [#422] pkg/services: Provide client options on cache creation
Because options are not used when client is already in cache
providing them to shared cache is misleading at best.
In the worst case `dial_timeout` is set randomly (because of race
condition) which can lead to one service having `dial_timeout` of
another. Thus we set default client creation options when cache is
created.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-03-15 14:07:53 +03:00
Leonard Lyubich
b644589bb4 [#378] cmd/node: Configure tombstone lifetime
Add `TOMBSTONE_LIFETIME` configuration value of the node which is measured
in NeoFS epoch and is set to 5 by default.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
b961570766 [#378] cmd/node: Subscribe shards on new epoch events
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
18add24727 [#378] cmd/node: Configure GC options
Add `GC_REMOVER_BATCH_SIZE` and `GC_REMOVER_SLEEP_INTERVAL` config values to
shard's config section.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Alex Vanin
9618c3e268 [#367] cmd/neofs-node: Don't update local node info for bootstrap
At startup, update only node info status. Leave all other
attributes from node configuration, so user can update them.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-04 14:16:22 +03:00
Leonard Lyubich
e9cbdc4a0d [#328] cmd/node: Add the ability to listen Container contract's events
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
198eb06032 [#339] cmd/node: Change the way the local NodeInfo structure is stored
Implement NodeState interface required by Netmap service. Make a single
point of updating the state of the node (for both Netmap and Control
services). Protect node info structure from data race.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-25 17:13:46 +03:00
Leonard Lyubich
3bf1519b1d [#323] cmd/node: Set BoltDB timeout option of Metabase component
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
cf00682065 [#323] cmd/node: Call Close method of the storage engine on app shutdown
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
9f41192bff [#323] cmd/node: Compose functions for closing components in app
Closing callback can be registered in app through onShutdown method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
707434efa9 [#323] cmd/node: Set default value of Blobovnicza's storage size limit
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
53f8f06f2d [#323] cmd/node: Set default value of small size limit cfg of BlobStor
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
147399f547 [#312] cmd/node: Implement HealthStatus method on node application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
83d4420a30 [#313] control: Rename HealthStatus enum to NetmapStatus
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
73e99fa590 [#306] cmd/node: Rename permitted_keys config value to authorized_keys
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
abd9952e46 [#306] Rename Private service to Control service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
df3746fa68 [#306] cmd/node: Switch health status on boot and shutdown
Implement HealthChecker on node app structure. Set health status to ONLINE
after node boot. Set health status to OFFLINE on shutdown.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
f3cac6cc31 [#306] cmd/node: Serve private node service in storage node app
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Alex Vanin
ca225fa3e8 [#304] cmd/neofs-node: Update morph endpoint config indent
Now it has symmetric indent for RPC and notification endpoints.

```
  morph:
    rpc_endpoint:
      - http://seed1.neo.org:20333
      - http://seed2.neo.org:20333
      - http://seed3.neo.org:20333
    notification_endpoint:
      - ws://seed1.neo.org:20333/ws
      - ws://seed-go.nspcc.ru:30333/ws
    dial_timeout: 10s
```

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Alex Vanin
842fb9248c [#304] cmd/neofs-node: Tidy default configuration and log output
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Alex Vanin
63ebe41991 [#304] cmd/neofs-node: Select random NEO endpoints from the list
Application should support several NEO endpoints so it can
switch between different RPC nodes when they fail. Application
iterates over endpoints in random order so the default list of
endpoints distribute workload kinda uniformly.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Alex Vanin
3774c5d69a [#304] cmd/neofs-node: Catch closing channel of listener endpoint
As in #72 storage application should behave the same way at remote
RPC node failures. The simplest way is to restart application. Later
we can reinitialize it without downtime.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Leonard Lyubich
9a86fff7e0 [#291] Remove some unused code from repository
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
Alex Vanin
feec559e47 [#254] Update neo-go to preview4 compatible version
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
a875d80491 Remove outdated code of metabase and localstore
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
f24daa10ff [#233] services/object: Implement new Get algorithm
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
3d77fdb347 [#222] Set write cache specific config on application level
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
4114028a63 [#222] Add shard amount check
There should be at least one correctly set up shard in node config.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
f45b7a048d [#222] Rename WithTreeRoot.. to WithRoot.. in blobstor opts
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
92c95150cd [#222] Use write cache options in storage app
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2eb83624cf [#220] cmd/neofs-node: Configure Shard's write cache usage
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
cd2228cd99 [#220] cmd/neofs-node: Support new configuration of storage components
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
a1164b2a5c [#211] cmd/neofs-node: Add SmallSizeLimit config value of BlobStor
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4ca1035aa4 [#186] cmd/neofs-node: Integrate new storage engine into application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
621840d542 [#193] cmd/neofs-node: Add dial timeouts to node configuration
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 15:34:30 +03:00
Leonard Lyubich
d286824d1d [#194] cmd/neofs-node: Use non-blocking worker pools for Object services
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:56 +03:00
Leonard Lyubich
58fcb35fb0 [#174] Use Marshal(JSON)/Unmarshal(JSON) methods for encoding/decoding
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Alex Vanin
eb13322bf0 [#166] Use async pools in object service handlers
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-13 15:54:00 +03:00
Leonard Lyubich
36d5af812e [#158] metabase: Add logger to DB configuration
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
60e4b5ddff [#158] metabase: Construct DB using options
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
8125b544b4 [#136] cmd/neofs-node: Use new metabase in app
Remove BoltDB bucket package. Construct meta.DB instance in node app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Alex Vanin
2bd827a478 [#134] Resend bootstrap txs periodically
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 10:53:11 +03:00
Leonard Lyubich
334e0e6f0f [#109] cmd/neofs-node: Activate Replicator
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 15:23:22 +03:00
Alex Vanin
0c3e6c7a04 [#40] cmd/neofs-node: Remove max object size value from config
Now max object size value is taken from global config in sidechain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-23 14:18:00 +03:00
Leonard Lyubich
d05b2ff500 [#83] cmd/neofs-node: Add network state option to response service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 10:54:48 +03:00
Leonard Lyubich
3065a7363b [#83] cmd/neofs-node: Use response services in application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 10:54:48 +03:00
Leonard Lyubich
a8481223d6 [#60] cmd/neofs-node: Implement primary network State instance
Define networkState structure and implement netmap.State interface on it.
Set epoch number on NewEpoch notification from chain. Provide
WithNetworkState option to Put service constructor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-22 11:54:08 +03:00
Leonard Lyubich
f52f643fe4 [#108] cmd/neofs-node: Add Policer worker to application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-21 14:42:51 +03:00
Leonard Lyubich
07fe49088c [#108] cmd/neofs-node: Listen new epoch notifications from morph
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-21 14:42:51 +03:00
Evgeniy Kulikov
53f7b58130 [#93] Add gRPC Reflection service
Added gRPC Reflection service that can be enabled
by settings `grpc.enable_reflect_service`.

Read more about at
https://github.com/grpc/grpc-go/blob/master/Documentation/server-reflection-tutorial.md

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-10-15 10:20:58 +03:00
Alex Vanin
dee1d81b04 [#84] Run netmap service in neofs-node app
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-09 09:15:18 +03:00
Alex Vanin
20c27d0542 [#84] Add API version to neofs-node config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-09 09:15:18 +03:00
Alex Vanin
a9e801cb22 [#71] Make GC configurable
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-05 14:03:55 +03:00
Leonard Lyubich
1d676fcfb2 [#69] object/acl: Add eACL components to service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 14:02:14 +03:00
Leonard Lyubich
9a604a50b9 [#70] cmd/neofs-node: Start asynchronous workers on app launch
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 09:36:29 +03:00
Alex Vanin
2ee24998ba [#68] Add go pprof to neofs-storage
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 19:47:08 +03:00
Alex Vanin
6ee908c2db [#65] Support FS bucket and BoltDB bucket
These buckets can be used to store blobs and metadata.
They will be removed as enhanced blob storage will be
implemented for neofs-node. To setup storage type
use `storage.object.type` and `storage.meta.type` params.

Available options:
  - inmemory (default)
  - boltdb
  - filesystem

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 14:48:00 +03:00
Alex Vanin
861bac3892 [#59] Use max msg size in transport server and splitter
For GRPC it is about 4 MiB.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
d2009c8731 [#59] Add grpc payload splitter in object service chain
GRPC has default message limit of 4MiB. Since every transmitted
neofs message has to be signed, then original message should
be split into transfer fit structures before signature service.

This commit introduce transport payload splitter for object
service pipeline. This splitter works with stream response
for methods:

  - object.Get
  - object.Range
  - object.Search

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
13080aa7a1 [#43] cmd/neofs-node: Fetch max object size from config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
e158497560 [#43] cmd/neofs-node: Support hostnames with dns, ipv4 and ipv6 addresses
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
8585e147d0 [#43] cmd/neofs-node: Use morph wrappers as data sources
Morph wrappers are being uses as sources for containers and
snapshots of network map.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
1914b4dbad [#43] cmd/neofs-node: Use object service in application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
480362b02f [#43] cmd/neofs-node: Make cfg provide network address source interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
1a169a1a9d [#49] Use zap.Logger in neofs-storage
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
df5fb23ba8 [#36] Move node attribute code into separate file
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
8e4416ad4b [#36] Add default capacity and price attributes
Default values for capacity and price will be 0, so storage node
won't be present in placement. Add these attributes if they are
not explicitly set up by user config.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
1759ff57f1 [#36] Parse attributes from node config
Well-known attributes set up with explicit configuration
values, other attributes set up from chain of attributes.

Chain of attributes is a string, that contains keys-value pairs,
divided by semicolon. Pairs itself divided by slash.

E.g. "StorageType:HDD/RPM:7200"

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
86b9aefcae [#28] Make storage node configurable
To run storage node at dev-env environment it should have
configurable parameters. To keep `cfg` structures we can
read configuration from env and yml config file with viper
and parse values such as script hashes, fees, keys into
`cfg` structures.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:35 +03:00
Alex Vanin
73b8ed4203 [#15] Add netmap contract and node info configs
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
c378c353d5 [#12] Serve gRPC session service in neofs-node app
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
8539f5c2cd [#11] services/container: Implement Neo:Morph executor and service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:24:45 +03:00
Leonard Lyubich
3308fcf56d [#11] neofs-node: Improve sources
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:24:20 +03:00
Leonard Lyubich
e6fedfbc69 [#11] Use Neo:Morph ServiceExecutor of Accounting service in app
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:23:54 +03:00
Leonard Lyubich
5cf622881f [#11] accounting: Implement signing service
Adds accounting Service implementation that verifiers request signatures and
signs responses from internal Service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:23:29 +03:00
Leonard Lyubich
65eb1181e9 [#11] Use gRPC handlers in neofs-node application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:23:17 +03:00
Leonard Lyubich
265c26150d [#11] Build simple neofs-node application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:23:02 +03:00