Commit graph

250 commits

Author SHA1 Message Date
Leonard Lyubich
8d81954afe [#168] cmd/cli: Print netmap in JSON format
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
cd4055d0b7 [#168] cmd/cli: Add snapshot command to netmap section
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
3e882d963f [#311] services/control: Implement NetmapSnapshot method
Implement methods required for signatures. Receive network map from netmap
storage, convert it to Control service message and return in response.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
73e99fa590 [#306] cmd/node: Rename permitted_keys config value to authorized_keys
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
0a0ee89665 [#306] control: Rename WithAllowedKeys function to WithAuthorizedKeys
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
abd9952e46 [#306] Rename Private service to Control service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
df3746fa68 [#306] cmd/node: Switch health status on boot and shutdown
Implement HealthChecker on node app structure. Set health status to ONLINE
after node boot. Set health status to OFFLINE on shutdown.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
e8bd2eac0d [#307] cmd/cli: Implement private command for private API access
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
f3cac6cc31 [#306] cmd/node: Serve private node service in storage node app
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Alex Vanin
ca225fa3e8 [#304] cmd/neofs-node: Update morph endpoint config indent
Now it has symmetric indent for RPC and notification endpoints.

```
  morph:
    rpc_endpoint:
      - http://seed1.neo.org:20333
      - http://seed2.neo.org:20333
      - http://seed3.neo.org:20333
    notification_endpoint:
      - ws://seed1.neo.org:20333/ws
      - ws://seed-go.nspcc.ru:30333/ws
    dial_timeout: 10s
```

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Alex Vanin
842fb9248c [#304] cmd/neofs-node: Tidy default configuration and log output
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Alex Vanin
63ebe41991 [#304] cmd/neofs-node: Select random NEO endpoints from the list
Application should support several NEO endpoints so it can
switch between different RPC nodes when they fail. Application
iterates over endpoints in random order so the default list of
endpoints distribute workload kinda uniformly.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Alex Vanin
3774c5d69a [#304] cmd/neofs-node: Catch closing channel of listener endpoint
As in #72 storage application should behave the same way at remote
RPC node failures. The simplest way is to restart application. Later
we can reinitialize it without downtime.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Leonard Lyubich
1e170c3812 [#234] services/object: Support netmap epoch and lookup dead in read ops
Support processing of NetmapEpoch and NetmapLookupDepth X-headers when
processing object read operations. Placement for operations
Get/Head/GetRange/GetRangeHash/Search is built for the epoch specified in
NetmapEpoch X-header (by default latest). Also the specified operations are
processed until success is achieved for network maps from the past up to
NetmapLookupDepth value. Behavior for default values (zero or missing) left
unchanged.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-12 19:19:11 +03:00
Alex Vanin
b62b19cb5a [#302] Fix goimports linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
e88c1f750d [#302] Fix golint else/return linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
0f3570bdeb [#302] Fix golint naming linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
74a68af5c2 [#302] Fix golint underscore linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
5e231c515a [#302] Fix whitespace linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Leonard Lyubich
d58e28afee [#298] morph/netmap: Use netmap.NodeState argument in UpdatePeerState
Replace custom NodeState enum with the one from netmap lib. Remove no longer
used NodeState enum.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:42:09 +03:00
Leonard Lyubich
b45360b933 [#298] cmd/node: Go offline on application shutdown
Transfer the state of the node to offline when the application is shut down.
Updating the state is done by calling UpdateState method of Netmap contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:42:09 +03:00
Leonard Lyubich
07a6e2cc43 [#290] cmd/cli: Trim 0x prefix from salt argument of object hash cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:40:59 +03:00
Leonard Lyubich
6df8c84fc2 [#290] cmd/cli: Add salt flag to object hash command
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:40:59 +03:00
Leonard Lyubich
9a86fff7e0 [#291] Remove some unused code from repository
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
Alex Vanin
df116852f2 [#289] Update neo-go version to v0.92.0
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-29 17:42:02 +03:00
Alex Vanin
7174abcc7c [#281] service/audit: Fix typos
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
0d7832f5e9 [#281] service/audit: Add separate pool for SG checks in PoR
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
3ec342b2c3 [#259] services/audit: Implement PDP check
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
310a4c3e4d [#275] innerring: Add storage group search timeout
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
782bcadd92 [#271] innerring: Implement GetSG and Head of audit communicator
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
ecb5227c27 [#255] cmd/ir: Set default values of audit configuration
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
87e1252065 [#265] innerring: Generate new audit event
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
f7ca4a8dce [#80] cmd/cli: Implement storagegroup commands
Implement PUT/GET/LIST/DELETE commands of storage group module.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
feec559e47 [#254] Update neo-go to preview4 compatible version
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
51b85b0a73 [#278] cli: Support request X-Headers
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-24 15:03:02 +03:00
Leonard Lyubich
e53bf574b5 [#279] container: Use new methods to work with container format
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-24 13:38:20 +03:00
Alex Vanin
7daf3727ee [#248] neofs-cli/object: Print split header values in Head command
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-14 14:46:59 +03:00
Leonard Lyubich
6814140a19 [#240] cmd/cli: Support raw flag in object GET/HEAD/RANGE commands
Support boolean raw flag. Print split information upon receipt.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-14 12:05:13 +03:00
Leonard Lyubich
09916f21b5 [#243] cmd/cli: Write tombstone address in Object.Delete cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
fe3906c295 [#243] object/delete: Implement new service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
611a29f682 [#241] object/search: Refactor service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
0e1f05ff45 [#239] object/head: Implement new service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
7e56427534 [#235] object/getrangehash: Implement new service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1d23483828 [#235] services/object: Implement new GetRange algorithm
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
a875d80491 Remove outdated code of metabase and localstore
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
869d9e571c [#233] object/head: Remove getting the right child header
With the update of the local storage engine, the headers of virtual objects
are directly given. In this regard, the step with obtaining the the right
child header is removed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
f24daa10ff [#233] services/object: Implement new Get algorithm
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
a488d202e1 [#230] engine: Rename WithAddress to WithAddresses for delete operations
Delete works with multiple addresses at once so it is convenient to
show it in parameter setter function.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
3d77fdb347 [#222] Set write cache specific config on application level
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
4114028a63 [#222] Add shard amount check
There should be at least one correctly set up shard in node config.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
351e4b4592 [#222] Support Inhume and Delete in object service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
f45b7a048d [#222] Rename WithTreeRoot.. to WithRoot.. in blobstor opts
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
92c95150cd [#222] Use write cache options in storage app
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2eb83624cf [#220] cmd/neofs-node: Configure Shard's write cache usage
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
cd2228cd99 [#220] cmd/neofs-node: Support new configuration of storage components
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
a1164b2a5c [#211] cmd/neofs-node: Add SmallSizeLimit config value of BlobStor
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4ca1035aa4 [#186] cmd/neofs-node: Integrate new storage engine into application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
49666f87f1 [#170] Implement --vote argument in neofs-ir
With this argument neofs-ir will be run as an CLI
application that initialize inner ring server, invokes
`vote` method on corresponding alphabet contract and
exits.

User can provide a comma-separated list of validator
public keys.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
0b65c64e98 [#170] Vote for validators on inner ring startup
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
4c0ca23b34 [#208] Print split ID in object.Head output of neofs-cli
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-30 10:44:15 +03:00
Leonard Lyubich
0168740c9a [#206] cli/container: Set default version of eACL table in set-eacl
Set version of parsed eACL table to using SDK version if unsupported version
was set in source data.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-25 14:35:27 +03:00
Leonard Lyubich
b4699b1e9f [#206] cli/container: Use GetEACLWithSignature in set-eacl polling
Replace GetEACL call with GetEACLWithSignature one in polling step of
set-eacl command of container CLI.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-25 14:35:27 +03:00
Leonard Lyubich
e8bc03d92f [#206] cli/container: Use GetEACLWithSignature client method
Use GetEACLWithSignature client method in get-eacl cmd of container CLI in
order to print eACL table and signature regardless of their correctness. The
ability to check the correctness of the signature will be added later.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-25 14:35:27 +03:00
Leonard Lyubich
621840d542 [#193] cmd/neofs-node: Add dial timeouts to node configuration
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 15:34:30 +03:00
Leonard Lyubich
400b65801d [#201] cli/container: Do not check eACL version
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:56:03 +03:00
Leonard Lyubich
d286824d1d [#194] cmd/neofs-node: Use non-blocking worker pools for Object services
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:56 +03:00
Leonard Lyubich
6a5c37d592 [#195] object/search: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
cb46e4b154 [#195] object/rangehash: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
af6484e3b1 [#195] object/range: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
cf2dc37a42 [#195] object/put: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
9fbfc0b5e4 [#195] object/head: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
e1e5a590e9 [#195] object/delete: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
ea36789b51 [#179] cmd/neofs-cli: Implement range command in Object section
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-23 13:23:31 +03:00
Leonard Lyubich
dbfb6fadd6 [#177] cmd/neofs-cli: Add object ID filter flag to search cmd
Add "oid" flag that allows to filter objects by identifier.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-23 09:54:59 +03:00
Alex Vanin
e8fe07edd0 [#184] Use SDK client cache in object.Rangehash
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
f85e88c4f8 [#184] Use SDK client cache in object.Range
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
d485a5967d [#184] Use SDK client cache in object.Search
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
7ba95dd5fc [#184] Use SDK client cache in object.Put
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
e9a6365333 [#184] Use SDK client cache in object.Head
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Leonard Lyubich
58fcb35fb0 [#174] Use Marshal(JSON)/Unmarshal(JSON) methods for encoding/decoding
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Leonard Lyubich
3de8febe57 [#174] Update to latest neofs-api-go changes
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Alex Vanin
e47c866d6a [#169] Add --ttl argument in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 12:50:21 +03:00
Alex Vanin
e580f2afd6 [#160] Update object CLI flags
* remove no-leaf
* rename leaf to phy
* remove no-root

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
eb13322bf0 [#166] Use async pools in object service handlers
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-13 15:54:00 +03:00
Alex Vanin
ded45e1fbc [#159] Add keyer to neofs-cli
Keyer prints information about private key, public key,
NEO3 Wallet, scripthash. It can generate new private key
or generate multisig address.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-09 18:59:21 +03:00
Leonard Lyubich
3c17e871e1 [#158] cmd/neofs-node: Handle objects from tombstone by metabase
Handle tombstone content by metabase instead of object GC queue.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
36d5af812e [#158] metabase: Add logger to DB configuration
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
60e4b5ddff [#158] metabase: Construct DB using options
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Alex Vanin
0021f1cc5f [#81] cmd/neofs-cli: Use hex encoding of node's public key
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 16:37:48 +03:00
Leonard Lyubich
25398262fc [#81] cmd/neofs-cli: Implement "nodeinfo" sub-cmd of "netmap" cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 16:37:48 +03:00
Leonard Lyubich
5d94594007 [#81] cmd/neofs-cli: Implement "epoch" sub-cmd of "netmap" cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 16:37:48 +03:00
Alex Vanin
65be09d3db [#155] Update neofs-api-go with refactored pkg/netmap
Refactored pkg/netmap package provides JSON converters for
NodeInfo and PlacementPolicy structures, that has been used
by client applications.

It also updates Node structure itself so it is a part of
grpc <-> v2 <-> pkg conversion chain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 09:55:05 +03:00
Alex Vanin
49da96d006 [#146] Set stdout as default output for CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-05 18:37:24 +03:00
Alex Vanin
64caa15556 [#148] Use base name as FileName attribute value
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-05 18:37:06 +03:00
Alex Vanin
54818d5a11 [#144] Support well-known application attributes for object in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-05 09:28:13 +03:00
Alex Vanin
83926647d7 [#144] Support well-known application attributes for container in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-05 09:28:13 +03:00
Leonard Lyubich
8125b544b4 [#136] cmd/neofs-node: Use new metabase in app
Remove BoltDB bucket package. Construct meta.DB instance in node app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Alex Vanin
97077294fc [#141] Fix double sidechain GAS emission on asset mint
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-03 16:28:01 +03:00
Alex Vanin
0c49c08609 [#139] Use storage node GAS emission in inner ring app
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-03 16:20:41 +03:00
Alex Vanin
83119c00ec [#127] Remove magic number from config
With neo-project/neo-modules#358 nodes support RPC to
return used magic number. Therefore client doesn't need
that configuration value any more.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 13:54:31 +03:00