Commit graph

883 commits

Author SHA1 Message Date
Pavel Karpy
53391f057e [#693] node/reputation: Log keys in hex format
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-13 15:12:21 +03:00
Alex Vanin
e2f7b3f1cc [#690] cmd/neofs-node: Fallback to wallet section if node key is not set
Some users want to specify only wallet section in the SN. It is not
possible if `Key` throws panic on empty value. Instead it should
fallback to wallet section. Panic is suitable if node's key is provided
but invalid.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 14:49:31 +03:00
Alex Vanin
6bf01a0a22 [#683] cmd/neofs-adm: Add CLI flags for morph commands
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 11:29:39 +03:00
Pavel Karpy
cf8f640726 [#675] cli/container: Support binary eACL format
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-09 09:36:02 +03:00
Alex Vanin
d189d60925 [#666] cmd/neofs-adm: Initial app structure with config init command
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-08 14:37:38 +03:00
Pavel Karpy
3a7d7bdecd [#665] cli: Set non-zero exit codes in err cases
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-06 18:10:38 +03:00
Pavel Karpy
8965e70463 [#665] cli/util: Add exit code utils
Add `errf`, `exitOnErr` and `exitOnErrCode` functions
that works with errors and exits with non-zero exit
codes on non-nil errors.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-06 18:10:38 +03:00
Pavel Karpy
75632a7d83 [#667] node: Add --version flag support
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-06 18:07:35 +03:00
Pavel Karpy
eadc3a4de9 [#667] ir: Make --version output same as in CLI
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-06 18:07:35 +03:00
Leonard Lyubich
d610346a7b [#660] cli/container: Use version.IsValid in parseEACL
Extended ACL input can have version later than CLI one, and it should not be
downgraded. But the version should be still adequate.

Set `pkg.SDKVersion` on false return of `version.IsValid`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-07-05 11:05:44 +03:00
Leonard Lyubich
dab33e0f9b [#652] Update Neo Go to v0.95.3
Add `keys.NEP2ScryptParams()` to `keys.NEP2Decrypt` call arguments.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-29 13:44:59 +03:00
Leonard Lyubich
56d4410913 [#638] Update to Go 1.16
Changes:

  * replace `iotuil` elements with the ones from `os` package;
  * replace `os.Filemode` with `fs.FileMode`;
  * use `signal.NotifyContext` instead of `NewGracefulContext` (removed).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-29 13:44:59 +03:00
Leonard Lyubich
61f48a2736 [#651] cmd/cli: Replace netmap snapshot from netmap section to control
There is a need to have all `Control` service-related commands in `control`
section.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-29 13:44:59 +03:00
Leonard Lyubich
81ddaeca16 [#607] config/example: Add all formats of node's network addresses
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
43eff09944 [#607] *: Do not use deprecated elements of code
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
119031c8c7 [#607] network: Make AddressGroup.WriteToNodeInfo method a function
Method implementation doesn't use any private logic.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
1e52e86bbc [#607] node/control: Make group address in NodeInfo message
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
163c24a2d2 [#607] cmd/node: Configure group of bootstrap addresses
There is a need to support multiple network addresses of the storage nodes.

Make `BootstrapAddress` to return `network.AddressGroup` (and rename).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
8060735732 [#607] cmd/node: Serve gRPC on multiple interfaces
Generalize single gRPC interface of the storage node to a group of
interfaces. Each interface calls the same RPC handler.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
d1eb9c3b0f [#607] cmd/node: Empty Control server address as disable
In previous implementation if Control listen endpoint was omitted in config
the gRPC listening endpoint was used instead.

Consider empty address as an option to disable the service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
c444118f47 [#607] cmd/node: Make reputationClientConstructor to accept AddressGroup
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
cede2b4ed7 [#607] reputation,container: Support address groups in ServerInfo
There is a need to support multiple server endpoints for reputation and
container transmission.

Replace `ServerInfo.Address` getter with `ServerInfo.IterateAddresses`
iterator.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
8ac3c62518 [#607] object/head: Make client constructor to work with group address
Make Object Head service to work with `AddressGroup` instead of `Address`
in order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
d0e48c949b [#607] object/search: Make client constructor to work with group address
Make Object Search service to work with `AddressGroup` instead of `Address`
in order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
ad14df07f6 [#607] object/get: Make client constructor to work with group address
Make Object Get service to work with `AddressGroup` instead of `Address` in
order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
8972f84672 [#607] object/put: Make client constructor to work with group address
Make Object Put service to work with `AddressGroup` instead of `Address` in
order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
6e5d7f84af [#607] network: Generalize LocalAddressSource to address group
Make `LocalAddressSource.LocalAddress` method to return `AddressGroup`. Make
`IsLocalAddress` function to accept parameter of type `AddressGroup`. Adopt
the application code with temporary `GroupFromAddress` helper.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
e11f50ec8e [#607] network: Make ClientCache to accept AddressGroup
Change type of the `ClientCache.Get` method's parameter to `AddressGroup`.
Use `GroupFromAddress` to call the method from the wrappers in order to no
change their interface.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
c82615667d [#607] network: Return group-address client from ClientCache
Add group-address `Client` implementation. Return instances of this
implementation from `ClientCache.Get` method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
3805b0f638 [#607] client: Overload Client interface
There is a need to generalize single-address client to group-address client.
To do this, we can re-implement `Client` interface from NeoFS API Go library
and still use it in the application code. There is a problem with method
`Raw` which must return single-address raw client. So as not to make changes
to API library we need to overload Client interface in order to support
`Raw` method in group-address client implementation.

Define `Client` interface in new `pkg/core/client` package. Completely
inherit API `Client` interface. Add `RawForAddress` method to build raw
client for the single node address. Adopt the application code that used Raw
method to work with new `Client`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
tivizi
1f461baf0b [#626] The default config file location
Signed-off-by: Tivizi Jing <tivizi@163.com>
2021-06-25 11:16:17 +03:00
tivizi
2da5a309f7 [#626] Support default config file
Signed-off-by: Tivizi Jing <tivizi@163.com>
2021-06-25 11:16:17 +03:00
Pavel Karpy
48827f42d3 [#643] pkg: Sync method names and commentaries to them
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-24 16:10:44 +03:00
Pavel Karpy
14afc6a1e5 [#635] cli: Do not enter password twice
Obtain key once in every cobra command to
pass it to `getOwnerID` and `initSession`
and do not ask to enter password more than
one time in `put` and `putSG` operations.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-23 10:22:28 +03:00
Pavel Karpy
02ca1c4cc1 [#624] cli: Do not print help on any failure
Change usage function to `Run`(does not return
err). Log errors with `cmd.PrintErrln`. Change
all `fmt.Print*` to `cmd.Print*`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-23 10:08:40 +03:00
Pavel Karpy
af72412b77 [#624] cli/object: Fix error messages
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-23 10:08:40 +03:00
Pavel Karpy
87ce64fbbb [#624] cli/root: Delete version command and add corresponding flag
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-23 10:08:40 +03:00
Pavel Karpy
e3b4216fa7 [#624] cli/netmap: Delete logging usage without agrs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-23 10:08:40 +03:00
Pavel Karpy
dc12202757 [#624] cli/util: Sync command description's capitalization
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-23 10:08:40 +03:00
Leonard Lyubich
adbbad0beb [#607] network: Do not work with Address pointers
`network.Address` structure in most cases created once and used read-only.

Replace `AddressFromString` function with `Address.FromString` method with
the same purpose and implementation. Make all libraries to work with value.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich
5de074f24c [#607] network: Implement WriteToNodeInfo method on Address
Implement `Address.WriteToNodeInfo` method which sets address of `NodeInfo`
structure. Use it in storage node application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich
e5504c7130 [#607] network: Do not use Address.String for address comparison
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich
47fe8911a3 [#607] network: Rename Address.HostAddrString method to HostAddr
Return tyype is clear from the method's signature and docs, there is no
point in reflecting it in the name.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich
35f81729e4 [#607] network: Do not return error from Address.HostAddrString method
Panic if internal `manet.DialArgs` call returns error since this is
unexpected according to `AddressFromString` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich
fdd123ad98 [#607] network: Prevent potential panic in Address.Equal method
Make `Address.Equal` method to accept value instead of pointer in order to
prevent NPE.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Pavel Karpy
6b176e8769 [#613] pkg/reputation: Move manager building to pkg
Move `managers` package to `pkg` since
it can be reused in other packages.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-17 18:30:06 +03:00
Evgenii Stratonikov
c172fcc11f [#610] neofs-cli: replace --key flag with --wif
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-17 10:36:35 +03:00
Evgenii Stratonikov
ad90b07ed5 [#610] neofs-cli: add --binary-key flag
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-17 10:36:35 +03:00
Evgenii Stratonikov
1b14b25e6c [#610] neofs-cli: add --wallet flag
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-17 10:36:35 +03:00
Evgenii Stratonikov
2b5998b820 [#610] neofs-cli: add --generate-key flag
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-17 10:36:35 +03:00
Leonard Lyubich
7cead1bc3a [#414] cmd/cli: Support IR health-check
Add `--ir` flag to `control healthcheck` to communicate with IR node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-15 15:21:52 +03:00
Leonard Lyubich
455fd952dd [#414] ir: Serve ControlService
Serve `ControlService` instance on configured endpoint (do not serve if not
specified). Read allowed keys from config.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-15 15:21:52 +03:00
Evgenii Stratonikov
41a30d6ec2 [#562] config/node: fallback to wallet if raw key is invalid
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-15 14:49:59 +03:00
Evgenii Stratonikov
2f020a500d [#562] config: parse key on config load
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-15 14:49:59 +03:00
Evgenii Stratonikov
5cab0026c3 [#562] pkg/morph: remove neofs-crypto uses
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-15 14:49:59 +03:00
Evgenii Stratonikov
fcdef227e4 [#562] cmd/neofs-cli: use NEP-6 wallet for keys
Encrypted NEP-2 is still supported.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-15 14:49:59 +03:00
Evgenii Stratonikov
3f07313604 [#562] cmd/neofs-node: use NEP-6 wallet for keys
Also use neo-go private key wrapper where possible, as it
already has methods for (un)marshaling.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-15 14:49:59 +03:00
Evgenii Stratonikov
1553967328 [#562] cmd/neofs-ir: use NEP-6 wallet for keys
Also remove neofs-crypto uses from `pkg/innerring`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-15 14:49:59 +03:00
Leonard Lyubich
946d4c4253 [#603] cmd/node: Remove no longer used BootstrapType enum
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-11 15:36:17 +03:00
Leonard Lyubich
6279b6343f [#603] cmd/node: Separate configuration and netmap node info
Config `NodeInfo` should be used for bootstrap. Separete local node info and
netmap one. Return configured `NodeInfo` if structure from netmap is
missing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-11 15:36:17 +03:00
Leonard Lyubich
5a4c3dfddf [#603] cmd/node: Parse node attributes when node info is constructed
Remove no longer needed `cfgNodeInfo.attributes` field.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-11 15:36:17 +03:00
Leonard Lyubich
4859bb2e1c [#603] cmd/node: Remove unused bootType field of cfgNodeInfo struct
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-11 15:36:17 +03:00
Alex Vanin
5ee7d7efcb [#598] cmd/neofs-ir: Update default config values
In testnet and neofs-dev-env we use timers with these
settings and they are looking good. So it makes sense
to make them default.

Storage node now can't configure bootstrap healthcheck
length and can't disable it. Inner ring default values
now adopt these changes too.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-09 16:20:37 +03:00
Alex Vanin
0368d5f2b2 [#598] cmd/neofs-ir: Print debug message in --version
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-09 16:20:37 +03:00
Alex Vanin
caafd973e4 [#598] misc: Remove global prefixes
New config package in storage node does not use
application prefix from misc package. Therefore
inner ring node can define prefix in local scope.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-09 16:20:37 +03:00
Alex Vanin
b8c8bf4ba2 [#587] cmd/neofs-cli: Add sign session-token command
Container commands in NeoFS CLI can use signed session token
to create, delete container and change extended ACL table.
This token should be signed the same way we sign bearer tokens.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-03 18:40:31 +03:00
Alex Vanin
bce92168c1 [#587] cmd/neofs-cli: Make session token parser reusable
containerSessionToken() actually parses any session token.
We can reuse this function to resign session token when it
takes filepath as an argument, instead of public variable.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-03 18:40:31 +03:00
Alex Vanin
a2547da5ae [#493] cmd/node: Fix linter errors in config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-02 16:07:42 +03:00
Alex Vanin
e9e986ac71 [#493] cmd/node: Remove viper from storage node
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-02 16:07:42 +03:00
Alex Vanin
cd947bb580 [#493] cmd/node: Add object service section to config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-02 16:07:42 +03:00
Alex Vanin
f40b84c99e [#493] cmd/node: Add replicator section to config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-02 16:07:42 +03:00
Alex Vanin
8c96494da0 [#493] cmd/node: Add policer section to config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-02 16:07:42 +03:00
Alex Vanin
27b4781e95 [#493] cmd/node: Add apiclient section to config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-02 16:07:42 +03:00
Alex Vanin
0f4e8d2362 [#496] cmd/node: Use new config for morph and mainchain configuration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-02 15:12:08 +03:00
Alex Vanin
4ef968aa06 [#496] cmd/node: Add mainchain section to config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-02 15:12:08 +03:00
Alex Vanin
cf5e371590 [#496] cmd/node: Add morph section to config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-02 15:12:08 +03:00
Alex Vanin
44b19c145f [#493] cmd/node: Use new config for control service configuration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-02 14:58:22 +03:00
Alex Vanin
c828848024 [#493] cmd/node: Add control section to config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-02 14:58:22 +03:00
Alex Vanin
161fca58eb [#493] cmd/node: Use new config for contracts configuration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-02 14:58:22 +03:00
Alex Vanin
0eea25375e [#493] cmd/node: Add contracts section to config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-02 14:58:22 +03:00
Pavel Karpy
8a0a75a6b2 [#579] cmd/node: Use new config for GRPC configuration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-02 14:02:33 +03:00
Pavel Karpy
7dbeb08c58 [#579] cmd/node: Add grpc section to config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-02 14:02:33 +03:00
Pavel Karpy
6ab7efb358 [#577] cmd/node: Use new config for node configuration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-02 13:28:15 +03:00
Pavel Karpy
42d18f1322 [#577] cmd/node: Add node section to config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-02 13:28:15 +03:00
Pavel Karpy
52c82ef46a [#577] cmd/node: Add tests for boolean type casting in config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-02 13:28:15 +03:00
Evgenii Stratonikov
b8a7c11e57 [#501] object/put: reduce TTL of the relayed request
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-02 10:31:14 +03:00
Leonard Lyubich
6ca7f4511c [#493] cmd/node: Use engineconfig pkg for storage engine construction
Use `engineconfig.IterateShards` in order to compose options of the shards.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-02 09:54:00 +03:00
Leonard Lyubich
e149eae7b2 [#493] node/config: Test storage engine's config with config/example
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-02 09:54:00 +03:00
Leonard Lyubich
f663a1c125 [#493] node/config: Implement sections of local object storage
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-02 09:54:00 +03:00
Leonard Lyubich
e26dc0a6e3 [#493] node/config: Fix corrupting of path to the subsection
In previous implementation `Config.Sub` method could lead to the violation
of the internal `path` slice because of `append`. This has been observed on
deeply nested subsections.

Fix `Config.Sub` to copy internal slice in order to prevent violations.
Cover problem case in test config files and unit test.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-02 09:54:00 +03:00
Leonard Lyubich
2bbd4d0ee3 [#493] node/config: Implement integer casters
Implement `Int` / `Uint` functions which casts value to `int64` / `uint64`.
Implement safe functions `IntSafe` / `UintSafe`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-02 09:54:00 +03:00
Alex Vanin
efcd12c71e [#493] cmd/node: Use new config for metrics configuration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-01 15:29:08 +03:00
Alex Vanin
3fbf5e05b2 [#493] cmd/node: Add metrics section to config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-01 15:29:08 +03:00
Alex Vanin
da8310f0e3 [#493] cmd/node: Use new config for profiler configuration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-01 15:29:08 +03:00
Alex Vanin
561c809fa5 [#493] cmd/node: Add profiler section to config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-01 15:29:08 +03:00
Alex Vanin
922d29ff88 [#493] cmd/node: Support duration type casting in config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-01 15:29:08 +03:00
Alex Vanin
25a13d3611 [#493] cmd/node: Add .env file for tests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-06-01 15:29:08 +03:00
Leonard Lyubich
db7a7f9bd3 [#493] cmd/node: Remove no longer needed cfgLogLevel constant
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
56109f941b [#493] cmd/node: Use new config implementation for logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
3b797d7957 [#493] node/config/logger: Simplify approach to read the level
Dedicated type `LoggerSection` turned out to be redundant since it doesn't
do a hidden logic and just uses `config.Config` API.

Remove `LoggerSection` type and implement `Level` which do the same.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
be6898a51d [#493] node/config/logger: Cover ENV variables with unit test
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
3fd4b32a4f [#493] node/config: Cover ENV variables with unit test
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
099ceeae50 [#493] node/config: Implement ENV variable key constructor
Add `internal.Env` function which converts path to config value to ENV
variable key.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
270b147205 [#493] node/config: Export ENV constants from internal package
Replace ENV prefix and separator to `internal` package in order to reuse
them for testing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
32dc80e7bc [#493] node/config: Always read the values of ENV variables
In previous implementation ENV values were read only if config filepath is
specified. From now ENVs are always read to `Config`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
1aa88159ca [#493] node/config: Change Sub implementation
In some cases viper doesn't interpret `section.value` as a subsection with
`section` name, but value is value still can be accessed through full
pathname.

Fix `Config.Sub` method implementation in order to always interpret
configuration like described above as a subsection. From now method never
returns nil, therefore an additional check has been removed from the `Value`
method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
e5c014bbfb [#493] node/config/logger: Write unit tests
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
b8a5f09174 [#493] node/config: Export useful functions into a separate test package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
c645781b98 [#493] node/config: Implement logger section
Create `logger` sub-package of `config` package. Implement `LoggerSection`
type of logger sub-section. Add `Level` method to read logger level config
value. Default level is `info`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
cbe3e0a271 [#493] node/config: Implement string caster
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
7e11bf9a55 [#493] cmd/node: Implement a basic configuration component
Create `config` package nearby storage node application. Implement `Config`
as a wrapper over `viper.Viper` that provides the minimum functionality
required by the application.

The constructor allows you to read the config from the file. Methods are
provided for reading subsections and values from the config tree. Helper
functions are implemented to cast a value to native Go types.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-01 11:45:38 +03:00
Leonard Lyubich
3dd10b6795 [#570] *: Remove usage of deprecated elements from API Go library
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 20:45:15 +03:00
Leonard Lyubich
9b2939d538 [#567] cmd/node: Close cached clients on shutdown
Call `CloseAll` on all `ClientCache` instances on application shutdown.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 13:11:37 +03:00
Leonard Lyubich
2e814941c0 [#560] cli/container: Support session tokens
Container sessions allow to perform some operations on behalf of another
user. There is a need to to attach session tokens to commands.

Add `session` flag to `put`, `delete` and `set-eacl` commands from
`container` section. It should be a path to the JSON-encoded session token.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 11:53:06 +03:00
Leonard Lyubich
e67fe80132 [#552] cmd/ir: Remove redundant if-statements before exitErr calls
`exitErr` function checks `err != nil` by itself.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 10:09:40 +03:00
Leonard Lyubich
6b1916a529 [#552] cmd/node: Implement error checker with details
Implement function `fatalOnErrDetails` similar to `fatalOnErr` but accepting
string details that are written to log output. Use the function everywhere
in application code without wrapping in an if-else statement.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 10:09:40 +03:00
Pavel Karpy
3e0eccb548 [#549] cli: Add TLS support to control service
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-27 17:07:04 +03:00
Pavel Karpy
077f1af5a7 [#549] node: Add TLS to bootstrap address
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-27 17:07:04 +03:00
Pavel Karpy
33bef46f31 [#549] network/cache: Change Get signature
Make network cache's `Get` method accept
`network.Address` argument instead of
string.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-27 17:07:04 +03:00
Pavel Karpy
e10981a7d3 [#549] grpc/server: Add TLS encryption
Add TLS to config. Add server side encryption
if it is configured so.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-27 17:07:04 +03:00
Leonard Lyubich
df197dc38b [#525] morph/container: Do not return signature from GetEACL method
In previous implementation wrapper over the Container contract's client
returned the signature of the eACL table in addition to itself. After recent
changes in API Go lib table carries its signature. Thus, it is redundant to
return the table signature separately.

Make `Wrapper.GetEACL` method to return only `eacl.Table` with error.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-27 12:03:49 +03:00
Leonard Lyubich
83c27f6e8a [#505] morph/container: Change get container API
Make `Get` method of the wrapper over Container contract's client to
accept binary container ID. Create `Get` function similar to the previous
`Get` variation. Use this function in Container service server in the place
where `Get` method was used.

Additionally implement `AsContainerSource` function which allows
to simply compose container Source interface from the wrapper.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-25 16:35:52 +03:00
Evgenii Stratonikov
2b2b2c2c45 [#496] Use single contract wrapper constructor
There is no need in a separate `New()` or `WrapClient()`

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-05-25 13:41:39 +03:00
Angira Kekteeva
7eab752923 [#116] *: Replace pkg policy by neofs-sdk policy
Processing transfer of policy pkg to neofs-sdk repository.

Replace local dependency in neofs-cli.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2021-05-24 09:49:15 +03:00
Alex Vanin
b5256ccf4c [#521] Fix issues with transition from pkg/errors pkg
Wrap functions at `pkg/errors` return nil if error argument
was nil. fmt.Errorf always returns error so we need to add
missing error checks to the code.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-19 19:09:35 +03:00
Evgenii Stratonikov
71b87155ef [#521] *: use stdlib errors package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-05-19 16:42:54 +03:00
Leonard Lyubich
43e575cec2 [#528] cli/container: Use AWS-style names in basic ACL keywords
When user creates a container using neofs-cli, he can set a pre-defined ACL.
Current keywords are confusing: for example, `public` really means that
everyone can write to the container, while the expectation is just public
visibility. Perform the following renames for a better understanding of the
purpose:

 * `readonly` -> `public-read`;

 * `public` -> `public-read-write`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-19 16:42:14 +03:00
Leonard Lyubich
8f9e7c2a63 [#528] cli: Define keyword constants for predefined basic ACL values
Define constants for `public`/`private`/`readonly` keywords of predefined
basic ACL values. Use constants instead of repeating string literals.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-19 16:42:14 +03:00
Alex Vanin
16f13bc0a5 [#522] Use HostAddrString as RPC endpoint instead of IPAddrString
To enable TLS support we can't operate with IP addresses directly.
Certificates are issued with host names so it is required to
pass them into RPC client. DNS resolving should be done by transport
layer and not be a part of node. Therefore `IPAddrString` usage is
removed from code.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-18 12:07:00 +03:00
Pavel Karpy
9fbc6f5efe [#527] reputation: Fix EigenTrust algorithm
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-18 10:29:25 +03:00
Leonard Lyubich
3622e4165a [#493] node: Make list of main chain's RPC endpoints optional
Currently main chain's client is unused, therefore, it is not advisable to
require mandatory setting of addresses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-14 11:02:16 +03:00
Leonard Lyubich
c36bca1ddc [#493] node: Remove unused worker pools for reading object operations
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Leonard Lyubich
f582475ea2 [#493] node: Replace object pool configurations to object section
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Leonard Lyubich
9e2b47d663 [#493] node: Set default contract addresses to empty strings
If default smart contract addresses are valid 20 byte strings, then omitting
the values in configuration will not be tracked at startup.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Leonard Lyubich
9920425db9 [#493] node: Abolish tombstone lifetime configuration
There is no need to change this value for now.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Leonard Lyubich
d36084c26a [#493] node: Replace relay config value to node section
Set default value to `false`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Leonard Lyubich
f90fc86ec8 [#493] node: Abolish re-bootstrap configurations
From now non-relay node always sends re-bootstrap transaction every 2 epochs
starting from the boot-up epoch.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Leonard Lyubich
7a64a70b12 [#492] node: Never go offline on application shutdown
Application can be shut down often, it is undesirable to send status changes
on a transaction every time. Control API should be used before shutdown for
this purpose.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Leonard Lyubich
10308101df [#493] node: Abolish some of the Policer's configurations
Make work scope and expansion rate the app-side constants since Policer
independently regulates the amount of work performed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Leonard Lyubich
80ef4492c1 [#493] node: Get rid of outdated object GC worker
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Leonard Lyubich
1f817d1cd2 [#493] node: Abolish configuration of pools of notification handlers
There is no need to use synchronous execution of notification handlers. Also
there is no understanding of how to assess the need to change the size of
the pools.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Leonard Lyubich
3ef5b0ff9c [#493] node: Do not add fee in smart contract calls
Calls to contracts by storage nodes do not lead to the accumulation of
multisignatures in the contract memory, so the call cost can always be
accurately calculated in advance without additional fee.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Leonard Lyubich
acd63afd48 [#493] node: Replace configuration of API clients to a separated section
Replace dial timeout of API client connection from `grpc` section to
`apiclient` since this value is semantically decoupled from the transport
protocol.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Leonard Lyubich
31eb6617fc [#493] node: Abolish non-usable gRPC server configurations
Reflect service is disabled, max message size frozen by constant.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-13 19:29:10 +03:00
Leonard Lyubich
6d7fff1870 [#493] ir: Set default config values of prometheus and pprof servers
Although the default values for configurable addresses are empty strings,
explicitly specifying the default values allows you to know about all
possible configurations.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-12 10:46:17 +03:00
Leonard Lyubich
ece6618560 [#493] ir: Use pre-allocation in initHTTPServers function
Number of servers to be created is known in advance.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-12 10:46:17 +03:00
Leonard Lyubich
c340d77b74 [#493] ir: Replace creation of HTTP servers into a separate function
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-12 10:46:17 +03:00
Leonard Lyubich
8d17dab86e [#493] Refactor serving of prometheus and pprof services
Rename `util/profiler` package to `httputil` and refactor it:

  * simplify utility HTTP server;

  * make more generic server's parameters in order to remove `viper.Viper`
    dependency;

  * use single constructor for creating the pprof and prometheus servers;

  * replace `enabled` config value with empty-check of the network address.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-12 10:46:17 +03:00
Leonard Lyubich
6339f1a468 [#493] node: Connect to main chain
Establish client connection with main chain node on storage node startup.
Client is configured simlarly to morph client.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-11 19:41:51 +03:00
Leonard Lyubich
1fa49bca9c [#493] logger: Write app version once on start
Writing application version in each log message seems pretty redundant.

Remove global `app_version` field from Node/IR loggers. Write version, build
time and debug flag once on start.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-11 12:35:12 +03:00
Leonard Lyubich
bec4507997 [#493] logger: Do not write app name in each log message of Node/IR
Writing application name in each log message seems pretty redundant.

Remove no longer used `misc.NodeName` and `misc.InnerRingName` constants.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-11 12:35:12 +03:00
Leonard Lyubich
db2e43387b [#493] logger: Simplify Logger's parameters
Replace `viper.Viper` parameter with `Prm` structure. Currently only logging
level can be parameterized through string setter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-11 12:35:12 +03:00
Leonard Lyubich
2a970b0125 [#493] logger: Do not set global fields in constructor
Remove setting of `app_name` and `app_version` fields in `NewLogger`
constructor. Set these fields in Node and IR application in already
constructed log and remove them from viper .

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-11 12:35:12 +03:00
Leonard Lyubich
d17526f8ac [#493] logger: Abolish non-usable options
Change logger's encoding to `console`, time encoding to `ISO8601TimeEncoder`
and leave all other options as they are in `zap.NewProductionConfig`.

Remove default values of no longer existing options in node/ir config.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-11 12:35:12 +03:00
Alex Vanin
d49bd4b94a Update neofs-api-go to v1.26.0
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-10 09:45:11 +03:00
Pavel Karpy
bd6d5e7f7b [#504] reputation/intermediate: Add reading alpha from global cfg
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-06 13:52:43 +03:00
Alex Vanin
dddbf0368c [#486] innerring: Add option to disable only main chain notary support
For N3 Testnet RC2 release inner ring app supports three modes:
- notary enabled in all chains (default),
- notary disabled in all chains,
- notary enabled only in side chain.

All notary related functions are moved to notary.go

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-05 12:45:16 +03:00
Alex Vanin
5a167f3991 [#486] innerring: Adopt disabled notary work flow
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-05 12:45:16 +03:00
Alex Vanin
1f3bb33db8 [#486] innerring: Add fee configuration
When notary disabled, inner ring should be able to
configure extra fee for vote collections inside the
contracts. Previously these values were hardcoded,
however we might want to change them depending on
a environment.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-05 12:45:16 +03:00
Alex Vanin
bd65e41257 [#486] innerring: Add notary support in main chain client
With `mainnet.notary_deposit=false` inner ring will ignore
notary deposit \ awaiting routines in the application start,
so it can run on the environments without notary support.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-05 12:45:16 +03:00
Pavel Karpy
20b7295087 [#488] reputation/router: Add commentaries and TODO
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
25ea5fea90 [#488] reputation: Initial Trusts using netMap
Make initial trust values depend on NetMap:
initial trust equals 1 / (`amount of storage nodes`).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
d1db54acf8 [#488] reputation: Change Writer interface
Includes:
- Delete first `ctx` argument in `Write` method.
- Move intermediate Initial trust struct and method
to `calculator` file.
- Change Alpha to 0.1.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
0d34d7c508 [#488] cmd/reputation: Collect all reputation wrappers
Init all wrappers eigenTrust algorithm needed in
`main` packages. Implement `SendIntermediateResult`
GRPC method of `reputationServer`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
eb74a9cafc [#488] reputation: Add commentaries
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
2c8c9f69c8 [#488] storage/calculator: Make alpha dynamic
Delete reading `alpha` from env var. Cover
retrieving `alpha` behind interface in
intermediate calculator. Add TODO to decide
if it is necessary to receive that param from
global config or not.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
ad4a55468e [#488] storage/config: Add wrapper and scripthash
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
ea781664cf [#488] reputation/eigentrust/calculator: Implement calc wrapper
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
d3c1fc7dda [#488] reputation/local/storage: Add trusting peer
Fix iteration logic after adding trusting peer
in `Trust` struct.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
b75074991b [#488] reputation/cmd: Add implementation
Change existing implementations and
add new in `main` package after defining
new interfaces.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
e69917b27a [#488] reputation/common: Move ServerInfo to common pkg
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
f7aa79f0b6 [#488] cmd/reputation/storage: Add consumers storage
Add consumer storage wrapper that implements
`WriterProvider` interface. Change field naming
in daughters storage package.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
e8885d72f4 [#488] reputation/remoteProvider: Implement intermediate remote
Move common remoteProvider code to cmd/reputation/common.
Hide WriterProvider initialization behind interface and
add implementation of that interface to local and
intermediate packages in cmd/reputation directory.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
49d477f466 [#488] reputation/managers: Implement route builders
Add implementation of Builder interface for
intermediate trusts. Move all code associated
with managers to `common` directory in `cmd`
and `pkg/services/reputation`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
b293ea3ba1 [#488] reputation/intermediate: Add implementation of
`DaughterTrustIteratorProvider` interface

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
35ec694b04 [#488] cmd/cfg: Update cfgReputation
Add `alpha` and `workerPool` to
`cfgReputation` structure.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Pavel Karpy
f6783f4f81 [#488] cmd/reputation: Add DaughterStorage
Add `DaughterStorage` init in main pkg
and start write all received daughters'
trusts to it.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-04 17:53:02 +03:00
Evgenii Stratonikov
59de521fd1 [#472] blobstor: implement write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-04-29 17:36:36 +03:00
Evgenii Stratonikov
a8a9f88d90 [#471] cmd: allow to start node as Relay node
Disable bootstrapping completely.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-04-27 17:15:36 +03:00
Evgenii Stratonikov
60d74bce67 [#466] cli: support NEP-2 and NEP-6 key formats
NEP-2 is contains WIF encrypted with password.
NEP-6 is format used for neo-go wallets.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-04-23 10:11:47 +03:00
Alex Vanin
e9b3488aab [#490] cmd/neofs-node: Do not send rebootstrap after node went offline
Control service turns storage node to online and offline modes. We can't
reuse `cfgNetmap.reBootstrapEnabled` because it represents configuration
value while control service manages state in runtime.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-22 09:02:58 +03:00
Pavel Karpy
ff7a8ae677 [#484] cmd/reputation: split it into subpackages
Split cmd/reputation into subpackages
for future refactoring.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-20 10:14:28 +03:00
Pavel Karpy
477682adb7 [#482] reputation/router: Move router to ./common
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-19 11:47:04 +03:00
Pavel Karpy
ac8441b718 [#482] reputation/router: Make route pkg independent
Make route package independent from
controller package. Add common interfaces
to `./common` directory.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-19 11:47:04 +03:00
Pavel Karpy
0a16aaacb1 [#482] reputation/router: Use trusting peer
In route building use `trusting` peer
field of `reputation.Trust` struct
instead of `reputation.Peer`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-19 11:47:04 +03:00
Alex Vanin
66ddff3498 [#479] cmd/neofs-node: Add eigen trust block timer
Eigen trust block timer ticks to start new round of
eigen trust calculations. Every epoch this timer
recalculates duration and starts again.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-19 09:35:54 +03:00
Pavel Karpy
f02de33f8b [#476] cmd/containter: Make network handlers async
Make container handlers that do network
communication execute asynchronously.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-15 15:11:34 +03:00
Pavel Karpy
a04f40d616 [#476] cmd/netmap: Make network handlers async
Make netmap handlers that do network
communication execute asynchronously.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-15 15:11:34 +03:00
Pavel Karpy
91e0f4b370 [#476] cmd/container: Add async handlers
Add `addContainerAsyncNotificationHandler`
func that passed handler to container's
worker pool similar to how it is done in
reputation handlers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-15 15:11:34 +03:00
Pavel Karpy
72699b4c26 [#476] reputation: Make reputation report async
Add handler closure over worker pool
in the event package.
Add `addNewEpochAsyncNotificationHandler`
function that uses that closure. Pass
the reputation report handler to worker
pool via using that function.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-15 15:11:34 +03:00
Pavel Karpy
7cd4e409eb [#476] cmd/cfg: Add worker pools
Add worker pools to Netmap and Container
config structures. Add its initialization
that depends on environmental variables(
sync/async; worker pool size).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-15 15:11:34 +03:00
Alex Vanin
f6f2e4d10f [#469] cmd/neofs-node: Don't send UpdateState tx at shutdown by default
if `shutdown.offline.enabled` config option set to true, then
send netmap.UpdateState(offline) tx at the shutdown. This config
option is not set by default.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-13 09:19:36 +03:00
Pavel Karpy
60cc3b3e16 [#460] reputation: Add Router to reputation server
Add `Router` to the reputation server. `Router` is
called on every incoming request and inits `Writer`
that sends `Trust`s to the next route point or
handle(logs in that implementation) them if current
node is the end point of the route.

Rename `onlyKeyRemoteServerInfo` struct for container
to separate it from the same implementation of the
same `ServerInfo` interface for reputation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-08 17:29:08 +03:00
Pavel Karpy
39f55611c6 [#460] reputation/local: Implement dependencies
Implement dependencies for local trust router
in `main` package. Managers are defined as
next node after the current one in HRW sorted
list of current NetMap.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-04-08 17:29:08 +03:00
Alex Vanin
1d68e74636 [#465] innerring: Add audit fee fetcher to settlement processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-08 17:01:00 +03:00
Alex Vanin
7cf48d4d91 [#452] Update neo-go to latest master
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-06 10:35:20 +03:00
Alex Vanin
b18da34b55 [#452] innerring: Use reputation processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-04-06 10:35:20 +03:00
Leonard Lyubich
732038ac28 [#444] cmd/node: Serve ReputationService on the node
Register reputation service server that:
  * verifies signatures of the requests and signs the responses;
  * attaches response meta header to the responses;
  * logs body structures of the requests (only as initial implementation, in
    future it will be replaces with local trust storage).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-04-01 14:54:22 +03:00
Alex Vanin
3965cc2296 [#447] innerring: Use alias type for slice of public keys
Alias type provide sort function so it is better to use it
everywhere where list of public keys is presented.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-26 12:38:42 +03:00
Alex Vanin
80bfd08a47 [#446] innerring: Use indexer to get relevant inner ring size and index
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-26 11:53:10 +03:00
Alex Vanin
97fe50d452 [#446] cmd/neofs-node: Get inner ring list from RoleManagement contract
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-26 11:53:10 +03:00
Leonard Lyubich
28fb8c971c [#428] cmd/node: Overload client constructor for Object service
There is a need to track the results of Object interactions with the remote
peers during node's lifetime. Each successful operation should increment the
number of satisfactory interactions with the remote peer, and the failed
ones, on the contrary, decrement. Collected numbers of satisfactory
interactions are going to be used for calculation of the normalized local
trust values according to original EigenTrust algorithm.

Implement wrapper over local trust storage that calls `Update` method on it
with the result of any object operation. The wrapper provides interface of
the remote client constructor. It is used as a `ClientConstructor` component
in Object service server of the app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Leonard Lyubich
a6f2394dec [#428] cmd/node: Trigger local trust controller on new epoch
Implement and call `initReputationService` func that constructs local trust
storage and controller, and subscribes the controller on new epoch
notification. Event handler calls `Controller.Report` method to process
collected values.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Leonard Lyubich
c4a1c70089 [#428] cmd/node: Implement logging writer provider
Implement simple `WriterProvider` building a `Writer` that writes incoming
data to the log. In the future, this action will be replaced by sending the
value to the manager nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Leonard Lyubich
d9f1491566 [#428] cmd/node: Implement iterator provider on local trust storage
Implement iterator provider on wrapper over the local trust storage.
Provided iterator passes normalized local trust values according to original
EigenTrust algorithm description. In the borderline case, normalized values
are set as 1/N, N - number of remote nodes in network map.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Leonard Lyubich
106884fc40 [#428] client: Hide client cache behind interface in dependent packages
Replace usage of `cache.ClientCache` type with interface with similar
signature. This will further allow overloading clients without affecting the
logic of dependent packages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Alex Vanin
518f375dfd [#441] cmd/neofs-node: Don't update access time in ttlNetCache
ttlNetCache should evict records after TTL duration. However if
data is often accessed and there are no LRU eviction (cache used
with small number of keys), then data will not be evicted ever.

This is a invalid behaviour for mutable data such as eACL.

Solution is to not update access time on every get, so the data
will be guarantee evicted after TTL duration.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-23 20:19:16 +03:00
Pavel Karpy
b9892edd6e [#416] innerring: Add gas threshold
On the `Deposit` events add gas balance check.
Make transfer only if the balance is greater
than the `GasTransferThreshold` that is defined
with environmental variable.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-03-19 13:57:58 +03:00
Leonard Lyubich
383d2494eb [#425] Adapt the Client's refactoring to the interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Leonard Lyubich
6204ca8308 [#425] cli: Support latest changes from API library
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Leonard Lyubich
718a2fad26 [#425] services: Define service interfaces that was removed from API lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Alex Vanin
7131e6f339 [#426] cmd/neofs-node: Fix metric option in local storage
`WithMetrics` can't make nil check without reflection so we
have to explicitly check if metrics enabled outside of
engine constructor.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
b8e10571c6 [#426] Put prometheus behind pkg/metrics
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
3c0e47e6fd [#426] cmd/neofs-node: Add metrics collector in node
Disabled by default.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
534ce03421 [#426] cmd/neofs-node: Refactor object service pipeline set up
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Evgenii Stratonikov
6679d59e89 [#422] pkg/services: Provide client options on cache creation
Because options are not used when client is already in cache
providing them to shared cache is misleading at best.
In the worst case `dial_timeout` is set randomly (because of race
condition) which can lead to one service having `dial_timeout` of
another. Thus we set default client creation options when cache is
created.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-03-15 14:07:53 +03:00
Evgenii Stratonikov
cc7287d6f7 [#422] pkg/services: Cache clients by address only
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-03-15 14:07:53 +03:00
Leonard Lyubich
55dec28bbb [#419] cmd/node: Cache network maps read from sidechain
Implement LRU cache of eACL tables read from sidechain. Use it as a netmap
storage in neofs-node app. Current cache size is set to 10 (constant).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-04 16:33:29 +03:00
Leonard Lyubich
f9342aa562 [#419] cmd/node: Cache eACL tables read from sidechain
Implement TTL cache of eACL tables read from sidechain. Use it as a eACL
storage in neofs-node app. Current cache size is set to 100, TTL is 30s
(constants).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-04 16:33:29 +03:00
Leonard Lyubich
34a51ed38b [#419] cmd/node: Cache containers read from sidechain
Implement LRU cache for values read from the network. Implement LRU cache of
containers read from sidechain. Use it as a container storage in neofs-node
app. Current cache size is set to 100, TTL is 30s (constants).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-04 16:33:29 +03:00
Leonard Lyubich
ed808c3f1b [#419] eacl: Replace eACL storage implementation to app-side
Replace `eacl.Storage` interface implementation from eACL lib to neofs-node
app package. Remove `eacl.WithMorphClient` option in order to abstract eACL
validator from eACL storage implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-04 16:33:29 +03:00
Alex Vanin
71dce97b76 [#404] innerring: Make notary deposit periodically
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
ccaf4f5d55 [#404] innerring: Enable notary support in morph client
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
948823c392 [#404] innerring: Get GAS script hash from neo-go client
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Leonard Lyubich
f1a4b8d322 [#217] cmd/node: Use local Inhume operation as a redundant copy callback
Exec `StorageEngine.Inhume` operation on redundant copy callback from Object
Policer with `MarkAsGarbage` parameter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
7a4e3efa95 [#217] storage engine: Transform Inhume to batch operation
Make `StorageEngine.Inhume` to process list of objects per single operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
53b4d6d6d0 [#396] ir: Support working with sidechain only
NeoFS contract can be deployed in sidechain instead of main chain.

Add `without_mainnet` config flag that can switch IR node to work with
sidechain only. By default this flag is unset.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:17:35 +03:00
Leonard Lyubich
ba85b9ff25 [#392] ir/config: Do not set default values of the alphabetic contracts
Previously we set default addresses of alphabetic contracts to empty strings
in inner ring application config. Recently number of using alphabetic
contracts became dynamic. In order to not enumerate default address for each
glagolitic letter, they are removed from config defaults.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
356962c4d9 [#392] ir: Set default amount of alphabetical contracts to 7
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
645bef9de9 [#401] cli: Add command to drop object from the node's local storage
Add `drop-objects` sub-command to `control` command that drops objects from
the node via `ControlService.DropObjects` API. Address list argument is
passed in NeoFS API string format of the object address.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 13:07:53 +03:00
Leonard Lyubich
aa24702ebe [#401] control: Serve DropObjects RPC
Re-compile protobuf definition of Control service. Implement required
messages on DropObjects RPC request and response messages. Implement
`DropObjects` method on Control service server of the node. Use
`StorageEngine.Delete` method as a deleted object handler on server.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 13:07:53 +03:00
Leonard Lyubich
2bb48642ee [#393] cli: Add dedicated flag to indicate object expiration
Add `--expires-on` flag to `put` sub-command of `object` command that
indicates object's expiration epoch. Set corresponding object attributes if
flag value is set.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 11:14:22 +03:00
Leonard Lyubich
cb9e6ea6fa [#398] cmd/cli: Add netinfo command to netmap section
Add `netinfo` sub-cmd of `netmap` cmd that read recent information about
NeoFS network. Info is read via NeoFS API NetmapService.NetworkInfo RPC.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 19:16:40 +03:00
Leonard Lyubich
9073e198b9 [#398] cmd/node: Serve NetmapService.NetworkInfo RPC
Implement `NetworkInfo` calls on full stack of Netmap services. Current
epoch is read from node local state, magic number is read via `MagicNumber`
call of morph client.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 19:16:40 +03:00
Leonard Lyubich
146abe8520 [#388] cmd/cli: Use bearer token when collecting storage group members
CLI `storagegroup put` cmd collects information about SG members via NeoFS
API ObjectService.Head RPC in order to compose SG structure. Bearer token
attached to the call was not used in communication, which could lead to data
access problems. These changes fix the described problem.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 16:20:21 +03:00
Leonard Lyubich
b644589bb4 [#378] cmd/node: Configure tombstone lifetime
Add `TOMBSTONE_LIFETIME` configuration value of the node which is measured
in NeoFS epoch and is set to 5 by default.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
b961570766 [#378] cmd/node: Subscribe shards on new epoch events
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
18add24727 [#378] cmd/node: Configure GC options
Add `GC_REMOVER_BATCH_SIZE` and `GC_REMOVER_SLEEP_INTERVAL` config values to
shard's config section.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Alex Vanin
d7bd312abd [#373] Update neo-go to pre-v0.93.0
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-11 17:31:56 +03:00
Leonard Lyubich
6cb906ab5f [#362] cli/object: Support JSON input of search filter
Consider single word of search filter expression as path to file with
protobuf JSON filters. Decode filters from file and add them to the rest.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
6039cd119c [#362] cli/object: Change query language of search filters
Make `--filters` flag to be repeated. Define new filter expression format:
 * `<key> <binary_op> <value>` for binary filters. Supported binary ops: `EQ` (`STRING_EQUAL`), `NE` (`STRING_NOT_EQUAL`).
 * `<key> <unary_op>` for unary filters. Supported unary ops: `NOPRESENT` (`NOT_PRESENT`).
Any other string expressions are considered invalid.

Additionally support shorthand flag `-f`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
3b9d4b4df1 [#362] cmd/node: Use const key to UN/LOCODE attribute from the API lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
37915b53cf [#362] Pull latest neofs-api-go lib
Adopt new names of constant keys to node attributes from API lib.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
e63f9faab8 [#316] cmd/node: Make UN-LOCODE attribute mandatory in the configuration
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
7a9b02049f [#316] cmd/node: Refactor working with well-known node attributes
Split the attributes into those that must be explicitly set in the
configuration, and those that, if absent, will be assigned a default value.
Support this logic in `addWellKnownAttributes` function. If no explicit
attribute is set, the application will panic.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
9a425924cf [#316] locode/boltdb: Replace all references to the City with Location
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
c384fbbf4e [#316] cmd/ir: Set default value of path to NeoFS LOCODE database
Make `locode.db.path` to be empty string by default in IR configuration.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
8a67eea854 [#371] cli/sg: Make output of saved SG address similar to saved object
Add container ID to save storage group output in `storagegroup put` command.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 10:58:08 +03:00
Leonard Lyubich
c6411e4f87 [#372] cli/sg: Support bearer token in storagegroup command
Add `--bearer` flag to storagegroup command similar to object command.
Attach parsed bearer token to all API requests of storagegroup command.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 10:58:08 +03:00
Leonard Lyubich
687af30c05 [#372] cli/sg: Support X-headers in storagegroup command
Attach X-headers to all requests of storagegroup commands (put,
get, list, delete).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 10:58:08 +03:00
Leonard Lyubich
cdab2fc520 [#316] cmd/cli: Close location database after locode info cmd execution
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
38466d5173 [#316] cmd/cli: Add section to work with UN/LOCODE
Add locode section to util cmd. Add generate command to locode section for
generating the NeoFS location databases. Add info command to locode section
to work with the entries from NeoFS location database.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Alex Vanin
9618c3e268 [#367] cmd/neofs-node: Don't update local node info for bootstrap
At startup, update only node info status. Leave all other
attributes from node configuration, so user can update them.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-04 14:16:22 +03:00
Alex Vanin
fd461bdb65 [#365] innerring: Produce distribute income events
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-03 15:17:58 +03:00
Alex Vanin
487c9b7589 [#363] Define global config and use it to fetch basic income rate
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-02 21:07:00 +03:00
Alex Vanin
be2ed6bf4c [#360] Run basic income collection in inner ring
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-02 21:07:00 +03:00
Leonard Lyubich
bd5c70131b [#328] cmd/node: Serve Container.AnnounceUsedSpace RPC
Register recently implemented handler of AnnounceUsedSpace RPC in node app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
b270c49b5c [#328] container/load: Process Start/Stop notification events from contract
Construct used space Controller on node's app-side. Call Controller.Start on
StartEstimation event from sidechain. Call Controller.Stop on
StopEstimation event from sidechain.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
e9cbdc4a0d [#328] cmd/node: Add the ability to listen Container contract's events
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
8419294f22 [#328] container/load: Implement AnnounceUsedSpace NeoFS API v2 RPC handler
Implement processing of AnnounceUsedSpace RPC from NeoFS API v2 method on
the app-side according to the following algorithm (except common steps for
each RPC):

 * check if the request sender is presented in container;
 * check route of the request (compose from signatures);
 * pass the value to Writer returned by WriterProvider.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
10902c0b82 [#328] container/load: Implement RemoteWriterProvider for controller
Define component on app-side that implements RemoteWriterProvider with next
logic:

 * if ServerInfo is nil, call internal dead end provider;
 * if address in ServerInfo equals to local node's address, do nothing;
 * otherwise create API client instance via client cache and return the
   wrapper over it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
1bed03995b [#328] container/load: Implement Iterator on local storage engine
Implement wrapper over StorageEngine on app-side that iterates over the
values returned by ListContainers+ContainerSize method pair. Component will
be used as the source of local metrics.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
910017b72f [#328] container/load: Implement PlacementBuilder for used space router
Implement wrapper over netmap and container storages on app-side that builds
placement vectors by next steps:

 * read container from storage;
 * read network map from storage by epoch number;
 * build container nodes according to placement policy;
 * build placement vectors with epoch-dependent pivot.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
6f552c8599 [#328] container/load: Implement WriterProvider on API client
Implement wrapper over API client on app-side that passes used space values
to it. Values are buffered on Put call and sent on Close call.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
fa685160f9 [#328] container/load: Implement Writer on Container contract's client
Implement wrapper over Container contract's client on app-side that passes
used space values to it. Uses a fixed public key, whose value will be the
key of the local storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Alex Vanin
5b550bff22 [#355] innerring: Produce container size estimation notifications
There are two notifications:
- start estimation notification produced at the beginning of the
  epoch,
- stop estimation notifications should be produced before
  basic audit settlement starts.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:23:37 +03:00
Leonard Lyubich
d01b4e1a2d [#324] ir: Measure GAS emission intervals in sidechain blocks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
19bb94cc04 [#324] ir: Measure epochs in sidechain blocks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Alex Vanin
e252f93dbf [#350] object/acl: Make best effort to classify request
Classifier looks at list of inner ring nodes and container
nodes from current and previous epoch to classify request.
Sometimes these checks might return error.

Consider there is a request from unknown key and container's
placement policy valid for current epoch and invalid for past
epoch. Classifier tries to find if key belongs to container
node from current epoch -- it is not. Then it tries to find if
key belongs to container node from past epoch and it throws
error, because placement policy is invalid for past epoch.

This is a legit case and classifier should ignore such errors
to provide best effort in matching. The only error classifier
should return is an error when request does not contain
public key to classify it.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-27 20:18:33 +03:00
Leonard Lyubich
198eb06032 [#339] cmd/node: Change the way the local NodeInfo structure is stored
Implement NodeState interface required by Netmap service. Make a single
point of updating the state of the node (for both Netmap and Control
services). Protect node info structure from data race.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-25 17:13:46 +03:00
Leonard Lyubich
3bf1519b1d [#323] cmd/node: Set BoltDB timeout option of Metabase component
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
cf00682065 [#323] cmd/node: Call Close method of the storage engine on app shutdown
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
9f41192bff [#323] cmd/node: Compose functions for closing components in app
Closing callback can be registered in app through onShutdown method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
707434efa9 [#323] cmd/node: Set default value of Blobovnicza's storage size limit
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
53f8f06f2d [#323] cmd/node: Set default value of small size limit cfg of BlobStor
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
0690a54299 [#315] cmd/cli: Add set-status command to control section
Add set-status command that uses SetNetmapStatus rpc of Control API.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 17:46:12 +03:00
Leonard Lyubich
fa0ea35f83 [#315] cmd/node: Implement node status updating in application
Call Netmap contract on SetNetmapStatus rpc of Control service (AddPeer
method if new status is ONLINE, UpdatePeerState method otherwise).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 17:46:12 +03:00
Leonard Lyubich
b418aa6130 [#312] cmd/cli: Add health status to healthcheck command output
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
4666a7797b [#312] cmd/node: Switch health status while the application is running
Switch health status to:
  * STARTING after the initialization of app resources and before bootup;
  * READY after bootup and until shutting down;
  * SHUTTING_DOWN before exiting the application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
147399f547 [#312] cmd/node: Implement HealthStatus method on node application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
71fcbd3ed4 [#313] cmd/node: Update netmap status in the right way
Update node network status via update status by getting the latest network
map and finding the node in it. This step is performed at the start of the
application and upon notification of a new epoch.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
83d4420a30 [#313] control: Rename HealthStatus enum to NetmapStatus
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Alex Vanin
a89567a88d [#317] morph/client: Return complete eACL signature from contract
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-15 16:25:49 +03:00
Leonard Lyubich
8d81954afe [#168] cmd/cli: Print netmap in JSON format
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
cd4055d0b7 [#168] cmd/cli: Add snapshot command to netmap section
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
3e882d963f [#311] services/control: Implement NetmapSnapshot method
Implement methods required for signatures. Receive network map from netmap
storage, convert it to Control service message and return in response.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
73e99fa590 [#306] cmd/node: Rename permitted_keys config value to authorized_keys
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
0a0ee89665 [#306] control: Rename WithAllowedKeys function to WithAuthorizedKeys
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
abd9952e46 [#306] Rename Private service to Control service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
df3746fa68 [#306] cmd/node: Switch health status on boot and shutdown
Implement HealthChecker on node app structure. Set health status to ONLINE
after node boot. Set health status to OFFLINE on shutdown.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
e8bd2eac0d [#307] cmd/cli: Implement private command for private API access
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Leonard Lyubich
f3cac6cc31 [#306] cmd/node: Serve private node service in storage node app
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-14 12:13:49 +03:00
Alex Vanin
ca225fa3e8 [#304] cmd/neofs-node: Update morph endpoint config indent
Now it has symmetric indent for RPC and notification endpoints.

```
  morph:
    rpc_endpoint:
      - http://seed1.neo.org:20333
      - http://seed2.neo.org:20333
      - http://seed3.neo.org:20333
    notification_endpoint:
      - ws://seed1.neo.org:20333/ws
      - ws://seed-go.nspcc.ru:30333/ws
    dial_timeout: 10s
```

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Alex Vanin
842fb9248c [#304] cmd/neofs-node: Tidy default configuration and log output
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Alex Vanin
63ebe41991 [#304] cmd/neofs-node: Select random NEO endpoints from the list
Application should support several NEO endpoints so it can
switch between different RPC nodes when they fail. Application
iterates over endpoints in random order so the default list of
endpoints distribute workload kinda uniformly.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Alex Vanin
3774c5d69a [#304] cmd/neofs-node: Catch closing channel of listener endpoint
As in #72 storage application should behave the same way at remote
RPC node failures. The simplest way is to restart application. Later
we can reinitialize it without downtime.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 19:20:27 +03:00
Leonard Lyubich
1e170c3812 [#234] services/object: Support netmap epoch and lookup dead in read ops
Support processing of NetmapEpoch and NetmapLookupDepth X-headers when
processing object read operations. Placement for operations
Get/Head/GetRange/GetRangeHash/Search is built for the epoch specified in
NetmapEpoch X-header (by default latest). Also the specified operations are
processed until success is achieved for network maps from the past up to
NetmapLookupDepth value. Behavior for default values (zero or missing) left
unchanged.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-12 19:19:11 +03:00
Alex Vanin
b62b19cb5a [#302] Fix goimports linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
e88c1f750d [#302] Fix golint else/return linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
0f3570bdeb [#302] Fix golint naming linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
74a68af5c2 [#302] Fix golint underscore linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Alex Vanin
5e231c515a [#302] Fix whitespace linter errors
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-12 18:47:02 +03:00
Leonard Lyubich
d58e28afee [#298] morph/netmap: Use netmap.NodeState argument in UpdatePeerState
Replace custom NodeState enum with the one from netmap lib. Remove no longer
used NodeState enum.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:42:09 +03:00
Leonard Lyubich
b45360b933 [#298] cmd/node: Go offline on application shutdown
Transfer the state of the node to offline when the application is shut down.
Updating the state is done by calling UpdateState method of Netmap contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:42:09 +03:00
Leonard Lyubich
07a6e2cc43 [#290] cmd/cli: Trim 0x prefix from salt argument of object hash cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:40:59 +03:00
Leonard Lyubich
6df8c84fc2 [#290] cmd/cli: Add salt flag to object hash command
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-11 18:40:59 +03:00
Leonard Lyubich
9a86fff7e0 [#291] Remove some unused code from repository
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-30 10:27:53 +03:00
Alex Vanin
df116852f2 [#289] Update neo-go version to v0.92.0
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-29 17:42:02 +03:00
Alex Vanin
7174abcc7c [#281] service/audit: Fix typos
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
0d7832f5e9 [#281] service/audit: Add separate pool for SG checks in PoR
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
3ec342b2c3 [#259] services/audit: Implement PDP check
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
310a4c3e4d [#275] innerring: Add storage group search timeout
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
782bcadd92 [#271] innerring: Implement GetSG and Head of audit communicator
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
ecb5227c27 [#255] cmd/ir: Set default values of audit configuration
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
87e1252065 [#265] innerring: Generate new audit event
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
f7ca4a8dce [#80] cmd/cli: Implement storagegroup commands
Implement PUT/GET/LIST/DELETE commands of storage group module.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-25 16:49:27 +03:00
Alex Vanin
feec559e47 [#254] Update neo-go to preview4 compatible version
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-25 16:49:27 +03:00
Leonard Lyubich
51b85b0a73 [#278] cli: Support request X-Headers
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-24 15:03:02 +03:00
Leonard Lyubich
e53bf574b5 [#279] container: Use new methods to work with container format
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-24 13:38:20 +03:00
Alex Vanin
7daf3727ee [#248] neofs-cli/object: Print split header values in Head command
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-14 14:46:59 +03:00
Leonard Lyubich
6814140a19 [#240] cmd/cli: Support raw flag in object GET/HEAD/RANGE commands
Support boolean raw flag. Print split information upon receipt.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-14 12:05:13 +03:00
Leonard Lyubich
09916f21b5 [#243] cmd/cli: Write tombstone address in Object.Delete cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
fe3906c295 [#243] object/delete: Implement new service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
611a29f682 [#241] object/search: Refactor service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
0e1f05ff45 [#239] object/head: Implement new service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
7e56427534 [#235] object/getrangehash: Implement new service processing
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
1d23483828 [#235] services/object: Implement new GetRange algorithm
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
a875d80491 Remove outdated code of metabase and localstore
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
869d9e571c [#233] object/head: Remove getting the right child header
With the update of the local storage engine, the headers of virtual objects
are directly given. In this regard, the step with obtaining the the right
child header is removed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
f24daa10ff [#233] services/object: Implement new Get algorithm
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
a488d202e1 [#230] engine: Rename WithAddress to WithAddresses for delete operations
Delete works with multiple addresses at once so it is convenient to
show it in parameter setter function.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
3d77fdb347 [#222] Set write cache specific config on application level
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
4114028a63 [#222] Add shard amount check
There should be at least one correctly set up shard in node config.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
351e4b4592 [#222] Support Inhume and Delete in object service
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
f45b7a048d [#222] Rename WithTreeRoot.. to WithRoot.. in blobstor opts
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
92c95150cd [#222] Use write cache options in storage app
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
2eb83624cf [#220] cmd/neofs-node: Configure Shard's write cache usage
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
cd2228cd99 [#220] cmd/neofs-node: Support new configuration of storage components
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
a1164b2a5c [#211] cmd/neofs-node: Add SmallSizeLimit config value of BlobStor
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Leonard Lyubich
4ca1035aa4 [#186] cmd/neofs-node: Integrate new storage engine into application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
49666f87f1 [#170] Implement --vote argument in neofs-ir
With this argument neofs-ir will be run as an CLI
application that initialize inner ring server, invokes
`vote` method on corresponding alphabet contract and
exits.

User can provide a comma-separated list of validator
public keys.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
0b65c64e98 [#170] Vote for validators on inner ring startup
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-12-10 17:22:37 +03:00
Alex Vanin
4c0ca23b34 [#208] Print split ID in object.Head output of neofs-cli
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-30 10:44:15 +03:00
Leonard Lyubich
0168740c9a [#206] cli/container: Set default version of eACL table in set-eacl
Set version of parsed eACL table to using SDK version if unsupported version
was set in source data.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-25 14:35:27 +03:00
Leonard Lyubich
b4699b1e9f [#206] cli/container: Use GetEACLWithSignature in set-eacl polling
Replace GetEACL call with GetEACLWithSignature one in polling step of
set-eacl command of container CLI.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-25 14:35:27 +03:00
Leonard Lyubich
e8bc03d92f [#206] cli/container: Use GetEACLWithSignature client method
Use GetEACLWithSignature client method in get-eacl cmd of container CLI in
order to print eACL table and signature regardless of their correctness. The
ability to check the correctness of the signature will be added later.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-25 14:35:27 +03:00
Leonard Lyubich
621840d542 [#193] cmd/neofs-node: Add dial timeouts to node configuration
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 15:34:30 +03:00
Leonard Lyubich
400b65801d [#201] cli/container: Do not check eACL version
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:56:03 +03:00
Leonard Lyubich
d286824d1d [#194] cmd/neofs-node: Use non-blocking worker pools for Object services
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:56 +03:00
Leonard Lyubich
6a5c37d592 [#195] object/search: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
cb46e4b154 [#195] object/rangehash: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
af6484e3b1 [#195] object/range: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
cf2dc37a42 [#195] object/put: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
9fbfc0b5e4 [#195] object/head: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
e1e5a590e9 [#195] object/delete: Add option to set logger
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-24 13:38:06 +03:00
Leonard Lyubich
ea36789b51 [#179] cmd/neofs-cli: Implement range command in Object section
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-23 13:23:31 +03:00
Leonard Lyubich
dbfb6fadd6 [#177] cmd/neofs-cli: Add object ID filter flag to search cmd
Add "oid" flag that allows to filter objects by identifier.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-23 09:54:59 +03:00
Alex Vanin
e8fe07edd0 [#184] Use SDK client cache in object.Rangehash
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
f85e88c4f8 [#184] Use SDK client cache in object.Range
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
d485a5967d [#184] Use SDK client cache in object.Search
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
7ba95dd5fc [#184] Use SDK client cache in object.Put
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Alex Vanin
e9a6365333 [#184] Use SDK client cache in object.Head
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-18 18:18:07 +03:00
Leonard Lyubich
58fcb35fb0 [#174] Use Marshal(JSON)/Unmarshal(JSON) methods for encoding/decoding
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Leonard Lyubich
3de8febe57 [#174] Update to latest neofs-api-go changes
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-17 11:56:00 +03:00
Alex Vanin
e47c866d6a [#169] Add --ttl argument in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 12:50:21 +03:00
Alex Vanin
e580f2afd6 [#160] Update object CLI flags
* remove no-leaf
* rename leaf to phy
* remove no-root

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
eb13322bf0 [#166] Use async pools in object service handlers
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-13 15:54:00 +03:00
Alex Vanin
ded45e1fbc [#159] Add keyer to neofs-cli
Keyer prints information about private key, public key,
NEO3 Wallet, scripthash. It can generate new private key
or generate multisig address.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-09 18:59:21 +03:00
Leonard Lyubich
3c17e871e1 [#158] cmd/neofs-node: Handle objects from tombstone by metabase
Handle tombstone content by metabase instead of object GC queue.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
36d5af812e [#158] metabase: Add logger to DB configuration
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
60e4b5ddff [#158] metabase: Construct DB using options
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Alex Vanin
0021f1cc5f [#81] cmd/neofs-cli: Use hex encoding of node's public key
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 16:37:48 +03:00
Leonard Lyubich
25398262fc [#81] cmd/neofs-cli: Implement "nodeinfo" sub-cmd of "netmap" cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 16:37:48 +03:00
Leonard Lyubich
5d94594007 [#81] cmd/neofs-cli: Implement "epoch" sub-cmd of "netmap" cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 16:37:48 +03:00
Alex Vanin
65be09d3db [#155] Update neofs-api-go with refactored pkg/netmap
Refactored pkg/netmap package provides JSON converters for
NodeInfo and PlacementPolicy structures, that has been used
by client applications.

It also updates Node structure itself so it is a part of
grpc <-> v2 <-> pkg conversion chain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 09:55:05 +03:00
Alex Vanin
49da96d006 [#146] Set stdout as default output for CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-05 18:37:24 +03:00
Alex Vanin
64caa15556 [#148] Use base name as FileName attribute value
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-05 18:37:06 +03:00
Alex Vanin
54818d5a11 [#144] Support well-known application attributes for object in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-05 09:28:13 +03:00
Alex Vanin
83926647d7 [#144] Support well-known application attributes for container in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-05 09:28:13 +03:00
Leonard Lyubich
8125b544b4 [#136] cmd/neofs-node: Use new metabase in app
Remove BoltDB bucket package. Construct meta.DB instance in node app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Alex Vanin
97077294fc [#141] Fix double sidechain GAS emission on asset mint
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-03 16:28:01 +03:00
Alex Vanin
0c49c08609 [#139] Use storage node GAS emission in inner ring app
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-03 16:20:41 +03:00
Alex Vanin
83119c00ec [#127] Remove magic number from config
With neo-project/neo-modules#358 nodes support RPC to
return used magic number. Therefore client doesn't need
that configuration value any more.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 13:54:31 +03:00
Alex Vanin
2bd827a478 [#134] Resend bootstrap txs periodically
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 10:53:11 +03:00
Alex Vanin
976ba06300 [#132] Use netmap cleaner in inner ring server
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-02 10:52:52 +03:00
Alex Vanin
b12d26f974 Update neofs-api-go to latest version
Handle errors provided by JSON encoders.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-30 11:28:49 +03:00
Alex Vanin
b93c146824 [#26] Add extnded ACL table converter in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-28 13:36:25 +03:00
Alex Vanin
c4e9e258fd [#120] Add utility command to sign bearer token in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-23 15:31:04 +03:00
Alex Vanin
e7f9da9139 [#120] Add --bearer argument in CLI
Bearer token should be stored in file and correctly
signed.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-23 15:31:04 +03:00
Leonard Lyubich
334e0e6f0f [#109] cmd/neofs-node: Activate Replicator
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 15:23:22 +03:00
Alex Vanin
0c3e6c7a04 [#40] cmd/neofs-node: Remove max object size value from config
Now max object size value is taken from global config in sidechain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-23 14:18:00 +03:00
Leonard Lyubich
23f00a3cd2 [#40] cmd/neofs-node: Re-implement max object size source
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 14:03:25 +03:00
Leonard Lyubich
d05b2ff500 [#83] cmd/neofs-node: Add network state option to response service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 10:54:48 +03:00
Leonard Lyubich
19f9c7eacb [#83] services: Remove setting of meta header from executing services
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 10:54:48 +03:00
Leonard Lyubich
3065a7363b [#83] cmd/neofs-node: Use response services in application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-23 10:54:48 +03:00
Alex Vanin
23ec33e821 [#106] Check bearer token lifetime
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-22 18:02:11 +03:00
Leonard Lyubich
0bdfd18469 [#60] cmd/neofs-node: Initialize network state on boot
Get current epoch number from morph client and store the value in local
state structure.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-22 11:54:08 +03:00
Leonard Lyubich
a8481223d6 [#60] cmd/neofs-node: Implement primary network State instance
Define networkState structure and implement netmap.State interface on it.
Set epoch number on NewEpoch notification from chain. Provide
WithNetworkState option to Put service constructor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-22 11:54:08 +03:00
Alex Vanin
c28641d290 [#95] Support --json flag in container.get command
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-21 18:22:59 +03:00
Alex Vanin
e40dc0412d [#26] Add get and set EACL commands in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-21 18:22:59 +03:00
Alex Vanin
1482b0c5d0 [#79] Simplify getCID and getOID functions in CLI
Parser from string is available in neofs-api-go.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-21 18:22:59 +03:00
Leonard Lyubich
f52f643fe4 [#108] cmd/neofs-node: Add Policer worker to application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-21 14:42:51 +03:00
Leonard Lyubich
07fe49088c [#108] cmd/neofs-node: Listen new epoch notifications from morph
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-21 14:42:51 +03:00
Alex Vanin
e6f04f7785 [#104] Update neofs-api-go with new protobuf API
Also update contains JSON converters for neofs-cli
and fixes bug in container.set-acl command of SDK.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-20 17:31:59 +03:00
Evgenii Stratonikov
4d2ff842a2 [#79] Implement object commands
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2020-10-19 14:35:39 +03:00
Alex Vanin
b7dfc73bbc [#25] Use comma separated container attributes in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-17 13:16:59 +03:00
Alex Vanin
feac279ced [#25] Add container.get command in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-17 13:16:59 +03:00
Alex Vanin
26e13f4b28 [#25] Add container.list-objects command in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-17 13:16:59 +03:00
Alex Vanin
c0a4343282 [#25] Use api-go stringers and parsers for ID types in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-17 13:16:59 +03:00
Alex Vanin
db0dd6363f [#25] Add container.create, list, delete commands to CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-15 10:25:41 +03:00
Alex Vanin
d83411016a [#25] Use printVerbose wrapper for one-line messages
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-15 10:25:41 +03:00
Evgeniy Kulikov
53f7b58130 [#93] Add gRPC Reflection service
Added gRPC Reflection service that can be enabled
by settings `grpc.enable_reflect_service`.

Read more about at
https://github.com/grpc/grpc-go/blob/master/Documentation/server-reflection-tutorial.md

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
2020-10-15 10:20:58 +03:00
Alex Vanin
dd48666357 [#72] Shutdown inner ring app if RPC node has been terminated
Adopt error channel from Listener interface.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-14 09:37:48 +03:00
Alex Vanin
a9d296faa8 [#87] Use viper to configure key and endpoint in CLI
With viper we can set up key and endpoint with config
and environmental variables with `NEOFS_CLI` prefix.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-13 14:02:48 +03:00
Alex Vanin
69dd95d748 [#78] Generate random key in CLI with --key new argument
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-13 14:02:48 +03:00
Alex Vanin
c2083d773c [#76] Add owner argument for accounting.balance command
With `--owner` argument user can look for balances of other
nodes by knowing their owner ID which is NEO3 compatible
address.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-13 14:02:48 +03:00
Alex Vanin
945bb723ed [#76] Do not print error message twice in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-13 14:02:48 +03:00
Alex Vanin
80cd4039a8 [#76] Implement accounting.balance command in CLI
It prints result as a float type value. With verbose flag
it prints decimal precision and value.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-13 14:02:48 +03:00
Alex Vanin
84928527d7 [#87] Add verbose global flag in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-13 14:02:48 +03:00
Alex Vanin
cb1ca511cb [#77] Add endpoint global argument in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-13 14:02:48 +03:00
Alex Vanin
fc700cbbdc [#78] Add private key global argument in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-13 14:02:48 +03:00
Alex Vanin
f1f20b49db [#86] Add alphabet contract processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-12 14:43:33 +03:00
Alex Vanin
4308a6f522 [#86] Run timer for gas emission event
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-12 14:43:33 +03:00
Alex Vanin
6bc787bb19 [#86] Add alphabet contract configuration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-12 14:43:33 +03:00
Alex Vanin
dee1d81b04 [#84] Run netmap service in neofs-node app
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-09 09:15:18 +03:00
Alex Vanin
20c27d0542 [#84] Add API version to neofs-node config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-09 09:15:18 +03:00
Alex Vanin
65a93af176 [#84] Fix loop variable capture issue in neofs-node workers
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-09 09:15:18 +03:00
Alex Vanin
0e7e0bd2d6 [#84] Remove mocks and debug code from neofs-node services
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-09 09:15:18 +03:00
Alex Vanin
a9e801cb22 [#71] Make GC configurable
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-05 14:03:55 +03:00
Leonard Lyubich
1d676fcfb2 [#69] object/acl: Add eACL components to service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 14:02:14 +03:00
Leonard Lyubich
30e6912c7b [#69] object/acl: Construct service from options
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 14:02:14 +03:00
Leonard Lyubich
e5898c9ca8 [#69] object/acl: Rename BasicChecker to Service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 14:02:14 +03:00
Leonard Lyubich
ab57fddb2b [#70] cmd/neofs-node: Add object GC to app
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 09:36:29 +03:00
Leonard Lyubich
798fca9354 [#70] core/object: Process a delete group at tombstone
Send object group to delete queue processor after tombstone content
validation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 09:36:29 +03:00
Leonard Lyubich
276ed6c04b [#70] cmd/neofs-node: Implement Del method on in-memory bucket
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 09:36:29 +03:00
Leonard Lyubich
9a604a50b9 [#70] cmd/neofs-node: Start asynchronous workers on app launch
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-05 09:36:29 +03:00
Alex Vanin
6fe6005e71 [#66] Check basic ACL in object service pipeline
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 19:47:49 +03:00
Alex Vanin
2ee24998ba [#68] Add go pprof to neofs-storage
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 19:47:08 +03:00
Alex Vanin
6ee908c2db [#65] Support FS bucket and BoltDB bucket
These buckets can be used to store blobs and metadata.
They will be removed as enhanced blob storage will be
implemented for neofs-node. To setup storage type
use `storage.object.type` and `storage.meta.type` params.

Available options:
  - inmemory (default)
  - boltdb
  - filesystem

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 14:48:00 +03:00
Alex Vanin
861bac3892 [#59] Use max msg size in transport server and splitter
For GRPC it is about 4 MiB.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
d2009c8731 [#59] Add grpc payload splitter in object service chain
GRPC has default message limit of 4MiB. Since every transmitted
neofs message has to be signed, then original message should
be split into transfer fit structures before signature service.

This commit introduce transport payload splitter for object
service pipeline. This splitter works with stream response
for methods:

  - object.Get
  - object.Range
  - object.Search

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
13080aa7a1 [#43] cmd/neofs-node: Fetch max object size from config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
e158497560 [#43] cmd/neofs-node: Support hostnames with dns, ipv4 and ipv6 addresses
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
8585e147d0 [#43] cmd/neofs-node: Use morph wrappers as data sources
Morph wrappers are being uses as sources for containers and
snapshots of network map.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
1914b4dbad [#43] cmd/neofs-node: Use object service in application
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
ea511d559c [#43] cmd/neofs-node: Refactor mocked object service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
a5ae128f34 [#43] cmd/neofs-node: Implement mocked max object size source
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
8dced272d2 [#43] cmd/neofs-node: Implement in-memory bucket
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
480362b02f [#43] cmd/neofs-node: Make cfg provide network address source interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
caedef82af [#43] cmd/neofs-node: Change object executor mocked calls
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00