Assume contracts path is tar.gz archive if it is not a directory.
In theory we could support raw tar but it has little value as our
distribution format is tar.gz and this is the primary use-case for this
feature (download and use).
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
In case we have multiple sections with similar structure (e.g. shards)
having defaults in a single place is easier to work with.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
There is no point to pass key storage in parameters because
it can be defined on the service level of application.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
```
> source config/example/node.env
config/example/node.env: строка 14: /dns4/s02.neofs.devenv/tcp/8081: Нет такого файла или каталога
config/example/node.env: строка 16: MSK: команда не найдена
config/example/node.env: строка 35: 028f42cfcb74499d7b15b35d9bff260a1c8d27de4f446a627406a382d8961486d6: команда не найдена
config/example/node.env: строка 48: https://rpc2.morph.fs.neo.org:40341: Нет такого файла или каталога
config/example/node.env: строка 49: wss://rpc2.morph.fs.neo.org:40341/ws: Нет такого файла или каталога
config/example/node.env: строка 53: https://rpc2.n3.nspcc.ru:30341: Нет такого файла или каталога
```
New implementation is still basic, but we don't need to parse any env
file, just have a correct example.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Note that we cannot mark `--rpc-endpoint` flag as required because
it can be taken from config.
Before:
```
Error: can't create N3 client: failed to get network magic: Post "": unsupported protocol scheme ""
```
Now:
```
Error: can't create N3 client: missing endpoint
```
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
```
Error: can't to initialize context: can't read alphabet wallets dir: open : no such file or directory
```
It needs neither contracts nor wallets, just N3 client.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Make the implementation of network info source (Netmap V2 service
dependency) to read MillisecondsPerBlock sidechain parameter and NeoFS
network parameters depending on the client version.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
SN tries to keep 1:3 proportion of GAS and
notary balances respectively. If that proportion
has been messed(means that notary balance is
lower than required) it sends half of its
GAS balance to the notary service.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
After storage engine started to limit number of PUT operations there is no
need to limited worker pool in Object Put service.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add `shard_pool_size` config to `storage` section. Set app default to 20.
Pass the value to `WithShardPoolSize` option.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Non-alphabet contract are always deployed from committee
so it makes sense to calculate hashes in one place.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Container contract uses actual NNS interface. This also aleviates some
pain related to update as neofs-adm code itself uses current NNS
version.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Container contract now calls NNS contract to register TLD for nice
names. This must be witnessed by a committee.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Do not log in options constructors. Also failure to
initialize compression module (possibly due to invalid options) is
certainly an error deserving proper treatment.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Implement `list` command for object listing. Support listing of the objects
from write-cache and blobovnicza.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement skeleton of `neofs-lens` app which is going to be used for working
with storage engine's data.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Also delete `ttl` and `xhdr` flags from
`accounting balance` command and refactor
command initialization.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
In previous implementation of reputation client constructor compared network
addresses to check the membership of the network map.
Replace network addresses comparison with public key comparison.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
There is a need to have the ability to expand the data needed for client
construction.
Replace `network.AddressGroup` parameter of client cache interfaces with
`client.NodeInfo`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation Object PUT used single pool of workers for local
and remote ops, but these ops are heterogeneous.
Use remote/local pool for remote/local operations in PUT service. At first
the pools are configured with the same size.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Contract owners are removed in neofs-contract v0.11.0.
Now side chain committee has rights to update contracts.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Implement `RefillMetabase` method to `shardconfig.Config` type which reads
`refill_metabase` config value. Pass the result to `WithRefillMetabase`
option.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add `size_limit` config value to write-cache section. Add `SizeLimit` method
of `writecache.Config` type. Pass its value to `WithMaxCacheSize` option on
app construction.
Reflect config update in examples. Cover new value in unit test.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Bootstrap process involves `addPeer` contract
call => it is necessary to make deposit before
booting up.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Create non-notary container wrapper and
use it in load announcing since it shouldn't
be invoked as notary request.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Storage Node needs to have notary deposit
for successful notary request sending.
Add notary deposit on startup(and wait for
its acceptance). Add notary deposit timer,
its config in `morph` section and env vars
for its tuning.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Prepare all listening structures for notary events:
rename(add prefix/suffix 'notification') all
notification specific handlers/parsers.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Some software components regulate the way of working with placement arrays
when a local node enters it. In the previous implementation, the locality
criterion was the correspondence between the announced network address
(group) and the address with which the node was configured. However, by
design, network addresses are not unique identifiers of storage nodes in the
system.
Change comparisons by network addresses to comparisons by keys in all
packages with the logic described above. Implement `netmap.AnnouncedKeys`
interface on `cfg` type in the storage node application.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Since morph `Client` works in multi-client mode, there is an error case when
we can not get network magic when all endpoints are unavailable.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
There is a need to work with a set of Neo RPC nodes in order not to depend
on the failure of some nodes while others are active.
Support "multi-client" mode of morph `Client` entity. If instance is not
"multi-client", it works as before. Constructor `New` creates multi-client,
and each method performs iterating over the fixed set of endpoints until
success. Opened client connections are cached (without eviction for now).
Storage (as earlier) and IR (from now) nodes can be configured with multiple
Neo endpoints. As above, `New` creates multi-client instance, so we don't
need initialization changes on app-side.
`Wait` and `GetDesignateHash` methods of `Client` return an error from now
to detect connection errors. `NotaryEnabled` method is removed as unused.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation Container service handlers didn't cache the
results of `Get` / `GetEACL` / `List` operations. As a consequence of this,
high load on the service caused neo-go client's connection errors. To avoid
this there is a need to use cache. Object service already uses `Get` and
`GetEACL` caches.
Implement cache of `List` results. Share already implemented cache of Object
service with the Container one. Provide new instance of read-only container
storage (defined as an interface)to morph executor's constructor on which
container service is based. Write operations remained unchanged.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Storage node should not try to register itself in network in relay mode.
Implement `needBootstrap` method which checks if node need to bootstrap.
Call `bootstrap` method in `bootstrapNode` function only on true return.
Skip re-bootstrap logic in new epoch event handler on false return.
Return an error if `ControlService.SetNetmapStatus` is called on relay
node.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
`reBootstrapEnabled` state var is not used. It is going to be used to decide
whether to bootstrap node or not.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add `refill-gas` sub-command to `morph` command which provides the ability
to refill storage node's GAS.
Command performs some actions from `generate-storage-wallet` runner, so
common code is moved to a separate function `refillGas`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
`--container-contract` flag must be used for deployments without NNS.
Our current testnet sidechain is like this, for example.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
`morph dump-containers` will dump all containers from the
contaner contract. JSON format is chosen to allow manual intervention.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Do not init caches for eACL, containers and netmap
if `disable_cache` config options is `true`, use
direct RPC calls instead.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Initial GAS can be provided both in config and as a CLI argument.
Generating wallet with 0 GAS is currently prohibited.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
If `init` is run on a dirty network (i.e. with some stages already
done), no transactions are really sent so clean up logs a bit.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Depending on having notary contract in sidechain get
IR list either from NeoFSAlphabet role either from
netmap contract.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Some users want to specify only wallet section in the SN. It is not
possible if `Key` throws panic on empty value. Instead it should
fallback to wallet section. Panic is suitable if node's key is provided
but invalid.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Add `errf`, `exitOnErr` and `exitOnErrCode` functions
that works with errors and exits with non-zero exit
codes on non-nil errors.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Extended ACL input can have version later than CLI one, and it should not be
downgraded. But the version should be still adequate.
Set `pkg.SDKVersion` on false return of `version.IsValid`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Changes:
* replace `iotuil` elements with the ones from `os` package;
* replace `os.Filemode` with `fs.FileMode`;
* use `signal.NotifyContext` instead of `NewGracefulContext` (removed).
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
There is a need to support multiple network addresses of the storage nodes.
Make `BootstrapAddress` to return `network.AddressGroup` (and rename).
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Generalize single gRPC interface of the storage node to a group of
interfaces. Each interface calls the same RPC handler.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation if Control listen endpoint was omitted in config
the gRPC listening endpoint was used instead.
Consider empty address as an option to disable the service.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
There is a need to support multiple server endpoints for reputation and
container transmission.
Replace `ServerInfo.Address` getter with `ServerInfo.IterateAddresses`
iterator.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make Object Head service to work with `AddressGroup` instead of `Address`
in order to support multiple addresses of the storage node.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make Object Search service to work with `AddressGroup` instead of `Address`
in order to support multiple addresses of the storage node.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make Object Get service to work with `AddressGroup` instead of `Address` in
order to support multiple addresses of the storage node.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make Object Put service to work with `AddressGroup` instead of `Address` in
order to support multiple addresses of the storage node.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make `LocalAddressSource.LocalAddress` method to return `AddressGroup`. Make
`IsLocalAddress` function to accept parameter of type `AddressGroup`. Adopt
the application code with temporary `GroupFromAddress` helper.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Change type of the `ClientCache.Get` method's parameter to `AddressGroup`.
Use `GroupFromAddress` to call the method from the wrappers in order to no
change their interface.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Add group-address `Client` implementation. Return instances of this
implementation from `ClientCache.Get` method.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
There is a need to generalize single-address client to group-address client.
To do this, we can re-implement `Client` interface from NeoFS API Go library
and still use it in the application code. There is a problem with method
`Raw` which must return single-address raw client. So as not to make changes
to API library we need to overload Client interface in order to support
`Raw` method in group-address client implementation.
Define `Client` interface in new `pkg/core/client` package. Completely
inherit API `Client` interface. Add `RawForAddress` method to build raw
client for the single node address. Adopt the application code that used Raw
method to work with new `Client`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Obtain key once in every cobra command to
pass it to `getOwnerID` and `initSession`
and do not ask to enter password more than
one time in `put` and `putSG` operations.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Change usage function to `Run`(does not return
err). Log errors with `cmd.PrintErrln`. Change
all `fmt.Print*` to `cmd.Print*`.
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
`network.Address` structure in most cases created once and used read-only.
Replace `AddressFromString` function with `Address.FromString` method with
the same purpose and implementation. Make all libraries to work with value.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement `Address.WriteToNodeInfo` method which sets address of `NodeInfo`
structure. Use it in storage node application.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Return tyype is clear from the method's signature and docs, there is no
point in reflecting it in the name.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Panic if internal `manet.DialArgs` call returns error since this is
unexpected according to `AddressFromString` implementation.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Serve `ControlService` instance on configured endpoint (do not serve if not
specified). Read allowed keys from config.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Also use neo-go private key wrapper where possible, as it
already has methods for (un)marshaling.
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Config `NodeInfo` should be used for bootstrap. Separete local node info and
netmap one. Return configured `NodeInfo` if structure from netmap is
missing.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In testnet and neofs-dev-env we use timers with these
settings and they are looking good. So it makes sense
to make them default.
Storage node now can't configure bootstrap healthcheck
length and can't disable it. Inner ring default values
now adopt these changes too.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
New config package in storage node does not use
application prefix from misc package. Therefore
inner ring node can define prefix in local scope.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Container commands in NeoFS CLI can use signed session token
to create, delete container and change extended ACL table.
This token should be signed the same way we sign bearer tokens.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
containerSessionToken() actually parses any session token.
We can reuse this function to resign session token when it
takes filepath as an argument, instead of public variable.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
In previous implementation `Config.Sub` method could lead to the violation
of the internal `path` slice because of `append`. This has been observed on
deeply nested subsections.
Fix `Config.Sub` to copy internal slice in order to prevent violations.
Cover problem case in test config files and unit test.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Dedicated type `LoggerSection` turned out to be redundant since it doesn't
do a hidden logic and just uses `config.Config` API.
Remove `LoggerSection` type and implement `Level` which do the same.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation ENV values were read only if config filepath is
specified. From now ENVs are always read to `Config`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In some cases viper doesn't interpret `section.value` as a subsection with
`section` name, but value is value still can be accessed through full
pathname.
Fix `Config.Sub` method implementation in order to always interpret
configuration like described above as a subsection. From now method never
returns nil, therefore an additional check has been removed from the `Value`
method.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Create `logger` sub-package of `config` package. Implement `LoggerSection`
type of logger sub-section. Add `Level` method to read logger level config
value. Default level is `info`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Create `config` package nearby storage node application. Implement `Config`
as a wrapper over `viper.Viper` that provides the minimum functionality
required by the application.
The constructor allows you to read the config from the file. Methods are
provided for reading subsections and values from the config tree. Helper
functions are implemented to cast a value to native Go types.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Container sessions allow to perform some operations on behalf of another
user. There is a need to to attach session tokens to commands.
Add `session` flag to `put`, `delete` and `set-eacl` commands from
`container` section. It should be a path to the JSON-encoded session token.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement function `fatalOnErrDetails` similar to `fatalOnErr` but accepting
string details that are written to log output. Use the function everywhere
in application code without wrapping in an if-else statement.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous implementation wrapper over the Container contract's client
returned the signature of the eACL table in addition to itself. After recent
changes in API Go lib table carries its signature. Thus, it is redundant to
return the table signature separately.
Make `Wrapper.GetEACL` method to return only `eacl.Table` with error.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make `Get` method of the wrapper over Container contract's client to
accept binary container ID. Create `Get` function similar to the previous
`Get` variation. Use this function in Container service server in the place
where `Get` method was used.
Additionally implement `AsContainerSource` function which allows
to simply compose container Source interface from the wrapper.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Wrap functions at `pkg/errors` return nil if error argument
was nil. fmt.Errorf always returns error so we need to add
missing error checks to the code.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
When user creates a container using neofs-cli, he can set a pre-defined ACL.
Current keywords are confusing: for example, `public` really means that
everyone can write to the container, while the expectation is just public
visibility. Perform the following renames for a better understanding of the
purpose:
* `readonly` -> `public-read`;
* `public` -> `public-read-write`.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Define constants for `public`/`private`/`readonly` keywords of predefined
basic ACL values. Use constants instead of repeating string literals.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
To enable TLS support we can't operate with IP addresses directly.
Certificates are issued with host names so it is required to
pass them into RPC client. DNS resolving should be done by transport
layer and not be a part of node. Therefore `IPAddrString` usage is
removed from code.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Currently main chain's client is unused, therefore, it is not advisable to
require mandatory setting of addresses.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
If default smart contract addresses are valid 20 byte strings, then omitting
the values in configuration will not be tracked at startup.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
From now non-relay node always sends re-bootstrap transaction every 2 epochs
starting from the boot-up epoch.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Application can be shut down often, it is undesirable to send status changes
on a transaction every time. Control API should be used before shutdown for
this purpose.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make work scope and expansion rate the app-side constants since Policer
independently regulates the amount of work performed.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
There is no need to use synchronous execution of notification handlers. Also
there is no understanding of how to assess the need to change the size of
the pools.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Calls to contracts by storage nodes do not lead to the accumulation of
multisignatures in the contract memory, so the call cost can always be
accurately calculated in advance without additional fee.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Replace dial timeout of API client connection from `grpc` section to
`apiclient` since this value is semantically decoupled from the transport
protocol.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Although the default values for configurable addresses are empty strings,
explicitly specifying the default values allows you to know about all
possible configurations.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Rename `util/profiler` package to `httputil` and refactor it:
* simplify utility HTTP server;
* make more generic server's parameters in order to remove `viper.Viper`
dependency;
* use single constructor for creating the pprof and prometheus servers;
* replace `enabled` config value with empty-check of the network address.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Establish client connection with main chain node on storage node startup.
Client is configured simlarly to morph client.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Writing application version in each log message seems pretty redundant.
Remove global `app_version` field from Node/IR loggers. Write version, build
time and debug flag once on start.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
