Commit graph

1329 commits

Author SHA1 Message Date
Leonard Lyubich
3cfb58aabd [#790] storage engine: Add common template of log messages
There is a need to keep track of each local storage change. Log messages are
the most convenient way to do it.

Implement function which writes log message about the completed writing
operation in storage engine.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-13 15:02:13 +03:00
Alex Vanin
b6dfa6c118 [#786] pkg/innerring: Use NNS to find contract script hashes missing in config
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin
f43f389399 [#786] morph/client: Add NNS contract address getter
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:35 +03:00
Alex Vanin
dc26a09ec3 [#812] pkg/innerring: Support notary notifications in reputation processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
a7f6a3df78 [#812] morph/client: Add underline client getter in reputation wrapper
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Alex Vanin
e2c2e27c60 [#812] morph/event: Expand reputation put event with notary notification
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:51:07 +03:00
Pavel Karpy
b303e49408 [#815] morph/event/netmap: Add updateState notary notification support
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Pavel Karpy
74afcbe409 [#815] pkg/morph/netmap: Add Morph() method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Pavel Karpy
00caed8d3d [#815] morph/event/netmap: Add addPeer notary notification support
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 14:50:52 +03:00
Alex Vanin
01df4ffa61 [#811] service/container: Hide cache invalidation logic in Writer interface
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Alex Vanin
4a1f0de8f4 [#811] service/container: Invalidate cache records on update requests
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-13 14:50:18 +03:00
Pavel Karpy
a8d41f596d [#822] event/notaryPreparator: Add unit test for correct NR
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 10:32:43 +03:00
Pavel Karpy
3b9ef4f63c [#822] event/notaryPreparator: Do not pass PACK opcode
Do not pass high level `PACK` opcode to
notary parsers. Add opcode amount check.
Delete `PACK` cases in notary parsers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 10:32:43 +03:00
Pavel Karpy
4f3de1a9af [#822] moprh/event: Add parsers from Op struct
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-13 10:32:43 +03:00
Leonard Lyubich
61b4baf736 [#778] services/object: Wrap last client's error into errIncompletePut
Make `errIncompletePut` to be a structure which wraps single client error.
Wrap error of the last client into `errIncompletePut` during placement
execution.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 16:49:37 +03:00
Leonard Lyubich
02f2a98bcc [#778] services/object: Return the composite error of PUT operation
In previous implementation Object service's handler returned const error in
case of failure (full or partial) of PUT operation. This did not even allow
us to roughly guess what the reason is. Not as a complete solution, but to
alleviate some cases where all nodes in a container return the same error,
it is suggested to return the error of the last server that responded.

Return latest server error from placement loop of `iteratePlacement` method
of `distributedTarget` type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 16:49:37 +03:00
Evgenii Stratonikov
5f86d54721 [#816] object/acl: fix eACL target processing
Ignore role if public keys are present.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-09-10 15:15:36 +03:00
Leonard Lyubich
7a13053fab [#818] object/acl: Fit sticky bit specification
In previous implementation sticky bit could disrupt access of container
nodes to replication. According to NeoFS specification sticky bit should not
affect the requests sent by nodes from SYSTEM group.

Add role check to `stickyBitCheck`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-10 15:14:59 +03:00
Pavel Karpy
ba77bb44e4 [#807] ir: Merge ContractProcessor and NotaryContractProcessor interfaces
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
539da27ccb [#807] morph/event/container: Add setEACL notary support
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
e03b44ffc1 [#807] morph/event/container: Add delete notary support
Includes:
- `delete` container notary notification;
- refactor in processor bindings;
- fixed conflicts in `put` notifications.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
45aac7bb7f [#807] event/listener: Add event information to error logs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
ffde45d164 [#807] morph/container: Add opcode check
Parsers return error if unexpected opcode
returns.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 18:37:45 +03:00
Pavel Karpy
6f41a979ca [#814] morph/event/container: Fix param amount check
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 14:43:29 +03:00
Pavel Karpy
b4059f652e [#814] morph/event: Fix NotaryPreparator parsing
Add checking for call flag in TX's script
and do not pass it to parsers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 14:43:29 +03:00
Pavel Karpy
1ed5323850 [#801] morph/wrappers: Sync commentary and struct names
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 14:07:57 +03:00
Pavel Karpy
d0160b23e5 [#801] linter: Fix errcheck warning
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 14:07:57 +03:00
Pavel Karpy
425c02b0ec [#801] morph/netmap: Add wrapper options
Add constructor options on wrapper level
of `netmap` contract.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-09 14:07:57 +03:00
Alex Vanin
1edf40f4d6 [#798] morph/subscriber: Remove RPC Init timeout
More convenient way is to fail straight away and
expect external restart.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Alex Vanin
005f54e61e [#798] pkg/innerring: Save latest processed block number
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Alex Vanin
2bcf22ad79 [#798] pkg/innerring: Store withoutMainNet flag in server
withoutMainNet flag will be reused in runtime server start
to enable main chain block number state saver.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Alex Vanin
67471a8914 [#798] morph/subscriber: Await for RPC initialization
Remote RPC node might be in the transition state and produce
events from the past. We should avoid listening such nodes.
To do that subscriber component can await minimal height of
remote node. If minimal height is not reached, then throw
error.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Alex Vanin
7f074a775e [#798] util/state: Implement persistent storage for application state
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-07 17:22:25 +03:00
Pavel Karpy
4a81781c0c [#770] logger: Delete spaces in log message keys
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
1fedd8f860 [#770] morph/event/container: Sync comment and method name
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
13f1273e82 [#770] ir/container: Support notary Put
Implement `NotaryContractProcessor` by IR
container processor. Add support for notary
`put` container operation. Do not parse `put`
non-notary notifications in notary enabled
environment.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
9921358f09 [#770] pkg/morph: Add NotarySignAndInvokeTX
Add `NotarySignAndInvokeTX` method to morph
client. This function allows invoking notary
request with passed main TX(not creating a
new one). It signs passed main TX with
client's key.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
f4a521cfd8 [#770] pkg/innerring: Make all morph wrappers Alphabet
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
79b350b628 [#770] morph: Support non-alpha notary request by wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
94d431e56e [#770] morph/client: Support non-alpha notary request
Add `NotaryInvokeNotAlpha` to low-level
client. It creates and sends notary request
that must be signed by Alphabet nodes, but
does not sign it by current node's private
key.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
b46adf188c [#770] pkg/morph/container: Add Morph() method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
46df288d33 [#770] pkg/morph: Add Morph() method
Add access to low-level morph client in
wrappers

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
4e5618aecb [#770] pkg/innerring: Add NotaryContractProcessor interface
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
d40f898ee5 [#770] morph/event/container: Add notary Put parser
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
c042f6a429 [#770] pkg/morph/event: Add notary notifications support
Add handlers and parsers functionality for listener.
Separate notification and notary events by files.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
8f2924d6cf [#770] pkg/morph/event: Add notary request preparator
Add preparator for notary requests. Is parses
raw notary requests, checks if it should be
handled by Alphabet node. If handling is required,
returns `NotaryEvent` that contains information
about contract scripthash, method name and
arguments of the call.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
8d1f2e66e6 [#770] pkg/innerring: Fix commentary to bind functions
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
d252aa4a3e [#770] pkg/morph: Rename all parsers and handlers structs/interfaces
Prepare all listening structures for notary events:
rename(add prefix/suffix 'notification') all
notification specific handlers/parsers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
c87bc70536 [#770] pkg/morph/event: Fix typo in AddPeer parser
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
f0cbf2e99d [#770] pkg/morph/subscriber: Add subscription for notary
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
e24489d33c [#770] pkg/morph: Fix commentaries for Listener interface
Refactor comments so they start with the names of
methods.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
c0e25d1706 [#770] morph/client: Fix comment for fallbackTime
`fallbackTime` is delta b/w `ValidUntilBlock` of
the main transaction and block when `fallback`
transaction is sent.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Pavel Karpy
3400a071f9 [#770] ir: Reuse read notary config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-09-07 12:55:01 +03:00
Leonard Lyubich
846dd8c89e [#645] network: Remove no longer used IsLocalAddress function
In recent changes, the locality criterion for a node has been changed to
compare public keys.

Remove no longer used `IsLocalAddress` function and `LocalAddressSource`
interface.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich
358e3ed8c4 [#645] *: Change the locality condition of the node from the placement
Some software components regulate the way of working with placement arrays
when a local node enters it. In the previous implementation, the locality
criterion was the correspondence between the announced network address
(group) and the address with which the node was configured. However, by
design, network addresses are not unique identifiers of storage nodes in the
system.

Change comparisons by network addresses to comparisons by keys in all
packages with the logic described above. Implement `netmap.AnnouncedKeys`
interface on `cfg` type in the storage node application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich
f14987c06d [#645] core: Define interface of the tool for working with netmap keys
Announced keys of storage nodes are required for many application components
to function.

Define a unified interface for the utility for working with public keys of
nodes. Add a method to check if the key has been advertised by the local
node in the application context.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich
22926e8f28 [#645] placement/traverser: Extend Node type with public key
There is a need to process public keys of the placement result.

Implement `Node.PublicKey` method which returns storage node's key announced
in netmap.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich
fe90456dcc [#645] placement: Overload result of Traverser.Next method
In previous implementation `placement.Traverser.Next` method returned slice
of `network.AddressGroup` elements. There is a need to process keys of
storage nodes besides network addresses for intra-container communication.

Wrap `network.AddressGroup` in a new type `placement.Node` that summarizes
the storage node information required for communication. Return slice of
`Node` instances from `Traverser.Next` method. Fix compilation breaks in
dependent packages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich
ba234699cd [#794] ir/neofs: Print tx IDs in LE on Withdraw and Deposit events
IDs come from NeoFS contract in big endian, but it is customary to write in
the node logs in little endian.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-06 18:11:38 +03:00
Leonard Lyubich
3f293cb55e [#762] storage/shard: Try to read headers from write-cache
Shard should try to read object headers from write-cache if it is enabled.

Extend `writecache.Cache` interface with `Head` method. Call the method in
`Shard.Head` if `Shard.hasWriteCache` returns true.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:33:17 +03:00
Leonard Lyubich
aa2151fbeb [#762] write-cache: Implement Head operation
Write cache should be able to execute HEAD operations according to spec.

Add simple implementation of `Head` method through the `Get` one. Leave
notes for future optimization.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:33:17 +03:00
Leonard Lyubich
81722c373b [#746] morph/client: Don't cache GAS contract address on NeoFS-side
Caching is performed inside `GetNativeContractHash` method of neo-go client,
so the additional cache level is redundant.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:04:38 +03:00
Leonard Lyubich
18c49c2cb9 [#746] morph/client: Break loop in iterateClients on NeoFS error
There is no need to continue iterating over Neo RPC endpoints in case of
some address-independent error (e.g. NeoFS logic error).

Unwrap and immediately return `neofsError` errors from loop in
`iterateClients`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:04:38 +03:00
Leonard Lyubich
a2b81f13b0 [#746] morph/client: Simplify loop control in iterateClients
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:04:38 +03:00
Leonard Lyubich
3b7e884e74 [#746] morph/client: Distinguish between neo-go errors and NeoFS logic
Implement `error` interface on new `neofsError` type which is a wrapper over
NeoFS-specific error. Wrap all `Client` errors except neo-go client API ones
into `neofsError`. Wrapped errors are going to be used for multi-endpoint
loop control.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:04:38 +03:00
Leonard Lyubich
85bd2a1cdf [#746] morph: Add error return of MagicNumber method
Since morph `Client` works in multi-client mode, there is an error case when
we can not get network magic when all endpoints are unavailable.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:04:38 +03:00
Leonard Lyubich
ad7ad12a0c [#746] morph: Implement and use multi-client
There is a need to work with a set of Neo RPC nodes in order not to depend
on the failure of some nodes while others are active.

Support "multi-client" mode of morph `Client` entity. If instance is not
"multi-client", it works as before. Constructor `New` creates multi-client,
and each method performs iterating over the fixed set of endpoints until
success. Opened client connections are cached (without eviction for now).

Storage (as earlier) and IR (from now) nodes can be configured with multiple
Neo endpoints. As above, `New` creates multi-client instance, so we don't
need initialization changes on app-side.

`Wait` and `GetDesignateHash` methods of `Client` return an error from now
to detect connection errors. `NotaryEnabled` method is removed as unused.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-02 11:04:38 +03:00
Alex Vanin
f9218bf84f [#787] util/attributes: Restore escape characters after caching the result
As soon as resulting list and cache operate with the attribute pointer,
we can add attribute structure immediately and set restored key and
value of the attribute later.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-01 14:29:26 +03:00
Alex Vanin
f09b7e48af [#787] util/attributes: Fix order of require.Equal arguments
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-01 14:29:26 +03:00
Alex Vanin
2d7e9f0c40 [#787] util/attributes: Return consistent list of attributes
Attributes are linked to each other through parents, so they can
be returned in any order. However, it will be better to return
the list in consistent order to reduce entropy.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-01 14:29:26 +03:00
Alex Vanin
d8a04726ad [#787] util/attributes: Reuse attribute struct to update parents list
Parser should reuse existing attributes from the cache to update list
of the parent links in it. Parent links should be unique.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-09-01 14:29:26 +03:00
Leonard Lyubich
e738699fcc [#676] services/container: Cache the results of read operations
In previous implementation Container service handlers didn't cache the
results of `Get` / `GetEACL` / `List` operations. As a consequence of this,
high load on the service caused neo-go client's connection errors. To avoid
this there is a need to use cache. Object service already uses `Get` and
`GetEACL` caches.

Implement cache of `List` results. Share already implemented cache of Object
service with the Container one. Provide new instance of read-only container
storage (defined as an interface)to morph executor's constructor on which
container service is based. Write operations remained unchanged.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-08-30 18:01:26 +03:00
Pavel Karpy
c54f524df9 [#773] writecache: Delete unused dbSize param
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-27 15:05:53 +03:00
Pavel Karpy
4ff98a7e2b [#760] metabase: Support COMMON_PREFIX matchtype
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-27 15:03:15 +03:00
Alex Vanin
d6439369ab [#768] object/acl: Support all eACL filter fields from the spec
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 19:01:03 +03:00
Alex Vanin
cadd94f08f [#766] Fix stylecheck import linter error
Remove redundant imports

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 17:57:15 +03:00
Alex Vanin
53f031e98c [#766] Fix misspell linter error
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 17:57:15 +03:00
Alex Vanin
30738549ef [#766] Fix makezero linter error
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 17:57:15 +03:00
Leonard Lyubich
abfcc7498c [#715] services/policer: Select pseudo-random list of objects to check
In previous implementation of Policer's job queue the same list of objects for
processing was selected at each iteration. This was caused by consistent
return of `engine.List` function.

Use `rand.Shuffle` function to compose pseudo-random list of all objects in
order to approximately evenly distribute objects to work.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-08-25 14:40:12 +03:00
Pavel Karpy
85a4682c0c [#759] pkg/innerring: Log notary status on startup
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-23 15:26:10 +03:00
Alex Vanin
300d8879a6 [#751] innerring: Use CalledByEntry scope in main chain invocations
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-06 17:29:27 +03:00
Alex Vanin
43dfccd9b3 [#751] morph/client: Add custom signer scope support
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-06 17:29:27 +03:00
ZhangTao1596
f121a73049 [#737] network/cache: handle request canceled in multiclient
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2021-08-05 14:47:41 +03:00
Pavel Karpy
3d981320c6 [#745] gc: Stop GC work on Shard's Close
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-05 14:44:05 +03:00
Alex Vanin
6527f9157c [#744] util/attributes: Support escaped symbols
To encode attributes with semicolon or slash, use
backslash as escaped character.

Example:
  User-Agent:NeoFS\/0.23
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-04 16:48:54 +03:00
Pavel Karpy
7a52bb31f1 [#740] innerring/notary: Fix reading notary config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 21:00:57 +03:00
Pavel Karpy
e8665f6cef [#730] node/morph: Use disable_cache config param
Do not init caches for eACL, containers and netmap
if `disable_cache` config options is `true`, use
direct RPC calls instead.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 09:40:10 +03:00
Pavel Karpy
c423aa432a [#730] node/config/morph: Add disable_cache
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 09:40:10 +03:00
Pavel Karpy
896c749b92 [#720] pkg/innerring: Check Notary availability automatically
Do not read `without_notary` config value from env.
Make morph client constructor return client without
notary support. Enabling notary support should be done
with public `EnableNotarySupport` method separately.

Notary availability is deducted with client. Further,
if notary is presented on chain its support is
enabled at the corresponding client.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 09:30:11 +03:00
Pavel Karpy
28aa0f521e [#735] pkg/network: Delete tls multiaddr implementation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-02 14:28:58 +03:00
Pavel Karpy
7a10d902be [#727] Use util.MkdirAllX instead of os.MkdirAll
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-30 16:42:22 +03:00
Pavel Karpy
51eeae3640 [#727] util: Add MkdirAllX func
Add `MkdirAllX` function that wraps `os.MkdirAll`
and adds `x` permission for user and group.
This is used to make all created dirs openable
for user and group regardless of the read config
permissions.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-30 16:42:22 +03:00
ZhangTao1596
01806db612 [#723] container/announcement/storage: Map announcements by epoch and container id
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2021-07-30 09:58:30 +03:00
Pavel Karpy
c41ec2e2e9 [#718] ir: Do not require processing hash in main notary disabled mode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-23 10:59:45 +03:00
Alex Vanin
bba470570f [#708] morph/client: Update last epoch block method name
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-22 20:40:09 +03:00
Alex Vanin
3561cf5873 [#712] moprh/client: Log VUB in notary invocations
For easier debug when notary transaction didn't execute.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-22 15:46:48 +03:00
Alex Vanin
62efa3f098 [#708] innerring: Synchronize initial epoch tick
When Inner Ring node starts, it should sync nearest epoch tick
event based on the block of the latest epoch. Otherwise epoch
ticking can be stopped, because ballots or notary transactions
are valid for limited period of time.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-22 15:20:34 +03:00
Alex Vanin
67b3682348 [#708] morph/timer: Add single tick timer
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-22 15:20:34 +03:00
Alex Vanin
7146afcd28 [#708] morph/client: Add epoch block getter from netmap
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-22 15:20:34 +03:00
Alex Vanin
7e43af3de3 [#708] morph/client: Add BlockCount method
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-22 15:20:34 +03:00
Pavel Karpy
dfbde0004e [#705] innerring/governance: Delete useless sort
Do not sort slice that is already sorted.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
c8d58d56b9 [#705] innerring/governance: Hide fetching ir keys behind interface
Add to governance processor IRFetcher interface.
Implementation must deduct if notary enabled or not.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
73ba35b379 [#705] pkg/innerring: Fix disable notary mode
Get innerring list and committee in innerring
indexer from interfaces, not from structs.
Passing IR fetcher to indexer constructor depends
on having Notary contract in sidechain(use NeoFS
role or netmap contract).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
8e66c67a74 [#705] pkg/innerring: Add IR keys fetchers
Add `IrFetcherWithNotary` and `IrFetcherWithoutNotary`
that can fetch IR keys with and without usage of
notary contract.
Both can be hidden behind
`InnerRingKeys() (keys.PublicKeys, error)` interface.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
8ea5744326 [#705] pkg/morph/netmap: Fix updateInnerRing method
Add initialization of `updateInnerRing` method name
in config on startup.
Rename var `setInnerRing` => `updateInnerRing`.
Rename method `SetInnerRing` => `UpdateInnerRing`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
84361beada [#705] pkg/morph/netmap/wrapper: Add GetInnerRingList method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
b3c81116b9 [#705] pkg/morph/netmap: Add innerRingList method
Add `innerRingList` method to `netmap` client
wrapper. Method parses current IR nodes list.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
00b487e3c3 [#705] morph/notary: Add ProbeNotary method to Client
`ProbeNotary` method checks if native `Notary`
contract is presented on chain.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
46a66a21ec [#705] pkg/util: Fix commentary error
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-22 15:12:06 +03:00
Pavel Karpy
795c49d16b [#706] ir: Do not require proxy hash in notary disabled mode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-21 15:32:17 +03:00
Alex Vanin
96da7ceb4f [#697] governance: Make best effort traversing main chain list of keys
We should go through every key in main chain list to merget lists
as fast as possible. Previously we drop main chain traversing as
soon as we have no more new keys to add. Instead we should try
to go for old keys in the list and add it as more as we can.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-15 10:49:21 +03:00
Alex Vanin
aed76f6b5a [#694] pkg/network: Use single function to get or create client instance
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 15:17:48 +03:00
Alex Vanin
38afb82926 [#694] pkg/network: Use client cache in raw client getter
Every raw client getter invocation produced new connection, that lead
to connection leak.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 15:17:48 +03:00
Alex Vanin
8112f73023 [#689] innerring: Remove debug output
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-13 13:11:29 +03:00
Pavel Karpy
c20eb15bd5 [#688] pkg/ir/governance: Add unsorted test case
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-12 18:46:09 +03:00
Pavel Karpy
bb3f16d487 [#688] pkg/ir/governance: Add public key sorting
Add sorting of passed public keys in `newAlphabetList`
function before Alphabet list merge to make returned
results more expected.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-12 18:46:09 +03:00
Pavel Karpy
361bff216a [#688] pkg/ir/governance: Check err type in tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-12 18:46:09 +03:00
Pavel Karpy
9da777ac8c [#681] ir: Do not require MainNet attributes
If `WITHOUT_MAINNET` environmental variable is
`true`:
- Do not read `NeoFS` and `processing`
script-hashes from envs;
- Do not init Governance processor;
- Do not init NeoFS processor.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-12 15:35:35 +03:00
Alex Vanin
24d1725cc8 [#666] innerring: Make GlagoliticLetter enum public for re-use
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-08 14:37:38 +03:00
Pavel Karpy
0bcc505682 [#668] pkg/audit: Skip audit for containers without SG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-07-06 21:19:18 +03:00
Leonard Lyubich
7c1432aa57 [#660] services/container: Don't check container format in Put
Storage node should only reflect container Put requests to sidechain w/o any
validation. All verification steps are performed by IR.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-07-05 11:05:44 +03:00
Leonard Lyubich
54d6f3de55 [#660] core/container: Use version.IsValid in CheckFormat
Container can have version later than the current lib version, and it should
not be considered as invalid. But it should still have adequate version.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-07-05 11:05:44 +03:00
Leonard Lyubich
8192933313 [#660] services/netmap: Don't call IsSupportedVersion in LocalNodeInfo
Clients can have version later than the server. The server should not deny
these clients.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-07-05 11:05:44 +03:00
Leonard Lyubich
69826ebd90 [#660] core: Implement function to check protocol version adequacy
Create `version` package and implement `IsValid` function which checks if
version is not earlier than the genesis one.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-07-05 11:05:44 +03:00
Alex Vanin
c90f054f35 [#658] morph/neofs: Add TryNotary() option
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-02 09:53:34 +03:00
Alex Vanin
c0093b2b79 [#658] morph/neofsid: Add TryNotary() option
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-02 09:53:34 +03:00
Alex Vanin
92451c08af [#658] innerring: Do not use notary in audit client
Audit client is used to send audit results and they should be
signed by inner ring node itself on order to be saved in
smart contract.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-07-02 09:53:34 +03:00
Pavel Karpy
a2b622d62e [#647] pkg/morph/netmap/wrapper: GetCandidates method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-29 22:29:23 +03:00
Pavel Karpy
d664a00e94 [#647] pkg/morph/netmap: Add netmapCandidates method
Add `netmapCandidates` method to `netmap` client
wrapper. Method parses node storages candidates
for the next epoch.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-29 22:29:23 +03:00
Pavel Karpy
403f836968 [#647] pkg/morph: Add constant var to err message
Use defined constant variable instead of
hardcoded number in error message.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-29 22:29:23 +03:00
Leonard Lyubich
dab33e0f9b [#652] Update Neo Go to v0.95.3
Add `keys.NEP2ScryptParams()` to `keys.NEP2Decrypt` call arguments.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-29 13:44:59 +03:00
Leonard Lyubich
56d4410913 [#638] Update to Go 1.16
Changes:

  * replace `iotuil` elements with the ones from `os` package;
  * replace `os.Filemode` with `fs.FileMode`;
  * use `signal.NotifyContext` instead of `NewGracefulContext` (removed).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-29 13:44:59 +03:00
Leonard Lyubich
8eadf18e01 [#607] services/control: Fix incorrect allocation in NetmapSnapshot
Fix make with length + subsequent append.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
f8ee6b1164 [#607] services/netmap: Support backward compatibility
Make Netmap service to return single network address on `LocalNodeInfo` RPC
in order to older clients.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
6f861b6489 [#607] network: Support URI address strings
Make Address.FromString method to parse URI addresses and enable TLS for TLS
schemes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
43eff09944 [#607] *: Do not use deprecated elements of code
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
cbe20a2bac [#607] network: Add helper function to iterate over all addresses
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
119031c8c7 [#607] network: Make AddressGroup.WriteToNodeInfo method a function
Method implementation doesn't use any private logic.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
359bbe319a [#607] network: Remove no longer needed GroupFromAddress function
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
e3eade619d [#607] network: Remove no longer used Address.AddTLS method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
b723c92ef8 [#607] network: Make Address.Equal method unexported
The method is used only in the same package.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
61cc70d899 [#607] network: Remove no longer used Address.WriteToNodeInfo method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
149367cab9 [#607] network: Sort addresses in AddressGroup.FromIterator method
Implement `sort.Interface` interface on `AddressGroup` and perform sorting
in `AddressGroup.FromIterator` method. Addresses with enabled TLS are "less"
in terms of slice position.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
1e52e86bbc [#607] node/control: Make group address in NodeInfo message
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
cede2b4ed7 [#607] reputation,container: Support address groups in ServerInfo
There is a need to support multiple server endpoints for reputation and
container transmission.

Replace `ServerInfo.Address` getter with `ServerInfo.IterateAddresses`
iterator.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
d6bb697726 [#607] container/announce: Use public keys as keys to server cache
In future server info will contain multiple endpoints whose string
representation will be worse suited to the cache key.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
44a2c81f8e [#607] reputation/router: Use public keys as keys to server cache
In future server info will contain multiple endpoints whose string
representation will be worse suited to the cache key.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
f7b42a1525 [#607] ir: Work with AddressGroup instead of single Address
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
9033cf5aa1 [#607] placement: Work with AddressGroup
Make `PlacementBuilder` implementations to work `network.AddressGroup` type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
b3dd9a3254 [#607] placement: Make traverser to return list of address groups
Make placement `Traverser.Next` method to return ``[]network.AddressGroup`
in order to support multiple addresses of the storeage nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00