Commit graph

1347 commits

Author SHA1 Message Date
Alex Vanin
f8ab7a9c60 [#426] service/object: Add object service metrics collector
Collect number of request per method in object service.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
ec07792c4e [#427] Update neo-go to pre-released v0.94.0 version
Should be compatible with NEO3 testnet.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 09:57:40 +03:00
Evgenii Stratonikov
364945a955 [#422] pkg/innerring: fix typo in config parameter name
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-03-15 14:07:53 +03:00
Evgenii Stratonikov
6679d59e89 [#422] pkg/services: Provide client options on cache creation
Because options are not used when client is already in cache
providing them to shared cache is misleading at best.
In the worst case `dial_timeout` is set randomly (because of race
condition) which can lead to one service having `dial_timeout` of
another. Thus we set default client creation options when cache is
created.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-03-15 14:07:53 +03:00
Evgenii Stratonikov
cc7287d6f7 [#422] pkg/services: Cache clients by address only
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-03-15 14:07:53 +03:00
Leonard Lyubich
55dec28bbb [#419] cmd/node: Cache network maps read from sidechain
Implement LRU cache of eACL tables read from sidechain. Use it as a netmap
storage in neofs-node app. Current cache size is set to 10 (constant).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-04 16:33:29 +03:00
Leonard Lyubich
f9342aa562 [#419] cmd/node: Cache eACL tables read from sidechain
Implement TTL cache of eACL tables read from sidechain. Use it as a eACL
storage in neofs-node app. Current cache size is set to 100, TTL is 30s
(constants).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-04 16:33:29 +03:00
Leonard Lyubich
34a51ed38b [#419] cmd/node: Cache containers read from sidechain
Implement LRU cache for values read from the network. Implement LRU cache of
containers read from sidechain. Use it as a container storage in neofs-node
app. Current cache size is set to 100, TTL is 30s (constants).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-04 16:33:29 +03:00
Leonard Lyubich
ed808c3f1b [#419] eacl: Replace eACL storage implementation to app-side
Replace `eacl.Storage` interface implementation from eACL lib to neofs-node
app package. Remove `eacl.WithMorphClient` option in order to abstract eACL
validator from eACL storage implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-04 16:33:29 +03:00
Alex Vanin
8c3864e6d6 [#404] innerring: Wait for deposit in initialization
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
ddf1ac0f28 [#404] morph/client: Add awaiting function
Awaiting function locks execution for N blocks.
Useful to wait for notary deposit.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
4b10e82685 [#404] innerring: Use notary invocations in inner ring
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
83980ccb85 [#404] morph/client: Support notary calls in wrappers
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
71dce97b76 [#404] innerring: Make notary deposit periodically
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
ccaf4f5d55 [#404] innerring: Enable notary support in morph client
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
779a495625 [#404] morph/client: Define notary client
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
948823c392 [#404] innerring: Get GAS script hash from neo-go client
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
883a2e3e77 [#404] Update neo-go to post preview5 version
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
6a50c6ce8d Update changelog for v0.16.0
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-26 18:29:07 +03:00
Alex Vanin
1d2832b61a Update neofs-api-go to v1.24.0
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-26 18:29:07 +03:00
Leonard Lyubich
bcfd33d9fb [#397] ir: Make UN-LOCODE attribute of the network map candidate optional
Previously UN-LOCODE attribute was required for each network map candidate.
In the absence of this attribute, the candidate was not allowed into the
network map. After revising the requirements for candidates, it was decided
not to require the mandatory installation of the attribute by candidates.

From now inner ring does not modify location attributes of the network map
candidate in the absence of UN-LOCODE attribute and does not block entry
into the network map for this criterion.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-25 14:36:11 +03:00
Alex Vanin
5f800458a3 [#410] pkg/policy: Parse strings in filter key
With UN/LOCODE support, storage node may have
`UN-LOCODE` attribute. Policy parser should support
both `Ident` and `Strings` as filter keys to parse
rules such as `FILTER "UN-LOCODE" EQ "RU LED" AS F`.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-25 11:57:26 +03:00
Leonard Lyubich
5900975d58 [#217] object/policer: Leave readability instead of performance comment
Right now we pass redundant copy to callback outside the for-loop through
the helpful boolean variable instead of calling it deeply nested.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
f1a4b8d322 [#217] cmd/node: Use local Inhume operation as a redundant copy callback
Exec `StorageEngine.Inhume` operation on redundant copy callback from Object
Policer with `MarkAsGarbage` parameter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
277e3ca20a [#217] policer: Handler redundant local copy of the object
Detect redundant local copy of the object in Object Policer. Add redundant
copy callback (`WithRedundantCopyCallback` option). Pass address of the
redundant copy to callback.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
35073fb61b [#217] storage engine: Add GC marking to Inhume operation
Add `InhumePrm.MarkAsGarbage` method which marks passed objects to be
removed from local storage. Update `InhumePrm.WithTarget` doc to prevent
conflicting use with the new method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
7a4e3efa95 [#217] storage engine: Transform Inhume to batch operation
Make `StorageEngine.Inhume` to process list of objects per single operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
53b4d6d6d0 [#396] ir: Support working with sidechain only
NeoFS contract can be deployed in sidechain instead of main chain.

Add `without_mainnet` config flag that can switch IR node to work with
sidechain only. By default this flag is unset.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:17:35 +03:00
Leonard Lyubich
ba85b9ff25 [#392] ir/config: Do not set default values of the alphabetic contracts
Previously we set default addresses of alphabetic contracts to empty strings
in inner ring application config. Recently number of using alphabetic
contracts became dynamic. In order to not enumerate default address for each
glagolitic letter, they are removed from config defaults.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
0bca98975a [#392] ir/alphabet: Rename constant corresponding to I/izhei letter
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
356962c4d9 [#392] ir: Set default amount of alphabetical contracts to 7
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
72935d0a80 [#392] ir: Upgrade to a dynamic number of alphabetical contracts
In previous implementation IR worked with exactly 7 alphabetic contracts
only. Actually number of contracts is limited to only the Glagolitic
alphabet.

Make IR to work with any valid number of alphabetic contracts. Change parser
of alphabetic contract addresses to read amount of processing contracts
before performance. Make Alphabet processor to use interface of the
alphabetic contract group. Use `alphabetContracts` type in IR `Server`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
1016ba3a5d [#392] ir: Implement type for group of alphabet contracts
Define `alphabetContracts` type that map glagolic letters to smart contract
addresses. Implement constructor and all methods which are going to be used
in IR processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
3430a6d101 [#392] ir: Define Glagolitsa alphabet type
Define `glagolicLetter` enumeration of the Glagolitsa alphabet letters.
Implement `configString` method that returns config-compatible string format
of the letters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
397c3b6757 [#392] ir: Remove bitsize from IR indices
There is no need to specify that IR indices are 32 bits in size.

Change return types of `Indexer` interface methods in audit and alphabet
packages. Support interface changes in `Server` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
645bef9de9 [#401] cli: Add command to drop object from the node's local storage
Add `drop-objects` sub-command to `control` command that drops objects from
the node via `ControlService.DropObjects` API. Address list argument is
passed in NeoFS API string format of the object address.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 13:07:53 +03:00
Leonard Lyubich
aa24702ebe [#401] control: Serve DropObjects RPC
Re-compile protobuf definition of Control service. Implement required
messages on DropObjects RPC request and response messages. Implement
`DropObjects` method on Control service server of the node. Use
`StorageEngine.Delete` method as a deleted object handler on server.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 13:07:53 +03:00
Leonard Lyubich
f2337baedc [#401] control: Add RPC to mark objects to be removed from the node
There is a need to define storage node's interface to remove objects from
the local storage.

Define `DropObjects` RPC in `ControlService`. Define `DropObjectsRequest` /
`DropObjectsResponse` structure of the request / response messages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 13:07:53 +03:00
Leonard Lyubich
2bb48642ee [#393] cli: Add dedicated flag to indicate object expiration
Add `--expires-on` flag to `put` sub-command of `object` command that
indicates object's expiration epoch. Set corresponding object attributes if
flag value is set.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 11:14:22 +03:00
Leonard Lyubich
cb9e6ea6fa [#398] cmd/cli: Add netinfo command to netmap section
Add `netinfo` sub-cmd of `netmap` cmd that read recent information about
NeoFS network. Info is read via NeoFS API NetmapService.NetworkInfo RPC.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 19:16:40 +03:00
Leonard Lyubich
9073e198b9 [#398] cmd/node: Serve NetmapService.NetworkInfo RPC
Implement `NetworkInfo` calls on full stack of Netmap services. Current
epoch is read from node local state, magic number is read via `MagicNumber`
call of morph client.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 19:16:40 +03:00
Leonard Lyubich
4c8d29ce46 [#398] morph/client: Add method to read magic number
Implement `Client.MagicNumber` method that returns magic number of the
network to which the underlying RPC node client is connected.

Network magic value is received via `GetNetwork` method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 19:16:40 +03:00
Leonard Lyubich
146abe8520 [#388] cmd/cli: Use bearer token when collecting storage group members
CLI `storagegroup put` cmd collects information about SG members via NeoFS
API ObjectService.Head RPC in order to compose SG structure. Bearer token
attached to the call was not used in communication, which could lead to data
access problems. These changes fix the described problem.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 16:20:21 +03:00
Alex Vanin
5f47580ff7 [#399] object/fmt: Fix test for tombstone epoch expiration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-19 16:20:10 +03:00
Alex Vanin
e6cdf3fbf5 [#399] object/fmt: Check expiration epoch in tombstone body and header
According to nspcc-dev/neofs-api#136 tombstone body should store the same
attribute as in object header. If they are different, then check is failed
with `errTombstoneExpiration`.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-19 16:20:10 +03:00
Leonard Lyubich
3ed0065455 [#395] metabase: Do not add tombstone-on-tombstone records to graveyard
Creating tombstones for tombstones is prohibited in NeoFS system. Metabase
graveyard contains records of the form {address: address}: key is an address
of inhumed object, value is an address of the tombstone. To prevent creation
tombstones for tombstones metabase must control incoming Inhume calls:

 * if Inhume target is a tombstone, then "grave" should not be added;
 * if {a1:a2} "grave" was created earlier and {a2: a3} "grave" came later,
   then first "grave" must be removed as tomb-on-tomb.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 16:19:26 +03:00
Leonard Lyubich
bc5e04f502 [#378] metabase: Calculate expiration once for each fake bucket
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
a6d5556dc1 [#378] shard/gc: Reuse addressFromKey func in all appropriate places
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
72f7df3339 [#378] shard/gc: Limit the accumulated batch of objects to be deleted
Change Shard's garbage remover to interrupt iterating over the metabase
graveyard when the buffer is full to the max size (`WithRemoverBatchSize`
Shard's option).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
f5f416268f [#378] metabase: Add the ability to break the loop in IterateOverGraveyard
Make `DB.IterateOverGraveyard` to immediately return nil if GraveHandler
returns ErrInterruptIterator.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00