Commit graph

855 commits

Author SHA1 Message Date
Leonard Lyubich
35073fb61b [#217] storage engine: Add GC marking to Inhume operation
Add `InhumePrm.MarkAsGarbage` method which marks passed objects to be
removed from local storage. Update `InhumePrm.WithTarget` doc to prevent
conflicting use with the new method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
7a4e3efa95 [#217] storage engine: Transform Inhume to batch operation
Make `StorageEngine.Inhume` to process list of objects per single operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
53b4d6d6d0 [#396] ir: Support working with sidechain only
NeoFS contract can be deployed in sidechain instead of main chain.

Add `without_mainnet` config flag that can switch IR node to work with
sidechain only. By default this flag is unset.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:17:35 +03:00
Leonard Lyubich
0bca98975a [#392] ir/alphabet: Rename constant corresponding to I/izhei letter
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
72935d0a80 [#392] ir: Upgrade to a dynamic number of alphabetical contracts
In previous implementation IR worked with exactly 7 alphabetic contracts
only. Actually number of contracts is limited to only the Glagolitic
alphabet.

Make IR to work with any valid number of alphabetic contracts. Change parser
of alphabetic contract addresses to read amount of processing contracts
before performance. Make Alphabet processor to use interface of the
alphabetic contract group. Use `alphabetContracts` type in IR `Server`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
1016ba3a5d [#392] ir: Implement type for group of alphabet contracts
Define `alphabetContracts` type that map glagolic letters to smart contract
addresses. Implement constructor and all methods which are going to be used
in IR processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
3430a6d101 [#392] ir: Define Glagolitsa alphabet type
Define `glagolicLetter` enumeration of the Glagolitsa alphabet letters.
Implement `configString` method that returns config-compatible string format
of the letters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
397c3b6757 [#392] ir: Remove bitsize from IR indices
There is no need to specify that IR indices are 32 bits in size.

Change return types of `Indexer` interface methods in audit and alphabet
packages. Support interface changes in `Server` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
aa24702ebe [#401] control: Serve DropObjects RPC
Re-compile protobuf definition of Control service. Implement required
messages on DropObjects RPC request and response messages. Implement
`DropObjects` method on Control service server of the node. Use
`StorageEngine.Delete` method as a deleted object handler on server.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 13:07:53 +03:00
Leonard Lyubich
f2337baedc [#401] control: Add RPC to mark objects to be removed from the node
There is a need to define storage node's interface to remove objects from
the local storage.

Define `DropObjects` RPC in `ControlService`. Define `DropObjectsRequest` /
`DropObjectsResponse` structure of the request / response messages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 13:07:53 +03:00
Leonard Lyubich
9073e198b9 [#398] cmd/node: Serve NetmapService.NetworkInfo RPC
Implement `NetworkInfo` calls on full stack of Netmap services. Current
epoch is read from node local state, magic number is read via `MagicNumber`
call of morph client.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 19:16:40 +03:00
Leonard Lyubich
4c8d29ce46 [#398] morph/client: Add method to read magic number
Implement `Client.MagicNumber` method that returns magic number of the
network to which the underlying RPC node client is connected.

Network magic value is received via `GetNetwork` method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 19:16:40 +03:00
Alex Vanin
5f47580ff7 [#399] object/fmt: Fix test for tombstone epoch expiration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-19 16:20:10 +03:00
Alex Vanin
e6cdf3fbf5 [#399] object/fmt: Check expiration epoch in tombstone body and header
According to nspcc-dev/neofs-api#136 tombstone body should store the same
attribute as in object header. If they are different, then check is failed
with `errTombstoneExpiration`.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-19 16:20:10 +03:00
Leonard Lyubich
3ed0065455 [#395] metabase: Do not add tombstone-on-tombstone records to graveyard
Creating tombstones for tombstones is prohibited in NeoFS system. Metabase
graveyard contains records of the form {address: address}: key is an address
of inhumed object, value is an address of the tombstone. To prevent creation
tombstones for tombstones metabase must control incoming Inhume calls:

 * if Inhume target is a tombstone, then "grave" should not be added;
 * if {a1:a2} "grave" was created earlier and {a2: a3} "grave" came later,
   then first "grave" must be removed as tomb-on-tomb.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 16:19:26 +03:00
Leonard Lyubich
bc5e04f502 [#378] metabase: Calculate expiration once for each fake bucket
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
a6d5556dc1 [#378] shard/gc: Reuse addressFromKey func in all appropriate places
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
72f7df3339 [#378] shard/gc: Limit the accumulated batch of objects to be deleted
Change Shard's garbage remover to interrupt iterating over the metabase
graveyard when the buffer is full to the max size (`WithRemoverBatchSize`
Shard's option).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
f5f416268f [#378] metabase: Add the ability to break the loop in IterateOverGraveyard
Make `DB.IterateOverGraveyard` to immediately return nil if GraveHandler
returns ErrInterruptIterator.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
ff442e9559 [#378] shard/gc: Avoid usage of the range-loop vars in go-routines
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
b8d1144839 [#378] object/delete: Set expiration epoch of the created tombstones
Make object delete service to use network information to calculate and set
expiration of the created tombstone.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
717f2beb47 [#378] shard: Collect expired tombstones in GC every epoch
Add new epoch event handler to GC that finds all expired tombstones and
marks them and underlying objects to be removed. Shard uses callbacks
provided by the storage engine to mark underlying objects.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
3d5169c4c9 [#378] metabase: Implement iterator over the objects under tombstones
Implement `DB.IterateCoveredByTombstones` method that iterates over graves
and handles all objects under one of the tombstones.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
dab45050b9 [#378] shard: Collect expired non-tombstone objects in GC every epoch
Add new epoch event handler to GC that finds all expired non-tombstone
objects and marks them to be removed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
89a22450e5 [#378] shard: Define new epoch event
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
182df23859 [#378] metabase: Implement iterator over expired objects
Implement `DB.IterateExpired` method that iterates over the objects in
metabase that are expired at particular epoch.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
a9a1acc880 [#378] shard: Control the completion of all handlers of the previous event
Group handlers of the particular event to a WaitGroup and wait for it before
the next event handling. This will ensure that all handlers complete and
prevent potential conflicts between past and present jobs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
4da41613c3 [#378] shard: Initialize GC processes on Init
`Shard.Init` method creates a new GC instance from shard configuration and
starts GC's workers through `init` call. In initial implementation GC
routines are indefinite and can be killed only with by application shutdown.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
13087dc3dd [#378] shard: Implement skeleton of internal GC
Shard's GC component consists of:
 * asynchronous remover that periodically wake up and removes all garbage
   objects from the shard, and goes to sleep for particular time interval;
 * external event listener that distributes jobs between workers;
 * group of workers that can handle a single job related to particular
   external event.

Remover and event listener represents go-routines which are started by
`init` method (calls from `Shard.Init`). In initial version all event
handlers are interrupted: this means that next event of the same type will
interrupt previous handling and start the new one.

GC is fully encapsulated in Shard. All GC configurations are reflected in
Shard's configuration.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
d72d37a322 [#378] metabase: Implement iterator over metabase graveyard
Implement `DB.IterateOverGraveyard` method that iterates over all graves and
passes passes their descriptors (new type `Grave`) to handler (new type
`GraveHandler`). `Grave` currently have buried object address and garbage
flag.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
3d7d074af4 [#377] shard: Support batch Inhume operation
Replace single target address in `InhumePrm` with the list of addresses.
Change corresponding parameter in `WithTarget` and `MarkAsGarbage` methods
to variadic.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
0d6d195d0d [#377] metabase: Support batch Inhume operation
Replace single target address in `InhumePrm` with the list of addresses.
Rename `WithAddress` method to `WithAddresses` and change parameter to
variadic.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
6ec7433e14 [#377] storage engine: Change the implementation of Delete operation
Make `StorageEngine.Delete` to execute `Inhume` operation with
`MarkAsGarbage` parameter on the `Shard` that holds the object. Searching of
the particular shard is performed through iterating over HRW-sorted shards.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
d3a0079d1d [#377] shard: Add Inhume parameter to mark the object as garbage
Implement `InhumePrm.MarkAsGarbage` method that leads to marking object as
garbage in metabase. Update `InhumePrm.WithTarget` doc indicating a conflict
with the new method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
30ac234c20 [#377] metabase: Add Inhume parameter to mark the object as garbage
Implement `InhumePrm.WithGCMark` method that marks the object as garbage in
graveyard. Update `InhumePrm.WithTombstoneAddress` doc indicating a conflict
with the new method. Update `Inhume` function doc about tombstone address
parameter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
7c3f828893 [#219] metabase: Count parent references in Delete operation
Delete operation of Metabase is performed on group of objects. The set being
removed can contain descendants of a common parent. In the case when all
descendants of a parent object are deleted, it must also be deleted from
the metabase. In the previous implementation, this was not done due to the
chosen approach to counting references to the parent.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-18 09:54:19 +03:00
Leonard Lyubich
38727c2930 [#368] object: Reject expired objects
The lifetime of an object can be limited by specifying a correspondin
well-known attribute. Node should refuse to save expired objects.

Checking objects in FormatValidator is extended with an expiration attribute
parsing step.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-15 19:04:33 +03:00
Leonard Lyubich
a2c2241356 [#379] storage engine: Inhume object in one shard
In previous implementation StorageEngine.Inhume operation forced Shard
.Inhume call on all internal shards. There is a need to inhume object in a
single shard. To achieve this, Inhume operation is performed in next steps:

 1. iterate over sorted shards, check object presence through Exists call;
 2. if object exists at any shard in step 1 => inhume it and return on
    success;
 3. if no shards contain the object => iterate over sorted shards again and
    try to inhume the object at first possible shard;
 4. if all Inhume calls are failed => return an error.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-15 18:58:18 +03:00
Leonard Lyubich
065c663bde Update neo-go to v0.93.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-15 10:49:38 +03:00
Alex Vanin
bd420b6002 [#373] innerring: Listen sidechain blocks for internal timer
Updated neo-go fixed bug in listener, so we can use sidechain
for block timer as it planned.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-11 17:31:56 +03:00
Alex Vanin
0bb3836e84 [#383] innerring: Sort node info attributes
All node info attribute transformations can't guarantee
the order of attributes. However it should be consistent
otherwise smart-contract won't be able to collect signatures
and approve transaction.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-11 14:11:43 +03:00
Alex Vanin
058ab0e533 [#383] innerring: Marshal node info back after validation
Node info validator may change node attributes, e.g. update
it with human-readable location attributes based on LOCODE.
So inner ring node should provide new node info binary to
smart contract.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-11 14:11:43 +03:00
Leonard Lyubich
a737a46988 [#362] ir/netmap: Use const keys to node attributes from the API lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
34248b00ed [#362] ir/netmap: Do not add LocationCode attribute to netmap candidate
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
977229eb5a [#316] locode/csv: Scan subdivision csv-table into memory
Scanning subdivision csv-table entries one-by-one takes significant time and
system resources. To speed up random access to table records, on the first
call, the table is pumped into memory (map). On subsequent calls, I/O
operations are not performed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 21:11:28 +03:00
Leonard Lyubich
307355f165 [#316] locode/airports: Scan csv table into memory
Scanning csv-table entries one-by-one takes significant time and system
resources. To speed up random access to table records, on the first call,
the table is pumped into memory (map). On subsequent calls, I/O operations
are not performed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 20:18:06 +03:00
Leonard Lyubich
687c7d3b4a [#316] locode/column: Fix LatitudeHemisphere.North method implementation
North method must return true if hemisphere char is `N`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 20:18:06 +03:00
Leonard Lyubich
19681693f7 [#316] ir/netmap: Rename City node attributes
Rename `City`/`CityCode` attribute keys to `Location`/`LocationCode`
respectively.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
9a425924cf [#316] locode/boltdb: Replace all references to the City with Location
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
0a87fec6c6 [#316] ir/netmap: Rename LOCODE attribute key
Rename key to LOCODE node attribute from `Locode` to `UN-LOCODE`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
46d60f3d52 [#316] ir/locode: Open LOCODE database in read-only mode
Inner ring Server does not modify LOCODE database (Put method), thus it is
better to open it in RO mode.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
0d2440649a [#316] locode/boltdb: Add option to enable read-only mode
Add ReadOnly function that returns Option that enables read-only mode in DB.
RO mode can be used by processes that won't modify the DB in order to not
acquire write flock.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
04b06ba539 [#316] locode/boltdb: Add default timeout to open underlying BoltDB instance
Set timeout option of BoltDB Open operation to 3s in order to prevent
indefinite waiting for file lock.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
97264acb26 [#316] ir: Use LOCODE validator as NodeValidator in Netmap processor
Implement DB interface required by LOCODE validator on new wrapper over the
LOCODE Bolt DB (Record on new wrapper over LOCODE Bolt DB entries).
Construct LOCODE validator and pass it to Netmap processor's constructor as
NodeValidator parameter.

Thus, candidates for a network map must set LOCODE attribute for which there
is an entry in the NeoFS location database.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
ff814aec26 [#316] ir: Add before-start and after-stop function to Server
Keep list of functions which are called first when the server starts (method
Server.Start). If any of the starters returns an error, the server will not
start. Such starters will mainly be used for resources that need to be
initialized after a successful server construction, but before its main work
(e.g. local files).

Keep list of functions which are called when the server stops (Server.Stop
method). Such closers will mainly be used for resources that need to be
released after server shutdown (e.g. initialized by starters).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
db703a5117 [#316] ir/netmap: Validate LOCODE attributes of network map candidates
Define a structure for dealing with the geographic location of nodes.
Implement VerifyAndUpdate (with the same purpose as NodeValidator interface)
that checks LOCODE attribute and fills other attributes of the location.
Technically the entity is a wrapper over the NeoFS location database: it
maps the node LOCODE to the database record from which the new attributes
are generated.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
cbfaceb956 [#316] ir/netmap: Validate network map candidates in Netmap processor
Define NodeValidator interface of the entity that checks and finalizes
NodeInfo structure. Add NodeValidator to Netmap processor. Pass NodeInfo
structures of network map candidates to NodeValidator in order to verify it
and prepare to final state required by network.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
f88e0866fe [#316] locode: Implement NeoFS location database based on BoltDB instance
Define NeoFS location database based on BoltDB. Implement methods to save
the record by key (Put) and to read the record by key (Get).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
9669afdfc7 [#316] locode: Implement polygons DB based on GeoJSON polygons
Define database of Earth's polygons in GeoJSON format. Implement resolving
of geo point to continent in which it is located through point-in-polygon
calculation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
d8c3a0e4f5 [#316] locode: Implement csv OpenFlights airport database
Implement airport database based on csv OpenFlights table. Implement
UN/LOCODE entry matcher. Implement country namespace.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
6829048124 [#316] locode: Implement csv UN/LOCODE table
Define csv UN/LOCODE table. Implement iterator over table entries. Implement
method that resolves country/subdivision code pair to subdivision name.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
cdd1274e1c [#316] locode: Define the API of location database
Define structure of keys and records of the location database. Define the
interfaces of all components necessary for the formation of the database.
Implement the function of filling the database.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
0be35859ed [#316] locode: Define UN/LOCODE table data types
Define the data types needed to work with LOCODE's in NeoFS (country code,
location code, coordinates). Implement string parsers for new types.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
137ef25d3e [#316] locode: Define structure of the entry of the UN/LOCODE table
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Alex Vanin
3775d61ccb [#365] settlement/basic: Use big.Int constructor for unification
Check if `new(big.Int)` will be efficient later and replace
all `big.NewInt()` in code or leave it as it is.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-03 15:17:58 +03:00
Alex Vanin
6a9a2b5d04 [#365] settlement/basic: Remove TxTable from context
TxTable used twice in context to transfer assets to
and from banking account. There is no need to store
instance of table persistently.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-03 15:17:58 +03:00
Alex Vanin
fd461bdb65 [#365] innerring: Produce distribute income events
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-03 15:17:58 +03:00
Alex Vanin
d77d49bd2a [#365] Provide distribute income event in settlement processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-03 15:17:58 +03:00
Alex Vanin
8e741a277d [#365] settlement/basic: Implement asset distribution
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-03 15:17:58 +03:00
Alex Vanin
4433448645 [#365] settlement/basic: Check amount of collected assets
Some transfers from container owners into bank account may
fail due to lack of assets, so we have to deal with remaining
amount of assets in banking account.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-03 15:17:58 +03:00
Leonard Lyubich
f85e5f0238 [#202] object/eacl: Fix algorithm for checking eACL table
Add SignWithRFC6979 option to signature verification function since eACL
table are signed by users with this option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-03 15:14:59 +03:00
Leonard Lyubich
ad348afcd6 [#202] object/eacl: Verify signature of eACL table
Since the contract started returning the table signature, it became
necessary to check its correctness.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-03 15:14:59 +03:00
Leonard Lyubich
ffbf6b922f [#361] metabase: Support new match types
Support STRING_NOT_EQUAL and NOT_PRESENT match types.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-03 10:32:15 +03:00
Alex Vanin
487c9b7589 [#363] Define global config and use it to fetch basic income rate
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-02 21:07:00 +03:00
Alex Vanin
be2ed6bf4c [#360] Run basic income collection in inner ring
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-02 21:07:00 +03:00
Alex Vanin
8c4bf81351 [#360] Use basic implement context in settlement processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-02 21:07:00 +03:00
Alex Vanin
a624bb881d [#360] Implement basic settlement context
Basic settlement context is a main structure that
implement logic of basic settlement phases: collecting
assets from container owners and then distributing them
to storage nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-02 21:07:00 +03:00
Alex Vanin
f45675b7a2 [#360] Share common parts of basic and audit settlements
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-02 21:07:00 +03:00
Leonard Lyubich
bd5c70131b [#328] cmd/node: Serve Container.AnnounceUsedSpace RPC
Register recently implemented handler of AnnounceUsedSpace RPC in node app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
7e586aef90 [#328] container/grpc: Implement AnnounceUsedSpace method
Add AnnounceUsedSpace method that converts gRPC request to v2 request and
passes it to v2 client.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
21133aee2f [#328] container/load: Implement route builder based on placement
Implement route Builder interface on wrapper over the container placement
builder, Component implies exactly one transfer to each of the most weight
nodes of the container (according to some weighing algorithm).
Implementation is planned for use when transferring local estimates of
storage nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
d48fb81193 [#328] container/load: Implement route controller
Implement a component for transmitting the value of the used container space
along a route defined in the system. Implement WriterProvider interface on
it. By implementation, it is the link between the route planner and the
point-to-point transmitter, and abstracts from the implementation of both of
them. In the future, this implementation will be used as a transmitter of
local estimates of storage nodes among themselves.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
1c6d37e821 [#328] morph/container: Add client methods related to size estimations
Implement API methods of the Container contracts client  corresponding
to the calls of the contract for storing the used space of the container
("putContainerSize", "getContainerSize", "listContainerSizes"). Extend
the wrapper over the client with methods abstracted from sidechen calls.
In particular, the method of storing the value will be used to record
the estimates of the used space of containers calculated by the network
participants.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
4415f8dc5b [#328] container/load: Implement simple Iterator and Writer providers
Implement functions to wrap Writer or Iterator. The resulting wrapper
provides WriterProvider or IteratorProvider interface respectively.
Such a wrapper can be used as a single storage instance provider
regardless of context.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
2622e11ae3 [#328] container/load: Implement local storage of used space announcements
Implement a component that stores the values of the used space of
containers. The storage allows you to write several values for a fixed
container and epoch number, and read the averaged estimates of all
accumulated values. All values are stored in memory. This component is
planned to be used as an accumulator of opinions from various network
participants about the fullness of the container.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
cc88320d6b [#328] container/load: Implement a metrics exchange controller
Implement a component that connects the value stores of the used space of
containers. Implement the Start/Stop operations on it, which will later
become the application handlers of the corresponding events from the
sidechain. The main task of the controller is to temporarily synchronize the
stages of calculating the global estimate of the used space in the
container. The details of the score calculation (the way of collecting /
transmitting local scores, the final score formula and writing to the
contract) are encapsulated in the dependency components, the controller is
abstracted from them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Alex Vanin
5b550bff22 [#355] innerring: Produce container size estimation notifications
There are two notifications:
- start estimation notification produced at the beginning of the
  epoch,
- stop estimation notifications should be produced before
  basic audit settlement starts.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:23:37 +03:00
Alex Vanin
6848a816f9 [#355] innerring: Refactor block timer constructors
This small refactoring adds `blocktimer.go` file with
all timer related function and constructors. This way
we can create all timers in one place (at the end of
innerring.Server constructor).

To do that we had to move timer reset into global
server state so it can be accessed by netmap
processor.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:23:37 +03:00
Alex Vanin
402192c8c4 [#355] innerring: Add fee to NoFee wrappers
There is no point of making separate `Fee` and `NoFee`
wrappers because all reading operations are free disregarding
of fee value in static client. However we can use these same
wrappers so send transaction.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:23:37 +03:00
Alex Vanin
70aa73cbd2 [#355] moprh/client: Add start/stop container estimation methods
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:23:37 +03:00
Alex Vanin
7a7a351f1f [#355] morph/event: Add container estimate size events
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:23:37 +03:00
Alex Vanin
db94ad5567 [#355] morph/event: Fix delete container event test
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:23:37 +03:00
Alex Vanin
244bcc5c04 [#357] Fix linter issues
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
819300f939 [#326] ir: Add debug message about transfer for audit transaction
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
d4bd726c25 [#326] ir: Make netmap processor to generate audit settlement events
Pass handler of audit settlement event to netmap event processor. Generate
AuditEvent in during new epoch processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
4204a9f920 [#326] ir: Implement settlement processor
Define a processor of events related to monetary transactions. Define
audit-related event. Provide an interface for processing the audit payout
event.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
aaeb13f7df [#326] ir: Change construction of Balance contact client
Use client with extraFee instead of readOnlyFee. Rename
NewNoFeeBalanceClient to NewBalanceClient since no-fee client is no longer
used.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
a6785c1585 [#326] ir/timers: Remove println from unit test
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
a6a5a84f36 [#326] settlement/audit: Prevent negative amount arguments of Exchanger
Do not pass zero transfers from the calculation table to Exchanger. Revert
transfers with negative amount since Exchanger interface requires positive
amounts of funds.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
2bca5879cd [#326] ir/settlement: Move the calculation of the minimum audit fee
Move control of the minimum cost of payment for the audit from the
implementation of the Exchanger to the place where the amount is calculated.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
87c2c3ecc6 [#326] ir: Calculate audit settlements on new epoch
Calculate payments to storage nodes for the passed audit when changing the
epoch. The calculation results are wrapped in a call to the Balance contract
(one transaction per user-to-user transfer).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
685b593af3 [#326] ir/settlement: Implement calculator of settlements for audit
Implement component that analyzes audit results and generates transactions
for payment of awards for successfully passed audit. When calculating the
total fee, the declared price of the node (attribute) and the total volume
of storage groups, which were successfully audited by the container, are
taken into account. In one call the calculator processes all audit results
for the previous epoch (relative to the calculated parameter).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
20ad0ae82a [#326] morph/balance: Provide interface to "transferX" method's invocations
Implement TransferX method on Balance contract client. Implement TransferX
method on Balance client's wrapper that encapsulates it's TransferX call.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
f7a9e43165 [#356] ir/blocktimer: Fix reset behavior
In previous implementation handler was not called more the one time after
Reset. It was caused by tick counter not being reset inside Reset method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Alex Vanin
de1f601765 [#337] engine: Add container listing for both engine and shard
Container listing already supported in the metabase for `engine.List`
operation. To get container statistics engine should provide both the
option to get container volume estimation and list of all containers.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:04:30 +03:00
Alex Vanin
3926e76c32 [#337] engine: Get all container sizes estimation from shards
Objects of one container can be split among shards, so engine
should iterate over all available shards to sum all size
estimations.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:04:30 +03:00
Alex Vanin
dc5a481f17 [#337] shard: Get container size estimation from metabase
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:04:30 +03:00
Alex Vanin
41578001e4 [#337] metabase: Keep container size estimation
Storage nodes keep container size estimation so they
can announce this info and hope for some basic income
settlements. This is also useful for monitoring.

Container size does not include non regular or inhumed
object sizes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
998a097767 [#324] ir: Listen new blocks from main chain
Listen to new blocks from mainnet until the required fix in neo-go lib is
released (https://github.com/nspcc-dev/neo-go/pull/1687).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
abc7885056 [#324] ir: Register block handler before starting the listening routine
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
44c899b560 [#324] morph/listener: Do not subscribe on new blocks without handlers
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
96f750d9c8 [#324] ir/alphabet: Write error of TransferGas call to log
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
a6517bc877 [#324] ir/netmap: Do not return on block timer restart error
Continue execution of the new epoch handler in case of a failed timer
restart.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
1664cf9869 [#324] morph/subscriber: Init ws client in Subscriber's constructor
Replace WSClient.Init call from the body of BlockNotifications method to
constructor New since Init should be called before working with client
according to neo-go docs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
ca41810859 [#324] ir: Change level of "new block" log message to debug
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
5e212f97d6 [#324] morph/listener: Prevent potential writing to nil channel
Write error returned by BlockNotifications() call ins listenLoop method body
to error channel only if it is procided. Otherwise, write debug log message.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
96febdb6a5 [#324] morph/subscriber: Make block channel in constructor
Replace block channel's initialization from BlockNotifications method body
to Subscriber's constructor in order to prevent potential writing to nil
channel.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
d01b4e1a2d [#324] ir: Measure GAS emission intervals in sidechain blocks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
19bb94cc04 [#324] ir: Measure epochs in sidechain blocks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
b5dc28f79c [#324] ir/timers: Complicate the logic of fractional block intervals
Call handler of the fractional block interval once between base interval
ticks by default. Add option to call handler of fractional block interval
multiple times (N times if fractional interval == BASE_INTERVAL / N).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
476349dd1c [#324] ir/timers: Implement chain block timer
Implement a timer that can tick chain blocks and perform actions at time
intervals in the blocks.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
23c220ae28 [#324] morph/listener: Register handlers of new chain blocks
Extend Listener with RegisterBlockHandler method. All block handlers are
called on each block read from Subscriber.BlockNotifications channel.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Leonard Lyubich
c65d1d9db9 [#324] morph/subscriber: Subscribe on new chain blocks
Call SubscribeForNewBlocks in Subscriber's constructor. Provide
BlockNotifications interface method that returns block channel. Write new
blocks to the channel on notification events with BlockEventID type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-29 11:04:30 +03:00
Alex Vanin
e252f93dbf [#350] object/acl: Make best effort to classify request
Classifier looks at list of inner ring nodes and container
nodes from current and previous epoch to classify request.
Sometimes these checks might return error.

Consider there is a request from unknown key and container's
placement policy valid for current epoch and invalid for past
epoch. Classifier tries to find if key belongs to container
node from current epoch -- it is not. Then it tries to find if
key belongs to container node from past epoch and it throws
error, because placement policy is invalid for past epoch.

This is a legit case and classifier should ignore such errors
to provide best effort in matching. The only error classifier
should return is an error when request does not contain
public key to classify it.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-27 20:18:33 +03:00
Alex Vanin
5d7451d388 [#349] innerring/audit: Use pivot in container placement
Pivot used to shuffle nodes in the CRUSH tree. This is
required argument. We use container ID value to select
container nodes, so `nil` value produces incorrect placements.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-27 11:18:36 +03:00
Leonard Lyubich
c77d346016 [#339] services/netmap: Use dynamic node information
Replace static NodeInfo structure with NodeState interface that provides
method to read node information in runtime.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-25 17:13:46 +03:00
Alex Vanin
e03d906cb7 [#334] engine: Make tests more predictable
There is a codecov issue because objects are not placed
in the engine the same way every unit test. Therefore
sometimes there are more coverage, sometimes there are
less. Seeded RNG should solve this issue for engine tests.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-21 11:00:23 +03:00
Alex Vanin
0745267411 [#47] innerring: Increase minimal extra fee on contract invocations
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-20 16:36:10 +03:00
Alex Vanin
8865252daf [#325] innerring: Fix storage node re-bootstrap approval
Netmap snapshot table caches bootstrap `AddPeer` txs so inner ring
node does not produce redundant approval txs for bootstrapped nodes.
However if node updates states to `Offline`, then such node should
be flagged in snapshot table, so re-bootstrap will actually produce
approval tx.

`ev.PublicKey.String()` returns uncompressed representation of the
node's public key, while snapshot contains compressed values.
Therefore the node was not flagged and re-bootstrap tx was not
approved by inner ring nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-18 18:54:07 +03:00
Leonard Lyubich
a9c1f88b8a [#323] storage/blobovnicza: Add Timeout option of BoltDB instances
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
06085ba8be [#323] storage/shard: Fix message of the error returned by Open method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
1ab630eb24 [#323] storage/shard: Fix message of the error returned by Init method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
b5a1c03fb9 [#323] storage/engine: Fix message of the error returned by Open method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-18 18:53:39 +03:00
Leonard Lyubich
619f8826e1 [#315] control: Implement SetNetmapStatus on Server
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 17:46:12 +03:00
Leonard Lyubich
f39d08bda7 [#315] control: Add SetNetmapStatus rpc
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 17:46:12 +03:00
Leonard Lyubich
e5dc4ede57 [#312] control: Add newline at the end of proto file
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
f327024ee7 [#312] control/server: Extended HealthChecker interface
Add HealthStatus method to HealthChecker interface that should return
current health status of the node application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
6ae51e41f9 [#312] control: Add health status field to health check response body
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
0abb8acef3 [#312] control: Define HealthStatus enum
Define enumeration of the statuses of storage node application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Leonard Lyubich
83d4420a30 [#313] control: Rename HealthStatus enum to NetmapStatus
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:52:08 +03:00
Alex Vanin
a89567a88d [#317] morph/client: Return complete eACL signature from contract
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-15 16:25:49 +03:00
Leonard Lyubich
c75a828adf [#311] control: Fix loss of parent attribute keys in netmap converter
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
6c9dab481c [#311] control: Cover StableMarshal methods with unit tests
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
5f22e7fac7 Add newline at end of Control service proto file
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
36d8e2d3b7 [#311] control: Implement JSON marshaler on Netmap message
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
dde24bc9e5 [#311] control: Rewrite methods for binary encoding and signatures
Rewrite marshalers and sizers using proto library from API repo.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
e9d4b2d25b [#311] Recompile protobuf files
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
3e882d963f [#311] services/control: Implement NetmapSnapshot method
Implement methods required for signatures. Receive network map from netmap
storage, convert it to Control service message and return in response.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
28777911fb [#311] services/control: Define NetmapSnapshot rpc
Add NetmapSnapshot rpc to ControlService protobuf definition. Recompile
proto files. Add required method to server structure.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00
Leonard Lyubich
88023f3655 [#311] services/control: Define Netmap structure
Define NodeInfo protobuf type. Define Netmap protobuf type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-01-15 16:11:50 +03:00