Commit graph

713 commits

Author SHA1 Message Date
Leonard Lyubich
a737a46988 [#362] ir/netmap: Use const keys to node attributes from the API lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
34248b00ed [#362] ir/netmap: Do not add LocationCode attribute to netmap candidate
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
977229eb5a [#316] locode/csv: Scan subdivision csv-table into memory
Scanning subdivision csv-table entries one-by-one takes significant time and
system resources. To speed up random access to table records, on the first
call, the table is pumped into memory (map). On subsequent calls, I/O
operations are not performed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 21:11:28 +03:00
Leonard Lyubich
307355f165 [#316] locode/airports: Scan csv table into memory
Scanning csv-table entries one-by-one takes significant time and system
resources. To speed up random access to table records, on the first call,
the table is pumped into memory (map). On subsequent calls, I/O operations
are not performed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 20:18:06 +03:00
Leonard Lyubich
687c7d3b4a [#316] locode/column: Fix LatitudeHemisphere.North method implementation
North method must return true if hemisphere char is `N`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 20:18:06 +03:00
Leonard Lyubich
19681693f7 [#316] ir/netmap: Rename City node attributes
Rename `City`/`CityCode` attribute keys to `Location`/`LocationCode`
respectively.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
9a425924cf [#316] locode/boltdb: Replace all references to the City with Location
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
0a87fec6c6 [#316] ir/netmap: Rename LOCODE attribute key
Rename key to LOCODE node attribute from `Locode` to `UN-LOCODE`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
46d60f3d52 [#316] ir/locode: Open LOCODE database in read-only mode
Inner ring Server does not modify LOCODE database (Put method), thus it is
better to open it in RO mode.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
0d2440649a [#316] locode/boltdb: Add option to enable read-only mode
Add ReadOnly function that returns Option that enables read-only mode in DB.
RO mode can be used by processes that won't modify the DB in order to not
acquire write flock.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
04b06ba539 [#316] locode/boltdb: Add default timeout to open underlying BoltDB instance
Set timeout option of BoltDB Open operation to 3s in order to prevent
indefinite waiting for file lock.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
97264acb26 [#316] ir: Use LOCODE validator as NodeValidator in Netmap processor
Implement DB interface required by LOCODE validator on new wrapper over the
LOCODE Bolt DB (Record on new wrapper over LOCODE Bolt DB entries).
Construct LOCODE validator and pass it to Netmap processor's constructor as
NodeValidator parameter.

Thus, candidates for a network map must set LOCODE attribute for which there
is an entry in the NeoFS location database.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
ff814aec26 [#316] ir: Add before-start and after-stop function to Server
Keep list of functions which are called first when the server starts (method
Server.Start). If any of the starters returns an error, the server will not
start. Such starters will mainly be used for resources that need to be
initialized after a successful server construction, but before its main work
(e.g. local files).

Keep list of functions which are called when the server stops (Server.Stop
method). Such closers will mainly be used for resources that need to be
released after server shutdown (e.g. initialized by starters).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
db703a5117 [#316] ir/netmap: Validate LOCODE attributes of network map candidates
Define a structure for dealing with the geographic location of nodes.
Implement VerifyAndUpdate (with the same purpose as NodeValidator interface)
that checks LOCODE attribute and fills other attributes of the location.
Technically the entity is a wrapper over the NeoFS location database: it
maps the node LOCODE to the database record from which the new attributes
are generated.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
cbfaceb956 [#316] ir/netmap: Validate network map candidates in Netmap processor
Define NodeValidator interface of the entity that checks and finalizes
NodeInfo structure. Add NodeValidator to Netmap processor. Pass NodeInfo
structures of network map candidates to NodeValidator in order to verify it
and prepare to final state required by network.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
f88e0866fe [#316] locode: Implement NeoFS location database based on BoltDB instance
Define NeoFS location database based on BoltDB. Implement methods to save
the record by key (Put) and to read the record by key (Get).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
9669afdfc7 [#316] locode: Implement polygons DB based on GeoJSON polygons
Define database of Earth's polygons in GeoJSON format. Implement resolving
of geo point to continent in which it is located through point-in-polygon
calculation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
d8c3a0e4f5 [#316] locode: Implement csv OpenFlights airport database
Implement airport database based on csv OpenFlights table. Implement
UN/LOCODE entry matcher. Implement country namespace.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
6829048124 [#316] locode: Implement csv UN/LOCODE table
Define csv UN/LOCODE table. Implement iterator over table entries. Implement
method that resolves country/subdivision code pair to subdivision name.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
cdd1274e1c [#316] locode: Define the API of location database
Define structure of keys and records of the location database. Define the
interfaces of all components necessary for the formation of the database.
Implement the function of filling the database.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
0be35859ed [#316] locode: Define UN/LOCODE table data types
Define the data types needed to work with LOCODE's in NeoFS (country code,
location code, coordinates). Implement string parsers for new types.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
137ef25d3e [#316] locode: Define structure of the entry of the UN/LOCODE table
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Alex Vanin
3775d61ccb [#365] settlement/basic: Use big.Int constructor for unification
Check if `new(big.Int)` will be efficient later and replace
all `big.NewInt()` in code or leave it as it is.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-03 15:17:58 +03:00
Alex Vanin
6a9a2b5d04 [#365] settlement/basic: Remove TxTable from context
TxTable used twice in context to transfer assets to
and from banking account. There is no need to store
instance of table persistently.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-03 15:17:58 +03:00
Alex Vanin
fd461bdb65 [#365] innerring: Produce distribute income events
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-03 15:17:58 +03:00
Alex Vanin
d77d49bd2a [#365] Provide distribute income event in settlement processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-03 15:17:58 +03:00
Alex Vanin
8e741a277d [#365] settlement/basic: Implement asset distribution
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-03 15:17:58 +03:00
Alex Vanin
4433448645 [#365] settlement/basic: Check amount of collected assets
Some transfers from container owners into bank account may
fail due to lack of assets, so we have to deal with remaining
amount of assets in banking account.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-03 15:17:58 +03:00
Leonard Lyubich
f85e5f0238 [#202] object/eacl: Fix algorithm for checking eACL table
Add SignWithRFC6979 option to signature verification function since eACL
table are signed by users with this option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-03 15:14:59 +03:00
Leonard Lyubich
ad348afcd6 [#202] object/eacl: Verify signature of eACL table
Since the contract started returning the table signature, it became
necessary to check its correctness.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-03 15:14:59 +03:00
Leonard Lyubich
ffbf6b922f [#361] metabase: Support new match types
Support STRING_NOT_EQUAL and NOT_PRESENT match types.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-03 10:32:15 +03:00
Alex Vanin
487c9b7589 [#363] Define global config and use it to fetch basic income rate
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-02 21:07:00 +03:00
Alex Vanin
be2ed6bf4c [#360] Run basic income collection in inner ring
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-02 21:07:00 +03:00
Alex Vanin
8c4bf81351 [#360] Use basic implement context in settlement processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-02 21:07:00 +03:00
Alex Vanin
a624bb881d [#360] Implement basic settlement context
Basic settlement context is a main structure that
implement logic of basic settlement phases: collecting
assets from container owners and then distributing them
to storage nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-02 21:07:00 +03:00
Alex Vanin
f45675b7a2 [#360] Share common parts of basic and audit settlements
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-02 21:07:00 +03:00
Leonard Lyubich
bd5c70131b [#328] cmd/node: Serve Container.AnnounceUsedSpace RPC
Register recently implemented handler of AnnounceUsedSpace RPC in node app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
7e586aef90 [#328] container/grpc: Implement AnnounceUsedSpace method
Add AnnounceUsedSpace method that converts gRPC request to v2 request and
passes it to v2 client.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
21133aee2f [#328] container/load: Implement route builder based on placement
Implement route Builder interface on wrapper over the container placement
builder, Component implies exactly one transfer to each of the most weight
nodes of the container (according to some weighing algorithm).
Implementation is planned for use when transferring local estimates of
storage nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
d48fb81193 [#328] container/load: Implement route controller
Implement a component for transmitting the value of the used container space
along a route defined in the system. Implement WriterProvider interface on
it. By implementation, it is the link between the route planner and the
point-to-point transmitter, and abstracts from the implementation of both of
them. In the future, this implementation will be used as a transmitter of
local estimates of storage nodes among themselves.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
1c6d37e821 [#328] morph/container: Add client methods related to size estimations
Implement API methods of the Container contracts client  corresponding
to the calls of the contract for storing the used space of the container
("putContainerSize", "getContainerSize", "listContainerSizes"). Extend
the wrapper over the client with methods abstracted from sidechen calls.
In particular, the method of storing the value will be used to record
the estimates of the used space of containers calculated by the network
participants.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
4415f8dc5b [#328] container/load: Implement simple Iterator and Writer providers
Implement functions to wrap Writer or Iterator. The resulting wrapper
provides WriterProvider or IteratorProvider interface respectively.
Such a wrapper can be used as a single storage instance provider
regardless of context.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
2622e11ae3 [#328] container/load: Implement local storage of used space announcements
Implement a component that stores the values of the used space of
containers. The storage allows you to write several values for a fixed
container and epoch number, and read the averaged estimates of all
accumulated values. All values are stored in memory. This component is
planned to be used as an accumulator of opinions from various network
participants about the fullness of the container.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Leonard Lyubich
cc88320d6b [#328] container/load: Implement a metrics exchange controller
Implement a component that connects the value stores of the used space of
containers. Implement the Start/Stop operations on it, which will later
become the application handlers of the corresponding events from the
sidechain. The main task of the controller is to temporarily synchronize the
stages of calculating the global estimate of the used space in the
container. The details of the score calculation (the way of collecting /
transmitting local scores, the final score formula and writing to the
contract) are encapsulated in the dependency components, the controller is
abstracted from them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-02 15:45:04 +03:00
Alex Vanin
5b550bff22 [#355] innerring: Produce container size estimation notifications
There are two notifications:
- start estimation notification produced at the beginning of the
  epoch,
- stop estimation notifications should be produced before
  basic audit settlement starts.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:23:37 +03:00
Alex Vanin
6848a816f9 [#355] innerring: Refactor block timer constructors
This small refactoring adds `blocktimer.go` file with
all timer related function and constructors. This way
we can create all timers in one place (at the end of
innerring.Server constructor).

To do that we had to move timer reset into global
server state so it can be accessed by netmap
processor.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:23:37 +03:00
Alex Vanin
402192c8c4 [#355] innerring: Add fee to NoFee wrappers
There is no point of making separate `Fee` and `NoFee`
wrappers because all reading operations are free disregarding
of fee value in static client. However we can use these same
wrappers so send transaction.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:23:37 +03:00
Alex Vanin
70aa73cbd2 [#355] moprh/client: Add start/stop container estimation methods
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:23:37 +03:00
Alex Vanin
7a7a351f1f [#355] morph/event: Add container estimate size events
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:23:37 +03:00
Alex Vanin
db94ad5567 [#355] morph/event: Fix delete container event test
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-01-29 11:23:37 +03:00