Commit graph

842 commits

Author SHA1 Message Date
Leonard Lyubich
106884fc40 [] client: Hide client cache behind interface in dependent packages
Replace usage of `cache.ClientCache` type with interface with similar
signature. This will further allow overloading clients without affecting the
logic of dependent packages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Leonard Lyubich
f25253738a [] reputation: Implement local trust storage
Implement in-memory `Storage` which is going to be used to submit the
results of interactions with network members. `Storage` also provides an
iterator interface, so the component can be used in `Controller`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Leonard Lyubich
29e974df93 [] reputation: Implement local trust controller
Implement `Controller` designed to analyze the values collected by the local
node (the source is hidden behind Writer/WriterProvider interfaces) and
transfer them to the destination (hidden behind Iterator / IteratorProvider
interface).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Leonard Lyubich
eadb3204f0 [] reputation: Define basic types
Define PeerID, TrustValue and Trust types and basic methods on them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Leonard Lyubich
8d5c17facd [] metabase: Fix saving of object payload in Put
Metabase should not store payloads of objects. Make Put operation to cut
object payload before saving binary object in metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-22 21:40:27 +03:00
Leonard Lyubich
e3f8470d7a [] metabase: Write unit tests for checking the absence of payloads
Metabase should not store payloads of objects. Set payload in generated test
object. Ascertain that objects returned by Get method have no payload.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-22 21:40:27 +03:00
Alex Vanin
5149a701f3 [] Remove notary extra fee
`client.CalculateNotaryFee` now calculates precise
fee value.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-22 11:45:35 +03:00
Pavel Karpy
b9892edd6e [] innerring: Add gas threshold
On the `Deposit` events add gas balance check.
Make transfer only if the balance is greater
than the `GasTransferThreshold` that is defined
with environmental variable.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-03-19 13:57:58 +03:00
Leonard Lyubich
383d2494eb [] Adapt the Client's refactoring to the interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Leonard Lyubich
b62a2a0f54 [] blobstor: Remove hard-code used for Blobovnicza test
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Leonard Lyubich
e6cde0ab28 [] client cache: fix the passing of arguments to Client constructor
Private key was removed from Client constructor New recently.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Leonard Lyubich
4640099a67 [] control: Implement service RPCs
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Leonard Lyubich
9eaba52660 [] grpc: Use new mechanism of message conversion
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Leonard Lyubich
718a2fad26 [] services: Define service interfaces that was removed from API lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Alex Vanin
892b4f4f12 [] service/object: Fix put duration
At object.Put operation node does not transfer any
data. It is done in the stream inside, so we should
register duration from `object.Put` invocation until
`CloseAndRecv` on the stream.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
b8e10571c6 [] Put prometheus behind pkg/metrics
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
980b774af2 [] engine: Support duration metrics
With `enable metrics` option, engine will collect
durations for all public methods.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
bc6e7a823b [] service/object: Add object size metrics
We can't rely on object size in the header because it
might be omitted on initial put or it can be 0xFF.. on
streaming data.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
0b93e8a029 [] service/object: Add request duration metrics
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
f8ab7a9c60 [] service/object: Add object service metrics collector
Collect number of request per method in object service.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
ec07792c4e [] Update neo-go to pre-released v0.94.0 version
Should be compatible with NEO3 testnet.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 09:57:40 +03:00
Evgenii Stratonikov
364945a955 [] pkg/innerring: fix typo in config parameter name
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-03-15 14:07:53 +03:00
Evgenii Stratonikov
6679d59e89 [] pkg/services: Provide client options on cache creation
Because options are not used when client is already in cache
providing them to shared cache is misleading at best.
In the worst case `dial_timeout` is set randomly (because of race
condition) which can lead to one service having `dial_timeout` of
another. Thus we set default client creation options when cache is
created.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-03-15 14:07:53 +03:00
Evgenii Stratonikov
cc7287d6f7 [] pkg/services: Cache clients by address only
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-03-15 14:07:53 +03:00
Leonard Lyubich
ed808c3f1b [] eacl: Replace eACL storage implementation to app-side
Replace `eacl.Storage` interface implementation from eACL lib to neofs-node
app package. Remove `eacl.WithMorphClient` option in order to abstract eACL
validator from eACL storage implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-04 16:33:29 +03:00
Alex Vanin
8c3864e6d6 [] innerring: Wait for deposit in initialization
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
ddf1ac0f28 [] morph/client: Add awaiting function
Awaiting function locks execution for N blocks.
Useful to wait for notary deposit.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
4b10e82685 [] innerring: Use notary invocations in inner ring
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
83980ccb85 [] morph/client: Support notary calls in wrappers
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
71dce97b76 [] innerring: Make notary deposit periodically
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
ccaf4f5d55 [] innerring: Enable notary support in morph client
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
779a495625 [] morph/client: Define notary client
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Alex Vanin
948823c392 [] innerring: Get GAS script hash from neo-go client
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-04 12:39:15 +03:00
Leonard Lyubich
bcfd33d9fb [] ir: Make UN-LOCODE attribute of the network map candidate optional
Previously UN-LOCODE attribute was required for each network map candidate.
In the absence of this attribute, the candidate was not allowed into the
network map. After revising the requirements for candidates, it was decided
not to require the mandatory installation of the attribute by candidates.

From now inner ring does not modify location attributes of the network map
candidate in the absence of UN-LOCODE attribute and does not block entry
into the network map for this criterion.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-25 14:36:11 +03:00
Alex Vanin
5f800458a3 [] pkg/policy: Parse strings in filter key
With UN/LOCODE support, storage node may have
`UN-LOCODE` attribute. Policy parser should support
both `Ident` and `Strings` as filter keys to parse
rules such as `FILTER "UN-LOCODE" EQ "RU LED" AS F`.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-25 11:57:26 +03:00
Leonard Lyubich
5900975d58 [] object/policer: Leave readability instead of performance comment
Right now we pass redundant copy to callback outside the for-loop through
the helpful boolean variable instead of calling it deeply nested.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
277e3ca20a [] policer: Handler redundant local copy of the object
Detect redundant local copy of the object in Object Policer. Add redundant
copy callback (`WithRedundantCopyCallback` option). Pass address of the
redundant copy to callback.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
35073fb61b [] storage engine: Add GC marking to Inhume operation
Add `InhumePrm.MarkAsGarbage` method which marks passed objects to be
removed from local storage. Update `InhumePrm.WithTarget` doc to prevent
conflicting use with the new method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
7a4e3efa95 [] storage engine: Transform Inhume to batch operation
Make `StorageEngine.Inhume` to process list of objects per single operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 20:59:14 +03:00
Leonard Lyubich
53b4d6d6d0 [] ir: Support working with sidechain only
NeoFS contract can be deployed in sidechain instead of main chain.

Add `without_mainnet` config flag that can switch IR node to work with
sidechain only. By default this flag is unset.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:17:35 +03:00
Leonard Lyubich
0bca98975a [] ir/alphabet: Rename constant corresponding to I/izhei letter
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
72935d0a80 [] ir: Upgrade to a dynamic number of alphabetical contracts
In previous implementation IR worked with exactly 7 alphabetic contracts
only. Actually number of contracts is limited to only the Glagolitic
alphabet.

Make IR to work with any valid number of alphabetic contracts. Change parser
of alphabetic contract addresses to read amount of processing contracts
before performance. Make Alphabet processor to use interface of the
alphabetic contract group. Use `alphabetContracts` type in IR `Server`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
1016ba3a5d [] ir: Implement type for group of alphabet contracts
Define `alphabetContracts` type that map glagolic letters to smart contract
addresses. Implement constructor and all methods which are going to be used
in IR processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
3430a6d101 [] ir: Define Glagolitsa alphabet type
Define `glagolicLetter` enumeration of the Glagolitsa alphabet letters.
Implement `configString` method that returns config-compatible string format
of the letters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
397c3b6757 [] ir: Remove bitsize from IR indices
There is no need to specify that IR indices are 32 bits in size.

Change return types of `Indexer` interface methods in audit and alphabet
packages. Support interface changes in `Server` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 19:16:52 +03:00
Leonard Lyubich
aa24702ebe [] control: Serve DropObjects RPC
Re-compile protobuf definition of Control service. Implement required
messages on DropObjects RPC request and response messages. Implement
`DropObjects` method on Control service server of the node. Use
`StorageEngine.Delete` method as a deleted object handler on server.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 13:07:53 +03:00
Leonard Lyubich
f2337baedc [] control: Add RPC to mark objects to be removed from the node
There is a need to define storage node's interface to remove objects from
the local storage.

Define `DropObjects` RPC in `ControlService`. Define `DropObjectsRequest` /
`DropObjectsResponse` structure of the request / response messages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-24 13:07:53 +03:00
Leonard Lyubich
9073e198b9 [] cmd/node: Serve NetmapService.NetworkInfo RPC
Implement `NetworkInfo` calls on full stack of Netmap services. Current
epoch is read from node local state, magic number is read via `MagicNumber`
call of morph client.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 19:16:40 +03:00
Leonard Lyubich
4c8d29ce46 [] morph/client: Add method to read magic number
Implement `Client.MagicNumber` method that returns magic number of the
network to which the underlying RPC node client is connected.

Network magic value is received via `GetNetwork` method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 19:16:40 +03:00
Alex Vanin
5f47580ff7 [] object/fmt: Fix test for tombstone epoch expiration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-19 16:20:10 +03:00
Alex Vanin
e6cdf3fbf5 [] object/fmt: Check expiration epoch in tombstone body and header
According to  tombstone body should store the same
attribute as in object header. If they are different, then check is failed
with `errTombstoneExpiration`.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-19 16:20:10 +03:00
Leonard Lyubich
3ed0065455 [] metabase: Do not add tombstone-on-tombstone records to graveyard
Creating tombstones for tombstones is prohibited in NeoFS system. Metabase
graveyard contains records of the form {address: address}: key is an address
of inhumed object, value is an address of the tombstone. To prevent creation
tombstones for tombstones metabase must control incoming Inhume calls:

 * if Inhume target is a tombstone, then "grave" should not be added;
 * if {a1:a2} "grave" was created earlier and {a2: a3} "grave" came later,
   then first "grave" must be removed as tomb-on-tomb.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 16:19:26 +03:00
Leonard Lyubich
bc5e04f502 [] metabase: Calculate expiration once for each fake bucket
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
a6d5556dc1 [] shard/gc: Reuse addressFromKey func in all appropriate places
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
72f7df3339 [] shard/gc: Limit the accumulated batch of objects to be deleted
Change Shard's garbage remover to interrupt iterating over the metabase
graveyard when the buffer is full to the max size (`WithRemoverBatchSize`
Shard's option).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
f5f416268f [] metabase: Add the ability to break the loop in IterateOverGraveyard
Make `DB.IterateOverGraveyard` to immediately return nil if GraveHandler
returns ErrInterruptIterator.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
ff442e9559 [] shard/gc: Avoid usage of the range-loop vars in go-routines
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
b8d1144839 [] object/delete: Set expiration epoch of the created tombstones
Make object delete service to use network information to calculate and set
expiration of the created tombstone.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
717f2beb47 [] shard: Collect expired tombstones in GC every epoch
Add new epoch event handler to GC that finds all expired tombstones and
marks them and underlying objects to be removed. Shard uses callbacks
provided by the storage engine to mark underlying objects.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
3d5169c4c9 [] metabase: Implement iterator over the objects under tombstones
Implement `DB.IterateCoveredByTombstones` method that iterates over graves
and handles all objects under one of the tombstones.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
dab45050b9 [] shard: Collect expired non-tombstone objects in GC every epoch
Add new epoch event handler to GC that finds all expired non-tombstone
objects and marks them to be removed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
89a22450e5 [] shard: Define new epoch event
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
182df23859 [] metabase: Implement iterator over expired objects
Implement `DB.IterateExpired` method that iterates over the objects in
metabase that are expired at particular epoch.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
a9a1acc880 [] shard: Control the completion of all handlers of the previous event
Group handlers of the particular event to a WaitGroup and wait for it before
the next event handling. This will ensure that all handlers complete and
prevent potential conflicts between past and present jobs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
4da41613c3 [] shard: Initialize GC processes on Init
`Shard.Init` method creates a new GC instance from shard configuration and
starts GC's workers through `init` call. In initial implementation GC
routines are indefinite and can be killed only with by application shutdown.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
13087dc3dd [] shard: Implement skeleton of internal GC
Shard's GC component consists of:
 * asynchronous remover that periodically wake up and removes all garbage
   objects from the shard, and goes to sleep for particular time interval;
 * external event listener that distributes jobs between workers;
 * group of workers that can handle a single job related to particular
   external event.

Remover and event listener represents go-routines which are started by
`init` method (calls from `Shard.Init`). In initial version all event
handlers are interrupted: this means that next event of the same type will
interrupt previous handling and start the new one.

GC is fully encapsulated in Shard. All GC configurations are reflected in
Shard's configuration.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
d72d37a322 [] metabase: Implement iterator over metabase graveyard
Implement `DB.IterateOverGraveyard` method that iterates over all graves and
passes passes their descriptors (new type `Grave`) to handler (new type
`GraveHandler`). `Grave` currently have buried object address and garbage
flag.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
3d7d074af4 [] shard: Support batch Inhume operation
Replace single target address in `InhumePrm` with the list of addresses.
Change corresponding parameter in `WithTarget` and `MarkAsGarbage` methods
to variadic.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
0d6d195d0d [] metabase: Support batch Inhume operation
Replace single target address in `InhumePrm` with the list of addresses.
Rename `WithAddress` method to `WithAddresses` and change parameter to
variadic.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
6ec7433e14 [] storage engine: Change the implementation of Delete operation
Make `StorageEngine.Delete` to execute `Inhume` operation with
`MarkAsGarbage` parameter on the `Shard` that holds the object. Searching of
the particular shard is performed through iterating over HRW-sorted shards.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
d3a0079d1d [] shard: Add Inhume parameter to mark the object as garbage
Implement `InhumePrm.MarkAsGarbage` method that leads to marking object as
garbage in metabase. Update `InhumePrm.WithTarget` doc indicating a conflict
with the new method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
30ac234c20 [] metabase: Add Inhume parameter to mark the object as garbage
Implement `InhumePrm.WithGCMark` method that marks the object as garbage in
graveyard. Update `InhumePrm.WithTombstoneAddress` doc indicating a conflict
with the new method. Update `Inhume` function doc about tombstone address
parameter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
7c3f828893 [] metabase: Count parent references in Delete operation
Delete operation of Metabase is performed on group of objects. The set being
removed can contain descendants of a common parent. In the case when all
descendants of a parent object are deleted, it must also be deleted from
the metabase. In the previous implementation, this was not done due to the
chosen approach to counting references to the parent.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-18 09:54:19 +03:00
Leonard Lyubich
38727c2930 [] object: Reject expired objects
The lifetime of an object can be limited by specifying a correspondin
well-known attribute. Node should refuse to save expired objects.

Checking objects in FormatValidator is extended with an expiration attribute
parsing step.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-15 19:04:33 +03:00
Leonard Lyubich
a2c2241356 [] storage engine: Inhume object in one shard
In previous implementation StorageEngine.Inhume operation forced Shard
.Inhume call on all internal shards. There is a need to inhume object in a
single shard. To achieve this, Inhume operation is performed in next steps:

 1. iterate over sorted shards, check object presence through Exists call;
 2. if object exists at any shard in step 1 => inhume it and return on
    success;
 3. if no shards contain the object => iterate over sorted shards again and
    try to inhume the object at first possible shard;
 4. if all Inhume calls are failed => return an error.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-15 18:58:18 +03:00
Leonard Lyubich
065c663bde Update neo-go to v0.93.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-15 10:49:38 +03:00
Alex Vanin
bd420b6002 [] innerring: Listen sidechain blocks for internal timer
Updated neo-go fixed bug in listener, so we can use sidechain
for block timer as it planned.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-11 17:31:56 +03:00
Alex Vanin
0bb3836e84 [] innerring: Sort node info attributes
All node info attribute transformations can't guarantee
the order of attributes. However it should be consistent
otherwise smart-contract won't be able to collect signatures
and approve transaction.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-11 14:11:43 +03:00
Alex Vanin
058ab0e533 [] innerring: Marshal node info back after validation
Node info validator may change node attributes, e.g. update
it with human-readable location attributes based on LOCODE.
So inner ring node should provide new node info binary to
smart contract.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-11 14:11:43 +03:00
Leonard Lyubich
a737a46988 [] ir/netmap: Use const keys to node attributes from the API lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
34248b00ed [] ir/netmap: Do not add LocationCode attribute to netmap candidate
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
977229eb5a [] locode/csv: Scan subdivision csv-table into memory
Scanning subdivision csv-table entries one-by-one takes significant time and
system resources. To speed up random access to table records, on the first
call, the table is pumped into memory (map). On subsequent calls, I/O
operations are not performed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 21:11:28 +03:00
Leonard Lyubich
307355f165 [] locode/airports: Scan csv table into memory
Scanning csv-table entries one-by-one takes significant time and system
resources. To speed up random access to table records, on the first call,
the table is pumped into memory (map). On subsequent calls, I/O operations
are not performed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 20:18:06 +03:00
Leonard Lyubich
687c7d3b4a [] locode/column: Fix LatitudeHemisphere.North method implementation
North method must return true if hemisphere char is `N`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 20:18:06 +03:00
Leonard Lyubich
19681693f7 [] ir/netmap: Rename City node attributes
Rename `City`/`CityCode` attribute keys to `Location`/`LocationCode`
respectively.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
9a425924cf [] locode/boltdb: Replace all references to the City with Location
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
0a87fec6c6 [] ir/netmap: Rename LOCODE attribute key
Rename key to LOCODE node attribute from `Locode` to `UN-LOCODE`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
46d60f3d52 [] ir/locode: Open LOCODE database in read-only mode
Inner ring Server does not modify LOCODE database (Put method), thus it is
better to open it in RO mode.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
0d2440649a [] locode/boltdb: Add option to enable read-only mode
Add ReadOnly function that returns Option that enables read-only mode in DB.
RO mode can be used by processes that won't modify the DB in order to not
acquire write flock.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
04b06ba539 [] locode/boltdb: Add default timeout to open underlying BoltDB instance
Set timeout option of BoltDB Open operation to 3s in order to prevent
indefinite waiting for file lock.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
97264acb26 [] ir: Use LOCODE validator as NodeValidator in Netmap processor
Implement DB interface required by LOCODE validator on new wrapper over the
LOCODE Bolt DB (Record on new wrapper over LOCODE Bolt DB entries).
Construct LOCODE validator and pass it to Netmap processor's constructor as
NodeValidator parameter.

Thus, candidates for a network map must set LOCODE attribute for which there
is an entry in the NeoFS location database.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
ff814aec26 [] ir: Add before-start and after-stop function to Server
Keep list of functions which are called first when the server starts (method
Server.Start). If any of the starters returns an error, the server will not
start. Such starters will mainly be used for resources that need to be
initialized after a successful server construction, but before its main work
(e.g. local files).

Keep list of functions which are called when the server stops (Server.Stop
method). Such closers will mainly be used for resources that need to be
released after server shutdown (e.g. initialized by starters).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
db703a5117 [] ir/netmap: Validate LOCODE attributes of network map candidates
Define a structure for dealing with the geographic location of nodes.
Implement VerifyAndUpdate (with the same purpose as NodeValidator interface)
that checks LOCODE attribute and fills other attributes of the location.
Technically the entity is a wrapper over the NeoFS location database: it
maps the node LOCODE to the database record from which the new attributes
are generated.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
cbfaceb956 [] ir/netmap: Validate network map candidates in Netmap processor
Define NodeValidator interface of the entity that checks and finalizes
NodeInfo structure. Add NodeValidator to Netmap processor. Pass NodeInfo
structures of network map candidates to NodeValidator in order to verify it
and prepare to final state required by network.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
f88e0866fe [] locode: Implement NeoFS location database based on BoltDB instance
Define NeoFS location database based on BoltDB. Implement methods to save
the record by key (Put) and to read the record by key (Get).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
9669afdfc7 [] locode: Implement polygons DB based on GeoJSON polygons
Define database of Earth's polygons in GeoJSON format. Implement resolving
of geo point to continent in which it is located through point-in-polygon
calculation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
d8c3a0e4f5 [] locode: Implement csv OpenFlights airport database
Implement airport database based on csv OpenFlights table. Implement
UN/LOCODE entry matcher. Implement country namespace.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
6829048124 [] locode: Implement csv UN/LOCODE table
Define csv UN/LOCODE table. Implement iterator over table entries. Implement
method that resolves country/subdivision code pair to subdivision name.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
cdd1274e1c [] locode: Define the API of location database
Define structure of keys and records of the location database. Define the
interfaces of all components necessary for the formation of the database.
Implement the function of filling the database.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00
Leonard Lyubich
0be35859ed [] locode: Define UN/LOCODE table data types
Define the data types needed to work with LOCODE's in NeoFS (country code,
location code, coordinates). Implement string parsers for new types.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-09 11:05:55 +03:00