Commit graph

956 commits

Author SHA1 Message Date
Leonard Lyubich
4c8d29ce46 [#398] morph/client: Add method to read magic number
Implement `Client.MagicNumber` method that returns magic number of the
network to which the underlying RPC node client is connected.

Network magic value is received via `GetNetwork` method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 19:16:40 +03:00
Leonard Lyubich
146abe8520 [#388] cmd/cli: Use bearer token when collecting storage group members
CLI `storagegroup put` cmd collects information about SG members via NeoFS
API ObjectService.Head RPC in order to compose SG structure. Bearer token
attached to the call was not used in communication, which could lead to data
access problems. These changes fix the described problem.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 16:20:21 +03:00
Alex Vanin
5f47580ff7 [#399] object/fmt: Fix test for tombstone epoch expiration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-19 16:20:10 +03:00
Alex Vanin
e6cdf3fbf5 [#399] object/fmt: Check expiration epoch in tombstone body and header
According to nspcc-dev/neofs-api#136 tombstone body should store the same
attribute as in object header. If they are different, then check is failed
with `errTombstoneExpiration`.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-19 16:20:10 +03:00
Leonard Lyubich
3ed0065455 [#395] metabase: Do not add tombstone-on-tombstone records to graveyard
Creating tombstones for tombstones is prohibited in NeoFS system. Metabase
graveyard contains records of the form {address: address}: key is an address
of inhumed object, value is an address of the tombstone. To prevent creation
tombstones for tombstones metabase must control incoming Inhume calls:

 * if Inhume target is a tombstone, then "grave" should not be added;
 * if {a1:a2} "grave" was created earlier and {a2: a3} "grave" came later,
   then first "grave" must be removed as tomb-on-tomb.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 16:19:26 +03:00
Leonard Lyubich
bc5e04f502 [#378] metabase: Calculate expiration once for each fake bucket
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
a6d5556dc1 [#378] shard/gc: Reuse addressFromKey func in all appropriate places
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
72f7df3339 [#378] shard/gc: Limit the accumulated batch of objects to be deleted
Change Shard's garbage remover to interrupt iterating over the metabase
graveyard when the buffer is full to the max size (`WithRemoverBatchSize`
Shard's option).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
f5f416268f [#378] metabase: Add the ability to break the loop in IterateOverGraveyard
Make `DB.IterateOverGraveyard` to immediately return nil if GraveHandler
returns ErrInterruptIterator.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
ff442e9559 [#378] shard/gc: Avoid usage of the range-loop vars in go-routines
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
b644589bb4 [#378] cmd/node: Configure tombstone lifetime
Add `TOMBSTONE_LIFETIME` configuration value of the node which is measured
in NeoFS epoch and is set to 5 by default.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
b8d1144839 [#378] object/delete: Set expiration epoch of the created tombstones
Make object delete service to use network information to calculate and set
expiration of the created tombstone.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
717f2beb47 [#378] shard: Collect expired tombstones in GC every epoch
Add new epoch event handler to GC that finds all expired tombstones and
marks them and underlying objects to be removed. Shard uses callbacks
provided by the storage engine to mark underlying objects.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
3d5169c4c9 [#378] metabase: Implement iterator over the objects under tombstones
Implement `DB.IterateCoveredByTombstones` method that iterates over graves
and handles all objects under one of the tombstones.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
b961570766 [#378] cmd/node: Subscribe shards on new epoch events
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
dab45050b9 [#378] shard: Collect expired non-tombstone objects in GC every epoch
Add new epoch event handler to GC that finds all expired non-tombstone
objects and marks them to be removed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
89a22450e5 [#378] shard: Define new epoch event
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
182df23859 [#378] metabase: Implement iterator over expired objects
Implement `DB.IterateExpired` method that iterates over the objects in
metabase that are expired at particular epoch.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
a9a1acc880 [#378] shard: Control the completion of all handlers of the previous event
Group handlers of the particular event to a WaitGroup and wait for it before
the next event handling. This will ensure that all handlers complete and
prevent potential conflicts between past and present jobs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
18add24727 [#378] cmd/node: Configure GC options
Add `GC_REMOVER_BATCH_SIZE` and `GC_REMOVER_SLEEP_INTERVAL` config values to
shard's config section.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
4da41613c3 [#378] shard: Initialize GC processes on Init
`Shard.Init` method creates a new GC instance from shard configuration and
starts GC's workers through `init` call. In initial implementation GC
routines are indefinite and can be killed only with by application shutdown.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
13087dc3dd [#378] shard: Implement skeleton of internal GC
Shard's GC component consists of:
 * asynchronous remover that periodically wake up and removes all garbage
   objects from the shard, and goes to sleep for particular time interval;
 * external event listener that distributes jobs between workers;
 * group of workers that can handle a single job related to particular
   external event.

Remover and event listener represents go-routines which are started by
`init` method (calls from `Shard.Init`). In initial version all event
handlers are interrupted: this means that next event of the same type will
interrupt previous handling and start the new one.

GC is fully encapsulated in Shard. All GC configurations are reflected in
Shard's configuration.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
d72d37a322 [#378] metabase: Implement iterator over metabase graveyard
Implement `DB.IterateOverGraveyard` method that iterates over all graves and
passes passes their descriptors (new type `Grave`) to handler (new type
`GraveHandler`). `Grave` currently have buried object address and garbage
flag.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
3d7d074af4 [#377] shard: Support batch Inhume operation
Replace single target address in `InhumePrm` with the list of addresses.
Change corresponding parameter in `WithTarget` and `MarkAsGarbage` methods
to variadic.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
0d6d195d0d [#377] metabase: Support batch Inhume operation
Replace single target address in `InhumePrm` with the list of addresses.
Rename `WithAddress` method to `WithAddresses` and change parameter to
variadic.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
6ec7433e14 [#377] storage engine: Change the implementation of Delete operation
Make `StorageEngine.Delete` to execute `Inhume` operation with
`MarkAsGarbage` parameter on the `Shard` that holds the object. Searching of
the particular shard is performed through iterating over HRW-sorted shards.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
d3a0079d1d [#377] shard: Add Inhume parameter to mark the object as garbage
Implement `InhumePrm.MarkAsGarbage` method that leads to marking object as
garbage in metabase. Update `InhumePrm.WithTarget` doc indicating a conflict
with the new method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
30ac234c20 [#377] metabase: Add Inhume parameter to mark the object as garbage
Implement `InhumePrm.WithGCMark` method that marks the object as garbage in
graveyard. Update `InhumePrm.WithTombstoneAddress` doc indicating a conflict
with the new method. Update `Inhume` function doc about tombstone address
parameter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-19 11:56:32 +03:00
Leonard Lyubich
7c3f828893 [#219] metabase: Count parent references in Delete operation
Delete operation of Metabase is performed on group of objects. The set being
removed can contain descendants of a common parent. In the case when all
descendants of a parent object are deleted, it must also be deleted from
the metabase. In the previous implementation, this was not done due to the
chosen approach to counting references to the parent.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-18 09:54:19 +03:00
Leonard Lyubich
38727c2930 [#368] object: Reject expired objects
The lifetime of an object can be limited by specifying a correspondin
well-known attribute. Node should refuse to save expired objects.

Checking objects in FormatValidator is extended with an expiration attribute
parsing step.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-15 19:04:33 +03:00
Leonard Lyubich
a2c2241356 [#379] storage engine: Inhume object in one shard
In previous implementation StorageEngine.Inhume operation forced Shard
.Inhume call on all internal shards. There is a need to inhume object in a
single shard. To achieve this, Inhume operation is performed in next steps:

 1. iterate over sorted shards, check object presence through Exists call;
 2. if object exists at any shard in step 1 => inhume it and return on
    success;
 3. if no shards contain the object => iterate over sorted shards again and
    try to inhume the object at first possible shard;
 4. if all Inhume calls are failed => return an error.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-15 18:58:18 +03:00
Leonard Lyubich
75609012c7 Update changelog and docs for release v0.15.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-15 10:49:38 +03:00
Leonard Lyubich
065c663bde Update neo-go to v0.93.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-15 10:49:38 +03:00
Leonard Lyubich
d936567249 Update neofs-api-go to v1.23.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-15 10:49:38 +03:00
Alex Vanin
bd420b6002 [#373] innerring: Listen sidechain blocks for internal timer
Updated neo-go fixed bug in listener, so we can use sidechain
for block timer as it planned.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-11 17:31:56 +03:00
Alex Vanin
d7bd312abd [#373] Update neo-go to pre-v0.93.0
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-11 17:31:56 +03:00
Alex Vanin
0bb3836e84 [#383] innerring: Sort node info attributes
All node info attribute transformations can't guarantee
the order of attributes. However it should be consistent
otherwise smart-contract won't be able to collect signatures
and approve transaction.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-11 14:11:43 +03:00
Alex Vanin
058ab0e533 [#383] innerring: Marshal node info back after validation
Node info validator may change node attributes, e.g. update
it with human-readable location attributes based on LOCODE.
So inner ring node should provide new node info binary to
smart contract.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-02-11 14:11:43 +03:00
Leonard Lyubich
6cb906ab5f [#362] cli/object: Support JSON input of search filter
Consider single word of search filter expression as path to file with
protobuf JSON filters. Decode filters from file and add them to the rest.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
6039cd119c [#362] cli/object: Change query language of search filters
Make `--filters` flag to be repeated. Define new filter expression format:
 * `<key> <binary_op> <value>` for binary filters. Supported binary ops: `EQ` (`STRING_EQUAL`), `NE` (`STRING_NOT_EQUAL`).
 * `<key> <unary_op>` for unary filters. Supported unary ops: `NOPRESENT` (`NOT_PRESENT`).
Any other string expressions are considered invalid.

Additionally support shorthand flag `-f`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
3b9d4b4df1 [#362] cmd/node: Use const key to UN/LOCODE attribute from the API lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
a737a46988 [#362] ir/netmap: Use const keys to node attributes from the API lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
34248b00ed [#362] ir/netmap: Do not add LocationCode attribute to netmap candidate
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
37915b53cf [#362] Pull latest neofs-api-go lib
Adopt new names of constant keys to node attributes from API lib.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-11 14:01:35 +03:00
Leonard Lyubich
977229eb5a [#316] locode/csv: Scan subdivision csv-table into memory
Scanning subdivision csv-table entries one-by-one takes significant time and
system resources. To speed up random access to table records, on the first
call, the table is pumped into memory (map). On subsequent calls, I/O
operations are not performed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 21:11:28 +03:00
Leonard Lyubich
307355f165 [#316] locode/airports: Scan csv table into memory
Scanning csv-table entries one-by-one takes significant time and system
resources. To speed up random access to table records, on the first call,
the table is pumped into memory (map). On subsequent calls, I/O operations
are not performed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 20:18:06 +03:00
Leonard Lyubich
687c7d3b4a [#316] locode/column: Fix LatitudeHemisphere.North method implementation
North method must return true if hemisphere char is `N`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 20:18:06 +03:00
Leonard Lyubich
e63f9faab8 [#316] cmd/node: Make UN-LOCODE attribute mandatory in the configuration
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
7a9b02049f [#316] cmd/node: Refactor working with well-known node attributes
Split the attributes into those that must be explicitly set in the
configuration, and those that, if absent, will be assigned a default value.
Support this logic in `addWellKnownAttributes` function. If no explicit
attribute is set, the application will panic.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00
Leonard Lyubich
19681693f7 [#316] ir/netmap: Rename City node attributes
Rename `City`/`CityCode` attribute keys to `Location`/`LocationCode`
respectively.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-02-10 14:05:03 +03:00