Commit graph

510 commits

Author SHA1 Message Date
Alex Vanin
ea41dec23f [#160] Store root and phy indexes as fake bucket tree
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
f1a3acd27a [#160] Fix metabase test for virtual object
Parent should not have split header.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
e580f2afd6 [#160] Update object CLI flags
* remove no-leaf
* rename leaf to phy
* remove no-root

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
32219bb294 [#160] Remove query match function
This function duplicates query processing that
is done in meta-storage now.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
03fed8ca59 [#160] Update metabase with new root and phy flags processing
Now root and phy (leaf) filters work like flags. They work with
any matcher and any value. So meta-storage sets `true` value for
all root and phy objects and puts them into separate bucket.

We also do not work with inversion anymore, so it either added
to the bucket or not. We don't need to store both options.
This is the reason `selectAll` function is changed a bit. Now
it performs some low-level parsing from primary bucket and root
bucket.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
2c50032831 [#160] Classify only regular objects as root
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
1c12f6bcbf [#160] Update neofs-api-go to latest version
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
b59af19dff [#171] Increase extra gas on ApprovePeer method
With latest changes in neo's execution costs, we need more
gas to add storage node to netmap.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-13 17:12:46 +03:00
Alex Vanin
eb13322bf0 [#166] Use async pools in object service handlers
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-13 15:54:00 +03:00
Alex Vanin
ded45e1fbc [#159] Add keyer to neofs-cli
Keyer prints information about private key, public key,
NEO3 Wallet, scripthash. It can generate new private key
or generate multisig address.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-09 18:59:21 +03:00
Leonard Lyubich
fcb35d82cf [#165] Revert "[#142] metabase: Store header value index in a tree leaf"
Revert commit 0faa40e4 to increase the disk space consumed by the
metabase in favor of the speed of index updates.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 18:58:58 +03:00
Leonard Lyubich
3c42f5b452 [#161] object/head: Inherit common parameters in HeadRelation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 17:19:34 +03:00
Leonard Lyubich
d38633e047 [#161] object/delete: Add address from request body to tombstone content
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 15:53:58 +03:00
Leonard Lyubich
3c17e871e1 [#158] cmd/neofs-node: Handle objects from tombstone by metabase
Handle tombstone content by metabase instead of object GC queue.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
49470ed613 [#158] metabase: Log errors inside DeleteObjects method implementation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
36d5af812e [#158] metabase: Add logger to DB configuration
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
60e4b5ddff [#158] metabase: Construct DB using options
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Leonard Lyubich
0cd05fdca5 [#158] metabase: Implement operation of deleting a group of objects
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 14:07:20 +03:00
Alex Vanin
0021f1cc5f [#81] cmd/neofs-cli: Use hex encoding of node's public key
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 16:37:48 +03:00
Leonard Lyubich
25398262fc [#81] cmd/neofs-cli: Implement "nodeinfo" sub-cmd of "netmap" cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 16:37:48 +03:00
Leonard Lyubich
5d94594007 [#81] cmd/neofs-cli: Implement "epoch" sub-cmd of "netmap" cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 16:37:48 +03:00
Leonard Lyubich
200fdbd361 [#149] metabase: Do not write virtual objects to the primary index
In the previous implementation of the metabase, it was necessary to write
virtual objects to the primary index to be able to select them. In this
approach, virtual objects can be obtained directly using Head operation.
This has a side effect in handling object operations that do not expect to
receive a virtual object header in a single operation. With recent changes,
it is no longer necessary to have records of virtual objects in the primary
index, so this no longer happens for system integrity.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 14:01:01 +03:00
Leonard Lyubich
5ad013c10b [#149] object/search: Return fixed error if relation not found
Define ErrRelationNotFound error in searchsvc package. Return
ErrRelationNotFound from RelationSearcher.SearchRelation method if search
result is empty.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 14:01:01 +03:00
Leonard Lyubich
2913aa0fd1 [#149] metabase: Add parent ID index
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 14:01:01 +03:00
Alex Vanin
65be09d3db [#155] Update neofs-api-go with refactored pkg/netmap
Refactored pkg/netmap package provides JSON converters for
NodeInfo and PlacementPolicy structures, that has been used
by client applications.

It also updates Node structure itself so it is a part of
grpc <-> v2 <-> pkg conversion chain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 09:55:05 +03:00
Alex Vanin
9f8eb0fd51 [#152] Use ID in SetConfig contract method invocation.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 09:26:00 +03:00
Alex Vanin
7c1776a281 [#152] Add ID field to update state event
NeoFS contract produces event with three arguments:
  - update ID,
  - config key,
  - config value.

Update ID is a unique shared by inner ring holders
byte sequence that is used to update NeoFS runtime
configuration.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-06 09:26:00 +03:00
Alex Vanin
49da96d006 [#146] Set stdout as default output for CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-05 18:37:24 +03:00
Alex Vanin
64caa15556 [#148] Use base name as FileName attribute value
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-05 18:37:06 +03:00
Alex Vanin
54818d5a11 [#144] Support well-known application attributes for object in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-05 09:28:13 +03:00
Alex Vanin
83926647d7 [#144] Support well-known application attributes for container in CLI
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-05 09:28:13 +03:00
Leonard Lyubich
62bd22a379 [#142] metabase: Fix false-positive select in absence of filtered header
Fix a bug in the selection when the object without some filtered header
added to the final result.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
3c39c5a90c [#142] metabase: Fix selection emptying due to deleted object
Fix a bug in the selection when removed object that matches search query
provoked the return of an empty result.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
7a8f322d59 [#142] metabase: Fix false-positive select
Fix a bug in the selection when an object could be added to the result after
a mismatch in the previous filter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
77e80f517f [#142] metabase: Store header value index in a tree leaf
In the previous implementation of the metabase, the unique value of the
header was assigned a bucket, the elements of which were leaves with a
key-address and an empty value. This approach was relatively efficient in
terms of write speed. However, a large number of buckets led to a rapid
increase in the database volume (~4GB for 100K objects with unique
attributes). An approach is presented with storing indexes on the value of
headers in the leaves of the tree, where the keys are the unique values ​​of
the header, and the values ​​are a serialized list of addresses (gob
encoding is temporarily used for serialization).

The new approach gave a good result in saving space (~350MB), however, it
significantly reduced the write speed with an increase in the number of
objects (~ 80x after 100K objects).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
e48f8a189e [#142] metabase: Replace exclusive select with the inclusive one
The previous metabase implementation took an exclusionary approach: filters
narrowed the set of all objects to those that match all filters. An
inclusive approach is presented. In it, when traversing the indexed headers,
the object becomes a candidate for selection. If at least one of the
subsequent filters is not passed, the object ceases to be a candidate. At
the end of the traversal, the remaining candidates are added to the
resulting sample. The borderline case of no filters is handled in a special
way: all stored objects are added to the resulting selection.

Presented inclusive approach showed better performance in most scenarios
(although not all).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
3bbf1d6a60 [#137] metabase: Do not select objects if there is no filterable attribute
In previous metabase implementation the absence of an attribute presented in
the search filter did not exclude the object from the result. Change this
behavior to exclude the object from the result.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
d9a5007c72 [#137] metabase: Add unit test for a nonexistent attribute
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
7704811654 [#137] metabase: Implement and useful test funcs for work with DB
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
fc2038e929 [#137] metabase: Implement Path method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
8125b544b4 [#136] cmd/neofs-node: Use new metabase in app
Remove BoltDB bucket package. Construct meta.DB instance in node app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
c0aa892161 [#136] localstorage: Make local storage to use new metabase
Replace meta Bucket with meta.DB instance in local storage implementation.
Adopt all dependent components to new local storage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
a61f8d44d1 [#135] metabase: Implement benchmarking tests
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
1db01725c9 [#131] metabase: Implement indexing by object properties
Process parent objects in Put method. Headers of parent object are stored as
regular leaf objects in metabase from now. Build indexes for ROOT, LEAF and
CHILDFREE properties.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Alex Vanin
97077294fc [#141] Fix double sidechain GAS emission on asset mint
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-03 16:28:01 +03:00
Alex Vanin
0c49c08609 [#139] Use storage node GAS emission in inner ring app
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-03 16:20:41 +03:00
Alex Vanin
d193f1087c [#139] Emit gas to the storage nodes at emission tick
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-03 16:20:41 +03:00
Leonard Lyubich
d74aa807bf Update go.sum
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 14:14:38 +03:00
Leonard Lyubich
a8e9d15f3f [#85] inner-ring/container: Check container format in process Put
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 14:14:38 +03:00
Leonard Lyubich
766eea4c8c [#85] services/container: Check container format in Put
Call CheckFormat function in container.Put handler for conducting initial
checks of the structure that are not performed by the smart contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 14:14:38 +03:00