In the previous implementation of the metabase, the unique value of the
header was assigned a bucket, the elements of which were leaves with a
key-address and an empty value. This approach was relatively efficient in
terms of write speed. However, a large number of buckets led to a rapid
increase in the database volume (~4GB for 100K objects with unique
attributes). An approach is presented with storing indexes on the value of
headers in the leaves of the tree, where the keys are the unique values of
the header, and the values are a serialized list of addresses (gob
encoding is temporarily used for serialization).
The new approach gave a good result in saving space (~350MB), however, it
significantly reduced the write speed with an increase in the number of
objects (~ 80x after 100K objects).
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
The previous metabase implementation took an exclusionary approach: filters
narrowed the set of all objects to those that match all filters. An
inclusive approach is presented. In it, when traversing the indexed headers,
the object becomes a candidate for selection. If at least one of the
subsequent filters is not passed, the object ceases to be a candidate. At
the end of the traversal, the remaining candidates are added to the
resulting sample. The borderline case of no filters is handled in a special
way: all stored objects are added to the resulting selection.
Presented inclusive approach showed better performance in most scenarios
(although not all).
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
In previous metabase implementation the absence of an attribute presented in
the search filter did not exclude the object from the result. Change this
behavior to exclude the object from the result.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Replace meta Bucket with meta.DB instance in local storage implementation.
Adopt all dependent components to new local storage.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Process parent objects in Put method. Headers of parent object are stored as
regular leaf objects in metabase from now. Build indexes for ROOT, LEAF and
CHILDFREE properties.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Call CheckFormat function in container.Put handler for conducting initial
checks of the structure that are not performed by the smart contract.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
With neo-project/neo-modules#358 nodes support RPC to
return used magic number. Therefore client doesn't need
that configuration value any more.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Neo-go does not use smartcontract.Parameter to return values
anymore, so it's convertes partly removed from neofs-node.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
There is an optimization to send ApprovePeer tx only in
case if `touch` returns false. It returns false if node
is not in the cache or it was flagged to removal.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
This event will be produced by new epoch handler. All
cleanups should be synchronous because smart contract can't
store inner ring votes forever. Therefore voting should be done
in similar time interval for all inner ring nodes.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Cleanup table is a cache for inner ring node to look for netmap
snapshot. It updates access time of bootstrapped nodes and will
be used to send `updateState` txs to clean netmap from unresponsive
nodes.
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
Implement Delete method on DB structure that adds deleted addresses to
tombstone index. Do not attach addresses from tombstone index to Select
result. Return error from Get method if address is presented in tombstone
index.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Implement bolt-based metabase that is going to be used in local object
storage. Implement Put/Get/Select methods.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Make Policer to call AddTask method of Replicator when an insufficient
number of copies of an object is detected in the container.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
