Commit graph

65 commits

Author SHA1 Message Date
Leonard Lyubich
a875d80491 Remove outdated code of metabase and localstore
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-12-11 17:19:37 +03:00
Alex Vanin
ea41dec23f [#160] Store root and phy indexes as fake bucket tree
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Alex Vanin
03fed8ca59 [#160] Update metabase with new root and phy flags processing
Now root and phy (leaf) filters work like flags. They work with
any matcher and any value. So meta-storage sets `true` value for
all root and phy objects and puts them into separate bucket.

We also do not work with inversion anymore, so it either added
to the bucket or not. We don't need to store both options.
This is the reason `selectAll` function is changed a bit. Now
it performs some low-level parsing from primary bucket and root
bucket.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-11-16 10:02:12 +03:00
Leonard Lyubich
fcb35d82cf [#165] Revert "[#142] metabase: Store header value index in a tree leaf"
Revert commit 0faa40e4 to increase the disk space consumed by the
metabase in favor of the speed of index updates.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-09 18:58:58 +03:00
Leonard Lyubich
200fdbd361 [#149] metabase: Do not write virtual objects to the primary index
In the previous implementation of the metabase, it was necessary to write
virtual objects to the primary index to be able to select them. In this
approach, virtual objects can be obtained directly using Head operation.
This has a side effect in handling object operations that do not expect to
receive a virtual object header in a single operation. With recent changes,
it is no longer necessary to have records of virtual objects in the primary
index, so this no longer happens for system integrity.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-06 14:01:01 +03:00
Leonard Lyubich
62bd22a379 [#142] metabase: Fix false-positive select in absence of filtered header
Fix a bug in the selection when the object without some filtered header
added to the final result.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
3c39c5a90c [#142] metabase: Fix selection emptying due to deleted object
Fix a bug in the selection when removed object that matches search query
provoked the return of an empty result.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
7a8f322d59 [#142] metabase: Fix false-positive select
Fix a bug in the selection when an object could be added to the result after
a mismatch in the previous filter.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
77e80f517f [#142] metabase: Store header value index in a tree leaf
In the previous implementation of the metabase, the unique value of the
header was assigned a bucket, the elements of which were leaves with a
key-address and an empty value. This approach was relatively efficient in
terms of write speed. However, a large number of buckets led to a rapid
increase in the database volume (~4GB for 100K objects with unique
attributes). An approach is presented with storing indexes on the value of
headers in the leaves of the tree, where the keys are the unique values ​​of
the header, and the values ​​are a serialized list of addresses (gob
encoding is temporarily used for serialization).

The new approach gave a good result in saving space (~350MB), however, it
significantly reduced the write speed with an increase in the number of
objects (~ 80x after 100K objects).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
e48f8a189e [#142] metabase: Replace exclusive select with the inclusive one
The previous metabase implementation took an exclusionary approach: filters
narrowed the set of all objects to those that match all filters. An
inclusive approach is presented. In it, when traversing the indexed headers,
the object becomes a candidate for selection. If at least one of the
subsequent filters is not passed, the object ceases to be a candidate. At
the end of the traversal, the remaining candidates are added to the
resulting sample. The borderline case of no filters is handled in a special
way: all stored objects are added to the resulting selection.

Presented inclusive approach showed better performance in most scenarios
(although not all).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
3bbf1d6a60 [#137] metabase: Do not select objects if there is no filterable attribute
In previous metabase implementation the absence of an attribute presented in
the search filter did not exclude the object from the result. Change this
behavior to exclude the object from the result.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
1db01725c9 [#131] metabase: Implement indexing by object properties
Process parent objects in Put method. Headers of parent object are stored as
regular leaf objects in metabase from now. Build indexes for ROOT, LEAF and
CHILDFREE properties.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-11-03 18:42:32 +03:00
Leonard Lyubich
20ed7c0d61 [#128] metabase: Implement Delete method
Implement Delete method on DB structure that adds deleted addresses to
tombstone index. Do not attach addresses from tombstone index to Select
result. Return error from Get method if address is presented in tombstone
index.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-29 17:34:41 +03:00
Leonard Lyubich
5f78a18a4f [#128] metabase: Do not create leaves in indexed header bucket
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-29 17:34:41 +03:00
Leonard Lyubich
85aacbbb10 [#128] localstorage: Implement primary object metabase
Implement bolt-based metabase that is going to be used in local object
storage. Implement Put/Get/Select methods.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-29 17:34:41 +03:00