TrueCloudLab/frostfs-node
19
1
Fork 27
Code Issues 91 Pull requests 6 Projects Releases 1 Activity Actions
4859 commits 8 branches 81 tags 52 MiB
Commit graph

4859 commits

This branch
This branch
All branches
Author SHA1 Message Date
Leonard Lyubich
cd2228cd99 [#220] cmd/neofs-node: Support new configuration of storage components 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
e758c246b7 [#220] localstorage: Implement Open/Init/Close methods 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
2963473c08 [#220] localstorage: Implement hrw.Hasher over Shard structure 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
88c1584e6a [#220] shard: Implement Open/Init/Close methods 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
d30454a572 [#220] shard: Use new metabase implementation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
d8d38d3476 [#220] blobstor: Add blobovnicza tree structure to BlobStor 
Add blobovnicza instance to BlobStor structure. Create blobovnicza tree in
BlobStor constructor. Implement Open/Init/Close methods.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
51ab6991d2 [#220] blobstor: Implement method to close blobovnicza tree 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
34a259d5c0 [#220] blobstor: Support zero shallow depth of blobovnicza tree 
There is a need to support single blobovnicza in blobovnicza tree. This can
be achieved with a width of 1, and a depth of 0 or 1. With depth = 1 one
redundant directory is created, inside which there is a blobovnicza. If the
depth is zero, the blobobnivza will be in the root path. Fix negative
capacity in iterateDeepest method with zero depth.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
8bff073061 [#220] blobstor: Fix hard-code in blobovnicza unit test 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
9d4fba923c [#199] Create metabase instance as storage engine component 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
8a293c76ed [#199] Ignore invalid search attributes 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
337e2f6a57 [#199] Add SplitID index in metabase 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
74d44beb99 [#199] Add CleanUp method to remove empty buckets from metabase 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
7fb87aac85 [#199] Support Delete operation in metabase 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
e42fa69735 [#199] Make option to get object from metabase without graveyard check 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
b45d523125 [#199] Split IsSmall check in metabase 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
5bfae833fe [#199] Update blobovniczaID on existed objects in metabase 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
54cd91adff [#199] Simplify tests 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
a8d76f2ebb [#199] Implement ObjectID selection filter in metabase 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
700bd7de01 [#199] Get virtual objects from metabase 
Virtual objects are obtained differently with
relative lookup in parent bucket.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
4e7d49791b [#199] Index parent first in metabase 
With exist check we should index parent first, because
as soon as child will be added to metabase, exist on
parent will return true even if it was not indexed yet.

Also this commit makes one db.Update instead of two for
parent and child.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
f6387a9b94 [#199] Make exist check on all primary buckets in metabase 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
e478c0d024 [#199] Refactor metabase internal structure 
Accoring to MetaBase-Plan-B storage engine specification.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
1c81d507fd [#218] blobstor: Inherit the root path to Blobovnicza from BlobStor 
Place the root of blobovnicza tree in a subdirectory of BlobStor with same
permissions. Abolish WithBlobovniczaRootPath and WithBlobovniczaPersmissions
options.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
4c3d231e82 [#218] blobstor: Implement blobovnicza tree 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
3acbd3011b [#218] blobovnicza: Сreate at least one range bucket 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
1b0d2c67d1 [#218] blobstor: Add blobovnicza ID value to DeleteSmallPrm 
Blobovnicza ID parameter provides the ability to specify particular
blobovnicza to delete object from. In this case only specified blobovnicza
is processed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
a97ab242eb [#218] blobstor: Add config values of blobovnicza tree 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
35ccda13a8 [#216] blobovnicza: Change ID stringer 
Replace hex encoding with slice to string conversion.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
07e998d846 [#216] blobovnicza: Implement main methods and logic 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
52b0c1f19a [#216] blobovnicza: Add configuration values 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
bce16de72a [#216] blobstor: Classify objects by size according to binary format 
In previous implementation objects were classified by size according to
payload size. From now they are classified by the size of their binary
representation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
40b09f9266 [#216] blobovnicza: Define Blobovnicza type 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
a1164b2a5c [#211] cmd/neofs-node: Add SmallSizeLimit config value of BlobStor 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
b9054e2ee0 [#211] blobstor: Add a branch for processing objects by size in Put 
Implement "big or small" property classifier (only the size of the payload
is temporarily considered). Save "big" objects in shallow dir. Save "small"
objects in shallow dir until the moment of implementation of blobovnicza.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
cfc770c3fe [#211] blobstor: Add SmallSizeLimit parameter 
Add numeric parameter of maximum size of the "small" objects.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
3028718691 [#211] blobovnicza: Refactor ID implementation 
Replace UUID implementation of ID with byte slice.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
53b114cf8b [#211] blobstor: Remove global lock 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
59f7cf9873 [#211] blobstor: Implement Exists method 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
1ba556f5e7 [#211] blobstor: Define Iterate operation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
3c0ef270f4 [#211] blobstor: Define Exists operation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
915243c69e [#211] blobstor: Define DeleteSmall operation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
65da2025e5 [#211] blobstor: Rename Delete to DeleteBig 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
b605ea3678 [#211] blobstor: Refactor Put operation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
7afcde659d [#211] blobstor: Define GetRangeSmall operation interface 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
0b130150ae [#211] blobstor: Rename GetRange to GetRangeBig 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
2ba4ebc8c9 [#211] blobstor: Implement no-op GetSmall operation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
2f61d8d63c [#211] localstorage: Define blobovnicza ID 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
6813f40665 [#211] blobstor: Refactor GetBig parameters and result 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
fb6857a1cb [#211] blobstor: Rename Get operation to GetBig 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00