ironbee/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
762 commits 15 branches 26 tags 17 MiB
Commit graph

605 commits

Author SHA1 Message Date
Alex Vanin
6977adec66 [#273] service/audit: Ignore coverage of very small objects 
PDP audit check is not quite working with very small objects, so
we try to build coverage with bigger objects.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
afb83c610c [#273] service/audit: Add cache of head responses in PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
9212864f42 [#258] services/audit: Implement PoP check 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
a5320408a5 [#271] service/audit: Implement PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
5f65ec0265 [#271] audit/report: Add PoR related setters 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
85773c419b [#271] service/object: Fill all fields of new short header structure 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
af7ae0ec4c [#271] acl: Add permission to get objects for inner ring 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
782bcadd92 [#271] innerring: Implement GetSG and Head of audit communicator 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
dd5e67ff0c [#271] service/audit: Add netmap structure to audit task 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
64b022a5ed [#271] placement: Add FlattenNodes function 
This function converts 2-dimension array of container nodes
into single dimension array. Useful when we need to iterate
over whole container.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
58033a3adc [#271] placement: Add BuildObjectPlacement function 
This function allows to shuffle prepared container nodes
according to HRW weights from network map.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
a8c26378b1 [#271] pkg/network: Add multiaddr to ipaddr converter 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
2ebcbe70fb [#255] ir/audit: Make task manager to return number of skipped tasks 
Add numeric return from TaskManager.Reset method that shows the number of
canceled tasks. This values will be used for assessment of the progress of
the audit.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
03e3afb0e8 [#255] services/audit: Define interface of container communicator 
Define interface of the container communicator which methods are going to be
used in audit checks. Make innerring Server to implement this interface.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
754cd8b579 [#255] ir: Replace Server.WriteReport method to another source file 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
54523d2949 [#255] ir: Replace audit client creation to invoke package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
76d4e53ea0 [#255] services/audit: Skip all tasks from previous epoch in audit processor 
Implement Reset method on audit task manager that cleans task queue.
Extended TaskManager interface with Reset method on IR side. Call Reset
method in audit processor before new audit start.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
33804e024d [#255] services/audit: Complete audit report after the last check 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
4dc09b19f3 [#255] ir: Make audit processor to push tasks to audit task manager 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
580c9c974a [#255] services/audit: Implement task manager 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
0f0be2377b [#255] services/audit: Implement audit executor without checks 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
babfbc18f2 [#255] services/audit: Implement Task structure 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
076f201807 [#255] services/audit: Define Report structure and Reporter interface 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
919f4364f1 [#269] morph/audit: Implement wrapper over Audit contract client 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
07da9d31f2 [#269] morph/client: Implement Audit contract client 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
8dd7c689f2 [#265] innerring: Select storage groups to audit 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
22cffbf529 [#265] innerring: Add client cache 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
3d3d058b05 [#265] innerring: Select containers to audit 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
87e1252065 [#265] innerring: Generate new audit event 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
8c4e033db3 [#80] core/object: Add SG-object format validation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
43ee94a572 [#80] storageroup: Implement function to collect structure from container 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
44466f4a29 [#80] storagegroup: Implement search query generator 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
6245e58d1f [#80] services/object: Implement iterators over split-tree leaves 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
bb85ac24eb [#254] innerring: Increase extra fee for container registration 
Some rough estimation until there will be p2p signature collection
inside the chain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
feec559e47 [#254] Update neo-go to preview4 compatible version 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
e53bf574b5 [#279] container: Use new methods to work with container format 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-24 13:38:20 +03:00
Leonard Lyubich
bbe8f34ffd [#253] object/eacl: Process only CID/OID object headers in RANGEHASH/DELETE 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 18:31:20 +03:00
Leonard Lyubich
9e27263df3 [#253] object/put: Fix loss of session token 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 18:31:20 +03:00
Leonard Lyubich
8e661d5c37 [#264] object/get: Prevent receiving payloads through GETRANGE when GET 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 16:42:17 +03:00
Leonard Lyubich
66f9532857 [#264] object/get: Check parent address in linking/last child 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 16:42:17 +03:00
Leonard Lyubich
ec21040542 [#264] transformer: Finalize parent header once 
In previous implementation parent object header finalized twice in size
limiter + formatter. On the one hand, this added redundant action, on the
other hand, it could provoke a difference in the headers of the linking and
the last part. Change formatter to finalize parent header if it does not
container the signature. Change size limiter to reuse parent header after
last child finalization in linking child.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 16:42:17 +03:00
Leonard Lyubich
edef9463d7 [#253] metabase: Add prefix to Graveyard and ToMoveIt bucket names 
In previous implementation DB.Containers method could return an error about
invalid container ID string format. This could happen if some of top-level
buckets had name w/o "_" substring.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-17 12:03:18 +03:00
Leonard Lyubich
49131f1bc7 [#247] object/eacl: Use address from session token in request validation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-15 13:21:13 +03:00
Leonard Lyubich
dba88c79b4 [#247] object/range: Process only object_id and container_id headers 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-15 13:21:13 +03:00
Leonard Lyubich
8654458b19 [#247] object/eacl: Use object ID from session token context 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-15 13:21:13 +03:00
Leonard Lyubich
168dcbdccd [#247] object/eacl: Process object address from request 
In previous implementation eACL validator didn't take into account container
and object ID fields of request bodies.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-15 13:21:13 +03:00
Alex Vanin
bbe700fa37 [#250] service/policer: Don't shrink node list at unknown error 
Every unknown error must not decrease shortage counter and must not
exclude faulty node from the node list, because this list will be used
later for replication.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-14 21:49:50 +03:00
Alex Vanin
e6f2d84736 [#246] blobovnivza: Fix deadlock on concurrent evict and open new blobovnicza 
Deadlock occurs when `getActivate` function opens new blobovnicza and that
invokes evict in LRU cache of open blobovniczas. `getActivate` makes
`activeMtx.Lock()` and then cache evict makes `activeMtx.RLock()` and deadlock
happens.

Fix contains two steps:
- add separate mutex to open blobovniczas (1),
- split single Lock outside of `updateAndGet` (2).

As for the (1) `bbolt.Open()` locks when it tries to open the same file from
two threads. So separate mutex will prevent that.

As for the (2) `updateAndGet` function contains from two parts. At first it
checks if required blobovnicza is ready and it returns it. In this case we can
use the simple RLock. But then there is an option when we should open new
blobovnicza and update map of active blobovniczas.

In this case we call `openBlobovnicza` without activeMtx lock. Cache evict
happens there and it won't cause deadlock.

Then we lock activeMtx to update the map of active blobovniczas. Concurrency can
happen there. However `openBlobovnicza` will not open the same blobovnicza twice,
so we can make one more check if opened blobovnicza was activated while thread was
locked in activeMtx. If so, then return active blobovnicza, else finish activation.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-14 13:37:45 +03:00
Leonard Lyubich
91bea44a1a [#245] blobovnicza: Fix initializing dimensional buckets 
In previous implementation Blobovnicza could incorrectly initialize
dimensional buckets: if SmallSizeLimit = 2 ^ X + Y && Y < 2 ^ X, then
largest dimensional bucket was [2 ^ (X - 1) : 2 ^ X]. This was caused by an
incorrect condition for stopping the iterator along the dimensional
boundaries.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-14 12:48:59 +03:00
Leonard Lyubich
fb50362dcc [#243] services/object: Share common parameters across services 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
a01262d8bd [#243] service/object: Fix object chain assembly for tombstone body 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
9265e31e65 [#243] services/object: Fix lost tokens when generating requests 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
fe3906c295 [#243] object/delete: Implement new service processing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
510e9ff2ec [#243] core/object: Replace test content of tombstone with an API structure 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
4f5d5c7e45 Write system path to BoltDB on blobovnicza closing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
627012e4fe [#241] object/search: Set container ID parameter of storage engine call 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
611a29f682 [#241] object/search: Refactor service processing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
3bfb18386b object/get: Replace code with function call 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
63bc3aab82 [#242] engine: Do not iterate over shards if cid is not set 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
9fb7190358 [#242] service/object: Support ContainerID field in local search 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
511b5f782a [#242] engine: Support ContainerID argument in shard selects 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
f585772746 [#242] shard: Support ContainerID argument in metabase selects 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
b97f818323 [#242] metabase: Make ContainerID a mandatory parameter for Select 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
0e1f05ff45 [#239] object/head: Implement new service processing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
9dd83bdf0d [#235] blobstor: Return object.ErrRangeOutOfBounds from shallow dir 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
41b9fa5b45 [#235] engine: Return OutOfBounds error 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
173d34a8a2 [#235] object/get: Set common operation parameters 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
7e56427534 [#235] object/getrangehash: Implement new service processing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
0fc5ea674c [#235] object/getrange: Fix incorrect processing of range from last child 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
397d912e19 [#235] object/head: Support raw flag in service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
1d23483828 [#235] services/object: Implement new GetRange algorithm 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
91d8e0a4de [#238] engine: Support raw flag in Head method 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
da42ee3a26 [#238] engine: Stop shard iteration at split info error in Head 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
69dee6ec6d [#238] shard: Support raw flag in Head method 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
bf7a390304 [#238] metabase: Support raw flag in Get method 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
c267a08886 [#236] engine: Stop shard iteration at split info error in GetRange 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
590745204c [#237] metabase: Structure parameters and results of all operations 
All parameters and resulting values of all metabase operations are
structured in new types. The most popular scenarios for using operations are
moved to auxiliary functions.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
a875d80491 Remove outdated code of metabase and localstore 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
869d9e571c [#233] object/head: Remove getting the right child header 
With the update of the local storage engine, the headers of virtual objects
are directly given. In this regard, the step with obtaining the the right
child header is removed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
f24daa10ff [#233] services/object: Implement new Get algorithm 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
26f03c6301 [#232] Fix SplitInfo update in existed virtual object 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
e0350efe00 [#231] services/policer: Use engine.List method 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
ee1e3931a5 [#231] engine: Add List method 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
a7087faa1a [#231] shard: Add List method to return all available objects 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
a1cb48f800 [#231] metabase: Add container lister 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
3a37eda410 [#230] engine: Stop on first SplitInfoError in Get method 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
a488d202e1 [#230] engine: Rename WithAddress to WithAddresses for delete operations 
Delete works with multiple addresses at once so it is convenient to
show it in parameter setter function.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
7c97bd1360 [#230] shard: Add test for getting SplitInfo 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
bf9e938a3b [#230] metabase: Return SplitInfoError in Exist method 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
e2de95e3f6 [#230] metabase: Index split info structures 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
eaae5a5dd7 [#229] blobovnicza: Store objects in a binary format 
In previous implementation Blobovnicza's stored objects in protocol format
which did not allow working with externally compressed objects. To achieve
this goal, operations Get and Put no longer work with the structure of the
object, but only with abstract binary data. Operation GetRange has become
incorrect in its original purpose to receive the payload range. In this
regard, BlobStor receives the payload range of the object through Get
operation. In the future either Blobovnicza will learn to compress objects
by itself, or the GetRange operation will be eliminated.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
3d77fdb347 [#222] Set write cache specific config on application level 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
af2719689d [#222] Do not compress data in blobovniczas 
Blobovnicza returns object, so we can't put compressed
data there. Compressed data won't be deserialized correctly.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
6bec50963b [#222] Add tests for shards 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
ce92df3543 [#222] Return nil in metabase.IsSmall if object is big 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
862f9a3f1a [#222] Put small objects in blobovnicza in blobStor 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
351e4b4592 [#222] Support Inhume and Delete in object service 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
21708d5408 [#222] Add Exists method in storage engine 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
f470022594 [#222] Update Select method in storage engine 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
25486194b9 [#222] Update Head method in storage engine 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00