dstepanov-yadro/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
726 commits 246 branches 26 tags 23 MiB
Commit graph

726 commits

This branch
This branch
All branches
Author SHA1 Message Date
Leonard Lyubich
33804e024d [#255] services/audit: Complete audit report after the last check 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
4dc09b19f3 [#255] ir: Make audit processor to push tasks to audit task manager 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
580c9c974a [#255] services/audit: Implement task manager 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
0f0be2377b [#255] services/audit: Implement audit executor without checks 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
babfbc18f2 [#255] services/audit: Implement Task structure 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
076f201807 [#255] services/audit: Define Report structure and Reporter interface 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
919f4364f1 [#269] morph/audit: Implement wrapper over Audit contract client 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
07da9d31f2 [#269] morph/client: Implement Audit contract client 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
8dd7c689f2 [#265] innerring: Select storage groups to audit 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
22cffbf529 [#265] innerring: Add client cache 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
3d3d058b05 [#265] innerring: Select containers to audit 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
87e1252065 [#265] innerring: Generate new audit event 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
8c4e033db3 [#80] core/object: Add SG-object format validation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
f7ca4a8dce [#80] cmd/cli: Implement storagegroup commands 
Implement PUT/GET/LIST/DELETE commands of storage group module.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
43ee94a572 [#80] storageroup: Implement function to collect structure from container 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
44466f4a29 [#80] storagegroup: Implement search query generator 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
6245e58d1f [#80] services/object: Implement iterators over split-tree leaves 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
bb85ac24eb [#254] innerring: Increase extra fee for container registration 
Some rough estimation until there will be p2p signature collection
inside the chain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
feec559e47 [#254] Update neo-go to preview4 compatible version 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
c539657567 Update changelog for v0.13.2 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-24 15:34:20 +03:00
Leonard Lyubich
0302dc7490 Pull neofs-api-go v1.21.2 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-24 15:34:20 +03:00
Leonard Lyubich
51b85b0a73 [#278] cli: Support request X-Headers 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-24 15:03:02 +03:00
Leonard Lyubich
e53bf574b5 [#279] container: Use new methods to work with container format 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-24 13:38:20 +03:00
Leonard Lyubich
9680dfbdea Update changelog for v0.13.1 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 18:58:38 +03:00
Leonard Lyubich
b56cfa4318 Pull neofs-api-go release v1.21.1 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 18:58:38 +03:00
Leonard Lyubich
bbe8f34ffd [#253] object/eacl: Process only CID/OID object headers in RANGEHASH/DELETE 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 18:31:20 +03:00
Leonard Lyubich
9e27263df3 [#253] object/put: Fix loss of session token 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 18:31:20 +03:00
Leonard Lyubich
8e661d5c37 [#264] object/get: Prevent receiving payloads through GETRANGE when GET 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 16:42:17 +03:00
Leonard Lyubich
66f9532857 [#264] object/get: Check parent address in linking/last child 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 16:42:17 +03:00
Leonard Lyubich
ec21040542 [#264] transformer: Finalize parent header once 
In previous implementation parent object header finalized twice in size
limiter + formatter. On the one hand, this added redundant action, on the
other hand, it could provoke a difference in the headers of the linking and
the last part. Change formatter to finalize parent header if it does not
container the signature. Change size limiter to reuse parent header after
last child finalization in linking child.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 16:42:17 +03:00
Leonard Lyubich
edef9463d7 [#253] metabase: Add prefix to Graveyard and ToMoveIt bucket names 
In previous implementation DB.Containers method could return an error about
invalid container ID string format. This could happen if some of top-level
buckets had name w/o "_" substring.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-17 12:03:18 +03:00
Leonard Lyubich
df3d30d1ec Update changelog for v0.13.0 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-15 13:48:18 +03:00
Leonard Lyubich
49131f1bc7 [#247] object/eacl: Use address from session token in request validation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-15 13:21:13 +03:00
Leonard Lyubich
dba88c79b4 [#247] object/range: Process only object_id and container_id headers 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-15 13:21:13 +03:00
Leonard Lyubich
8654458b19 [#247] object/eacl: Use object ID from session token context 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-15 13:21:13 +03:00
Leonard Lyubich
168dcbdccd [#247] object/eacl: Process object address from request 
In previous implementation eACL validator didn't take into account container
and object ID fields of request bodies.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-15 13:21:13 +03:00
Alex Vanin
bbe700fa37 [#250] service/policer: Don't shrink node list at unknown error 
Every unknown error must not decrease shortage counter and must not
exclude faulty node from the node list, because this list will be used
later for replication.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-14 21:49:50 +03:00
Alex Vanin
7daf3727ee [#248] neofs-cli/object: Print split header values in Head command 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-14 14:46:59 +03:00
Alex Vanin
e6f2d84736 [#246] blobovnivza: Fix deadlock on concurrent evict and open new blobovnicza 
Deadlock occurs when `getActivate` function opens new blobovnicza and that
invokes evict in LRU cache of open blobovniczas. `getActivate` makes
`activeMtx.Lock()` and then cache evict makes `activeMtx.RLock()` and deadlock
happens.

Fix contains two steps:
- add separate mutex to open blobovniczas (1),
- split single Lock outside of `updateAndGet` (2).

As for the (1) `bbolt.Open()` locks when it tries to open the same file from
two threads. So separate mutex will prevent that.

As for the (2) `updateAndGet` function contains from two parts. At first it
checks if required blobovnicza is ready and it returns it. In this case we can
use the simple RLock. But then there is an option when we should open new
blobovnicza and update map of active blobovniczas.

In this case we call `openBlobovnicza` without activeMtx lock. Cache evict
happens there and it won't cause deadlock.

Then we lock activeMtx to update the map of active blobovniczas. Concurrency can
happen there. However `openBlobovnicza` will not open the same blobovnicza twice,
so we can make one more check if opened blobovnicza was activated while thread was
locked in activeMtx. If so, then return active blobovnicza, else finish activation.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-14 13:37:45 +03:00
Leonard Lyubich
91bea44a1a [#245] blobovnicza: Fix initializing dimensional buckets 
In previous implementation Blobovnicza could incorrectly initialize
dimensional buckets: if SmallSizeLimit = 2 ^ X + Y && Y < 2 ^ X, then
largest dimensional bucket was [2 ^ (X - 1) : 2 ^ X]. This was caused by an
incorrect condition for stopping the iterator along the dimensional
boundaries.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-14 12:48:59 +03:00
Leonard Lyubich
6814140a19 [#240] cmd/cli: Support raw flag in object GET/HEAD/RANGE commands 
Support boolean raw flag. Print split information upon receipt.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-14 12:05:13 +03:00
Leonard Lyubich
fb50362dcc [#243] services/object: Share common parameters across services 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
a01262d8bd [#243] service/object: Fix object chain assembly for tombstone body 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
9265e31e65 [#243] services/object: Fix lost tokens when generating requests 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
09916f21b5 [#243] cmd/cli: Write tombstone address in Object.Delete cmd 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
fe3906c295 [#243] object/delete: Implement new service processing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
510e9ff2ec [#243] core/object: Replace test content of tombstone with an API structure 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
4f5d5c7e45 Write system path to BoltDB on blobovnicza closing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
627012e4fe [#241] object/search: Set container ID parameter of storage engine call 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
611a29f682 [#241] object/search: Refactor service processing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00