dkirillov/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
1803 commits 15 branches 26 tags 16 MiB
Commit graph

433 commits

Author SHA1 Message Date
Leonard Lyubich
c75a828adf [#311] control: Fix loss of parent attribute keys in netmap converter 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
6c9dab481c [#311] control: Cover StableMarshal methods with unit tests 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
5f22e7fac7 Add newline at end of Control service proto file 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
36d8e2d3b7 [#311] control: Implement JSON marshaler on Netmap message 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
dde24bc9e5 [#311] control: Rewrite methods for binary encoding and signatures 
Rewrite marshalers and sizers using proto library from API repo.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
e9d4b2d25b [#311] Recompile protobuf files 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
3e882d963f [#311] services/control: Implement NetmapSnapshot method 
Implement methods required for signatures. Receive network map from netmap
storage, convert it to Control service message and return in response.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
28777911fb [#311] services/control: Define NetmapSnapshot rpc 
Add NetmapSnapshot rpc to ControlService protobuf definition. Recompile
proto files. Add required method to server structure.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
88023f3655 [#311] services/control: Define Netmap structure 
Define NodeInfo protobuf type. Define Netmap protobuf type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
f311be3ab6 [#234] Fix ineffassign linter remark 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
0a0ee89665 [#306] control: Rename WithAllowedKeys function to WithAuthorizedKeys 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
abd9952e46 [#306] Rename Private service to Control service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
44a0fb5a69 [#306] private: Define and use HealthChecker interface to get the status 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
c1b8a4815f [#306] private: Implement server of gRPC private node service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
e75ddb0549 [#306] private: Implement setters on generated proto messages 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
85ec633938 [#306] Define and compile proto files for private node service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
1e170c3812 [#234] services/object: Support netmap epoch and lookup dead in read ops 
Support processing of NetmapEpoch and NetmapLookupDepth X-headers when
processing object read operations. Placement for operations
Get/Head/GetRange/GetRangeHash/Search is built for the epoch specified in
NetmapEpoch X-header (by default latest). Also the specified operations are
processed until success is achieved for network maps from the past up to
NetmapLookupDepth value. Behavior for default values (zero or missing) left
unchanged.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-12 19:19:11 +03:00
Leonard Lyubich
2f4d90025f [#234] core/netmap: Extend Source interface 
Add GetNetMapByEpoch method. Add Epoch method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-12 19:19:11 +03:00
Alex Vanin
e88c1f750d [#302] Fix golint else/return linter errors 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2021-01-12 18:47:02 +03:00
Alex Vanin
74a68af5c2 [#302] Fix golint underscore linter errors 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2021-01-12 18:47:02 +03:00
Alex Vanin
04c0c1b8f5 [#302] Fix errcheck linter errors 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2021-01-12 18:47:02 +03:00
Alex Vanin
5e231c515a [#302] Fix whitespace linter errors 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2021-01-12 18:47:02 +03:00
Leonard Lyubich
3a7f6701d8 [#290] object/rangehash: Apply salt to data before hashing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-11 18:40:59 +03:00
Leonard Lyubich
0e2c761481 [#290] object/rangehash: Add salt to RangeHashPrm structure 
Add binary salt field to RangeHashPrm struct. Implement field setter. Set
salt from the request in v2 service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-11 18:40:59 +03:00
Leonard Lyubich
9a86fff7e0 [#291] Remove some unused code from repository 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-30 10:27:53 +03:00
Leonard Lyubich
a51211eda7 [#291] Remove unused code from pkg/services/object/head 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-30 10:27:53 +03:00
Leonard Lyubich
3260e9263e [#291] Remove unused pkg/services/object/rangehash packages 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-30 10:27:53 +03:00
Leonard Lyubich
d299d94049 [#291] Remove unused pkg/services/object/search/query packages 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-30 10:27:53 +03:00
Leonard Lyubich
1dd1762163 [#285] object/put: Fix the loss of X-headers when forwarding objects 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-29 12:22:05 +03:00
Leonard Lyubich
2897e83fb2 [#285] object/eacl: Validate X-headers from the requests, not the responses 
In previous implementation of eACL service v2 the response X-headers were
validated at the stage of re-checking eACL. This provoked a mismatch of
records in the eACL table with requests. Fix this behavior by checking the
headers from the request, not the response.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-29 12:22:05 +03:00
Leonard Lyubich
c69f867af1 [#285] services/object: Add X-Headers to client call options 
Forward request X-headers to client calls during internal processing of
Object operations on the node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-29 12:22:05 +03:00
Alex Vanin
7174abcc7c [#281] service/audit: Fix typos 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
e5108cf135 [#281] service/audit: Run each SG check in separate routing at PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
0d7832f5e9 [#281] service/audit: Add separate pool for SG checks in PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
8e72abaab7 [#281] service/audit: Make report structure threadsafe 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
44d9f95137 [#276] service/audit: Count Head requests and retries in PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
9273a89f61 [#276] service/audit: Use random nodes for object.Head in PoR 
Since PoR audit check uses object.Head with large TTL values to
make real proof of retrievability, we may use random remote
nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
10b548275a [#276] innerring: Use Head with TTL in PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
3ec342b2c3 [#259] services/audit: Implement PDP check 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
6977adec66 [#273] service/audit: Ignore coverage of very small objects 
PDP audit check is not quite working with very small objects, so
we try to build coverage with bigger objects.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
afb83c610c [#273] service/audit: Add cache of head responses in PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
9212864f42 [#258] services/audit: Implement PoP check 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
a5320408a5 [#271] service/audit: Implement PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
5f65ec0265 [#271] audit/report: Add PoR related setters 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
85773c419b [#271] service/object: Fill all fields of new short header structure 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
af7ae0ec4c [#271] acl: Add permission to get objects for inner ring 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
dd5e67ff0c [#271] service/audit: Add netmap structure to audit task 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
64b022a5ed [#271] placement: Add FlattenNodes function 
This function converts 2-dimension array of container nodes
into single dimension array. Useful when we need to iterate
over whole container.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
58033a3adc [#271] placement: Add BuildObjectPlacement function 
This function allows to shuffle prepared container nodes
according to HRW weights from network map.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
2ebcbe70fb [#255] ir/audit: Make task manager to return number of skipped tasks 
Add numeric return from TaskManager.Reset method that shows the number of
canceled tasks. This values will be used for assessment of the progress of
the audit.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
03e3afb0e8 [#255] services/audit: Define interface of container communicator 
Define interface of the container communicator which methods are going to be
used in audit checks. Make innerring Server to implement this interface.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
76d4e53ea0 [#255] services/audit: Skip all tasks from previous epoch in audit processor 
Implement Reset method on audit task manager that cleans task queue.
Extended TaskManager interface with Reset method on IR side. Call Reset
method in audit processor before new audit start.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
33804e024d [#255] services/audit: Complete audit report after the last check 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
580c9c974a [#255] services/audit: Implement task manager 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
0f0be2377b [#255] services/audit: Implement audit executor without checks 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
babfbc18f2 [#255] services/audit: Implement Task structure 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
076f201807 [#255] services/audit: Define Report structure and Reporter interface 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
43ee94a572 [#80] storageroup: Implement function to collect structure from container 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
44466f4a29 [#80] storagegroup: Implement search query generator 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
6245e58d1f [#80] services/object: Implement iterators over split-tree leaves 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
e53bf574b5 [#279] container: Use new methods to work with container format 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-24 13:38:20 +03:00
Leonard Lyubich
bbe8f34ffd [#253] object/eacl: Process only CID/OID object headers in RANGEHASH/DELETE 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 18:31:20 +03:00
Leonard Lyubich
9e27263df3 [#253] object/put: Fix loss of session token 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 18:31:20 +03:00
Leonard Lyubich
8e661d5c37 [#264] object/get: Prevent receiving payloads through GETRANGE when GET 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 16:42:17 +03:00
Leonard Lyubich
66f9532857 [#264] object/get: Check parent address in linking/last child 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 16:42:17 +03:00
Leonard Lyubich
ec21040542 [#264] transformer: Finalize parent header once 
In previous implementation parent object header finalized twice in size
limiter + formatter. On the one hand, this added redundant action, on the
other hand, it could provoke a difference in the headers of the linking and
the last part. Change formatter to finalize parent header if it does not
container the signature. Change size limiter to reuse parent header after
last child finalization in linking child.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-18 16:42:17 +03:00
Leonard Lyubich
49131f1bc7 [#247] object/eacl: Use address from session token in request validation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-15 13:21:13 +03:00
Leonard Lyubich
dba88c79b4 [#247] object/range: Process only object_id and container_id headers 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-15 13:21:13 +03:00
Leonard Lyubich
8654458b19 [#247] object/eacl: Use object ID from session token context 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-15 13:21:13 +03:00
Leonard Lyubich
168dcbdccd [#247] object/eacl: Process object address from request 
In previous implementation eACL validator didn't take into account container
and object ID fields of request bodies.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-15 13:21:13 +03:00
Alex Vanin
bbe700fa37 [#250] service/policer: Don't shrink node list at unknown error 
Every unknown error must not decrease shortage counter and must not
exclude faulty node from the node list, because this list will be used
later for replication.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-14 21:49:50 +03:00
Leonard Lyubich
fb50362dcc [#243] services/object: Share common parameters across services 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
a01262d8bd [#243] service/object: Fix object chain assembly for tombstone body 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
9265e31e65 [#243] services/object: Fix lost tokens when generating requests 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
fe3906c295 [#243] object/delete: Implement new service processing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
510e9ff2ec [#243] core/object: Replace test content of tombstone with an API structure 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
627012e4fe [#241] object/search: Set container ID parameter of storage engine call 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
611a29f682 [#241] object/search: Refactor service processing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
3bfb18386b object/get: Replace code with function call 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
9fb7190358 [#242] service/object: Support ContainerID field in local search 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
0e1f05ff45 [#239] object/head: Implement new service processing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
173d34a8a2 [#235] object/get: Set common operation parameters 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
7e56427534 [#235] object/getrangehash: Implement new service processing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
0fc5ea674c [#235] object/getrange: Fix incorrect processing of range from last child 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
397d912e19 [#235] object/head: Support raw flag in service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
1d23483828 [#235] services/object: Implement new GetRange algorithm 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
869d9e571c [#233] object/head: Remove getting the right child header 
With the update of the local storage engine, the headers of virtual objects
are directly given. In this regard, the step with obtaining the the right
child header is removed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
f24daa10ff [#233] services/object: Implement new Get algorithm 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
e0350efe00 [#231] services/policer: Use engine.List method 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
351e4b4592 [#222] Support Inhume and Delete in object service 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
953387a1e5 [#186] object/acl: Use new storage engine for work 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
046206f670 [#186] object/head: Use new storage engine for work 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
2be8f154a0 [#186] object/put: Use new storage engine for work 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
14442a0801 [#186] object/range: Use new storage engine for work 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
9b32f131c0 [#186] object/rangehash: Use new storage engine for work 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
1bce2a4e55 [#186] object/search: Use new storage engine for work 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
aa9eb2eaf2 [#186] policer: Use new storage engine for work 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
1b867e53d7 [#186] replicator: Use new storage engine for work 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
6cddc91cbf [#227] transformer: Inherit parent header in linking object 
In previous implementation linking object inherited only ID of the parent
object. From now it inherits full header like last child.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-02 14:53:04 +03:00
Leonard Lyubich
b8e8e1e80d [#227] transformer: Do not inherit attribute in generated objects 
In previous implementation child objects inherited parent attributes after
split-transformation, which was redundant. From now attributes are not
inherited.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-02 14:53:04 +03:00
Leonard Lyubich
5470d94416 [#223] placement: Fix local and single-success placement traversal 
In previous implementation placement traverser processed incorrectly with
local placement build. Also entity incorrectly traversed the placement
vectors for fixed number read operations until success. The erroneous
behavior was due to the use of a vector number of successes instead of
a scalar number in these scenarios.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-01 19:36:46 +03:00
Leonard Lyubich
d94a0eb25e [#221] eacl: Get rid of deprecated methods usage 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-01 11:08:47 +03:00
Alex Vanin
f96630581a [#208] Set unique splitID in children 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-30 10:44:15 +03:00
Alex Vanin
6280d075b9 [#208] Remove childfree search attribute 
With updated specification of object related operation
we don't have this search attribute any more and we
should not use functions related to this attribute.

This commit breaks object service logic, however it will
be fixed later.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-30 10:44:15 +03:00
Leonard Lyubich
706bdf736e [#209] eacl: Support object ID filter in eACL mechanism 
Add object ID header to the list of processing object headers in eACL
validation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-25 11:59:15 +03:00
Leonard Lyubich
fddc50fd85 [#203] Replace ErrEACLNotFound to core library 
ErrEACLNotFound error was defined in implementation package. EACL validator
checked this error after the call of eACL storage interface method. Replace
ErrEACLNotFound to core container library. in order to: on the one hand not
use an implementation error, on the other hand, to be able to reuse a
generic type error (404).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 20:19:20 +03:00
Leonard Lyubich
9148980bd0 [#193] services/object: Support client options in all Object services 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 15:34:30 +03:00
Alex Vanin
cf1ea983e5 [#203] Do not fail eACL check on EACLNotFound error 
Now morph library returns error if there is not eACL in
sidechain storage. However in this case eACL check should
be passed since it is the same as having empty eACL table.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-24 15:32:26 +03:00
Leonard Lyubich
46dab77705 [#195] services/object: Write debug log messages on worker pool errors 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Leonard Lyubich
fa6e4a3ca4 [#195] services/object: Write debug log messages on internal service errors 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Leonard Lyubich
6a5c37d592 [#195] object/search: Add option to set logger 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Leonard Lyubich
cb46e4b154 [#195] object/rangehash: Add option to set logger 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Leonard Lyubich
af6484e3b1 [#195] object/range: Add option to set logger 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Leonard Lyubich
cf2dc37a42 [#195] object/put: Add option to set logger 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Leonard Lyubich
9fbfc0b5e4 [#195] object/head: Add option to set logger 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Leonard Lyubich
e1e5a590e9 [#195] object/delete: Add option to set logger 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Alex Vanin
dbf6c9efef [#190] Use request sender owner in sticky bit check 
Sticky bit checks if object owner and request owner are the
same. Container owner should not used in this check.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-19 19:07:16 +03:00
Alex Vanin
2148e282ec [#190] Rename owner to cnrOwner in object request info 
`owner` field may be misused as request sender owner, however
it is a owner of a container for that request. New naming
should be clear.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-19 19:07:16 +03:00
Alex Vanin
f0537b35c1 [#190] Add isOwnerFromKey helper function in ACL 
This function takes public key and returns true if
owner id was produced by this key.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-19 19:07:16 +03:00
Alex Vanin
a14bb6292b [#182] Reuse search filter in policer 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-19 17:59:46 +03:00
Alex Vanin
2e605b2435 [#182] Limit policer object filter to physical stored objects 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-19 17:59:46 +03:00
Alex Vanin
cf85fa9fab [#180] Return isInnerRing flag in request classifier 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-19 15:17:18 +03:00
Alex Vanin
6f841e319d [#180] Make separate basic ACL check for inner ring requests 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-19 15:17:18 +03:00
Alex Vanin
e8fe07edd0 [#184] Use SDK client cache in object.Rangehash 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-18 18:18:07 +03:00
Alex Vanin
f85e88c4f8 [#184] Use SDK client cache in object.Range 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-18 18:18:07 +03:00
Alex Vanin
d485a5967d [#184] Use SDK client cache in object.Search 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-18 18:18:07 +03:00
Alex Vanin
7ba95dd5fc [#184] Use SDK client cache in object.Put 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-18 18:18:07 +03:00
Alex Vanin
e9a6365333 [#184] Use SDK client cache in object.Head 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-18 18:18:07 +03:00
Leonard Lyubich
1caf15463e [#174] Update to neofs-api-go v1.20.0 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-17 11:56:00 +03:00
Leonard Lyubich
58fcb35fb0 [#174] Use Marshal(JSON)/Unmarshal(JSON) methods for encoding/decoding 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-17 11:56:00 +03:00
Leonard Lyubich
3de8febe57 [#174] Update to latest neofs-api-go changes 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-17 11:56:00 +03:00
Alex Vanin
32219bb294 [#160] Remove query match function 
This function duplicates query processing that
is done in meta-storage now.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-16 10:02:12 +03:00
Leonard Lyubich
3c42f5b452 [#161] object/head: Inherit common parameters in HeadRelation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-09 17:19:34 +03:00
Leonard Lyubich
d38633e047 [#161] object/delete: Add address from request body to tombstone content 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-09 15:53:58 +03:00
Leonard Lyubich
5ad013c10b [#149] object/search: Return fixed error if relation not found 
Define ErrRelationNotFound error in searchsvc package. Return
ErrRelationNotFound from RelationSearcher.SearchRelation method if search
result is empty.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-06 14:01:01 +03:00
Alex Vanin
65be09d3db [#155] Update neofs-api-go with refactored pkg/netmap 
Refactored pkg/netmap package provides JSON converters for
NodeInfo and PlacementPolicy structures, that has been used
by client applications.

It also updates Node structure itself so it is a part of
grpc <-> v2 <-> pkg conversion chain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-06 09:55:05 +03:00
Leonard Lyubich
c0aa892161 [#136] localstorage: Make local storage to use new metabase 
Replace meta Bucket with meta.DB instance in local storage implementation.
Adopt all dependent components to new local storage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-03 18:42:32 +03:00
Leonard Lyubich
766eea4c8c [#85] services/container: Check container format in Put 
Call CheckFormat function in container.Put handler for conducting initial
checks of the structure that are not performed by the smart contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-03 14:14:38 +03:00
Leonard Lyubich
b48a4ede02 [#125] services/eacl: Use latest object header keys 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-29 19:25:54 +03:00
Leonard Lyubich
8d931b81a6 [#125] object/search: Use latest search filter keys 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-29 19:25:54 +03:00
Leonard Lyubich
f34ad9e730 [#125] services/eacl: Fix undefined method usage 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-29 19:25:54 +03:00
Alex Vanin
d08c1c76c1 [#122] Reduce precision from balance contract to Fixed8. 
Fixed8 won't overflow int64 for values less than 92 billion
that is suitable for GAS.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-29 10:14:59 +03:00
Leonard Lyubich
f66c7958e7 [#109] services/policer: Assign tasks to Replicator 
Make Policer to call AddTask method of Replicator when an insufficient
number of copies of an object is detected in the container.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 15:23:22 +03:00
Leonard Lyubich
2d46baa4a5 [#109] services: Implement Replicator service 
Implement Replicator service that performs background work to replicate
local object to remote nodes in the container. Replicator is going to be
used by Policer.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 15:23:22 +03:00
Leonard Lyubich
53efa18e14 [#109] object/put: Implement remote object sender 
Define RemoteSender structure with PutObject method that puts object to a
remote node locally.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 15:23:22 +03:00
Leonard Lyubich
968033deed [#40] object/put: Assign zero return of MaxObjectSize invalid 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 14:03:25 +03:00
Leonard Lyubich
7fdb14cf8a [#83] services/response: Set epoch number from network state 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Leonard Lyubich
19f9c7eacb [#83] services: Remove setting of meta header from executing services 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Leonard Lyubich
0341773318 [#83] services: Implement response sub-service for each service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Leonard Lyubich
6bede7d836 [#83] services/util: Implement response service 
Create response package. Implement response Service that sets values of
response meta header.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Leonard Lyubich
1cc7983c4e [#83] services/util: Add meta header methods to ResponseMessage 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Leonard Lyubich
71a06f9e01 [#83] services/util: Define type of response message interface 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Alex Vanin
7464254680 [#106] Put simplest bearer token check first 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-22 18:02:11 +03:00
Alex Vanin
23ec33e821 [#106] Check bearer token lifetime 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-22 18:02:11 +03:00
Alex Vanin
bb455af05f [#106] Ignore bearer token if basic ACL restrict it 
There is a bit to allow or deny bearer token check for
each object service method. If this bit is not set then
ignore bearer token and use extended ACL table from
sidechain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-22 18:02:11 +03:00
Alex Vanin
89cd2ad463 [#106] Process bearer token in ACL service 
If bearer token is presented in the request then check
if it is a valid one and then use it to process extended
ACL checks.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-22 18:02:11 +03:00
Alex Vanin
094248690b [#115] Make ACL classifier errors transparent for client 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-22 11:55:28 +03:00
Alex Vanin
ca552f53c6 [#115] Check session token validity 
Malicious user can stole public session key and use
it by sending request from it's own scope. To prevent
this each session token is signed and signature private
key must be corresponded with owner id in token. Therefore
malicious node cannot impersonate request without private
key to sign token.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-22 11:55:28 +03:00
Leonard Lyubich
16a5107ef1 [#60] object/put: Provide network State interface to formatter 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-22 11:54:08 +03:00
Leonard Lyubich
b627814dd8 [#60] object/transformer: Set creation epoch number in new objects 
Set value of CreationEpoch object field to the value from network State.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-22 11:54:08 +03:00
Leonard Lyubich
4a56f82571 [#60] object/transformer: Group parameters of NewFormatTarget func 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-22 11:54:08 +03:00
Leonard Lyubich
2541ed4b8f [#88] object/eacl: Use String() methods to calculate ID values 
Replace hex encoding of IDs with String() call (base58) in eACL processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 18:53:04 +03:00
Leonard Lyubich
5318abcf38 [#88] object/search: Use String() methods to calculate ID values 
Replace hex encoding of IDs with String() call (base58) in search query
processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 18:53:04 +03:00
Leonard Lyubich
0dab4b7581 [#108] services: Implement Policer service 
Implement Policer service that performs background work to check compliance
with the placement policy for local objects in the container. In the initial
implementation, the selection of the working queue of objects is
simplified, and there is no transfer of the result to the replicator.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 14:42:51 +03:00
Leonard Lyubich
f6e56aa956 [#108] placement: Implement Builder from netmap source 
Implement placement.Builder interface on netmap.Source wrapper.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 14:42:51 +03:00
Leonard Lyubich
5017ff0e4a [#108] object/head: Export remote header retrieval utility 
Export remote head functionality in headsvc package. Refactor head service
to use RemoteHeader.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 14:42:51 +03:00
Leonard Lyubich
5ad0df7794 [#108] object/head: Return 404 error if header was not found 
Define ErrNotFound error in headsvc package. Return ErrNotFound from Head
method if the header was not found in the container.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 14:42:51 +03:00
Alex Vanin
ae0dd9e051 [#106] Pass bearer token through generated requests 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-20 18:05:29 +03:00
Alex Vanin
9e08b41a6f [#102] Set split header in left object 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-20 09:59:09 +03:00
Alex Vanin
719075ca97 [#99] Fix no-root search matcher 
Wrong boolean operation order made matcher return false
on `non-root` search query with non-regular objects. Instead
it should return true for `non-root` query and false for `root`
query.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-16 13:45:35 +03:00
Alex Vanin
1332a6d3a8 [#92] Provide session token to all produced requests 
If object service produces new request, the should contain
session token. This is the only way for node to grant access
for a private container.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-15 10:20:10 +03:00
Alex Vanin
2d5cb378a7 [#84] Add netmap service executor and signer 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-09 09:15:18 +03:00
Alex Vanin
0e7e0bd2d6 [#84] Remove mocks and debug code from neofs-node services 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-09 09:15:18 +03:00
Alex Vanin
87fc4f5df7 [#82] Use morph wrapper in container service 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-08 11:22:50 +03:00
Alex Vanin
cd34145969 [#73] Use request owner public key in eACL check 
Classifier fetches public key of the request owner
and owner itself. Extended ACL check should rely on
this public key, because it might be extracted from
session token.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-05 18:02:37 +03:00
Alex Vanin
7a2654719e [#71] Return only regular objects in root object search 
Root search applies for user objects, so it should not
return tombstones and storage groups.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-05 14:03:55 +03:00
Alex Vanin
11262bed4a [#71] Broadcast tombstone to container 
With one tombstone for split objects we can't simply
place it in container. We should inform all nodes that
store split objects of removed original object.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-05 14:03:55 +03:00
Leonard Lyubich
9cdf7d3896 [#69] object/acl: Check eACL rules in ACL service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
1d676fcfb2 [#69] object/acl: Add eACL components to service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
a7782cf1f9 [#69] object/acl: Extended requestInfo structure 
Add container identifier field. Add send public key field.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
6c3c872ee4 [#69] object/acl: Define access denied error 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
0f52444ae9 [#69] object/acl: Change basic ACL type in requestInfo 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
30e6912c7b [#69] object/acl: Construct service from options 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
e5898c9ca8 [#69] object/acl: Rename BasicChecker to Service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
0d5495e997 [#70] object manager: Implement an example object garbage collector 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Leonard Lyubich
798fca9354 [#70] core/object: Process a delete group at tombstone 
Send object group to delete queue processor after tombstone content
validation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Leonard Lyubich
2b16edebc9 [#70] object/put: Fix NPE caused by nil FormatValidator 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Alex Vanin
801999c577 [#66] Impersonate object service verb from session token 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 19:47:49 +03:00
Alex Vanin
afeebd310c [#66] Use session token of object header at put ACL check 
Owner of the request is stored in session token most of the times.
Put request contains session token in the object body, so we have
to fetch it from there.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 19:47:49 +03:00
Leonard Lyubich
69a69cdbee [#67] object/eacl: Implement eACL validator 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 19:46:45 +03:00
Leonard Lyubich
44fcd2f212 [#64] object/delete: Change the formation of tombstone 
Make delete service to write list of child object addresses to tombstone
payload.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 19:46:27 +03:00
Alex Vanin
861bac3892 [#59] Use max msg size in transport server and splitter 
For GRPC it is about 4 MiB.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
d2009c8731 [#59] Add grpc payload splitter in object service chain 
GRPC has default message limit of 4MiB. Since every transmitted
neofs message has to be signed, then original message should
be split into transfer fit structures before signature service.

This commit introduce transport payload splitter for object
service pipeline. This splitter works with stream response
for methods:

  - object.Get
  - object.Range
  - object.Search

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
64691e6248 [#62] object/transformer: Fix incorrect relation init stage 
In previous implementation InitRelations call in payload size limiter was
called in write chunk method. This provoked clearing the split header in
children starting from the second.

Replace InitRelations call to the 1st child allocating stage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
e158497560 [#43] cmd/neofs-node: Support hostnames with dns, ipv4 and ipv6 addresses 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
f930993e3a [#43] pkg/network: Do not panic at multiaddr to net.Addr conversion 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
6824a6f67b [#61] object/search: Support non-root and non-leaf filters 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
51e373c3f0 [#61] object/search: Support latest search filters 
Refactor query to match object and its parents in a single call. Support
KeyRoot and KeyLeaf filters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
f89c848e84 [#61] object/search: Filter objects by container ID from request 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
1654df4d97 [#61] Update to latest neofs-api-go changes 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00