aarifullin/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
773 commits 130 branches 26 tags 52 MiB
Commit graph

282 commits

Author SHA1 Message Date
Leonard Lyubich
6c9dab481c [#311] control: Cover StableMarshal methods with unit tests 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
5f22e7fac7 Add newline at end of Control service proto file 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
36d8e2d3b7 [#311] control: Implement JSON marshaler on Netmap message 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
dde24bc9e5 [#311] control: Rewrite methods for binary encoding and signatures 
Rewrite marshalers and sizers using proto library from API repo.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
e9d4b2d25b [#311] Recompile protobuf files 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
3e882d963f [#311] services/control: Implement NetmapSnapshot method 
Implement methods required for signatures. Receive network map from netmap
storage, convert it to Control service message and return in response.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
28777911fb [#311] services/control: Define NetmapSnapshot rpc 
Add NetmapSnapshot rpc to ControlService protobuf definition. Recompile
proto files. Add required method to server structure.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
88023f3655 [#311] services/control: Define Netmap structure 
Define NodeInfo protobuf type. Define Netmap protobuf type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-15 16:11:50 +03:00
Leonard Lyubich
f311be3ab6 [#234] Fix ineffassign linter remark 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
0a0ee89665 [#306] control: Rename WithAllowedKeys function to WithAuthorizedKeys 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
abd9952e46 [#306] Rename Private service to Control service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
44a0fb5a69 [#306] private: Define and use HealthChecker interface to get the status 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
c1b8a4815f [#306] private: Implement server of gRPC private node service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
e75ddb0549 [#306] private: Implement setters on generated proto messages 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
85ec633938 [#306] Define and compile proto files for private node service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-14 12:13:49 +03:00
Leonard Lyubich
1e170c3812 [#234] services/object: Support netmap epoch and lookup dead in read ops 
Support processing of NetmapEpoch and NetmapLookupDepth X-headers when
processing object read operations. Placement for operations
Get/Head/GetRange/GetRangeHash/Search is built for the epoch specified in
NetmapEpoch X-header (by default latest). Also the specified operations are
processed until success is achieved for network maps from the past up to
NetmapLookupDepth value. Behavior for default values (zero or missing) left
unchanged.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-12 19:19:11 +03:00
Leonard Lyubich
2f4d90025f [#234] core/netmap: Extend Source interface 
Add GetNetMapByEpoch method. Add Epoch method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-12 19:19:11 +03:00
Alex Vanin
e88c1f750d [#302] Fix golint else/return linter errors 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2021-01-12 18:47:02 +03:00
Alex Vanin
74a68af5c2 [#302] Fix golint underscore linter errors 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2021-01-12 18:47:02 +03:00
Alex Vanin
04c0c1b8f5 [#302] Fix errcheck linter errors 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2021-01-12 18:47:02 +03:00
Alex Vanin
5e231c515a [#302] Fix whitespace linter errors 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2021-01-12 18:47:02 +03:00
Leonard Lyubich
3a7f6701d8 [#290] object/rangehash: Apply salt to data before hashing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-11 18:40:59 +03:00
Leonard Lyubich
0e2c761481 [#290] object/rangehash: Add salt to RangeHashPrm structure 
Add binary salt field to RangeHashPrm struct. Implement field setter. Set
salt from the request in v2 service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2021-01-11 18:40:59 +03:00
Leonard Lyubich
9a86fff7e0 [#291] Remove some unused code from repository 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-30 10:27:53 +03:00
Leonard Lyubich
a51211eda7 [#291] Remove unused code from pkg/services/object/head 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-30 10:27:53 +03:00
Leonard Lyubich
3260e9263e [#291] Remove unused pkg/services/object/rangehash packages 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-30 10:27:53 +03:00
Leonard Lyubich
d299d94049 [#291] Remove unused pkg/services/object/search/query packages 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-30 10:27:53 +03:00
Leonard Lyubich
1dd1762163 [#285] object/put: Fix the loss of X-headers when forwarding objects 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-29 12:22:05 +03:00
Leonard Lyubich
2897e83fb2 [#285] object/eacl: Validate X-headers from the requests, not the responses 
In previous implementation of eACL service v2 the response X-headers were
validated at the stage of re-checking eACL. This provoked a mismatch of
records in the eACL table with requests. Fix this behavior by checking the
headers from the request, not the response.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-29 12:22:05 +03:00
Leonard Lyubich
c69f867af1 [#285] services/object: Add X-Headers to client call options 
Forward request X-headers to client calls during internal processing of
Object operations on the node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-29 12:22:05 +03:00
Alex Vanin
7174abcc7c [#281] service/audit: Fix typos 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
e5108cf135 [#281] service/audit: Run each SG check in separate routing at PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
0d7832f5e9 [#281] service/audit: Add separate pool for SG checks in PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
8e72abaab7 [#281] service/audit: Make report structure threadsafe 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
44d9f95137 [#276] service/audit: Count Head requests and retries in PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
9273a89f61 [#276] service/audit: Use random nodes for object.Head in PoR 
Since PoR audit check uses object.Head with large TTL values to
make real proof of retrievability, we may use random remote
nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
10b548275a [#276] innerring: Use Head with TTL in PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
3ec342b2c3 [#259] services/audit: Implement PDP check 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
6977adec66 [#273] service/audit: Ignore coverage of very small objects 
PDP audit check is not quite working with very small objects, so
we try to build coverage with bigger objects.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
afb83c610c [#273] service/audit: Add cache of head responses in PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
9212864f42 [#258] services/audit: Implement PoP check 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
a5320408a5 [#271] service/audit: Implement PoR 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
5f65ec0265 [#271] audit/report: Add PoR related setters 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
85773c419b [#271] service/object: Fill all fields of new short header structure 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
af7ae0ec4c [#271] acl: Add permission to get objects for inner ring 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
dd5e67ff0c [#271] service/audit: Add netmap structure to audit task 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
64b022a5ed [#271] placement: Add FlattenNodes function 
This function converts 2-dimension array of container nodes
into single dimension array. Useful when we need to iterate
over whole container.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Alex Vanin
58033a3adc [#271] placement: Add BuildObjectPlacement function 
This function allows to shuffle prepared container nodes
according to HRW weights from network map.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
2ebcbe70fb [#255] ir/audit: Make task manager to return number of skipped tasks 
Add numeric return from TaskManager.Reset method that shows the number of
canceled tasks. This values will be used for assessment of the progress of
the audit.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00
Leonard Lyubich
03e3afb0e8 [#255] services/audit: Define interface of container communicator 
Define interface of the container communicator which methods are going to be
used in audit checks. Make innerring Server to implement this interface.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-25 16:49:27 +03:00