Commit graph

3997 commits

Author SHA1 Message Date
Leonard Lyubich
3692f708ca [#51] object/put: Fix incorrect splited object streaming
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
1a169a1a9d [#49] Use zap.Logger in neofs-storage
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
2fc2f8792e [#48] Make IR initial config errors more readable
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
be353ad69f [#48] Use more extra fee at container registration
Large inner ring requires more gas to make container registration.
Container contract makes balance transfers for each inner ring node
and it require extra gas to execute. This estimation should be enough
for seven inner ring nodes.

Later there should be heuristic evaluations for this: #47

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
6f4b477195 [#44] services/object: Refactor for minor optimization
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
3692032e30 [#44] object/head: Fix incorrect placement traversal
Add object identifier option to placement traverser of object.Head service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
a433f9dd85 [#42] object/transformer: Fix incorrect checksums of parent object
In previous implementations object size limiter left checksums of parent
object payload unset. This was caused by the closure that written calculated
checksums to the child object instead of parent one.

Construct payload checksum writers in separate function in order to prevent
closure bug.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
3038c6cc5c [#37] Implement eACL related morph wrapper functions
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
14f85184ab [#37] Implement container related morph wrapper functions
Container get wrapper implements container.Source interface
so it can be used in object service as container storage.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
d6a9c06c25 [#39] service/object: Complicate Head service logic
Add a header recovery step through finding and getting the header of the
rightmost child.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
753a6a2de5 [#39] service/object: Implement right child searcher on Search service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Leonard Lyubich
39c324bd6d [#39] service/object: Add query filters to search right child
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
df5fb23ba8 [#36] Move node attribute code into separate file
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
8e4416ad4b [#36] Add default capacity and price attributes
Default values for capacity and price will be 0, so storage node
won't be present in placement. Add these attributes if they are
not explicitly set up by user config.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
1759ff57f1 [#36] Parse attributes from node config
Well-known attributes set up with explicit configuration
values, other attributes set up from chain of attributes.

Chain of attributes is a string, that contains keys-value pairs,
divided by semicolon. Pairs itself divided by slash.

E.g. "StorageType:HDD/RPM:7200"

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:36 +03:00
Alex Vanin
28534a509d [#36] Add attribute parser
Attribute parser converts strings of attribute chain into
one-dimension array of NodeInfo attributes, that used
in network map.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
05f3963975 [#38] service/object: Implement simplified object Head service
Implement Head service w/o linking object processing and restoration from
split-chain.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
21fc85540a [#38] localstore: Finalize object to meta converter
Use CutPayload method of RawObject in order to make object meta structure
from source object.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
2f57855775 [#38] sdk/object: Add CutPayload method to RawObject
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
becb9148c1 [#38] sdk/object: Add Object to RawObject converter
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
dc88934034 [#34] Update go.sum
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
09084a7bff [#34] service/object: Implement object Search distributed service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
a5ebdd1891 [#34] services/object: Replace local placement to util
Replace local placement tool in order to reuse it in different object
services.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
ded0350c5e [#34] localstore: Store object identifier in meta storage
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Alex Vanin
99d380c20b [#35] Implement netmap source interface in morph wrapper
Netmap source interface used by object service to build
placement based on current or previous network map.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:35 +03:00
Alex Vanin
92fb384c09 [#35] Add snapshot method for morph/netmap client
With snapshot method node can get previous network
maps from previous epoch.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
c410b23c97 [#33] Update to latest neofs-api-go version
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
57f8d3745d [#33] service/object: Implement object Put distributed service
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
dcfb6a6b3a [#33] placement: Remote UseNetworkMap option from traverser
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
12e5e4e2d8 [#33] placement: Add public constructor for netmap placement builder
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
900949c648 [#33] network: Implement tool to work with local address
Implement read-only local network address source. Implement function for
check if network address is local.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
d3349a293a [#33] core/object: Add to-SDK object converting method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
3d25d6c217 [#33] core/netmap: Define read-only storage interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
030375af63 [#33] container: Define read-only storage interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
d52bdb27c5 [#33] util: Define worker pool interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
d3b5ff9526 [#33] session/storage: Change Get method signature
In previous implementation session Storage.Get method took a v2 owner
identifier as an argument.

Change method signature to as an argument owner identifier from NeoFS SDK.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
4024b3c30e [#33] services/session: Fix signature service constructor
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
c312e11f6e [#33] network: Add network endpoint getter to Address
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
73ee5105ff [#31] placement: Fix confused selectors and replicas
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
f7c685f682 [#31] placement: Fix incorrect selectors processing
In previous implementation traverser worked like all counts of all selectors
are equal to counts of corresponding replicas. Make traverser to take into
account select count of all replicas.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
e7925fbc1c [#31] placement: Implement container placement traverser
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
44def45ff4 [#31] network: Make the address a separate type
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
d6a4965841 [#31] network: Remove unused dial functions
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
121137c62b [#30] object_manager: Implement object transformers
Define object writer interface. Implement payload size limiter that
restricts payload size of the object. Implement object format moulder that
fill verification fields and finalizes object structure.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
0a6130a924 [#30] core/object: Make Object constructors similar to RawObject
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
2975b61abd [#30] core/object: Remove no longer used MarshalStableV2 method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
87487cde00 [#30] localstorage: Refactor source code
Move the serialization of the address and object into separate functions and
use them in methods. Use recently implemented RawObject constructor and
method in metaFromObject function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
4326ff56a7 [#30] core/object: Remove redundant Address type
Remove Address type. Makes Address method of the Object to return NeoFS SDK
Address type. Makes local storage to work with NeoFS SDK object address.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
8366c146d7 [#30] core/object: Implement raw object
Define RawObject type that inherits RawObject from NeoFS SDK. Implement
converters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00
Leonard Lyubich
9c1c023f05 [#30] Update to latest neofs-api-go and fix conflicts
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2020-10-02 11:25:35 +03:00