Commit graph

241 commits

Author SHA1 Message Date
Leonard Lyubich
358e3ed8c4 [#645] *: Change the locality condition of the node from the placement
Some software components regulate the way of working with placement arrays
when a local node enters it. In the previous implementation, the locality
criterion was the correspondence between the announced network address
(group) and the address with which the node was configured. However, by
design, network addresses are not unique identifiers of storage nodes in the
system.

Change comparisons by network addresses to comparisons by keys in all
packages with the logic described above. Implement `netmap.AnnouncedKeys`
interface on `cfg` type in the storage node application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Leonard Lyubich
fe90456dcc [#645] placement: Overload result of Traverser.Next method
In previous implementation `placement.Traverser.Next` method returned slice
of `network.AddressGroup` elements. There is a need to process keys of
storage nodes besides network addresses for intra-container communication.

Wrap `network.AddressGroup` in a new type `placement.Node` that summarizes
the storage node information required for communication. Return slice of
`Node` instances from `Traverser.Next` method. Fix compilation breaks in
dependent packages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-09-07 09:53:18 +03:00
Alex Vanin
d6439369ab [#768] object/acl: Support all eACL filter fields from the spec
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-08-26 19:01:03 +03:00
Pavel Karpy
e8665f6cef [#730] node/morph: Use disable_cache config param
Do not init caches for eACL, containers and netmap
if `disable_cache` config options is `true`, use
direct RPC calls instead.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 09:40:10 +03:00
Pavel Karpy
c423aa432a [#730] node/config/morph: Add disable_cache
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-08-03 09:40:10 +03:00
Leonard Lyubich
43eff09944 [#607] *: Do not use deprecated elements of code
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
9033cf5aa1 [#607] placement: Work with AddressGroup
Make `PlacementBuilder` implementations to work `network.AddressGroup` type.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
b3dd9a3254 [#607] placement: Make traverser to return list of address groups
Make placement `Traverser.Next` method to return ``[]network.AddressGroup`
in order to support multiple addresses of the storeage nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
8ac3c62518 [#607] object/head: Make client constructor to work with group address
Make Object Head service to work with `AddressGroup` instead of `Address`
in order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
d0e48c949b [#607] object/search: Make client constructor to work with group address
Make Object Search service to work with `AddressGroup` instead of `Address`
in order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
ad14df07f6 [#607] object/get: Make client constructor to work with group address
Make Object Get service to work with `AddressGroup` instead of `Address` in
order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
8972f84672 [#607] object/put: Make client constructor to work with group address
Make Object Put service to work with `AddressGroup` instead of `Address` in
order to support multiple addresses of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
6e5d7f84af [#607] network: Generalize LocalAddressSource to address group
Make `LocalAddressSource.LocalAddress` method to return `AddressGroup`. Make
`IsLocalAddress` function to accept parameter of type `AddressGroup`. Adopt
the application code with temporary `GroupFromAddress` helper.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Leonard Lyubich
3805b0f638 [#607] client: Overload Client interface
There is a need to generalize single-address client to group-address client.
To do this, we can re-implement `Client` interface from NeoFS API Go library
and still use it in the application code. There is a problem with method
`Raw` which must return single-address raw client. So as not to make changes
to API library we need to overload Client interface in order to support
`Raw` method in group-address client implementation.

Define `Client` interface in new `pkg/core/client` package. Completely
inherit API `Client` interface. Add `RawForAddress` method to build raw
client for the single node address. Adopt the application code that used Raw
method to work with new `Client`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-28 15:52:50 +03:00
Pavel Karpy
48827f42d3 [#643] pkg: Sync method names and commentaries to them
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-06-24 16:10:44 +03:00
Leonard Lyubich
adbbad0beb [#607] network: Do not work with Address pointers
`network.Address` structure in most cases created once and used read-only.

Replace `AddressFromString` function with `Address.FromString` method with
the same purpose and implementation. Make all libraries to work with value.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich
47fe8911a3 [#607] network: Rename Address.HostAddrString method to HostAddr
Return tyype is clear from the method's signature and docs, there is no
point in reflecting it in the name.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich
35f81729e4 [#607] network: Do not return error from Address.HostAddrString method
Panic if internal `manet.DialArgs` call returns error since this is
unexpected according to `AddressFromString` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Leonard Lyubich
95ccbbc2f9 [#607] network: Accept value instead of pointer in IsLocalAddress
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-18 18:09:50 +03:00
Evgenii Stratonikov
9142c778c7 [#562] pkg: remove nspcc-dev/neofs-crypto dependency
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-15 14:49:59 +03:00
Leonard Lyubich
096acb2a44 [#580] v2/object/put: Check payload size of the relayed objects
NeoFS network dictates the limitation of the maximum size of the "finished"
objects. The payload size of an object is recorded in its header. When
writing finished objects, the nodes must check:

  * satisfying the constraint;
  * matching the value in the header to the number of payload bytes.

Provide value returned by `MaxSizeSource` component to `validatingTarget`.
Check max size value during the stream of the "finished" objects. Check
header value during the streaming and on-close. Check payload size in v2
relay scenario.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-06-09 12:08:37 +03:00
Evgenii Stratonikov
b8a7c11e57 [#501] object/put: reduce TTL of the relayed request
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-02 10:31:14 +03:00
Evgenii Stratonikov
3468491224 [#501] object/put: relay requests for signed objects
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-02 10:31:14 +03:00
Evgenii Stratonikov
a422f42ca9 [#501] object/put: refactor distributed target
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-06-02 10:31:14 +03:00
Leonard Lyubich
3e1463cc76 [#570] *: Use generator of test container IDs from API Go lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 20:45:15 +03:00
Leonard Lyubich
3dd10b6795 [#570] *: Remove usage of deprecated elements from API Go library
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 20:45:15 +03:00
Leonard Lyubich
70a7354e9d [#570] *: Use new Equal method of owner.ID type
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-31 20:45:15 +03:00
Pavel Karpy
33bef46f31 [#549] network/cache: Change Get signature
Make network cache's `Get` method accept
`network.Address` argument instead of
string.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-05-27 17:07:04 +03:00
Alex Vanin
975ab0dff7 [#561] acl: Add tests for request meta header traverser
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-27 11:21:14 +03:00
Alex Vanin
a025e6250e [#561] acl: Fetch session token from original request meta header
As it explained in previous commit, session token also should
be presented in original meta header but can be omitted in higher
layers.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-27 11:21:14 +03:00
Alex Vanin
d368afffe5 [#561] acl: Fetch bearer token from original request meta header
Request meta headers are organized in a layers, where
upper layers re-sign down layers. Bearer token should be
a part of original meta header and it can be omitted in
upper layers. Therefore we need to traverse over linked list
of meta header to the original meta header to get bearer token.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-27 11:21:14 +03:00
Leonard Lyubich
b587b23e79 [#543] object/search: Fix potential NPE on request forwarder
Request forwarding callback should be called only if set since it is an
optional parameter.

Call `forwarder` function only if it is non-nil.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-20 11:33:24 +03:00
Leonard Lyubich
b2002ffc6d [#543] object/get: Fix potential NPE on request forwarder
Request forwarding callback should be called only if set since it is an
optional parameter. In GetRangeHash forwarder is never set.

Call `forwarder` function only if it is non-nil. Remove no longer needed
`hashOnly` option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-20 11:33:24 +03:00
Alex Vanin
b5256ccf4c [#521] Fix issues with transition from pkg/errors pkg
Wrap functions at `pkg/errors` return nil if error argument
was nil. fmt.Errorf always returns error so we need to add
missing error checks to the code.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-19 19:09:35 +03:00
Evgenii Stratonikov
71b87155ef [#521] *: use stdlib errors package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-05-19 16:42:54 +03:00
Leonard Lyubich
72e2bc8fb6 [#532] object/hash: Fix NPE during request forwarding
In current implementation `Object.GetRangeHash` RPC handler forwards range
requests for payload data. Missing request forwarder of the original request
caused NPE during execution.

Do not call request forwarder if payload range hash is requested.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-18 18:05:18 +03:00
Alex Vanin
16f13bc0a5 [#522] Use HostAddrString as RPC endpoint instead of IPAddrString
To enable TLS support we can't operate with IP addresses directly.
Certificates are issued with host names so it is required to
pass them into RPC client. DNS resolving should be done by transport
layer and not be a part of node. Therefore `IPAddrString` usage is
removed from code.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-05-18 12:07:00 +03:00
Leonard Lyubich
969af596b4 [#431] object/search: Remove no longer used unexported methods
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-04 13:56:55 +03:00
Leonard Lyubich
7366d62e41 [#431] object/range: Fix payload slice allocation in re-sign callback
Allocate capacity instead of length of the slice to write the object payload
range since each chunk is written through `append`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-04 13:56:55 +03:00
Leonard Lyubich
36088949fc [#431] services/object: Re-sign original read requests during forwarding
In previous implementation node's Object Get/Head/GetRange V2 service
handlers created a new request for each RPC. Now original requests are
re-signed according to API specification. Logical handler abstracts from
this version-dependent logic through `RequestForwarder` callback.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-04 13:56:55 +03:00
Leonard Lyubich
e6b30aed36 [#431] object/search: Re-sign original requests during forwarding
In previous implementation node's Object Search V2 service handler created a
new request for each RPC. Now original requests are re-signed according to
API specification. Logical handler abstracts from this version-dependent
logic through `RequestForwarder` callback.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-05-04 13:56:55 +03:00
Leonard Lyubich
106884fc40 [#428] client: Hide client cache behind interface in dependent packages
Replace usage of `cache.ClientCache` type with interface with similar
signature. This will further allow overloading clients without affecting the
logic of dependent packages.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-24 10:11:52 +03:00
Leonard Lyubich
383d2494eb [#425] Adapt the Client's refactoring to the interface
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Leonard Lyubich
718a2fad26 [#425] services: Define service interfaces that was removed from API lib
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-03-17 15:00:37 +03:00
Alex Vanin
892b4f4f12 [#426] service/object: Fix put duration
At object.Put operation node does not transfer any
data. It is done in the stream inside, so we should
register duration from `object.Put` invocation until
`CloseAndRecv` on the stream.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
b8e10571c6 [#426] Put prometheus behind pkg/metrics
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
bc6e7a823b [#426] service/object: Add object size metrics
We can't rely on object size in the header because it
might be omitted on initial put or it can be 0xFF.. on
streaming data.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
0b93e8a029 [#426] service/object: Add request duration metrics
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Alex Vanin
f8ab7a9c60 [#426] service/object: Add object service metrics collector
Collect number of request per method in object service.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-03-17 10:58:00 +03:00
Evgenii Stratonikov
6679d59e89 [#422] pkg/services: Provide client options on cache creation
Because options are not used when client is already in cache
providing them to shared cache is misleading at best.
In the worst case `dial_timeout` is set randomly (because of race
condition) which can lead to one service having `dial_timeout` of
another. Thus we set default client creation options when cache is
created.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-03-15 14:07:53 +03:00