carpawell/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
970 commits 27 branches 26 tags 21 MiB
Commit graph

155 commits

Author SHA1 Message Date
Leonard Lyubich
611a29f682 [#241] object/search: Refactor service processing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
0e1f05ff45 [#239] object/head: Implement new service processing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
7e56427534 [#235] object/getrangehash: Implement new service processing 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
1d23483828 [#235] services/object: Implement new GetRange algorithm 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
a875d80491 Remove outdated code of metabase and localstore 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
869d9e571c [#233] object/head: Remove getting the right child header 
With the update of the local storage engine, the headers of virtual objects
are directly given. In this regard, the step with obtaining the the right
child header is removed.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
f24daa10ff [#233] services/object: Implement new Get algorithm 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
a488d202e1 [#230] engine: Rename WithAddress to WithAddresses for delete operations 
Delete works with multiple addresses at once so it is convenient to
show it in parameter setter function.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
3d77fdb347 [#222] Set write cache specific config on application level 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
4114028a63 [#222] Add shard amount check 
There should be at least one correctly set up shard in node config.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
351e4b4592 [#222] Support Inhume and Delete in object service 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
f45b7a048d [#222] Rename WithTreeRoot.. to WithRoot.. in blobstor opts 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Alex Vanin
92c95150cd [#222] Use write cache options in storage app 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
2eb83624cf [#220] cmd/neofs-node: Configure Shard's write cache usage 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
cd2228cd99 [#220] cmd/neofs-node: Support new configuration of storage components 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
a1164b2a5c [#211] cmd/neofs-node: Add SmallSizeLimit config value of BlobStor 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
4ca1035aa4 [#186] cmd/neofs-node: Integrate new storage engine into application 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-12-11 17:19:37 +03:00
Leonard Lyubich
621840d542 [#193] cmd/neofs-node: Add dial timeouts to node configuration 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 15:34:30 +03:00
Leonard Lyubich
d286824d1d [#194] cmd/neofs-node: Use non-blocking worker pools for Object services 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:56 +03:00
Leonard Lyubich
6a5c37d592 [#195] object/search: Add option to set logger 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Leonard Lyubich
cb46e4b154 [#195] object/rangehash: Add option to set logger 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Leonard Lyubich
af6484e3b1 [#195] object/range: Add option to set logger 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Leonard Lyubich
cf2dc37a42 [#195] object/put: Add option to set logger 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Leonard Lyubich
9fbfc0b5e4 [#195] object/head: Add option to set logger 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Leonard Lyubich
e1e5a590e9 [#195] object/delete: Add option to set logger 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-24 13:38:06 +03:00
Alex Vanin
e8fe07edd0 [#184] Use SDK client cache in object.Rangehash 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-18 18:18:07 +03:00
Alex Vanin
f85e88c4f8 [#184] Use SDK client cache in object.Range 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-18 18:18:07 +03:00
Alex Vanin
d485a5967d [#184] Use SDK client cache in object.Search 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-18 18:18:07 +03:00
Alex Vanin
7ba95dd5fc [#184] Use SDK client cache in object.Put 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-18 18:18:07 +03:00
Alex Vanin
e9a6365333 [#184] Use SDK client cache in object.Head 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-18 18:18:07 +03:00
Leonard Lyubich
58fcb35fb0 [#174] Use Marshal(JSON)/Unmarshal(JSON) methods for encoding/decoding 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-17 11:56:00 +03:00
Alex Vanin
eb13322bf0 [#166] Use async pools in object service handlers 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-13 15:54:00 +03:00
Leonard Lyubich
3c17e871e1 [#158] cmd/neofs-node: Handle objects from tombstone by metabase 
Handle tombstone content by metabase instead of object GC queue.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-09 14:07:20 +03:00
Leonard Lyubich
36d5af812e [#158] metabase: Add logger to DB configuration 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-09 14:07:20 +03:00
Leonard Lyubich
60e4b5ddff [#158] metabase: Construct DB using options 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-09 14:07:20 +03:00
Leonard Lyubich
8125b544b4 [#136] cmd/neofs-node: Use new metabase in app 
Remove BoltDB bucket package. Construct meta.DB instance in node app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-03 18:42:32 +03:00
Alex Vanin
2bd827a478 [#134] Resend bootstrap txs periodically 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-02 10:53:11 +03:00
Leonard Lyubich
334e0e6f0f [#109] cmd/neofs-node: Activate Replicator 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 15:23:22 +03:00
Alex Vanin
0c3e6c7a04 [#40] cmd/neofs-node: Remove max object size value from config 
Now max object size value is taken from global config in sidechain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-23 14:18:00 +03:00
Leonard Lyubich
23f00a3cd2 [#40] cmd/neofs-node: Re-implement max object size source 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 14:03:25 +03:00
Leonard Lyubich
d05b2ff500 [#83] cmd/neofs-node: Add network state option to response service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Leonard Lyubich
19f9c7eacb [#83] services: Remove setting of meta header from executing services 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Leonard Lyubich
3065a7363b [#83] cmd/neofs-node: Use response services in application 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Alex Vanin
23ec33e821 [#106] Check bearer token lifetime 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-22 18:02:11 +03:00
Leonard Lyubich
0bdfd18469 [#60] cmd/neofs-node: Initialize network state on boot 
Get current epoch number from morph client and store the value in local
state structure.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-22 11:54:08 +03:00
Leonard Lyubich
a8481223d6 [#60] cmd/neofs-node: Implement primary network State instance 
Define networkState structure and implement netmap.State interface on it.
Set epoch number on NewEpoch notification from chain. Provide
WithNetworkState option to Put service constructor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-22 11:54:08 +03:00
Leonard Lyubich
f52f643fe4 [#108] cmd/neofs-node: Add Policer worker to application 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 14:42:51 +03:00
Leonard Lyubich
07fe49088c [#108] cmd/neofs-node: Listen new epoch notifications from morph 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 14:42:51 +03:00
Evgeniy Kulikov
53f7b58130 [#93] Add gRPC Reflection service 
Added gRPC Reflection service that can be enabled
by settings `grpc.enable_reflect_service`.

Read more about at
https://github.com/grpc/grpc-go/blob/master/Documentation/server-reflection-tutorial.md

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-10-15 10:20:58 +03:00
Alex Vanin
dee1d81b04 [#84] Run netmap service in neofs-node app 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-09 09:15:18 +03:00
Alex Vanin
20c27d0542 [#84] Add API version to neofs-node config 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-09 09:15:18 +03:00
Alex Vanin
65a93af176 [#84] Fix loop variable capture issue in neofs-node workers 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-09 09:15:18 +03:00
Alex Vanin
0e7e0bd2d6 [#84] Remove mocks and debug code from neofs-node services 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-09 09:15:18 +03:00
Alex Vanin
a9e801cb22 [#71] Make GC configurable 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-05 14:03:55 +03:00
Leonard Lyubich
1d676fcfb2 [#69] object/acl: Add eACL components to service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
30e6912c7b [#69] object/acl: Construct service from options 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
e5898c9ca8 [#69] object/acl: Rename BasicChecker to Service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
ab57fddb2b [#70] cmd/neofs-node: Add object GC to app 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Leonard Lyubich
798fca9354 [#70] core/object: Process a delete group at tombstone 
Send object group to delete queue processor after tombstone content
validation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Leonard Lyubich
276ed6c04b [#70] cmd/neofs-node: Implement Del method on in-memory bucket 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Leonard Lyubich
9a604a50b9 [#70] cmd/neofs-node: Start asynchronous workers on app launch 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Alex Vanin
6fe6005e71 [#66] Check basic ACL in object service pipeline 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 19:47:49 +03:00
Alex Vanin
2ee24998ba [#68] Add go pprof to neofs-storage 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 19:47:08 +03:00
Alex Vanin
6ee908c2db [#65] Support FS bucket and BoltDB bucket 
These buckets can be used to store blobs and metadata.
They will be removed as enhanced blob storage will be
implemented for neofs-node. To setup storage type
use `storage.object.type` and `storage.meta.type` params.

Available options:
  - inmemory (default)
  - boltdb
  - filesystem

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 14:48:00 +03:00
Alex Vanin
861bac3892 [#59] Use max msg size in transport server and splitter 
For GRPC it is about 4 MiB.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
d2009c8731 [#59] Add grpc payload splitter in object service chain 
GRPC has default message limit of 4MiB. Since every transmitted
neofs message has to be signed, then original message should
be split into transfer fit structures before signature service.

This commit introduce transport payload splitter for object
service pipeline. This splitter works with stream response
for methods:

  - object.Get
  - object.Range
  - object.Search

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
13080aa7a1 [#43] cmd/neofs-node: Fetch max object size from config 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
e158497560 [#43] cmd/neofs-node: Support hostnames with dns, ipv4 and ipv6 addresses 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
8585e147d0 [#43] cmd/neofs-node: Use morph wrappers as data sources 
Morph wrappers are being uses as sources for containers and
snapshots of network map.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
1914b4dbad [#43] cmd/neofs-node: Use object service in application 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
ea511d559c [#43] cmd/neofs-node: Refactor mocked object service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
a5ae128f34 [#43] cmd/neofs-node: Implement mocked max object size source 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
8dced272d2 [#43] cmd/neofs-node: Implement in-memory bucket 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
480362b02f [#43] cmd/neofs-node: Make cfg provide network address source interface 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
caedef82af [#43] cmd/neofs-node: Change object executor mocked calls 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
1a169a1a9d [#49] Use zap.Logger in neofs-storage 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
df5fb23ba8 [#36] Move node attribute code into separate file 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
8e4416ad4b [#36] Add default capacity and price attributes 
Default values for capacity and price will be 0, so storage node
won't be present in placement. Add these attributes if they are
not explicitly set up by user config.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
1759ff57f1 [#36] Parse attributes from node config 
Well-known attributes set up with explicit configuration
values, other attributes set up from chain of attributes.

Chain of attributes is a string, that contains keys-value pairs,
divided by semicolon. Pairs itself divided by slash.

E.g. "StorageType:HDD/RPM:7200"

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
3ed02d5f64 [#28] Wrap bootstrap errors with prefix message 
Generic morph invocation errors are more useful, if they have
extra info about invocation context.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Alex Vanin
86b9aefcae [#28] Make storage node configurable 
To run storage node at dev-env environment it should have
configurable parameters. To keep `cfg` structures we can
read configuration from env and yml config file with viper
and parse values such as script hashes, fees, keys into
`cfg` structures.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Alex Vanin
aa98592ac4 [#15] Fix wrapper for add peer contract method 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Alex Vanin
ee9551992c [#15] Use api-go definition of NodeInfo in node 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Alex Vanin
a76a97ec01 [#15] Add bootstrap routine to node application 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Alex Vanin
73b8ed4203 [#15] Add netmap contract and node info configs 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Alex Vanin
80f10dab7b [#16] Fix balanceOf wrapper for api request 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
b5671c1971 [#13] neofs-node: Implement Object.GetRangeHash executor 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
7765671502 [#13] neofs-node: Implement Object.Delete executor 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
7e91fffd39 [#13] neofs-node: Implement Object.Head executor 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
3c773cfa67 [#13] neofs-node: Implement Object.GetRange executor 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
d90f53d311 [#13] neofs-node: Implement Object.Put executor 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
5caabf4208 [#13] neofs-node: Implement Object.Get executor 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
cd19183d3e [#13] neofs-node: Use execution Object service in app 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
eb874092b5 [#13] neofs-node: Use signing Object service in app 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
c378c353d5 [#12] Serve gRPC session service in neofs-node app 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:35 +03:00
Leonard Lyubich
8539f5c2cd [#11] services/container: Implement Neo:Morph executor and service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:24:45 +03:00
Leonard Lyubich
3308fcf56d [#11] neofs-node: Improve sources 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:24:20 +03:00
Leonard Lyubich
e6fedfbc69 [#11] Use Neo:Morph ServiceExecutor of Accounting service in app 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:23:54 +03:00
Leonard Lyubich
3ae7f7e914 [#11] Use accounting service executor in neofs-node app 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:23:42 +03:00
Leonard Lyubich
5cf622881f [#11] accounting: Implement signing service 
Adds accounting Service implementation that verifiers request signatures and
signs responses from internal Service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:23:29 +03:00