TrueCloudLab/frostfs-node
19
1
Fork 27
Code Issues 91 Pull requests 7 Projects Releases 1 Activity Actions
504 commits 8 branches 83 tags 53 MiB
Commit graph

380 commits

Author SHA1 Message Date
Leonard Lyubich
d9a5007c72 [#137] metabase: Add unit test for a nonexistent attribute 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-03 18:42:32 +03:00
Leonard Lyubich
7704811654 [#137] metabase: Implement and useful test funcs for work with DB 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-03 18:42:32 +03:00
Leonard Lyubich
fc2038e929 [#137] metabase: Implement Path method 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-03 18:42:32 +03:00
Leonard Lyubich
8125b544b4 [#136] cmd/neofs-node: Use new metabase in app 
Remove BoltDB bucket package. Construct meta.DB instance in node app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-03 18:42:32 +03:00
Leonard Lyubich
c0aa892161 [#136] localstorage: Make local storage to use new metabase 
Replace meta Bucket with meta.DB instance in local storage implementation.
Adopt all dependent components to new local storage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-03 18:42:32 +03:00
Leonard Lyubich
a61f8d44d1 [#135] metabase: Implement benchmarking tests 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-03 18:42:32 +03:00
Leonard Lyubich
1db01725c9 [#131] metabase: Implement indexing by object properties 
Process parent objects in Put method. Headers of parent object are stored as
regular leaf objects in metabase from now. Build indexes for ROOT, LEAF and
CHILDFREE properties.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-03 18:42:32 +03:00
Alex Vanin
97077294fc [#141] Fix double sidechain GAS emission on asset mint 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-03 16:28:01 +03:00
Alex Vanin
0c49c08609 [#139] Use storage node GAS emission in inner ring app 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-03 16:20:41 +03:00
Alex Vanin
d193f1087c [#139] Emit gas to the storage nodes at emission tick 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-03 16:20:41 +03:00
Leonard Lyubich
a8e9d15f3f [#85] inner-ring/container: Check container format in process Put 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-03 14:14:38 +03:00
Leonard Lyubich
766eea4c8c [#85] services/container: Check container format in Put 
Call CheckFormat function in container.Put handler for conducting initial
checks of the structure that are not performed by the smart contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-03 14:14:38 +03:00
Leonard Lyubich
a3e0a9f74c [#85] core/container: Implement CheckFormat function 
Implement function that conducts sanity checks of the container data.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-11-03 14:14:38 +03:00
Alex Vanin
83119c00ec [#127] Remove magic number from config 
With neo-project/neo-modules#358 nodes support RPC to
return used magic number. Therefore client doesn't need
that configuration value any more.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-02 13:54:31 +03:00
Alex Vanin
174efc9df3 [#124] Update neo-go to pre-preview4 version 
Neo-go does not use smartcontract.Parameter to return values
anymore, so it's convertes partly removed from neofs-node.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-02 13:54:31 +03:00
Alex Vanin
c7975dbe87 [#132] Use SDK defined netmap structure in cleanup table 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-02 10:52:52 +03:00
Alex Vanin
976ba06300 [#132] Use netmap cleaner in inner ring server 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-02 10:52:52 +03:00
Alex Vanin
94957dd38c [#132] Generate and handle netmap cleanup event 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-02 10:52:52 +03:00
Alex Vanin
5c7de27546 [#132] Add netmap cleanup event processor 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-02 10:52:52 +03:00
Alex Vanin
6fcaa13e0b [#132] Flag all gracefully shutdown nodes in cleaner cache 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-02 10:52:52 +03:00
Alex Vanin
118f7fb309 [#132] Put new storage peers in netmap cleaner cache 
There is an optimization to send ApprovePeer tx only in
case if `touch` returns false. It returns false if node
is not in the cache or it was flagged to removal.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-02 10:52:52 +03:00
Alex Vanin
0c42db2c3c [#132] Add cleanup table in netmap processor 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-02 10:52:52 +03:00
Alex Vanin
9b97e95f82 [#132] Add internal event to process cleanup 
This event will be produced by new epoch handler. All
cleanups should be synchronous because smart contract can't
store inner ring votes forever. Therefore voting should be done
in similar time interval for all inner ring nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-02 10:52:52 +03:00
Alex Vanin
dfc2e81717 [#132] Add cleanup table in inner ring netmap processor 
Cleanup table is a cache for inner ring node to look for netmap
snapshot. It updates access time of bootstrapped nodes and will
be used to send `updateState` txs to clean netmap from unresponsive
nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-02 10:52:52 +03:00
Alex Vanin
c80bce9438 [#132] Implement netmap snapshot getter for inner ring 
Later it should be merged with morph/client getters.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-11-02 10:52:52 +03:00
Leonard Lyubich
b48a4ede02 [#125] services/eacl: Use latest object header keys 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-29 19:25:54 +03:00
Leonard Lyubich
8d931b81a6 [#125] object/search: Use latest search filter keys 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-29 19:25:54 +03:00
Leonard Lyubich
f34ad9e730 [#125] services/eacl: Fix undefined method usage 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-29 19:25:54 +03:00
Leonard Lyubich
20ed7c0d61 [#128] metabase: Implement Delete method 
Implement Delete method on DB structure that adds deleted addresses to
tombstone index. Do not attach addresses from tombstone index to Select
result. Return error from Get method if address is presented in tombstone
index.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-29 17:34:41 +03:00
Leonard Lyubich
2d319aa29c [#128] metabase: Use static byte prefix for calculating non-empty keys 
It is no longer necessary to make different prefixes to generate non-empty
keys for buckets.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-29 17:34:41 +03:00
Leonard Lyubich
5f78a18a4f [#128] metabase: Do not create leaves in indexed header bucket 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-29 17:34:41 +03:00
Leonard Lyubich
85aacbbb10 [#128] localstorage: Implement primary object metabase 
Implement bolt-based metabase that is going to be used in local object
storage. Implement Put/Get/Select methods.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-29 17:34:41 +03:00
Alex Vanin
d08c1c76c1 [#122] Reduce precision from balance contract to Fixed8. 
Fixed8 won't overflow int64 for values less than 92 billion
that is suitable for GAS.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-29 10:14:59 +03:00
Alex Vanin
03f52bca01 [#122] Return big.Int in balanceOf function 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-29 10:14:59 +03:00
Alex Vanin
e3c060b739 [#122] Add BigIntFromStackItem 
Neo-go can return big.Int values on stack after contract execution.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-29 10:14:59 +03:00
Alex Vanin
330c4b42d9 [#122] Use balance precision converter in inner ring 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-28 13:36:59 +03:00
Alex Vanin
90984ee219 [#122] Add converter interface in balance and neofs processors 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-28 13:36:59 +03:00
Alex Vanin
b6012977bc [#122] Implement balance precision getter for inner ring 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-28 13:36:59 +03:00
Alex Vanin
7db40eafe0 [#122] Add precision converter 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-28 13:36:59 +03:00
Leonard Lyubich
f66c7958e7 [#109] services/policer: Assign tasks to Replicator 
Make Policer to call AddTask method of Replicator when an insufficient
number of copies of an object is detected in the container.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 15:23:22 +03:00
Leonard Lyubich
2d46baa4a5 [#109] services: Implement Replicator service 
Implement Replicator service that performs background work to replicate
local object to remote nodes in the container. Replicator is going to be
used by Policer.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 15:23:22 +03:00
Leonard Lyubich
53efa18e14 [#109] object/put: Implement remote object sender 
Define RemoteSender structure with PutObject method that puts object to a
remote node locally.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 15:23:22 +03:00
Leonard Lyubich
968033deed [#40] object/put: Assign zero return of MaxObjectSize invalid 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 14:03:25 +03:00
Leonard Lyubich
43533c991e [#40] morph/netmap: Implement MaxObjectSize method on wrapper 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 14:03:25 +03:00
Leonard Lyubich
90c38fc5e5 [#40] morph/client: Implement read config method 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 14:03:25 +03:00
Leonard Lyubich
7fdb14cf8a [#83] services/response: Set epoch number from network state 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Leonard Lyubich
19f9c7eacb [#83] services: Remove setting of meta header from executing services 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Leonard Lyubich
0341773318 [#83] services: Implement response sub-service for each service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Leonard Lyubich
6bede7d836 [#83] services/util: Implement response service 
Create response package. Implement response Service that sets values of
response meta header.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Leonard Lyubich
1cc7983c4e [#83] services/util: Add meta header methods to ResponseMessage 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Leonard Lyubich
71a06f9e01 [#83] services/util: Define type of response message interface 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-23 10:54:48 +03:00
Alex Vanin
7464254680 [#106] Put simplest bearer token check first 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-22 18:02:11 +03:00
Alex Vanin
23ec33e821 [#106] Check bearer token lifetime 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-22 18:02:11 +03:00
Alex Vanin
bb455af05f [#106] Ignore bearer token if basic ACL restrict it 
There is a bit to allow or deny bearer token check for
each object service method. If this bit is not set then
ignore bearer token and use extended ACL table from
sidechain.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-22 18:02:11 +03:00
Alex Vanin
89cd2ad463 [#106] Process bearer token in ACL service 
If bearer token is presented in the request then check
if it is a valid one and then use it to process extended
ACL checks.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-22 18:02:11 +03:00
Alex Vanin
094248690b [#115] Make ACL classifier errors transparent for client 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-22 11:55:28 +03:00
Alex Vanin
ca552f53c6 [#115] Check session token validity 
Malicious user can stole public session key and use
it by sending request from it's own scope. To prevent
this each session token is signed and signature private
key must be corresponded with owner id in token. Therefore
malicious node cannot impersonate request without private
key to sign token.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-22 11:55:28 +03:00
Leonard Lyubich
dae94aa230 [#60] morph/netmap: Implement Epoch method on netmap client wrapper 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-22 11:54:08 +03:00
Leonard Lyubich
20fb8547b6 [#60] morph/netmap: Implement Epoch method on Client 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-22 11:54:08 +03:00
Leonard Lyubich
16a5107ef1 [#60] object/put: Provide network State interface to formatter 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-22 11:54:08 +03:00
Leonard Lyubich
b627814dd8 [#60] object/transformer: Set creation epoch number in new objects 
Set value of CreationEpoch object field to the value from network State.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-22 11:54:08 +03:00
Leonard Lyubich
7c081e4f15 [#60] core/netmap: Define interface of current network properties 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-22 11:54:08 +03:00
Leonard Lyubich
4a56f82571 [#60] object/transformer: Group parameters of NewFormatTarget func 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-22 11:54:08 +03:00
Leonard Lyubich
2541ed4b8f [#88] object/eacl: Use String() methods to calculate ID values 
Replace hex encoding of IDs with String() call (base58) in eACL processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 18:53:04 +03:00
Leonard Lyubich
5318abcf38 [#88] object/search: Use String() methods to calculate ID values 
Replace hex encoding of IDs with String() call (base58) in search query
processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 18:53:04 +03:00
Leonard Lyubich
0dab4b7581 [#108] services: Implement Policer service 
Implement Policer service that performs background work to check compliance
with the placement policy for local objects in the container. In the initial
implementation, the selection of the working queue of objects is
simplified, and there is no transfer of the result to the replicator.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 14:42:51 +03:00
Leonard Lyubich
f6e56aa956 [#108] placement: Implement Builder from netmap source 
Implement placement.Builder interface on netmap.Source wrapper.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 14:42:51 +03:00
Leonard Lyubich
5017ff0e4a [#108] object/head: Export remote header retrieval utility 
Export remote head functionality in headsvc package. Refactor head service
to use RemoteHeader.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 14:42:51 +03:00
Leonard Lyubich
5ad0df7794 [#108] object/head: Return 404 error if header was not found 
Define ErrNotFound error in headsvc package. Return ErrNotFound from Head
method if the header was not found in the container.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-21 14:42:51 +03:00
Alex Vanin
ae0dd9e051 [#106] Pass bearer token through generated requests 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-20 18:05:29 +03:00
Alex Vanin
e6f04f7785 [#104] Update neofs-api-go with new protobuf API 
Also update contains JSON converters for neofs-cli
and fixes bug in container.set-acl command of SDK.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-20 17:31:59 +03:00
Alex Vanin
9e08b41a6f [#102] Set split header in left object 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-20 09:59:09 +03:00
Leonard Lyubich
54bdeb60a1 [#22] Support string type in stack parameter converter 
Set type of stack parameter to StringType in type-switch statement of
toStackParameter function.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-19 12:27:56 +03:00
Alex Vanin
643e81254c [#25] Add placement policy QL encoder 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-17 13:16:59 +03:00
Evgenii Stratonikov
d7c53debb5 [#46] Make SELECT attribute optional 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2020-10-16 16:15:27 +03:00
Alex Vanin
719075ca97 [#99] Fix no-root search matcher 
Wrong boolean operation order made matcher return false
on `non-root` search query with non-regular objects. Instead
it should return true for `non-root` query and false for `root`
query.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-16 13:45:35 +03:00
Alex Vanin
1332a6d3a8 [#92] Provide session token to all produced requests 
If object service produces new request, the should contain
session token. This is the only way for node to grant access
for a private container.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-15 10:20:10 +03:00
Alex Vanin
dd48666357 [#72] Shutdown inner ring app if RPC node has been terminated 
Adopt error channel from Listener interface.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-14 09:37:48 +03:00
Alex Vanin
ccbb9ce6ab [#72] Add ListenWithError method in Listener interface 
Listen and ListenWithError methods check if subscriber channel
has been closed. If so, ListenWithError passes error message
into provided channel.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-14 09:37:48 +03:00
Alex Vanin
ca006245d2 [#72] Close subscription channel if RPC was terminated 
RPC node closes websocket notification channel if it was terminated
or something wrong happened. Subscriber has to check this condition
and alert about this in upper context by closing it's own channel.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-14 09:37:48 +03:00
Alex Vanin
d3d8f00757 [#72] Init inner ring configuration before run 
This way we can re-initialize application in any given
moment, which is useful for reconnects.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-14 09:37:48 +03:00
Alex Vanin
ade185191b [#47] Use less extra gas on netmap contract methods 
* 1.0 extra gas for new epoch vote
* 0.5 extra gas for new peer approve

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-12 14:43:33 +03:00
Alex Vanin
b7ee05088c [#86] Use alphabet processor in inner ring app 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-12 14:43:33 +03:00
Alex Vanin
f1f20b49db [#86] Add alphabet contract processor 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-12 14:43:33 +03:00
Alex Vanin
4308a6f522 [#86] Run timer for gas emission event 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-12 14:43:33 +03:00
Alex Vanin
6bc787bb19 [#86] Add alphabet contract configuration 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-12 14:43:33 +03:00
Alex Vanin
dcb384d551 [#86] Specify inner ring list index in global state 
Inner ring index will be used to access alphabet contracts.
First seven inner ring nods communicating with one alphabet
contract, depending on their index.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-12 14:43:33 +03:00
Alex Vanin
2d5cb378a7 [#84] Add netmap service executor and signer 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-09 09:15:18 +03:00
Alex Vanin
f92dc5b27c [#84] Add GRPC layer of netmap service 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-09 09:15:18 +03:00
Alex Vanin
0e7e0bd2d6 [#84] Remove mocks and debug code from neofs-node services 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-09 09:15:18 +03:00
Alex Vanin
b5aef1011b [#74] Update test of netmap.UpdateState event parser 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-09 09:15:18 +03:00
Alex Vanin
e3f8e350f9 [#82] Fix conversion of container id values from smart-contract 
Smart-contract stores container ids as a raw bytes, not marshaled
protobuf structures.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-08 11:22:50 +03:00
Alex Vanin
7d51688d2c [#82] Return length check of serialized container 
In #37 we've decided to remove length check, because smart contract would
fail on casting `nil` value from storage to `[]byte` producing FAULT state.
Apparently it does not fail, so we have to check length explicitly.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-08 11:22:50 +03:00
Alex Vanin
87fc4f5df7 [#82] Use morph wrapper in container service 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-08 11:22:50 +03:00
Alex Vanin
204eaab5a9 [#75] Fix order of update peer state method arguments 
This method has node status first and public key second.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-05 18:58:35 +03:00
Alex Vanin
f40f1ca0c0 [#74] Fix order of netmap.UpdateState event arguments 
This event has node status first and public key second.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-05 18:31:13 +03:00
Alex Vanin
cd34145969 [#73] Use request owner public key in eACL check 
Classifier fetches public key of the request owner
and owner itself. Extended ACL check should rely on
this public key, because it might be extracted from
session token.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-05 18:02:37 +03:00
Alex Vanin
7a2654719e [#71] Return only regular objects in root object search 
Root search applies for user objects, so it should not
return tombstones and storage groups.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-05 14:03:55 +03:00
Alex Vanin
11262bed4a [#71] Broadcast tombstone to container 
With one tombstone for split objects we can't simply
place it in container. We should inform all nodes that
store split objects of removed original object.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-05 14:03:55 +03:00
Leonard Lyubich
9cdf7d3896 [#69] object/acl: Check eACL rules in ACL service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
1d676fcfb2 [#69] object/acl: Add eACL components to service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
a7782cf1f9 [#69] object/acl: Extended requestInfo structure 
Add container identifier field. Add send public key field.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
6c3c872ee4 [#69] object/acl: Define access denied error 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
0f52444ae9 [#69] object/acl: Change basic ACL type in requestInfo 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
30e6912c7b [#69] object/acl: Construct service from options 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
e5898c9ca8 [#69] object/acl: Rename BasicChecker to Service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 14:02:14 +03:00
Leonard Lyubich
0d5495e997 [#70] object manager: Implement an example object garbage collector 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Leonard Lyubich
798fca9354 [#70] core/object: Process a delete group at tombstone 
Send object group to delete queue processor after tombstone content
validation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Leonard Lyubich
2b16edebc9 [#70] object/put: Fix NPE caused by nil FormatValidator 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-05 09:36:29 +03:00
Alex Vanin
801999c577 [#66] Impersonate object service verb from session token 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 19:47:49 +03:00
Alex Vanin
afeebd310c [#66] Use session token of object header at put ACL check 
Owner of the request is stored in session token most of the times.
Put request contains session token in the object body, so we have
to fetch it from there.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 19:47:49 +03:00
Leonard Lyubich
69a69cdbee [#67] object/eacl: Implement eACL validator 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 19:46:45 +03:00
Leonard Lyubich
44fcd2f212 [#64] object/delete: Change the formation of tombstone 
Make delete service to write list of child object addresses to tombstone
payload.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 19:46:27 +03:00
Leonard Lyubich
33ca88f85f [#64] core/object: Claim TombstoneContent in tombstone payload 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 19:46:27 +03:00
Leonard Lyubich
21bdc82fb5 [#64] core/object: Implement tombstone content messages 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 19:46:27 +03:00
Alex Vanin
6ee908c2db [#65] Support FS bucket and BoltDB bucket 
These buckets can be used to store blobs and metadata.
They will be removed as enhanced blob storage will be
implemented for neofs-node. To setup storage type
use `storage.object.type` and `storage.meta.type` params.

Available options:
  - inmemory (default)
  - boltdb
  - filesystem

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 14:48:00 +03:00
Alex Vanin
861bac3892 [#59] Use max msg size in transport server and splitter 
For GRPC it is about 4 MiB.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
d2009c8731 [#59] Add grpc payload splitter in object service chain 
GRPC has default message limit of 4MiB. Since every transmitted
neofs message has to be signed, then original message should
be split into transfer fit structures before signature service.

This commit introduce transport payload splitter for object
service pipeline. This splitter works with stream response
for methods:

  - object.Get
  - object.Range
  - object.Search

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
64691e6248 [#62] object/transformer: Fix incorrect relation init stage 
In previous implementation InitRelations call in payload size limiter was
called in write chunk method. This provoked clearing the split header in
children starting from the second.

Replace InitRelations call to the 1st child allocating stage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
e158497560 [#43] cmd/neofs-node: Support hostnames with dns, ipv4 and ipv6 addresses 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
f930993e3a [#43] pkg/network: Do not panic at multiaddr to net.Addr conversion 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
6824a6f67b [#61] object/search: Support non-root and non-leaf filters 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
51e373c3f0 [#61] object/search: Support latest search filters 
Refactor query to match object and its parents in a single call. Support
KeyRoot and KeyLeaf filters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
f89c848e84 [#61] object/search: Filter objects by container ID from request 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
1654df4d97 [#61] Update to latest neofs-api-go changes 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
f251645def [#58] object/delete: Process linking object in Delete service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
16252ad09a [#58] object/search: Add object-with-children filter 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
624e8cd3cb [#58] object/search: Refactor RelationSearcher implementation 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
4bcfed37ca [#58] object/head: Generalize RelationSearcher interface method 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
6eb353c804 [#58] object/put: Validate payload content after filling 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
2abb03dbd1 [#58] core/object: Decouple the payload content validation method 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
107f3097e4 [#58] core/object: Add content validation to FormatValidator 
Add content validation step to FormatValidator. Check tombstone payload
correctness.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
017afbf0e3 [#58] services/object: Implement Delete service 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
b24adeae89 [#58] object/transformer: Inherit type of parent object 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
39ddb3a3f4 [#45] object/search: Fix double write of local result 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
08b9ae547a [#45] object/search: Add filtering parent objects 
In previous implementation object.Search services allowed to search only
physically stored objects. This limitation did not allow getting the ID of
the split object.

Extend search execution logic with parent object filtering. Parent objects
that passed filters are now included in the result

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
88459963fb [#57] services/object: Sign requests with session key 
Use key storage in object services in order to sign requests with private
session key within user session.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
68178b8d74 [#57] core/object: Fix validation of linking object 
Linking object carries only identifier of the parent object. Fix Validate
method of FormatValidator to not validate parent object of the linking
object.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
be322835af [#57] services/object: Implement private key storage 
Implement storage that provides access to local node key and session keys
through session token.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
2da323c4b9 [#57] services/object: Add session token to common parameters 
Add session token field to CommonPrm. Remove session token field from Put
parameters.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
39c17253be [#57] services/object: Combine common service parameters 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
8cddbe58a6 [#56] object/transformer: Write session token to object body 
Add session token argument to object formatter constructor which is written
to the object. Pass session token from trusted object Put.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
a4b9560ef6 [#56] object/put: Validate object format in untrusted Put 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
ede033256d [#56] core/object: Implement format validator 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Leonard Lyubich
12d57af998 [#56] Update to latest neofs-api-go changes 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
fc74e9b40c [#32] Remove recover from basic ACL checks 
Basic ACL checker gets request field via getters that are
NPE-free, therefore we don't need to worry about function
invocations on nil structures.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
4a8de3263d [#32] Use less v2 specific structures in basic ACL checker 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
c5a44e0a05 [#32] Add tests for basic ACL helper 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
91fef72bb6 [#32] Make basic ACL check in all object request 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00
Alex Vanin
49ee9a14a1 [#32] Add basic ACL helper 
Basic ACL helper provides functions for simple access to
bit fields of basic ACL.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2020-10-02 11:25:36 +03:00