Commit graph

1373 commits

Author SHA1 Message Date
Alex Vanin
5b6be7bc1c [#948] engine: Fix test file path
Before: `<t.Name()>.<id>.blobstor`
After:  `<t.Name()>/<id>.blobstor`
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
1950724a04 [#948] shard: Add ListWithCursor method
Interface for metabase.ListWithCursor method invocation.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
1a829a521f [#948] metabase: Add ListWithCursor method
ListWithCursor allows listing physically stored objects
from metabase with small chunks. Cursor tracks last
processed object, therefore new chunks are returned
on each request.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Pavel Karpy
60e3ea978f [#971] *: Group all 3rd parties imports
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
7ac3145980 [#971] *: Add optional parameters to audit morph client calls
Adapt all audit wrapper calls to new
structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
8a2f5c980b [#971] morph/audit: Add optional parameters
Add optional parameters to the client call
signature.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
ed4810a020 [#971] *: Add optional parameters to balance morph client calls
Adapt all balance wrapper calls to new
structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
644baf4985 [#971] morph/balance: Add optional parameters
Add optional parameters to the client call
signature.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
dbf3a2f2fb [#971] *: Add optional parameters to container morph client calls
Adapt all container wrapper calls to new
structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
af33dd65b2 [#971] morph/container: Add optional parameters
Add optional parameters to the client call
signature.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
822d73fb02 [#971] *: Add notification TX hash to neofsid morph client calls
Add hash of the TX that generated notification
to neofsid event structures. Adapt all
neofsid wrapper calls to new structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
404c62c5c4 [#971] morph/neofsid: Add optional parameters
Add optional parameters to the client call
signature.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
bad739258e [#971] *: Add notification TX hash to neofs/netmap morph client calls
Add hash of the TX that generated notification
to neofs/netmap event structures. Adapt all
neofs/netmap wrapper calls to new structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
c25f5a86ae [#971] morph/netmap: Add optional parameters
Add optional parameters to the client call
signature. Group parameters of a client call
into struct to improve future codebase
support.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
3114be39d0 [#971] morph/neofs: Add optional parameters
Add optional parameters to the client call
signature. Group parameters of a client call
into struct to improve future codebase
support.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
1db6d316c2 [#971] morph/client: Adapt signature changes in wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
3849d13e0b [#971] morph/client: Group Invoke and TestInvoke params
Also:
- add `validUntilBlock` and nonce calculation;
- pass notification hashes to notary calls.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
8e1f187822 [#971] ir: Drop CLI validators voting functionality
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
c167ae26f9 [#971] morph/event: Change notification parser's signature
Parsers should have original notification
structure to be able to construct internal
event structure that contains necessary
for unique nonce calculation information.
So notification parsers take raw notification
structure instead of slice of stack items.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Pavel Karpy
3666ae7ad2 [#971] morph/subscriber: Adapt extended notification structure
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-19 09:58:03 +03:00
Alex Vanin
c9f2804885 [#976] innerring: Ignore non-HALT netmap.AddPeer notary invocations
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-18 09:57:16 +03:00
Alex Vanin
93dce149ba [#976] morph/client: Add script verifier
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-18 09:57:16 +03:00
Alex Vanin
cb36f8b50e [#868] blobstor: Speed up compression test
Compression test does not require wide blobovnicza tree.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-16 13:35:34 +03:00
Alex Vanin
e4087893a1 [#966] engine: Fix test after neofs-api-go replacement
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-15 13:06:10 +03:00
Evgenii Stratonikov
95893927aa *: replace neofs-api-go with neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-12 17:29:09 +03:00
Leonard Lyubich
6c0b29e3e3 [#922] storage engine: Prevent any operations after first Close call
Make `BlockExecution` / `ResumeExecution` to not release per-shard worker
pools. Make `StorageEngine.Close` to block these methods and any
data-related operations. It is still releases the pools.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
d6457ee485 [#922] shard: Allow multiple gc.stop() calls
Use `sync.Once` to prevent locks of stopping GC. It will also allow to
safely call `Shard.Close` multiple  times.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
10f0bd91d6 [#922] engine: Change interface of container operations
Add `error` to return. Improve docs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
a537334f33 [#922] engine: Fix typos and improve naming related to exec blocks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
8f8b638b95 [#922] control: Add MAINTENANCE netmap status
Add `MAINTENANCE` value to `NetmapStatus` enum in Control API. The status is
going to be used to toggle maintenance mode of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
ec04e787aa [#922] storage engine: Support operation blocking
There is a need to disable execution of local data operation on storage
engine in runtime. If storage engine ops are blocked, node will act like
always but all local object operations will be denied.

Implement `BlockExecution` / `ResumeExecution` methods on `StorageEngine`
which blocks / resumes the execution of data ops. Wait for the completion of
all operations executed at the time of the call. Return error passed to
`BlockExecution` from all data-related methods until `ResumeExecution` call.
Make `Close` to block operations as well.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
cea1de3a27 [#922] storage engine: Remove unused Exists method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
68565d9617 [#938] ir/netmap: Call AddPeer method if existing candidate was updated
In previous implementation IR handler of `AddPeer` notification didn't send
registration to contract if existing peer changed has changed its
information. as a consequence, the network map members could not update the
information without going into offline.

Change `processAddPeer` handler to check if
  * candidate in the network map is a brand new
  * or information about the network map member was changed
and call `AddPeer` method if so.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 18:05:30 +03:00
Leonard Lyubich
0ec8f529ab [#842] object/delete: Set tombstone local node as tombstone owner
All objects in NeoFS must have owner ID. In previous implementation Object
Delete service handler set owner ID from request session token. If removal
was executed w/o a session, object with tombstone was prepared incorrectly.
In order to fix this node should set its own ID and become an owner of the
tombstone object.

Extend `NetworkInfo` interface required by Object.Delete handler with
`LocalNodeID` method which returns `owner.ID` of the local node. Implement
the method on `networkState` component of storage node application which is
updated on each node state change in NeoFS network map. Set owner returned
by `LocalNodeID` call as tombstone object's owner in Delete handler.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 16:18:02 +03:00
Leonard Lyubich
d421022547 [#842] object/acl: Check owner and key only if sticky bit is set
In previous implementation node returns "access denied" on Object.Put with
object with unset owner. Although object owner must be set, its absence
should not be considered as access error. The same applies to sender key.

Check owner ID and public key emptiness only if sticky bit is set.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 16:18:02 +03:00
Pavel Karpy
6cab1635d4 [#936] ir: Do not require number of Alphabet contracts to be set
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-10 16:11:57 +03:00
Pavel Karpy
b492201a84 [#936] morph/nns: Add record existence check
Also, add ErrNNSRecordNotFound error that
indicates that required hash is not presented
in `NNS` contract.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-10 16:11:57 +03:00
Leonard Lyubich
b3708fc530 [#957] services/object: Refactor usage of NeoFS API client
The client needs of the Object service are limited and change not often.
Interface changes of the client library should not affect the operation of
various service packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the Object service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-03 18:30:09 +03:00
Alex Vanin
a437ffc3ed [#910] innerring: Make notary deposit on notification instead of timer
Timer is not suitable for notary deposits because it can never fire
in case of desynchronization or external epoch changes. Notary deposits
must be handled on new epoch event.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-02 11:28:02 +03:00
Alex Vanin
45f244eb77 [#955] locode: Use the nearest polygon when continent can't be found
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-02 10:30:23 +03:00
Evgenii Stratonikov
1462824ab8 [#947] writecache: refactor object persisting
a1696a8 introduced some logic which in some situations prevented big objects
to be persisted in FSTree. In this commit a refactoring is done with the
goal of simplifying the code and also checking #866 issue.

1. Split a monstrous function into multiple simple ones: memory objects
   can only be small and for writing through the cache we can do a dispatch
   in `Put` itself.
2. Determine objects to be put in database before the actual update
   as setting up a transaction has non-zero overhead.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-01 15:52:56 +03:00
Leonard Lyubich
88e37ea372 [#946] ir: Refactor usage of NeoFS API client
The client needs of the IR application are very limited and rarely change.
Interface changes of the client library should not affect the operation of
various application packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the IR application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-01 11:41:57 +03:00
Alex Vanin
fbed86da2c [#943] service/object: Remove KeyStore from CommonPrm
There is no point to pass key storage in parameters because
it can be defined on the service level of application.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
c30aa20b04 [#943] service/object: Refactor private key fetching during execution
`CommonPrm` structure has private key for remote operations.
It obtained in the beginning of request processing. However,
not every operation triggers remote calls. Therefore, key
might not be used. It is important to avoid early key fetching
because `TokenStore` now returns error if session token does not
exist. This is valid case when container nodes receive request with
session token (for ACL pass) and they should process request locally.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
2fbdcbdee1 [#943] service/object: Check session token expiration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
508a28fdc0 [#943] service/object: Do not sign put service message with session token
This is invalid operation for storage nodes that receive part of split
object. While object is signed by session token, the message itself
should be signed by the node key.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
46fa07b7cc [#943] service/object: Return error if requested session token is not available
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
2271944064 [#945] metabase: Use cleanup instead of defer in tests
Cleanup is more idiomatic way to clean test data

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 16:10:01 +03:00
Alex Vanin
e41aba610d [#945] metabase: Fix containers listing
Container listing should not ignore tombstone and
storage group objects which are not stored in
primary buckets.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 16:10:01 +03:00
Pavel Karpy
8956f015fc [#941] morph/notary: Delete useless "low till" check
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 19:16:05 +03:00