Commit graph

2181 commits

Author SHA1 Message Date
Evgenii Stratonikov
cbf8ca12b0 [#859] config: rename refill_metabase to resync_metabase
This flag enables metabase content synchronization with blobstor on
start.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-15 13:14:24 +03:00
Alex Vanin
e4087893a1 [#966] engine: Fix test after neofs-api-go replacement
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-15 13:06:10 +03:00
Alex Vanin
9a4d4c03b7 [#918] config/testnet: Add control service endpoint
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-15 13:06:10 +03:00
Leonard Lyubich
68a454fc56 [#969] config/example: Add note about shard config in relay node
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-15 06:47:25 +03:00
Leonard Lyubich
fb89d29574 [#969] node: Do not require shard config in relay mode
Relay storage node doesn't exec local object operations, so it doesn't need
shard configuration.

Add `required` bool parameter to `engineconfig.IterateShards`. Make it to
panic if it is `true`, and immediately return otherwise. Pass `false` if
node is configured as relay in app (it also prevents panic).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-15 06:47:25 +03:00
Evgenii Stratonikov
95893927aa *: replace neofs-api-go with neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-12 17:29:09 +03:00
Leonard Lyubich
6c0b29e3e3 [#922] storage engine: Prevent any operations after first Close call
Make `BlockExecution` / `ResumeExecution` to not release per-shard worker
pools. Make `StorageEngine.Close` to block these methods and any
data-related operations. It is still releases the pools.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
d6457ee485 [#922] shard: Allow multiple gc.stop() calls
Use `sync.Once` to prevent locks of stopping GC. It will also allow to
safely call `Shard.Close` multiple  times.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
10f0bd91d6 [#922] engine: Change interface of container operations
Add `error` to return. Improve docs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
a537334f33 [#922] engine: Fix typos and improve naming related to exec blocks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
6b1ce99c35 [#922] node: Block data ops of storage engine via Control API
There is a need to block execution of local object storage operations if
node is put into maintenance mode (resume if the node is taken out of
maintenance mode).

Call `BlockExecution` method if `ControlService.SetNetmapStatus` was called
with `MAINTENANCE` status. Call `ResumeExecution` if it was called with
another status.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
29644e9bc2 [#922] cli: Support MAINTENANCE netmap status
Accept `maintenance` netmap status in `control set-status` command and
resolve it to `control.NetmapStatus_MAINTENANCE` value.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
8f8b638b95 [#922] control: Add MAINTENANCE netmap status
Add `MAINTENANCE` value to `NetmapStatus` enum in Control API. The status is
going to be used to toggle maintenance mode of the storage node.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
ec04e787aa [#922] storage engine: Support operation blocking
There is a need to disable execution of local data operation on storage
engine in runtime. If storage engine ops are blocked, node will act like
always but all local object operations will be denied.

Implement `BlockExecution` / `ResumeExecution` methods on `StorageEngine`
which blocks / resumes the execution of data ops. Wait for the completion of
all operations executed at the time of the call. Return error passed to
`BlockExecution` from all data-related methods until `ResumeExecution` call.
Make `Close` to block operations as well.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
cea1de3a27 [#922] storage engine: Remove unused Exists method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-12 17:28:38 +03:00
Leonard Lyubich
68565d9617 [#938] ir/netmap: Call AddPeer method if existing candidate was updated
In previous implementation IR handler of `AddPeer` notification didn't send
registration to contract if existing peer changed has changed its
information. as a consequence, the network map members could not update the
information without going into offline.

Change `processAddPeer` handler to check if
  * candidate in the network map is a brand new
  * or information about the network map member was changed
and call `AddPeer` method if so.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 18:05:30 +03:00
Leonard Lyubich
0ec8f529ab [#842] object/delete: Set tombstone local node as tombstone owner
All objects in NeoFS must have owner ID. In previous implementation Object
Delete service handler set owner ID from request session token. If removal
was executed w/o a session, object with tombstone was prepared incorrectly.
In order to fix this node should set its own ID and become an owner of the
tombstone object.

Extend `NetworkInfo` interface required by Object.Delete handler with
`LocalNodeID` method which returns `owner.ID` of the local node. Implement
the method on `networkState` component of storage node application which is
updated on each node state change in NeoFS network map. Set owner returned
by `LocalNodeID` call as tombstone object's owner in Delete handler.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 16:18:02 +03:00
Leonard Lyubich
d421022547 [#842] object/acl: Check owner and key only if sticky bit is set
In previous implementation node returns "access denied" on Object.Put with
object with unset owner. Although object owner must be set, its absence
should not be considered as access error. The same applies to sender key.

Check owner ID and public key emptiness only if sticky bit is set.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-10 16:18:02 +03:00
Pavel Karpy
5e5211305c [#936] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-10 16:11:57 +03:00
Pavel Karpy
6cab1635d4 [#936] ir: Do not require number of Alphabet contracts to be set
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-10 16:11:57 +03:00
Pavel Karpy
b492201a84 [#936] morph/nns: Add record existence check
Also, add ErrNNSRecordNotFound error that
indicates that required hash is not presented
in `NNS` contract.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-10 16:11:57 +03:00
Leonard Lyubich
e500efb9b3 [#957] services/reputation: Refactor usage of NeoFS API client
The client needs of the Reputation service are limited and change not often.
Interface changes of the client library should not affect the operation of
various service packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the Reputation service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-03 18:30:09 +03:00
Leonard Lyubich
b3708fc530 [#957] services/object: Refactor usage of NeoFS API client
The client needs of the Object service are limited and change not often.
Interface changes of the client library should not affect the operation of
various service packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the Object service.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-03 18:30:09 +03:00
Leonard Lyubich
bbc2b873ab [#950] cli: Refactor usage of NeoFS API client
The client needs of the CLI application are limited and change not often.
Interface changes of the client library should not affect the operation of
various application packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the CLI application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-03 18:26:36 +03:00
Leonard Lyubich
a3414b36dd [#958] cli/locode: Open RO database in info command
CLI `util locode info` command does not write operations. Opening a
`locodebolt.DB` instance in RW mode is redundant.

Provide `locodebolt.ReadOnly()` option to `locodebolt.DB` constructor in
order to create READ flock over BoltDB file.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-03 10:19:45 +03:00
Alex Vanin
cee4f3142f Release v0.26.1
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-02 17:08:37 +03:00
Alex Vanin
a437ffc3ed [#910] innerring: Make notary deposit on notification instead of timer
Timer is not suitable for notary deposits because it can never fire
in case of desynchronization or external epoch changes. Notary deposits
must be handled on new epoch event.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-02 11:28:02 +03:00
Alex Vanin
45f244eb77 [#955] locode: Use the nearest polygon when continent can't be found
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-02 10:30:23 +03:00
Evgenii Stratonikov
1462824ab8 [#947] writecache: refactor object persisting
a1696a8 introduced some logic which in some situations prevented big objects
to be persisted in FSTree. In this commit a refactoring is done with the
goal of simplifying the code and also checking #866 issue.

1. Split a monstrous function into multiple simple ones: memory objects
   can only be small and for writing through the cache we can do a dispatch
   in `Put` itself.
2. Determine objects to be put in database before the actual update
   as setting up a transaction has non-zero overhead.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-01 15:52:56 +03:00
Evgenii Stratonikov
088c894f44 [#733] neofs-adm: read contracts on start
Fail early and reduce disk operations when reading from archive.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-01 15:17:18 +03:00
Evgenii Stratonikov
0b6350d463 [#733] neofs-adm: fetch contracts release from Github
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-01 15:17:18 +03:00
Evgenii Stratonikov
c1e1b65ad9 [#733] neofs-adm: allow to read contract from tar archive
Assume contracts path is tar.gz archive if it is not a directory.
In theory we could support raw tar but it has little value as our
distribution format is tar.gz and this is the primary use-case for this
feature (download and use).

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-11-01 15:17:18 +03:00
Leonard Lyubich
88e37ea372 [#946] ir: Refactor usage of NeoFS API client
The client needs of the IR application are very limited and rarely change.
Interface changes of the client library should not affect the operation of
various application packages, if they do not change their requirements for
the provided functionality. To localize the use of the base client and
facilitate further support, an auxiliary package is implemented that will
only be used by the IR application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-01 11:41:57 +03:00
Evgenii Stratonikov
49c9dbfba8 [#877] config/engine: use default section for shards
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-28 15:04:14 +03:00
Evgenii Stratonikov
cb9bf00ceb [#877] neofs-node/config: allow to provide default values
In case we have multiple sections with similar structure (e.g. shards)
having defaults in a single place is easier to work with.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-28 15:04:14 +03:00
Evgenii Stratonikov
118c3b3cfe [#940] neofs-adm: print contract version in dump-hashes
```
nns         (unknown):  7061fbd31562664b58f422c3dee4acfd70dba8af
alphabet 0  (v0.12.1):  2392438eb31100857c0f161c66791872b249aa13
alphabet 1  (v0.12.1):  83ef4226d5d6519ca9c99a5de13b1b5ca223a6ad
alphabet 2  (v0.12.1):  6250927beaa9aa5a00171379dcb7187b0c91d17d
alphabet 3  (v0.12.1):  1d6a2519ba41a139b2ced1bfd5013938271a7578
alphabet 4  (v0.12.1):  b65fc7a3c31cf57a90d7eb1c0e9909e4ca69133c
alphabet 5  (v0.12.1):  f95b6ff8cd3b027c9911c18115518ad8c5d2f591
alphabet 6  (v0.12.1):  5b17c579bf56884fd68af152432b3b5aee7aee76
audit       (v0.12.1):  85fe181f4aa3cbdc94023d97c69001ece0730398
balance     (v0.12.1):  dc1ec98d9d0c5f9dfade16144defe08cffc5ca55
container   (v0.12.1):  1b6e68d299b570e1cb7e86eadfdc06aa2e8e0cc5
neofsid     (v0.12.1):  0a64ce753653cc97c0467e1334d9d3678ca8c682
netmap      (v0.12.1):  7c5bdb23e36cc7cce95bf42f3ab9e452c2501df1
reputation  (v0.12.1):  7ad824fd1eeb1565be2cee3889214b9aa605d2fc
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-28 09:35:52 +03:00
Alex Vanin
fbed86da2c [#943] service/object: Remove KeyStore from CommonPrm
There is no point to pass key storage in parameters because
it can be defined on the service level of application.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
c30aa20b04 [#943] service/object: Refactor private key fetching during execution
`CommonPrm` structure has private key for remote operations.
It obtained in the beginning of request processing. However,
not every operation triggers remote calls. Therefore, key
might not be used. It is important to avoid early key fetching
because `TokenStore` now returns error if session token does not
exist. This is valid case when container nodes receive request with
session token (for ACL pass) and they should process request locally.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
2fbdcbdee1 [#943] service/object: Check session token expiration
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
508a28fdc0 [#943] service/object: Do not sign put service message with session token
This is invalid operation for storage nodes that receive part of split
object. While object is signed by session token, the message itself
should be signed by the node key.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
46fa07b7cc [#943] service/object: Return error if requested session token is not available
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 20:25:25 +03:00
Alex Vanin
0f598289f6 [#942] neofs-cli: Return endpoint flag to control netmap-snapshot command
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 19:37:19 +03:00
Alex Vanin
2271944064 [#945] metabase: Use cleanup instead of defer in tests
Cleanup is more idiomatic way to clean test data

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 16:10:01 +03:00
Alex Vanin
e41aba610d [#945] metabase: Fix containers listing
Container listing should not ignore tombstone and
storage group objects which are not stored in
primary buckets.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-10-27 16:10:01 +03:00
Pavel Karpy
8956f015fc [#941] morph/notary: Delete useless "low till" check
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 19:16:05 +03:00
Pavel Karpy
78aab096fb [#941] morph/notary: Add dynamic notary deposit till calculation
Do not set `till` as some constant:
use maximum of two values instead:
1. currentDepositTill;
2. currentHeight+epochDuration+constant.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 19:16:05 +03:00
Evgenii Stratonikov
0057eeb0f7 [#901] *: release worker pools where possible
Some of the pools are initialized during config initialization,
so it isn't possible currently to release them in one place.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Evgenii Stratonikov
10afd26354 [#901] engine: release shard pools in Close
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Evgenii Stratonikov
0beaed2ef4 [#901] util: implement Release for PseudWorkerPool
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-10-25 18:53:36 +03:00
Pavel Karpy
616013cb8a [#933] write-cache: Use real number of objects in counters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-10-25 18:49:10 +03:00