Commit Graph

1666 Commits (088df0e2a9ae02fc7b8c7bc720d3092e5efe4fb2)

Author SHA1 Message Date
Leonard Lyubich 088df0e2a9 [#1369] version: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Leonard Lyubich 7a57d7b076 [#1369] audit: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Elizaveta Chichindaeva cc7a723d77 [#1320] English Check
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-05-11 10:40:02 +03:00
Leonard Lyubich cd545f0160 [#1351] cli: Fix connection scheme loss during endpoint parsing
In previous implementation NeoFS CLI app used `network.Address.HostAddr`
as a server URI, which caused scheme loss since host address doesn't
contain it.

Rename `HostAddr` to `URIAddr` and make it to return URI address with
`grpcs` scheme if TLS is enabled. Make `TLSEnabled` unexported since it
was used to provide default `tls.Config` only (it is used by default in
SDK).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-05 10:43:27 +03:00
Evgenii Stratonikov a4769d8624 *: use `require.ErrorIs` where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:47:57 +03:00
Evgenii Stratonikov 3c39e6df11 [#1319] blobstor: Compress big objects in a streaming fashion
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov 1219ff89d4 [#1319] services/object_manager: Fix error message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov 057d53459b [#1319] services/object: Remove `Write` method from `local` and `remote` targets
They work with prepared objects only.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov 4ea03c01b5 [#1319] services/object: Store payload directly in `distributedTarget`
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Pavel Karpy 9da7df4d42 [#1318] metabase: Init metabase in test
`Init()` creates buckets. It prevents NPE in tests.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy 486ec8a2e5 [#1318] metabase: Reuse comparing func from `require`
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy 1caeb0b648 [#1318] metabase: Add missing error check in tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy 6f2723b47e [#1318] node: Fix linter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy 2aec5736e0 [#1318] node/gc: Fix tombstones handling
Handle only expired tombstones.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy 24ca8ca5c7 [#1318] object: Change default ttl
This is need to reuse object service without passing any API v2 structs.
Also, fix comment.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy 7799f8e4cf [#1318] engine: Change tombstone clear process
- Delete objects physically on tombstone's arrival;
- Store information about tombstones in the Graveyard;
- Clear Graveyard every epoch based on the information about TS in the
network.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy e4cfeec449 [#1318] service: Add tombstone checker service
The service fetches tombstones from the network via object service, every
request is handled in the following order:
1. checks local LRU cache;
2. checks local storage engine;
3. tries to find object in the placement nodes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy 2583f608e8 [#1318] metabase: Add `DropGraves` method
Method clean up Graveyard and should be called by the routine that handles
tombstone expiration.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy daab30c391 [#1318] metabase: Add iteration offset
Add offset element to the iterations over deleted objects (both the
Graveyard and the Garbage buckets).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy fe8076e60a [#1318] metabase: Add immediate object deletion
Mark objects with GC immediately after a Tombstone is received.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy 8107c8d1a9 [#1318] metabase: Separate buckets with TS and GC marks
It allows storing information about object in both ways at the same time:
1. Metabase should know if an object is covered by a tombstone (that is
not expired yet);
2. It should be possible to physically delete objects covered by a
tombstone immediately (mark with GC) but keep tombstone knowledge.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Evgenii Stratonikov 5736b834c3 [#1286] services/object: Validate object when created with sesssion token
All fields set by a user should be verified.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:53:22 +03:00
Evgenii Stratonikov c6a9c5cd8c [#1283] services/object: Disallow creating objects without a session token
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:53:22 +03:00
Pavel Karpy bd27837364 [#1321] node: Register GC event channel before shard init
Morph "NewEpoch" event handling was registered in a closure over
`addNewEpochNotificationHandler` func. That may lead to the data race:
if a shard was initialized before the event registration, everything works
as planned, but if registration was made earlier, it was not able to
include GC handlers since a shard has not called `eventChanInit` yet and,
therefore, it has not registered handler yet.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-25 10:39:37 +03:00
Evgenii Stratonikov 6c08ec9ca2 [#1312] morph/netmap: Fix panic during parameter parsing
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-12 17:37:12 +03:00
Pavel Karpy b6e5aac604 [#1170] pkg/morph: Delete unused function
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy 01e69f2f7a [#1170] pkg/morph: Add worker pool
Add worker pool to the listener to prevent blocking. It is used only for
notary notifications and new block events handling since it uses RPC
calls. That may lead to the deadlock state: neo-go cannot send RPC until
notification channel is read but notification channel cannot be read since
neo-go client cannot send RPC.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy 4ed5b1ceef [#1170] pkg/morph: Invalidate cache on RPC switch
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy 6508136204 [#1170] ir, node: Stop the app on WS connection lost
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy 2427593da6 [#1170] pkg/morph: Add inactive mode callback
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy 424cc6d495 [#1170] go.mod: Update `neo-go`
github.com/nspcc-dev/neo-go v0.98.0 => v0.98.2

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy fdf1338d65 [#1170] config: Delete notification endpoints
Also, change `http` to `ws` in examples.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy 3e45b4a085 [#1170] pkg/morph/subscriber: Adopt new WS client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy 402f488bec [#1170] pkg/morph: Change HTTP for WS client
Updated client now supports subscription to chain notifications and RPC
switch between provided RPC endpoints.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Evgenii Stratonikov 622ea4818f [#1250] *: Remove io/ioutil imports
It is deprecated starting from go1.16.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 17:23:28 +03:00
Evgenii Stratonikov e8bf18c0b4 [#1208] innerring: Fix race condition between handling new epoch and block
Before resetting the timer, ensure the block tick is processed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 11:52:24 +03:00
Evgenii Stratonikov 4770cb8bf6 [#1208] innerring: Disallow to tick timer twice on the same height
Provide current heights as an argument to ticker.
Zero height disables any checks, thus corresponding to the old
behaviour. If non-zero height is used, ignore the tick if the height
is less than the timer tick state.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 11:52:24 +03:00
Evgenii Stratonikov c0e65dadaf [#1292] morph/event: Make listener methods use pointer receiver
After `Listen` has raw receiver and calls `listen` which has pointer
receiver, leading to ineffectual assign to `started` field.
Always use `listener` by pointer, to avoid similar bugs.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 11:48:57 +03:00
Leonard Lyubich 629a4f79cd [#1293] metabase: Fix out-of-range panic in `freePotentialLocks`
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-05 15:38:51 +03:00
Alex Vanin e62e02815b [#1291] listener: Stop only once
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-05 14:46:40 +03:00
Alex Vanin 77f4a5844b [#1291] innerring: Don't stop the same listener twice
When IR node configured without main chain, both
`morphListener` and `mainnetListener` are pointing
into single listener component. We should not call
`Stop()` twice, because it may trigger channel
closing in neo-go or other components and it
can throw panic.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-05 14:46:40 +03:00
Evgenii Stratonikov 923db59149 [#1213] morph/client: Remember last succesfully used client
Prevent attemts to connect to the failing endpoint multiple times.
This speeds up all invocations a bit and removes excessive error logs.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov 9bbb136e4a [#1213] morph/client: Protect cached values with a mutex
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov a8d2001b35 [#1213] morph/client: Use a separate cache for every client
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov 61f60b8461 [#1213] morph/client: Unlock mutex to perform network requests
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov 48eb87d32c [#1149] writecache: Prevent corruption in `Head`
Also, remove optimization comments:
1. Having to maintain an execute the same logic for headers as for
   objects is quite inefficient, as it increases memory footprint.
2. Unmarshaling object is a cheap operation if data slice is in memory.
3. For unmarshaling header-only, I think we need SDK support.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:40:25 +03:00
Evgenii Stratonikov ff1912aa2a services/acl: check session token expiration epoch
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:37:29 +03:00
Evgenii Stratonikov 68903c9fd9 [#1143] shard: Support degraded mode in `Get` and `GetRange`
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov f058cead8f [#1143] shard: Handle some errors in degraded mode
If metabase is corrupted for some reason, failback to
checking blobstor directly.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov 08e7914729 [#1143] blobstor: Implement `existsSmall` check
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00