Commit graph

2322 commits

Author SHA1 Message Date
Evgenii Stratonikov
1e786233bf [#1559] local_object_storage: Provide readOnly flag to Open
We should be able to reopen storage in readonly in runtime.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Leonard Lyubich
e38b0aa4ba [#1549] engine: Disable shard on blobovnicza init failure
There is a need to support working w/o shard if it has problems with
blobovnicza tree.

Make `BlobStor.Init` to return new `ErrInitBlobovniczas` error. Remove
shard from storage engine's shard set if it returned this error from
`Init` call. So if some of the shards (but not all) return this error,
the node will be able to continue working without them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
40a56c6b42 [#1559] engine: Do not count logical errors as storage ones
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
c8911d72d0 [#1559] shard: Do not consult metabase in a degraded mode
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
339864b720 [#1559] local_object_storage: Move shard.Mode to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
d8ba954aff [#1559] shard: Use Set prefix for parameter setting
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
7b882b26d8 [#1559] shard: Remove public functions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
34d319fed2 [#1559] metabase: Use Set prefix for parameter setting
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
f58234aa2f [#1559] metabase: Remove public functions
Reduce public interface of this package. Later each result will contain
an additional status, so it makes more sense to use the same functions
and result processing everywhere.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
30c7925b3c [#1609] morph/client: Retry connecting to failed endpoint during the switch
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 16:08:42 +03:00
Evgenii Stratonikov
7410827db8 [#1609] config: Allow to prioritize N3 endpoints
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 16:08:42 +03:00
Pavel Karpy
581a9901c9 [#1614] metrics: Add health metrics
Also, rename metrics structure since it collects not only storage metrics
now.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-21 16:07:47 +03:00
Evgenii Stratonikov
b549cc314c [#1607] services/tree: allow to customize some parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
a4adb79db7 [#1607] pilorama: Enable tree service explicitly
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
d62723f038 [#1505] pilorama: Provide timeout to bbolt.Open
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
26041f18bf [#1505] pilorama: Allow to customize database parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f80e52fbea [#1334] services/tree: Document *.proto files
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
bfdd68dcb3 [#1333] services/control: Allow to synchronize local trees
Do not check that a node indeed belongs to the container, because the
synchronization will fail in this case anyway.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
c7b10598f9 [#1333] engine: Increase error counter for pilorama errors
1. Modifying operations are not expected to fail, unless the shard is
   read-only.
2. `Get*` operations should increase error counter too, unless the
   error is `ErrTreeNotFound`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
5e843a73f9 [#1333] services/control: Return pilorama info in ListShards RPC
Do not return backend type from the service for now, because memory
backend is expected to vanish.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8f4ee1aded [#1333] local_object_storage: Support ReadOnly mode in pilorama
The tricky part here is the engine itself: we stop iteration on
`ErrReadOnly` because it is better to synchronize the shard later than
to have partial trees stored in 2 shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
735931c842 [#1481] pilorama: Fix TreeApply
Current implementation prevents invalid operations to become valid at
some later point (consider adding a child to the non-existent parent and
then adding the parent). This seems to diverge from the paper algorithm
and complicates implementation. Make it simpler.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
5ffbeb76e6 [#1456] services/tree: Wait some time before reconnecting after failure
In case node is down or failing for some reason, we can expect `Dial` to
fail. In case we actively try to replicate and `Dial` always takes 2
seconds, replication-related channels quickly become full. That affects
latency of all other write operations.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
39f47f61c6 [#1445] services/tree: Cache the list of container nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
a2edfec0c3 [#1446] services/tree: Use grpc.WithInsecure only for nodes without TLS
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad3038d16d [#1444] pilorama: Fix TreeMove in bbolt backend
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
33d8fb187a [#1427] services/tree: Parallelize replicator
Before this commit the replication channel was quickly filled under
heavy load. This lead to the continuously increasing latency for all
write operations. Now it looks better.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8027b7bb6b [#1444] pilorama: Optimize internal encoding/decoding
```
name                      old time/op    new time/op    delta
ApplySequential/bbolt-8     55.5µs ± 4%    55.5µs ± 3%     ~     (p=1.000 n=10+7)
ApplyReorderLast/bbolt-8     108µs ± 6%     112µs ± 8%     ~     (p=0.077 n=9+9)

name                      old alloc/op   new alloc/op   delta
ApplySequential/bbolt-8     28.8kB ± 3%    27.7kB ± 6%   -3.79%  (p=0.005 n=10+10)
ApplyReorderLast/bbolt-8    41.4kB ± 5%    38.9kB ± 5%   -6.19%  (p=0.001 n=10+9)

name                      old allocs/op  new allocs/op  delta
ApplySequential/bbolt-8        262 ± 2%       235 ±10%  -10.41%  (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8       684 ± 6%       616 ± 7%  -10.04%  (p=0.000 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
b04f712773 [#1446] services/tree: Cache connections to the container nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
4437cd7113 [#1442] pilorama: Generate timestamp based on node position in the container
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3caa982283 [#1442] services/tree: Fix log message for failed Apply
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
01d2e06a9b [#1401] services/tree: Marshal public key once
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
1f5a650b05 [#1401] services/tree: Retransmit queries to container nodes
Also fix a bug with replicator using the multiaddress instead of
<host>:<port> format expected by gRPC library.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3312924b82 [#1431] pilorama: Use Batch for write transactions
Helps a lot in case of concurrent request flow.

```
name                      old time/op    new time/op    delta
ApplySequential/bbolt-8     78.0µs ± 9%    59.8µs ± 4%  -23.39%  (p=0.000 n=10+9)
ApplyReorderLast/bbolt-8     143µs ± 5%     113µs ±15%  -21.06%  (p=0.000 n=10+10)

name                      old alloc/op   new alloc/op   delta
ApplySequential/bbolt-8     56.9kB ± 8%    28.9kB ± 3%  -49.22%  (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8    87.3kB ± 3%    40.9kB ±10%  -53.16%  (p=0.000 n=10+10)

name                      old allocs/op  new allocs/op  delta
ApplySequential/bbolt-8        224 ±11%       262 ± 5%  +16.93%  (p=0.000 n=9+10)
ApplyReorderLast/bbolt-8       518 ± 4%       674 ±11%  +30.09%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f0a67f948d [#1431] pilorama: Cache attributes in the index
Currently to find a node by path we iterate over all the children on
each level. This is far from optimal and scales badly with the number of
nodes on a single level. Thus we introduce "indexed attributes" for
which an additional information is stored and which can be use in
`*ByPath` operations. Currently this set only includes `FileName`
attribute but this may change in future.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f5d35571d0 [#1431] engine: Add benchmark for Select vs TreeGetByPath
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f4bc1c601a [#1329] services/tree: Synchronize from the last stored height
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
835170e452 [#1329] pilorama: Allow to benchmark all tree backends
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
536857ea5a [#1329] services/tree: Implement GetOpLog RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
886baf3136 [#1426] services/tree: Do not replicate to a local node
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
7703dd5d7f [#1419] pilorama: Create new nodes in path if needed
Consider a node `{FileName: "dir", Attribute: "xxx"}`. In case we add
a new node by path `["dir", "file.txt"]`, create a new intermediate node
with a single attribute.

`GetByPath` now also considers only nodes with a single attribute while building a path.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad48918a97 [#1406] pilorama: Return parent from TreeGetMeta
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Denis Kirillov
b30f14978d [#1404] services/tree: Fix nodeId in GetSubTree
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
45321b3f6d [#1328] services/tree: Fix grpc import path
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
16e3421825 [#1328] services/tree: Implement access control
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
94b9e13431 [#1328] services/tree: Fix proto field numbers
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8844d9b2db [#1344] pilorama: Document errors for Get* methods
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
d8ad68d613 [#1344] engine: Log errors in Tree* operations
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
910db42748 [#1344] pilorama: Use require.ErrorIs in tests
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
aea855e8f3 [#1326] services/tree: Implement GetSubTree RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
62154da17c [#1324] services/tree: Implement Object Tree Service
Object Tree Service allows changing trees assotiated with
the container in runtime.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
46f4ce2773 [#1324] engine: Implement Forest interface for storage engine
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8cf71b7f1c [#1324] local_object_storage: Implement tree service backend
In this commit we implement algorithm for CRDT trees from
https://martin.klepmann.com/papers/move-op.pdf

Each tree is identified by the ID of a container it belongs to
and the tree name itself. Essentially, it is a sequence of operations
which should be applied in chronological order to get a usual tree
representation.

There are 2 backends for now: bbolt database and in-memory.
In-memory backend is here for debugging and will eventually act
as a memory-cache for the on-disk database.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
11c9df6f00 [#1611] shard: Print shard ID in component logs
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 15:57:53 +03:00
Pavel Karpy
9f7a22e2aa [#1561] object: Return OUT_OF_RANGE status
Replace `ErrRangeOutOfBounds` error from `pkg/core/object` package with
`ObjectOutOfRange` from `apistatus` package. That error is returned by
storage node's server as NeoFS API statuses.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-19 13:29:45 +03:00
Evgenii Stratonikov
4a8dfd0ce4 [#1606] Remove deprecated functions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 12:41:06 +03:00
Evgenii Stratonikov
6cc180391e [#1606] go.mod: Update neo-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 12:41:06 +03:00
Pavel Karpy
cb0bb7207c [#1461] shard: Add a separate ErrLockObjectRemoval
Do not return `meta.ErrLockObjectRemoval` from shard's methods, add shard's
own error for that.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
558cc1193a [#1461] engine: Clarify force removal
Document force removal behaviour in all the Storage engine parts.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
51afcc1182 [#1461] engine, policer: Force remove objects w/o container
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
18ec5d7c8e [#1461] meta: Return error on lock object removal
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
b5c56d459a [#1461] engine: Add force lock removal tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
63c00e785d [#1461] shard: Fix option naming
`WitDeletedLockCallback` => `WithDeletedLockCallback`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
5122be34e7 [#1461] shard: Add lock tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
eaf96bccf7 [#1461] meta: Add lock tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
c6cf8e5c0b [#1461] engine: Fix LOCK test
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
9c5ef3bab8 [#1461] node: Allow force LOCK removal
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
25826bd96d [#1461] node: Do not accept lock object w/o expiration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
fed9e6679d [#1461] node: Unlock locked object on its lock removal
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Evgenii Stratonikov
72c044e2eb [#1599] engine: Parallelize shard initialization
Shard is intended to be used as a separate failure domain,
which usually resides on a separate disk. Thus, sequential
initialization is bound by IO and this change speeds up thing a bit.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-15 13:44:54 +03:00
Pavel Karpy
a421344727 [#1605] node: Stream forwarder requests
Do not hold objects in memory in the forwarded Get/GetRange requests.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-15 11:16:35 +03:00
Pavel Karpy
89ebc278b8 [#1596] *: Remove .gitkeep files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-14 15:10:57 +03:00
Evgenii Stratonikov
c4b86cf1f1 [#1592] Remove debug builds
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-14 10:39:51 +03:00
Pavel Karpy
4a316ceae9 [#1365] morph: Do not return errors if config key is missing
Return default values instead of casting errors in `HomomorphicHashDisabled`
method.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
89118e9da0 [#1365] node: Calculate object homomorphic hash flexibly
Do not calculate and do not write homomorphic hash for containers that were
configured to store objects without hash.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
ae4740f99f [#1365] ir: Check homomorphic hash flexibly in audit
Do not perform that check if it was turned off for the container being
checked.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
8a61d33c65 [#1365] ir: Check homomorphic hash setting on ContainerPut
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
fa9c910648 [#1365] morph: Add HomomorphicHashDisabled config getter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
ae86d2907c [#1365] cli: Calculate homomorphic hash flexibly
Do not use homomorphic hash in storage group for containers that have
`homomorphic_hashing_disabled` set to `true`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
1e034c8d48 [#1532] node: Support signature status codes
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 18:22:59 +03:00
Pavel Karpy
5d5d21e3d0 [#1490] node: Require SG members to be unique
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 17:35:46 +03:00
Evgenii Stratonikov
0504c3e0c6 [#1266] object/acl: Check bearer token container ID
If the container ID is not nil and not equal to the container ID in the
request, consider bearer token invalid.

See also nspcc-dev/neofs-api#207.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-12 12:25:02 +03:00
Pavel Karpy
1658242e00 [#1590] node: Smart memory allocation in GetRange
Allocate memory only if a node chosen as the forwarded request receiver
has responded with a successful status.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-08 12:04:03 +03:00
Pavel Karpy
fed5e76e7f [#1590] node: Check status of forwarded requests
After fixing version fields in forwarded requests, a node does not check
statuses since errors are not covered by direct call error checks.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-08 12:04:03 +03:00
Leonard Lyubich
fdf62e8562 [#1586] Upgrade NeoFS SDK Go to rc#5
Error checkers now support wrapped errors so there is no need to
explicitly unwrap errors in `Policer`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-07 14:23:41 +03:00
Leonard Lyubich
9a11a75b77 [#1570] Upgrade NeoFS SDK Go with changed reputation API
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-06 18:21:24 +03:00
Pavel Karpy
9a6da336db [#1581] node: Do not lose API version on forwarding
Forwarded requests contained zero version in their meta header. It did not
allow responding with API statuses (`v0.0` version considered to be older
than `v2.11`) to the forwarding node and, therefore, did not allow analyzing
responses.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 17:06:24 +03:00
Pavel Karpy
c8506b247e [#1582] *: Fix linter warnings
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 11:56:01 +03:00
Pavel Karpy
a153e040cb [#1402] ir: Do not use map for combining SG ID and SG object
Also, add corresponding structure for that.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
ec07fda97b [#1402] pkg: Move SGSource to the core directory
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
27304455bf [#1402] ir: Filter expired SGs in the audit process
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
7864959d0c [#1402] ir: Move GetSG interface
`auditor` does not need to request SG: processor will fetch that info before
audit context initialization.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
6370c2e160 [#1402] ir: Make ClientCache not depend on audit.Task
That allows using `ClientCache` for storage group searching before task
context is initialized. Also, that makes it more general purpose.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
5f658c499c [#1402] ir: Collect SG from the object
Do not use `Marshal()` with object's payload. Use `ReadFromObject` func from
SDK instead. That allows checking both attributes and SG body's expiration
epoch.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Leonard Lyubich
c165d1a9b5 [#1556] Upgrade NeoFS SDK Go with changed container API
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-05 11:26:06 +03:00
Evgenii Stratonikov
f699e82ea7 [#1560] morph/client: Remove customFees type
It is private, is a simple map and there is no complex logic to be wrapped in methods.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-04 09:29:01 +03:00
Evgenii Stratonikov
0ccea802e9 [#1560] morph/client: Perform RPC switch and restore in one step
Otherwise we could switch infinitely if subscription restore has failed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-04 09:29:01 +03:00
Evgenii Stratonikov
ee27c66952 [#1555] util/gendoc: Remove trailing dot from the help message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-28 13:46:02 +03:00
Leonard Lyubich
305dd7598f [#1533] acl: Upgrade NeoFS SDK Go with refactored basic ACL
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-25 13:57:21 +03:00
Pavel Karpy
e812d78672 [#1537] node: Remove "trust" from non-reputation logs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-23 10:06:36 +03:00
Leonard Lyubich
b67974a8d3 [#xxx] Upgrade NeoFS SDK Go with changed container sessions
After recent changes in NeoFS SDK Go library session tokens aren't
embedded into `container.Container` and `eacl.Table` structures.

Group value, session token and signature in a structure for container
and eACL.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-22 16:38:57 +03:00
Evgenii Stratonikov
633b4e7d2d [#1483] metabase: Add VERSION.md
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
6f243a2a76 [#1483] metabase: Store version
The main problem is to distinguish the case of initial initialization
and update from version 0. We can't do this at `Open`, because of
`resync_metabase` flag. Thus, the following approach was taken:
1. During `Open` check whether the metabase was initialized.
2. Check for the version in `Init` or write the new one if the metabase
   is new.
3. Update version in `Reset`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
7df50297cd [#1520] shard: Ignore errors on metabase refill
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:25:28 +03:00
Evgenii Stratonikov
78ea450c25 [#1502] shard: Process locks on metabase refill
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 11:07:26 +03:00
Evgenii Stratonikov
972ca83e23 [#1524] writecache: Add some bolt parameters to the configuration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
07e06249d5 [#1524] metabase: Add some bolt parameters to the configuration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
0408acc40e services/control: Autogenerate stable marshalers
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 16:52:58 +03:00
Leonard Lyubich
13e74fce8a [#1513] ir/settlement: Do not allocate intermediate slice of nodes
After recent changes `buildContainer` method returns two-dimensional
slice of `NodeInfo` so there is no need to flatten it to build slice of
`common.NodeInfo`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
c408a6a0db [#1513] morph/netmap: Use constant states in unit tests
Use values of the node state enumeration from Netmap contract instead of
numeric literals.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
820acebb7d [#1513] morph/netmap: Return result by value from net config reader
Make `ReadNetworkConfiguration` method to return `NetworkConfiguration`
by value in order to follow storage engine improvements and prevent heap
escaping.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
808ba87e82 [#1513] morph/netmap: Use node state constants from Netmap contract
`Netmap` contract exports enumeration of the node states.

Replace using literals and constants from NeoFS API Go V2 with the
values provided by contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
21d2f8f861 [#1513] Upgrade NeoFS SDK Go with changed netmap package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Evgenii Stratonikov
81f925d5a0 [#1516] metabase: Optimize ListWithCursor for long listings
Cache buckets outside of the main loop and allocate memory for the
resulting offset only once.

```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       6.45µs ±14%    5.79µs ±11%  -10.24%  (p=0.002 n=10+10)
ListWithCursor/10_items-8     20.9µs ±17%    17.3µs ± 9%  -17.27%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     153µs ±12%     131µs ± 9%  -14.63%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       2.31kB ± 0%    1.91kB ± 0%  -17.46%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     6.94kB ± 0%    5.50kB ± 0%  -20.78%  (p=0.000 n=8+8)
ListWithCursor/100_items-8    53.3kB ± 0%    41.5kB ± 0%  -22.18%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         40.0 ± 0%      34.0 ± 0%  -15.00%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        121 ± 0%       100 ± 0%  -17.36%  (p=0.000 n=10+10)
ListWithCursor/100_items-8       930 ± 0%       758 ± 0%  -18.49%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
a93373fe71 [#1516] metabase: Cache graveyard buckets in ListWithCursor
```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       6.40µs ±13%    6.45µs ±14%     ~     (p=0.739 n=10+10)
ListWithCursor/10_items-8     30.9µs ±21%    20.9µs ±17%  -32.49%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     274µs ±27%     153µs ±12%  -44.09%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       2.26kB ± 0%    2.31kB ± 0%   +2.46%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     10.8kB ± 0%     6.9kB ± 0%  -36.07%  (p=0.000 n=8+8)
ListWithCursor/100_items-8    96.8kB ± 0%    53.3kB ± 0%  -44.98%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         39.0 ± 0%      40.0 ± 0%   +2.56%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        192 ± 0%       121 ± 0%  -36.98%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     1.72k ± 0%     0.93k ± 0%  -45.93%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>

name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       5.23µs ±19%    5.26µs ±15%     ~     (p=0.853 n=10+10)
ListWithCursor/10_items-8     27.2µs ±15%    18.0µs ±19%  -33.80%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     250µs ±13%     139µs ±15%  -44.27%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       1.99kB ± 0%    2.04kB ± 0%   +2.82%  (p=0.000 n=8+8)
ListWithCursor/10_items-8     10.3kB ± 0%     6.4kB ± 0%  -37.83%  (p=0.000 n=8+10)
ListWithCursor/100_items-8    93.9kB ± 0%    50.4kB ± 0%  -46.37%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         35.0 ± 0%      36.0 ± 0%   +2.86%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        184 ± 0%       113 ± 0%  -38.59%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     1.67k ± 0%     0.88k ± 0%  -47.29%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
af4db8a73b [#1516] metabase: Cache address key and do not decode address twice
```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       10.6µs ± 1%     6.4µs ±13%  -39.62%  (p=0.000 n=7+10)
ListWithCursor/10_items-8     75.3µs ± 2%    30.9µs ±21%  -58.97%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     726µs ± 2%     274µs ±27%  -62.28%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       3.19kB ± 0%    2.26kB ± 0%  -29.21%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     20.7kB ± 0%    10.8kB ± 0%  -47.68%  (p=0.000 n=10+8)
ListWithCursor/100_items-8     196kB ± 0%      97kB ± 0%  -50.65%  (p=0.000 n=7+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         55.0 ± 0%      39.0 ± 0%  -29.09%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        346 ± 0%       192 ± 0%  -44.51%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     3.25k ± 0%     1.72k ± 0%  -47.13%  (p=0.000 n=9+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
504f45e9ee [#1516] metabase: Add benchmark for ListWithCursor
```
name                        time/op
ListWithCursor/1_item-8     10.6µs ± 1%
ListWithCursor/10_items-8   75.3µs ± 2%
ListWithCursor/100_items-8   726µs ± 2%

name                        alloc/op
ListWithCursor/1_item-8     3.19kB ± 0%
ListWithCursor/10_items-8   20.7kB ± 0%
ListWithCursor/100_items-8   196kB ± 0%

name                        allocs/op
ListWithCursor/1_item-8       55.0 ± 0%
ListWithCursor/10_items-8      346 ± 0%
ListWithCursor/100_items-8   3.25k ± 0%
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Pavel Karpy
36f4929e52 [#1507] node: Do not handle object concurrently by the policer
Cache object that are being processed. That prevents concurrent
object handling when there is a few number of objects and object handling
takes more time that the policer needs for starting that object handling one
more time.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-15 20:43:32 +03:00
Pavel Karpy
256165045b [#1508] node: Do not replicate object twice
If placement contains two vectors with intersecting nodes it was possible to
send the object to the nodes twice.
Also optimizes requests: do not ask about storing the object twice from the
same node.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-15 20:33:04 +03:00
Pavel Karpy
df8a3807fe [#1508] node: Remove unused replicator code
The node does not support asynchronous object replication anymore, so it
does not need to have replicator worker, channel and `AddTask` function.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-15 20:33:04 +03:00
Leonard Lyubich
2e4a1cb6df [#1518] Upgrade NeoFS SDK Go with changed subnet package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-15 17:02:49 +03:00
Evgenii Stratonikov
f602d05b0a [#1494] *: Fix linter warnings
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
bbf8b8e74d [#1494] services/object: Do not ignore bearer token decode errors
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
795d1e0789 [#1494] go.mod: Update neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
9269ed344d [#1510] placement: Cache container nodes
In case we have lots of objects in a single container,
`GetContainerNodes` invoked indirectly by a policer can be seen in
pprof.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-14 20:17:43 +03:00
Pavel Karpy
8c77387982 [#1509] morph: Log WS client closing error
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-10 12:53:44 +03:00
Evgenii Stratonikov
38772e1a2e writecache: provide timeout to bbolt.Open
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-09 14:19:47 +03:00
Evgenii Stratonikov
91ed0d20ff [#1417] go.mod: Update neo-go to v0.99.0
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-08 10:27:49 +03:00
Leonard Lyubich
fa009db140 [#1464] ir/container: Fix verifying the operations within sessions
In previous implementation `verifySignature` method of container
processor worked incorrectly for operations without a key and with
session: processor tried to verify signature with one of the bound owner
keys instead of session one.

Use `VerifySessionDataSignature` method to check the signature if
session is used. Refactor `verifySignature` a bit with session check
highlighting for readability.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-07 23:26:30 +03:00
Leonard Lyubich
0937513c14 [#1485] ir/container: Accept eACL only if extension is allowed
In order to extend container ACL `F` bit must be set in basic ACL.

Make `Container` contract processor to deny eACL tables bound to
non-extendable containers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-07 16:34:15 +03:00
Evgenii Stratonikov
af7d15cc1a [#1487] innerring: Log errors in processors
Log errors for network operations. The only places where we are not
interested in errors are `Submit` in pool and unmarshaling.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-07 14:11:58 +03:00
Pavel Karpy
010253a97a [#1460] blobovnicza: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
0e4a1beecf [#1460] blobstor: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
08bf8a68f1 [#1460] engine: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
7b6363f4c6 [#1460] shard: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
9b2932609b [#1460] meta: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
a580429996 [#1460] meta: Add a benchmark on Get operation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
b0c7b7851a [#1418] blobstor: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
14366bbd89 [#1418] engine: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
5f57db6bf8 [#1418] shard: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
6d4d920bad [#1418] meta: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Leonard Lyubich
72708296cc Upgrade NeoFS SDK Go to v1.0.0-rc.4 and NeoFS API Go to v2.12.2
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 17:17:50 +03:00
Leonard Lyubich
0e28902b0f [#1471] eacl: Fix incorrect request denial with incomplete object header
Node shouldn't perform eACL verification during GET/HEAD request
processing until full object header is received. Otherwise, for some
eACL tables request may be falsely rejected.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Leonard Lyubich
c89035d544 [#1471] eacl: Add testcase with incomplete list of object headers
Scenario:
 * HEAD request of some object
 * 1st eACL record allows op for objects with specific user attribute
 * 2nd eACL record forbids op by object ID
 * node doesn't store the requested object locally

With this scenario node shouldn't deny request.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Leonard Lyubich
f9504c1cba [#1471] eacl: Don't process object headers twice in response processing
It is redundant to process object headers in responses w/o object field
since result will be the same.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Leonard Lyubich
8b94cb22d3 [#1471] eacl: Prevent aborting on local storage failure
Request processing should not be interrupted in case of local storage
failure since error case in normal for relay nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Leonard Lyubich
a97ea2aa21 [#1471] eacl: Add testcase with local storage failure
ACL service should not deny request on local storage failure since in
this case relay nodes won't be able to continue the operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Pavel Karpy
fdd54b0a03 [#1441] ir: Do not validate subnet removal
It is useless process since subnet owner is able to delete subnet without an
Alphabet approval. The Alphabet should only validate netmap state after
removal:
1. Update nodes' attributes if they were included in the deleted subnet;
2. Remove nodes without any subnet entrance.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 14:36:58 +03:00
Evgenii Stratonikov
feef9a98f7 [#1462] writecache: remove useless logs
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
2ae7c94cd6 [#1462] *: Remove log.With invocations
`log.With` is suitable during initialization, but in other places it induces
some overhead, even when branches with logging are not taken.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
43e776dfb1 [#1462] writecache: Persist objects in batches
`Update` becomes a botleneck under a heavy load.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
6ad87e7959 [#1462] writecache: Fill flush marks during startup
Some of the objects are already flushed, don't do it twice.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
f2a7503964 [#1462] fstree: Allow to fetch file content lazily
If we should process address based on some condition, there is no need
to read file content in memory.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
54d4503701 [#1462] writecache: Rename flushWorkersCount
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
0148209168 [#1462] writecache: Use Timer instead of Ticker in flush
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
5073a37930 [#1433] metabase: Optimize Select
For some filters we can scan only a subset of keys instead of checking
each key.

```
name                       old time/op  new time/op  delta
Select/string_equal-8      49.3µs ± 4%  11.0µs ± 4%  -77.68%  (p=0.000 n=10+10)
Select/string_not_equal-8  7.01ms ± 5%  7.06ms ±10%     ~     (p=0.971 n=10+10)
Select/common_prefix-8      118µs ± 6%    79µs ± 5%  -33.04%  (p=0.000 n=10+9)
Select/unknown-8           21.3µs ± 4%   3.2µs ± 4%  -84.88%  (p=0.000 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
5c48588c64 [#1433] metabase: Add more tests for Select
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
ea76c989ca [#1433] metabase: Add a benchmark for Select
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
b602fff01f [#1463] blobovnicza: Remove pointers from struct members
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 13:42:54 +03:00
Evgenii Stratonikov
d9f0ac8909 [#1463] blobovnicza: Determine db size via os.Stat
Currently we use `(*bbolt.Bucket).Stats().KeyN` for estimating database
size. However, it iterates over all pages in bucket and thus heavily
depends on the bucket size. This commit replaces initial size estimation
with a single `os.Stat` call.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 13:42:54 +03:00
Pavel Karpy
021aa97965 [#1418] blobovnicza: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
281befec67 [#1418] blobstor: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
babd382ba5 [#1418] engine: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
6e752f36dc [#1418] shard: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
e265ce2d52 [#1418] meta: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
1c100fb4b0 [#1418] writecache: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Evgenii Stratonikov
d84d52924a [#1467] pkg/util: Allow to load default account from a wallet
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-02 16:13:20 +03:00
Leonard Lyubich
ff576d125d [#1454] container: Fix reading session token from request origin
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Leonard Lyubich
1c30414a6c [#1454] Upgrade NeoFS SDK Go module with new IDs
Core changes:
 * avoid package-colliding variable naming
 * avoid using pointers to IDs where unnecessary
 * avoid using `idSDK` import alias pattern
 * use `EncodeToString` for protocol string calculation and `String` for
  printing

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Leonard Lyubich
cc6209e8a0 [#1449] object/delete: Fix NPE in collectChain
`head.previous` method returns (nil, nil) if object has no previous
neighbor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-30 17:59:16 +03:00
Leonard Lyubich
96cdc04705 [#1449] policer: Unwrap status HEAD response
Helper function `client.IsErrObjectNotFound` doesn't support error
unwrapping, so we need to do it on caller side.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-30 17:59:16 +03:00
Leonard Lyubich
e96ea4635c [#1449] policer: Fix selection of new storage candidates
`Policer` should pass list of selected candidates into `WithNodes`
method of `replicator.Task`. In previous implementation `processNodes`
method passed an opposite list: failed nodes and/or the local one.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-30 17:59:16 +03:00
Evgenii Stratonikov
855cbf5a3a [#1159] services/container: Remove ContainerWithContext struct
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-30 15:40:22 +03:00
Leonard Lyubich
2bcc0051ab [#1423] session: Get session issuer from token structure
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Leonard Lyubich
2dec3a0a93 [#1423] ir/container: Fix error message related to container removal
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Leonard Lyubich
883f2f1ac6 [#1423] local storage: Activate skipped unit tests after fix
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Leonard Lyubich
4c8ec20e32 [#1423] session: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Pavel Karpy
57c5fccc8c [#1428] node/acl: Make OID optional
Not all the NeoFS requests must contain OID in their bodies (or must NOT
contain them at all). Do not pass object address in helper functions, pass
CID and OID separately instead.
Also, fixed NPE in the ACL service: updated SDK library brought errors
when working with `Put` and `Search` requests without OID fields.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-25 12:11:03 +03:00
Pavel Karpy
d69eb2aaf3 [#1428] node/acl: Make container ID as required param
Change pointer to value in request information since requests could not
exist without container ID.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-25 12:11:03 +03:00
Evgenii Stratonikov
c57114def3 [#1422] morph/client: Fetch group key properly
NNS contract `resolve` method returns an array.
`neofs-adm` is already up to date.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-23 19:24:46 +03:00
Leonard Lyubich
86552cf3ae engine: make corruptSubDir to ignore blobovnicza dir
There is a dir for Blobovnicza tree in BlobStor, so we need to ignore
it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:33:14 +03:00
Leonard Lyubich
bb25ecbd15 [#1400] owner: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:33:14 +03:00
Leonard Lyubich
f8ac4632f8 [#1335] policer: Prevent potential object loss
In previous implementation `Policer` considered local object copy as
redundant on processing single placement vector.

Make `Policer` to call redundant copy callback after full placement
processing. Also fix 404 error parsing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:24:23 +03:00
Pavel Karpy
9f62d25b50 [#1224] ir: Delete nodes that do not have subnets
Also, update their subnet entrance on subnet removal events.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Pavel Karpy
b3e1e90c01 [#1224] ir: Listen to subnet removal events
IR must listen to subnet removal notifications always (regardless of notary
mode).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Pavel Karpy
71f18ba9ec [#1224] morph: Call Alphabet methods flexibly
Call `UpdateStateIR` and `AddPeerIR` method instead of `UpdateState` and
`AddPeer` if calling client is configured as Alphabet in notary enabled
environment.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Pavel Karpy
a129fc98da [#1224] morph: Add configuration getters
Includes `WithNotary` and `IsAlpha` methods to the `StaticClient`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Pavel Karpy
f0c9a68ad3 [#1224] morph: Delete wrong comment
Current implementation does not use MurmurHash3.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Evgenii Stratonikov
df48ddcb3d Revert "[#1141] morph/client: Allow to use more integer types as arguments"
This reverts commit 9349f422fd9a29ec529ef0648bbfabd1f9ff7651.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 16:45:15 +03:00
Evgenii Stratonikov
4761857fb3 [#1396] cmd: Allow to autogenerate documentation for all commands
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:35:22 +03:00
Pavel Karpy
4be5dce848 [#1405] ir: Fix settlement log message
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-20 12:40:12 +03:00
Evgenii Stratonikov
3f2889a4e9 [#1392] eacl/v2: Use EncodeToString where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
63cdc16088 [#1386] eacl/v2: Do not allocate intermediate slices for XHeaders
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
915f87959e [#1386] eacl/v2: Allocate []eaclSDK.Header slice immediately
There is no point in having intermediate `[]sessionSDK.XHeader`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
b24589b62d [#1386] eacl/v2: Use raw structs where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
3f4475f97b [#1386] services/object: Fail eACL check if the request is invalid
Parse all headers beforehand and reject invalid requests.
Another approach would be to remember the error and check
it after `CalculateAction`, which is a bit faster.
The rule of thumb here is "first validate, then use".

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Pavel Karpy
f99a0498da [#1403] util: Add details to errors
Make error messages more descriptive when parsing LOCODE from string errors
appear.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-20 10:53:27 +03:00
Leonard Lyubich
aeb9884218 [#1389] crypto: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-19 10:12:14 +03:00
Leonard Lyubich
5e50ddd7f5 [#1389] storagegroup: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-19 10:12:14 +03:00
Evgenii Stratonikov
0f376a5d83 [#1388] services/control: Extend tests for ShardInfo marshaling
It is nice to have different paths for different components and also
check that the information returned is different for different shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-17 13:48:07 +03:00
Leonard Lyubich
f15e6e888f [#1377] oid, cid: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-16 15:33:22 +03:00
Evgenii Stratonikov
f65898a354 [#1141] morph/client: Use boolean type directly
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:09:35 +03:00
Evgenii Stratonikov
230a5cd037 [#1141] morph/client: Remove unneeded argument conversions
Client supports `util.Uint160` directly, no need to convert it to bytes.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:09:35 +03:00
Evgenii Stratonikov
1e8391d216 [#1141] morph/client: Allow to use more integer types as arguments
Also, use `*big.Int` as integer value, see nspcc-dev/neo-go#2413.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:09:35 +03:00
Leonard Lyubich
3a188bb2e5 [#1371] bearer: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 15:16:17 +03:00
Leonard Lyubich
ae92074272 [#1369] checksum: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Leonard Lyubich
088df0e2a9 [#1369] version: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Leonard Lyubich
7a57d7b076 [#1369] audit: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Elizaveta Chichindaeva
cc7a723d77 [#1320] English Check
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-05-11 10:40:02 +03:00
Leonard Lyubich
cd545f0160 [#1351] cli: Fix connection scheme loss during endpoint parsing
In previous implementation NeoFS CLI app used `network.Address.HostAddr`
as a server URI, which caused scheme loss since host address doesn't
contain it.

Rename `HostAddr` to `URIAddr` and make it to return URI address with
`grpcs` scheme if TLS is enabled. Make `TLSEnabled` unexported since it
was used to provide default `tls.Config` only (it is used by default in
SDK).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-05 10:43:27 +03:00
Evgenii Stratonikov
a4769d8624 *: use require.ErrorIs where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:47:57 +03:00
Evgenii Stratonikov
3c39e6df11 [#1319] blobstor: Compress big objects in a streaming fashion
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov
1219ff89d4 [#1319] services/object_manager: Fix error message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov
057d53459b [#1319] services/object: Remove Write method from local and remote targets
They work with prepared objects only.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov
4ea03c01b5 [#1319] services/object: Store payload directly in distributedTarget
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Pavel Karpy
9da7df4d42 [#1318] metabase: Init metabase in test
`Init()` creates buckets. It prevents NPE in tests.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
486ec8a2e5 [#1318] metabase: Reuse comparing func from require
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
1caeb0b648 [#1318] metabase: Add missing error check in tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
6f2723b47e [#1318] node: Fix linter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
2aec5736e0 [#1318] node/gc: Fix tombstones handling
Handle only expired tombstones.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
24ca8ca5c7 [#1318] object: Change default ttl
This is need to reuse object service without passing any API v2 structs.
Also, fix comment.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
7799f8e4cf [#1318] engine: Change tombstone clear process
- Delete objects physically on tombstone's arrival;
- Store information about tombstones in the Graveyard;
- Clear Graveyard every epoch based on the information about TS in the
network.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
e4cfeec449 [#1318] service: Add tombstone checker service
The service fetches tombstones from the network via object service, every
request is handled in the following order:
1. checks local LRU cache;
2. checks local storage engine;
3. tries to find object in the placement nodes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
2583f608e8 [#1318] metabase: Add DropGraves method
Method clean up Graveyard and should be called by the routine that handles
tombstone expiration.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
daab30c391 [#1318] metabase: Add iteration offset
Add offset element to the iterations over deleted objects (both the
Graveyard and the Garbage buckets).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
fe8076e60a [#1318] metabase: Add immediate object deletion
Mark objects with GC immediately after a Tombstone is received.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
8107c8d1a9 [#1318] metabase: Separate buckets with TS and GC marks
It allows storing information about object in both ways at the same time:
1. Metabase should know if an object is covered by a tombstone (that is
not expired yet);
2. It should be possible to physically delete objects covered by a
tombstone immediately (mark with GC) but keep tombstone knowledge.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Evgenii Stratonikov
5736b834c3 [#1286] services/object: Validate object when created with sesssion token
All fields set by a user should be verified.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:53:22 +03:00
Evgenii Stratonikov
c6a9c5cd8c [#1283] services/object: Disallow creating objects without a session token
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:53:22 +03:00
Pavel Karpy
bd27837364 [#1321] node: Register GC event channel before shard init
Morph "NewEpoch" event handling was registered in a closure over
`addNewEpochNotificationHandler` func. That may lead to the data race:
if a shard was initialized before the event registration, everything works
as planned, but if registration was made earlier, it was not able to
include GC handlers since a shard has not called `eventChanInit` yet and,
therefore, it has not registered handler yet.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-25 10:39:37 +03:00
Evgenii Stratonikov
6c08ec9ca2 [#1312] morph/netmap: Fix panic during parameter parsing
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-12 17:37:12 +03:00
Pavel Karpy
b6e5aac604 [#1170] pkg/morph: Delete unused function
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
01e69f2f7a [#1170] pkg/morph: Add worker pool
Add worker pool to the listener to prevent blocking. It is used only for
notary notifications and new block events handling since it uses RPC
calls. That may lead to the deadlock state: neo-go cannot send RPC until
notification channel is read but notification channel cannot be read since
neo-go client cannot send RPC.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
4ed5b1ceef [#1170] pkg/morph: Invalidate cache on RPC switch
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
6508136204 [#1170] ir, node: Stop the app on WS connection lost
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
2427593da6 [#1170] pkg/morph: Add inactive mode callback
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
424cc6d495 [#1170] go.mod: Update neo-go
github.com/nspcc-dev/neo-go v0.98.0 => v0.98.2

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
fdf1338d65 [#1170] config: Delete notification endpoints
Also, change `http` to `ws` in examples.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
3e45b4a085 [#1170] pkg/morph/subscriber: Adopt new WS client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Pavel Karpy
402f488bec [#1170] pkg/morph: Change HTTP for WS client
Updated client now supports subscription to chain notifications and RPC
switch between provided RPC endpoints.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-08 11:52:18 +03:00
Evgenii Stratonikov
622ea4818f [#1250] *: Remove io/ioutil imports
It is deprecated starting from go1.16.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 17:23:28 +03:00
Evgenii Stratonikov
e8bf18c0b4 [#1208] innerring: Fix race condition between handling new epoch and block
Before resetting the timer, ensure the block tick is processed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 11:52:24 +03:00
Evgenii Stratonikov
4770cb8bf6 [#1208] innerring: Disallow to tick timer twice on the same height
Provide current heights as an argument to ticker.
Zero height disables any checks, thus corresponding to the old
behaviour. If non-zero height is used, ignore the tick if the height
is less than the timer tick state.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 11:52:24 +03:00
Evgenii Stratonikov
c0e65dadaf [#1292] morph/event: Make listener methods use pointer receiver
After `Listen` has raw receiver and calls `listen` which has pointer
receiver, leading to ineffectual assign to `started` field.
Always use `listener` by pointer, to avoid similar bugs.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-07 11:48:57 +03:00
Leonard Lyubich
629a4f79cd [#1293] metabase: Fix out-of-range panic in freePotentialLocks
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-05 15:38:51 +03:00
Alex Vanin
e62e02815b [#1291] listener: Stop only once
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-05 14:46:40 +03:00
Alex Vanin
77f4a5844b [#1291] innerring: Don't stop the same listener twice
When IR node configured without main chain, both
`morphListener` and `mainnetListener` are pointing
into single listener component. We should not call
`Stop()` twice, because it may trigger channel
closing in neo-go or other components and it
can throw panic.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-04-05 14:46:40 +03:00
Evgenii Stratonikov
923db59149 [#1213] morph/client: Remember last succesfully used client
Prevent attemts to connect to the failing endpoint multiple times.
This speeds up all invocations a bit and removes excessive error logs.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov
9bbb136e4a [#1213] morph/client: Protect cached values with a mutex
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov
a8d2001b35 [#1213] morph/client: Use a separate cache for every client
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov
61f60b8461 [#1213] morph/client: Unlock mutex to perform network requests
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-01 17:07:12 +03:00
Evgenii Stratonikov
48eb87d32c [#1149] writecache: Prevent corruption in Head
Also, remove optimization comments:
1. Having to maintain an execute the same logic for headers as for
   objects is quite inefficient, as it increases memory footprint.
2. Unmarshaling object is a cheap operation if data slice is in memory.
3. For unmarshaling header-only, I think we need SDK support.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:40:25 +03:00
Evgenii Stratonikov
ff1912aa2a services/acl: check session token expiration epoch
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:37:29 +03:00
Evgenii Stratonikov
68903c9fd9 [#1143] shard: Support degraded mode in Get and GetRange
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
f058cead8f [#1143] shard: Handle some errors in degraded mode
If metabase is corrupted for some reason, failback to
checking blobstor directly.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
08e7914729 [#1143] blobstor: Implement existsSmall check
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
aa0cc1f824 [#1143] blobovnicza: Copy object data in Get
Data returned from `*bbolt.Bucket.Get()` is only valid for the lifetime
of the transaction.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Evgenii Stratonikov
6472a170eb [#1143] shard: Introduce explicit Degraded mode
`Degraded` mode is set automatically after error counter is over the
threshold. `ReadOnly` mode can still be set by an administrator.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-31 15:33:22 +03:00
Alex Vanin
7ed84d1755 [#1278] acl: Return netmap.Source interface
Application can provide cached netmap source in this case.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-30 14:22:12 +03:00
Evgenii Stratonikov
cf119e4ca9 [#1163] services/audit: Randomize the order of PDP checks
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-30 10:56:33 +03:00
Evgenii Stratonikov
882236a03b [#1239] morph/client: Remove intermediate conversion in morph client
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-29 16:02:05 +03:00
Evgenii Stratonikov
6936195afa [#1239] util/attributes: Remove excessive slice copy during parsing
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-29 16:02:05 +03:00
Evgenii Stratonikov
2ad8016d75 [#1239] innerring: Use pointer-less slices for object IDs
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-29 16:02:05 +03:00
Pavel Karpy
6ec104d686 [#1255] node/session: Rename constant
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
90a8c52bdb [#1255] object: Add persistent storage usage
Use persistent storage usage in the node if it was configured so.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
016eaa25f3 [#1255] node/session: Add encryption tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
01ed366e99 [#1255] node/session: Add encryption
Add `WithEncryption` option that passes ECDSA key to the persistent session
storage. It uses 32 bytes from marshalled ECDSA key in ASN.1 DER from in
AES-256 algorithm encryption in Galois/Counter Mode.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
a884ad56d9 [#1255] node/session: Add persistent tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
455b9fb325 [#1255] node/session: Add persistent session storage
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Pavel Karpy
929c9851a6 [#1255] node/session: Create separate dir for in-memory storage
Move in-memory session storage to the separate directory of `storage`. It is
done for future support of different kind of session storages.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-29 09:35:10 +03:00
Evgenii Stratonikov
2a69aaf976 [#1157] network/cache: Optimize client fetch from multiClient
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 18:11:20 +03:00
Evgenii Stratonikov
a4261243fc [#1157] network/cache: Cache multiclients based on public key only
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 18:11:20 +03:00
Evgenii Stratonikov
de5a2f6574 [#1262] metabase: Remove list index in place Delete
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
43867a3093 [#1262] metabase: Do not allocate intermediate slices for indices
```
name              old alloc/op   new alloc/op   delta
Put/parallel-8       123kB ± 4%     119kB ± 3%  -2.72%  (p=0.006 n=10+9)
Put/sequential-8     170kB ± 1%     168kB ± 1%  -1.42%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op  delta
Put/parallel-8         473 ± 1%       469 ± 0%  -0.87%  (p=0.000 n=10+10)
Put/sequential-8       792 ± 0%       787 ± 0%  -0.58%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
0e9b6be3fd [#1262] metabase: Remove intermediate allocations in decodeList
```
name              old time/op    new time/op    delta
Put/parallel-8      1.57ms ±11%    1.51ms ± 3%   -4.06%  (p=0.043 n=9+10)
Put/sequential-8    5.16ms ± 2%    5.16ms ± 3%     ~     (p=1.000 n=9+10)

name              old alloc/op   new alloc/op   delta
Put/parallel-8       126kB ± 4%     123kB ± 4%   -2.54%  (p=0.016 n=8+10)
Put/sequential-8     171kB ± 1%     170kB ± 1%     ~     (p=0.182 n=9+10)

name              old allocs/op  new allocs/op  delta
Put/parallel-8         565 ± 2%       473 ± 1%  -16.18%  (p=0.000 n=9+10)
Put/sequential-8       819 ± 1%       792 ± 0%   -3.34%  (p=0.000 n=9+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
d45df614fb [#1262] metabase: Optimize decodeList
Prevent additional allocation during `append` in `Put`.

```
name              old alloc/op   new alloc/op   delta
Put/parallel-8       131kB ± 1%     126kB ± 4%  -3.87%  (p=0.005 n=8+8)
Put/sequential-8     172kB ± 1%     171kB ± 1%  -0.73%  (p=0.028 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
456e1584d6 [#1262] metabase: Add benchmarks for Put
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
f1223b46df [#1262] blobovnicza: Make helper in Put function idempotent
`Batch` can execute the function multiple times leading to multiple
increases of a size approximation.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-28 17:08:11 +03:00
Evgenii Stratonikov
ad92493b86 [#1268] blobstor: Cleanup zstd encoders/decoders
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-25 14:05:06 +03:00
Alex Vanin
7e06d0aa69 [#1253] ir: Call UpdateStateIR method to remove dead storage node
Alphabet nodes in notary enabled environment cannot call `UpdateState`
method to remove unwanted storage nodes from the network map,
because this method checks witness of the storage node.

To force storage node state update, alphabet nodes should invoke
new method `UpdateStateIR` which is similar to `AddPeerIR`.

State update initiated by the storage node itself is processed
the same way as before -- alphabet nods resign such transaction.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-24 11:52:19 +03:00
Alex Vanin
e81081e0e0 [#1253] ir: Update AddPeer method name for notary enabled env
`Register` was renamed to `AddPeerIR` for consistency with
`UpdateState` changes in
https://github.com/nspcc-dev/neofs-contract/pull/227

This is protocol breaking change for notary enabled environment.
Luckily, there is no notary enabled environments anywhere except
of neofs-dev-env, so we can do such thing. We should avoid such
changes in the future, though.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-24 11:52:19 +03:00
Evgenii Stratonikov
414ba6e0a2 [#1244] nats: Split client creation into 2 stages
Create and connect to an endpoint using separate functions.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-24 11:51:49 +03:00
Evgenii Stratonikov
e2062013cf network: remove unused constants
Fix linter complaints. These constants are unused after
nspcc-dev/neofs-node#1232.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-22 11:55:19 +03:00
Evgenii Stratonikov
2848001dfb [#1246] object/acl: Return more concise description for eACL errors
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-21 19:20:01 +03:00
Leonard Lyubich
318639e5bf [#1247] *: Clarify docs about returned errors from apistatus package
`apistatus` package provides types which implement build-in `error`
interface. Add `error of type` pattern when documenting these errors in
order to clarify how these errors should be handled (e.g. `errors.Is` is
not good).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
459bdcf04b [#1247] object/acl: Return ObjectAccessDenied status error
Return `apistatus.ObjectAccessDenied` error on access violation from ACL
service. Write reason in format of the errors from the previous
implementation. These errors are returned by storage node's server as
NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
967650f2ed [#1247] container: Return ContainerNotFound status error
Replace `core/container.ErrNotFound` error returned by `Source.Get`
interface method with `apistatus.ContainerNotFound` status error. This
error is returned by storage node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
70ffdf3478 [#1247] object: Return NOT_FOUND and ALREADY_REMOVED statuses
Replace `ErrNotFound`/`ErrAlreadyRemoved` error from
`pkg/core/object` package with `ObjectNotFound`/`ObjectAlreadyRemoved`
one from `apistatus` package. These errors are returned by storage
node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
f32c9670ad [#1247] object: Return status errors from util.KeyStorage
Return `SessionTokenExpired`/`SessionTokenNotFound` error from
`apistatus` package if private session token is expired/missing. These
errors are returned by storage node's server as NeoFS API statuses.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Leonard Lyubich
a4ee59977e [#1247] session: Remove unused storage.ErrNotFound error
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-17 16:34:00 +03:00
Evgenii Stratonikov
737d40f2d9 [#1152] network: Reuse URI parsing function from api-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-17 16:33:08 +03:00
Pavel Karpy
16543a1904 [#1243] node/object: Hide engine behind interface
Do not use engine as explicit dependency in the put service. This allows
wrapping storage engine.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 14:27:52 +03:00
Pavel Karpy
d02720c910 [#1243] nats: Fix comments
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-17 14:27:52 +03:00
Pavel Karpy
c3db12d71b [#1210] reputation: Resolve race condition
Make all epoch independent in reputation process. Do not reset any timers
related to reputation. Make it possible to finish iteration after the
unexpected `NewEpoch` event.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-16 17:18:30 +03:00
Pavel Karpy
13af4e6046 [#1210] reputation: Improve debug logs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-16 17:18:30 +03:00
Alex Vanin
0adb29c035 [#1214] Fix metabase put logging
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 17:17:18 +03:00
Alex Vanin
362cda53d2 [#1238] Call String() explicitly instead of using stringer pointer
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Alex Vanin
8f476f3c4d [#1238] Adopt neofs-node for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Alex Vanin
9fad29dfe0 [#1238] Adopt neofs-ir for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Alex Vanin
7f3195b197 [#1238] Adopt neofs-cli for non pointer slices in SDK
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-16 15:29:31 +03:00
Leonard Lyubich
98c84670e3 [#1175] core/object: Check emptiness of LOCK members
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
9feb5f9405 [#1175] engine: Fix AddShard implementation
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
9dff07200c [#1181] local storage: Process expired locks similar to tombstones
There is a need to process expired `LOCK` objects similar to `TOMBSTONE`
ones: we collect them on `Shard`, notify all other shards about
expiration so they could unlock the objects, and only after that mark
lockers as garbage.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
ebd84f6dd4 [#1175] object/put: Perform additional container broadcast of LOCKs
`LOCK` objects should be broadcast across container like `TOMBSTONE`
ones.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
323dea95c6 [#1181] Do not use deprecated elements
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
e3f4e48d87 [#1175] Fix typo in error return docs
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
8a2a096680 [#1175] services/sign: Convert error to status return
In previous implementation `SignService` converted all `error` values to
`INTERNAL` server failure status. That was done for simplification only.
There is a need to transmit status errors as corresponding status
messages.

Make `SignService` to unwrap errors and convert them to status message
during writing to the response. Non-status errors are converted to
`INTERNAL` server failures. Status errors can also be wrapped in the
depths of the executable code, so `SignService` tries to unwrap them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
ced854bc2e [#1175] object/fmt: Handle LOCK objects
Make `FormatValidator.ValidateContent` to verify payload of `LOCK`
objects. Pass locked objects to `Locker` interface. Require from
`Locker.Lock` to return `apistatus.IrregularObjectLock` error on a
corresponding condition.

Also add error return to `DeleteHandler.DeleteObjects` method. Require
from method to return `apistatus.ObjectLocked` error on a corresponding
condition. Adopt implementations.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
d163008b63 [#1175] storage engine: Fix Lock implementation
`Lock` must return `apistatus.IrregularObjectLock` error instead of
`apistatus.ObjectLocked` one.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
07107c8bfd [#1175] metabase: Update locker list during Inhume
`Inhume` operation can potentially mark lockers as garbage. There is a
need to update locker list in locked bucket.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
26b8e33ed7 [#1175] metabase: Fix encoding of updated locker list
Even single locker must be encoded using `encodeList`, or subsequent
`decodeList` call will fail.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
88cefa7c5f [#1175] storage engine: Return status error on Delete of locked object
Make `StorageEngine.Delete` to forward first encountered
`apistatus.ObjectLocked` error during shard processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
97b6ec8150 [#1175] storage engine: Return status error on Inhume of locked object
Make `StorageEngine.Inhume` to forward first encountered
`apistatus.ObjectLocked` error during shard processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
545861d1bc [#1175] storage engine: Implement Lock operation
Implement `StorageEngine.Lock` method which works similar to `Inhume`
but calls `Lock` on the processing shards.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
6ed85ff1e1 [#1175] shard: Implement Lock operation
Implement `Shard.Lock` method which required rw mode and calls `Lock` on
underlying metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
e21eedee06 [#1175] shard: Return status error on Inhume of locked object
Make `Shard.Inhume` to forward `apistatus.ObjectLocked` returned by
underlying metabase.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
b585791d6e [#1175] metabase: Return status error on Lock of irregular object
Make `DB.Lock` to return `apistatus.IrregularObjectLock` if at least one
of the locked objects is irregular (not of type REGULAR).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
23fcacd3f2 [#1175] metabase: Return status error on Inhume of locked object
Make `DB.Inhume` to return `apistatus.ObjectLocked` if at least one of
the inhumed objects is locked.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
303eb2e078 [#1176] metabase: Ignore locked objs in IterateCoveredByTombstones
Make `DB.IterateCoveredByTombstones` to not pass locked objects to the
handler. The method is used by GC, therefore it will not consider locked
objects as candidates for deletion even if their tombstone is expired.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
dba3e58dc5 [#1176] metabase: Ignore locked objects in IterateExpired
Make `DB.IterateExpired` to not pass locked objects to the handler. The
method is used by GC, therefore it will not consider them as candidates
for deletion.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
9508633a7e [#1175] metabase: Work with LOCK objects
After introduction of LOCK objects (of type `TypeLock`) complicated
extended its behavior:
  * create `lockers` container bucket (LCB) during PUT;
  * remove object from LCB during DELETE;
  * look up object in LCB during EXISTS;
  * get object from LCB during GET;
  * list objects from LCB during LIST with cursor;
  * select objects from LCB during SELECT with '*'.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
9f13674a10 [#1175] metabase: Implement LOCK operation
Implement `DB.Lock` method which marks list of the objects as locked by
another object. Only regular objects can be locked.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
14d27455f3 [#1175] metabase: Add container bucket for LOCKs
Create class of container buckets with `LOCKED` suffix. Put identifiers
of the objects of type `LOCK` to these buckets during `DB.Put`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Leonard Lyubich
e90ac9c6f9 [#1181] metabase: Remove unused CleanUp method
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-15 13:03:23 +03:00
Alex Vanin
b6720d5f97 [#1231] Update new SDK Client interface
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-14 13:18:48 +03:00
Alex Vanin
e38d7dda6b [#1230] replicator: Improve error log
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-10 19:19:18 +03:00
Leonard Lyubich
14ea7b2c47 [#1219] object/delete: Change approach of the tombstone broadcast
After `putsvc.Service` started to support additional container broadcast
of the saved objects there is no more need to perform broadcast of
tombstone object in `deletesvc.Service`.

Make `putsvc.Service` to perform additional broadcast of `TOMBSTONE`
objects. Remove `broadcastTombstone` stage from `deletesvc.execCtx`,
from now it is encapsulated in `saveTombstone` stage. Remove no longer
needed `putsvc.PutInitPrm.WithTraverseOption` method.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-05 16:23:21 +03:00
Leonard Lyubich
a55af18ad1 [#1219] object/put: Support additional container broadcast
There are several cases when we need to spread the object around the
container after its primary placement (e.g. objects of type TOMBSTONE).
It'd be convenient to support this feature in `putsvc.Service`.

Add additional stage of container broadcast after the object is stored.
This stage is carried out no more than once and does not affect the
outcome of the main PUT operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-05 16:23:21 +03:00
Evgenii Stratonikov
0bf59522f7 [#1191] object/acl: check session token verb
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-05 16:03:16 +03:00
Pavel Karpy
79c6f52a27 [#1183] node: Add NATS client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 15:57:24 +03:00
Pavel Karpy
03b601b594 [#1183] node: Init notificator on startup
It uses logger as a notification writer. NATS messaging will be implemented
later.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 11:41:18 +03:00
Pavel Karpy
90ff066940 [#1183] node/notificator: Implement notificator
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 11:41:18 +03:00
Pavel Karpy
cf17686321 [#1111] control: Rename aliaces
`objectSDKAddress` -> `addressSDK`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 10:06:14 +03:00
Pavel Karpy
99b31e3235 [#1111] object/acl: Refactor service
Make all operations that related to `neofs-api-go` library be placed in `v2`
packages. They parse all v2-versioned structs info `neofs-sdk-go`
abstractions and pass them to the corresponding `acl`/`eacl` packages. `v2`
packages are the only packages that do import `neofs-api-go` library. `eacl`
and `acl` provide public functions that only accepts `sdk` structures.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-05 10:06:14 +03:00
Leonard Lyubich
7ccd1625af [#1214] *: Use single Object type in whole project
Remove `Object` and `RawObject` types from `pkg/core/object` package.
Use `Object` type from NeoFS SDK Go library everywhere. Avoid using the
deprecated elements.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-04 17:45:00 +03:00
Evgenii Stratonikov
19ad349b27 [#1204] shard: Save ID in the metabase
`AddShard` must return shard id, so we temporarily open metabase
there.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 17:13:13 +03:00
Evgenii Stratonikov
1fe9cd4d36 [#1186] engine: Allow to skip metabase in GetRange
Similarly to `Get`. Also fix a bug where `ErrNotFound` is returned
instead of `ErrRangeOutOfBounds`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
2b5550ccf6 [#1186] blobstor: Fix comment
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
b33fb0f739 [#1186] blobstor: Unify errors for Get and GetRangeBig
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
69e1e6ca20 [#1186] engine: Read object directly from blobstor in case of conflicts
Metabase is expected to contain actual information about objects stored
in shard. If the object is present in metabase but is missing from
blobstor, peform an additional attempt to fetch it directly without
consulting metabase. Such a situation is unexpected, so error counter
is increased for the shard which has the object in the metabase. We
don't increase error counter for the shard which has the object in
blobstor, because some garbage can be expected there. In this
implementation there is no overhead for objects which are really
missing, i.e. are not present in any metabase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 16:07:25 +03:00
Evgenii Stratonikov
6e6f3648d2 [#1188] metabase: log PUT / DELETE operations
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 15:21:35 +03:00
Evgenii Stratonikov
ce76325840 [#1146] engine: remove TODO and add a comment
Parallel check seems to be slower, so remove TODO.
```
name \ time/op      old          new           pool
Exists/2_shards-40  52.3µs ±23%   91.6µs ± 6%   84.2µs ± 1%
Exists/4_shards-40  72.7µs ±11%  121.8µs ± 8%  116.9µs ± 5%

name \ alloc/op     old          new           pool
Exists/2_shards-40  5.00kB ± 0%   5.03kB ± 0%   5.16kB ± 0%
Exists/4_shards-40  9.89kB ± 0%   9.93kB ± 1%  10.19kB ± 0%

name \ allocs/op    old          new           pool
Exists/2_shards-40     112 ± 1%      115 ± 1%      117 ± 1%
Exists/4_shards-40     207 ± 1%      211 ± 1%      216 ± 1%
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 12:31:41 +03:00
Evgenii Stratonikov
dab6bcabe4 [#1146] engine: Add benchmarks for exists
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-04 12:31:41 +03:00
Pavel Karpy
0e0a675f35 [#1212] morph: Fix NPE in multi client
Share multi cache between single clients.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-03-04 12:16:11 +03:00
Alex Vanin
cbc816d57d [#1211] node: Fix object get init error check
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Alex Vanin
bc45a79d1f [#1211] innerring: Fix object get init error check
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Alex Vanin
ca17eb0eb4 [#1211] innerring: Fix objectIDs list reader
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-03-04 11:47:24 +03:00
Leonard Lyubich
b5bdcfc076 [#1194] client: Support request X-headers
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-03-03 15:38:38 +03:00
Evgenii Stratonikov
e10b8f53d6 [#1151] morph/client: Cache notary transaction heights
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-03-01 10:35:54 +03:00
Leonard Lyubich
e0dce1043a [#1195] Adopt recent changes in NeoFS SDK
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-02-28 10:43:03 +03:00
ZhangTao1596
dd0e10d306 [#1180] ir/internal: Fix audit range hash type
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2022-02-14 12:36:02 +03:00
Evgenii Stratonikov
47cd10a4d7 [#767] services/object: replace duplicate link in TODO
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:59:27 +03:00
Evgenii Stratonikov
e21d054fe2 [#749] morph/client: set group signer scope
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:26:27 +03:00
Evgenii Stratonikov
fa947b85a6 [#749] morph/client: reuse TestInvokePrm in InvokePrm
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:26:27 +03:00
Evgenii Stratonikov
9e2f7ac371 [#749] morph/client: allow to fetch contract group key
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:26:27 +03:00
Evgenii Stratonikov
4838d3bb80 [#749] morph/client: cache NNS contract hash
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 13:26:27 +03:00
Evgenii Stratonikov
050a4bb2b0 [#1115] *: link TODOs to corresponding issues
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
692790a899 [#1115] services/object: reuse errors from core service
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
4caa330ddc [#1115] morph/client: reuse errors from container contract
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
f9d3111825 [#1115] blobovnicza: remove GetRange method
It is unused and incorrect as described in FIXME.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
c1530dec5e [#1115] shard/gc: provide parsed addresses in HandleExpiredTombstones
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-11 12:58:59 +03:00
Evgenii Stratonikov
0695ec4125 [#1133] services/session: remove expired tokens every epoch
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 14:58:32 +03:00
Pavel Karpy
1667ec9e6d [#1131] *: Adopt SDK changes
`object.Address` has been moved to `object/address`
`object.ID` has been moved to `object/id`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-02-08 09:45:38 +03:00
Evgenii Stratonikov
1c821d6c36 [#625] morph/client: remove unused interface
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
c34cfa1f35 [#625] client/container: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
819d80a7a9 [#625] client/audit: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
767ee5c0cd [#625] client/reputation: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
97d18bc515 [#625] client/netmap: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
8c5c3ac9e8 [#625] client/neofsid: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
8474abb911 [#625] client/neofs: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
6f50fefbea [#625] client/balance: remove intermediate wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-08 09:43:54 +03:00
Evgenii Stratonikov
ed50cf6207 [#1118] services/control: return error counter in ListShards
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Evgenii Stratonikov
7fb15fa1d0 [#1118] services/control: allow to reset error counter in SetShardMode
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Evgenii Stratonikov
6ad2624552 [#1118] engine: allow to set error threshold
There are certain errors which are not expected during usual node
operation and which tell us that something is wrong with the shard.
To prevent possible data corruption, move shard in read-only mode after
amount of errors exceeded some threshold. By default no actions are performed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 15:14:27 +03:00
Leonard Lyubich
4f3323f084 [#1101] *: Adopt interface changes of API client from SDK library
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-02-03 14:00:14 +03:00
Evgenii Stratonikov
674f520da7 [#1132] *: Use path/filepath package when working with files
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-03 10:12:07 +03:00
Evgenii Stratonikov
0decb95591 [#625] morph/client: make all constructors return no error
In some places we panic, in some we return error, in some (audit) just return a client.
However in all of the places static client is created immediately before
the sugared-client creation.

This commit makes all constructors to just return a client for the sake
of code simplification and unification.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-01 15:47:54 +03:00
Evgenii Stratonikov
3c5b62d839 [#625] morph/client: make method names constant
We don't use custom names and the only place where custom method option
is used it provides the default name and can be omitted.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-02-01 15:47:54 +03:00
Evgenii Stratonikov
476528361e [#1023] *: fix linter errors
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 13:01:58 +03:00
Evgenii Stratonikov
eb99b65134 [#1023] morph/event: remove notaryRequest from subnet.Delete
Subnet removal is done by owner.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 13:01:58 +03:00
Evgenii Stratonikov
f96bb8d3b8 [#1023] innerring/subnet: fix subnet removal event processing
Caught by `unused` linter.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 13:01:58 +03:00
Evgenii Stratonikov
a82ac0edcd [#1023] services/object: remove TTL from Get parameters
It was unused an should be provided as a `CallOption` instead.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 13:01:58 +03:00
Evgenii Stratonikov
5a03a14940 [#1094] metabase: move Select filters sanity check outside of transaction
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
651adf46c6 [#1094] blobovnicza: calculate size in Init properly
If pre-existing blobovnicza is initialized, it's size should be updated
even if all buckets are in place.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
418b553920 [#1094] shard: remove TODO for shard mode description
It was added back in 2fb379b7 when we had many shard modes. Now we have
only two and comments for constants are rather descriptive.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
ed7c732676 [#1094] shard: unify collection of expired objects
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
fbb95cff14 [#1094] services/object: remove some FIXME comments
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
1bdbd6ed22 [#1094] eacl/v2: preallocate slice in headersFromObject
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:40 +03:00
Evgenii Stratonikov
18cfd8b042 [#1086] services/control: implement RestoreShard RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
2ec4a3c897 [#1086] neofs-cli: add command for shard dump
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
0e60b1d6c9 [#1086] services/control: implement DumpShard RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Evgenii Stratonikov
aa53418119 [#1086] services/control: fix comments
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-31 12:36:14 +03:00
Leonard Lyubich
20b3ff84b3 [#1110] ir/container: Fix check of SetEACL operation
In previous implementation IR incorrectly verified `SetEACL` event of
`Container` contract. The incorrect behavior could be reproduced in two
ways:
  1. Create container using session, and perform `SetEACL` operation
  with a key that is different from the session one.
  2. Create container using session, and perform `SetEACL` w/o a
  session, but sign it using session key from the `Put` operation.

The problem was in the `checkSetEACL` validation method of IR container
processor. It always used session token used for container creation
during session ownership check.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-27 16:37:24 +03:00
Evgenii Stratonikov
adb80bebb2 [#1085] shard: rename Evacuate to Dump
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
d06425c852 [#1085] shard: allow to ignore errors in Evacuate
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
36eebb5932 [#1085] writecache: allow to ignore errors during iteration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
cd75638ce3 [#1085] blobstor: allow to ignore errors during iteration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
c954f0e71b [#1085] fstree: allow to ignore errors during iteration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
e53ad2f468 [#1085] blobovnicza: allow to ignore errors during iteration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
9c60ab893c [#1085] shard: allow to ignore errors during restore
We could also ignore errors during evacuate, but this requires
unmarshaling objects first which slowers the process considerably.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
0ef3d5ab03 [#1085] writecache: persist in-memory objects before shutdown
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
3215210d0c [#1085] shard: Close components in reverse order
Write-cache should be closed before blobstor and metabase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
b432ec3a03 [#1085] shard/test: add object integrity check for Evacuate
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
c0f6c988f0 [#1085] shard: dump data from write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
ad01aaf8bf [#1085] writecache: add read-only mode
In read-only mode modifying operations are immediately returned with
error and all background operations are suspended.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
9f963e001b [#1085] writecache: set metaOnly flag for receiving from priority channel
This can lead to objects being flushed twice.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Evgenii Stratonikov
7eedb23eb7 [#1085] shard: allow to dump/restore objects
Dump contains magic and a list of objects prefixed by object size in bytes.
We can't use proto-marshaled list because this requires having all dump
in memory. Using TAR induces 512 byte overhead for each object which can
be a problem in some cases.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-26 15:28:13 +03:00
Leonard Lyubich
87a45983a1 [#1084] accounting: Keep balance precision from contract
Do not convert balance response of Balance contract to Fixed8.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-26 14:10:02 +03:00
Pavel Karpy
fee2f5a330 [#1100] *: Adopt new SDK's owner.ID API
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 17:58:30 +03:00
Leonard Lyubich
4b5404047c [#1098] object/search: Fix error loss in SignService.Search
Error loss led to incorrect response formation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-21 14:54:22 +03:00
Pavel Karpy
202dbfb95d [#1096] eacl: Rename eacl import aliases
Since `Validator` has been moved to SDK it is more readable to see `eaclSDK`
than `acl` import.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 10:47:33 +03:00
Pavel Karpy
ed156cd738 [#1096] eacl: Use validator from SDK
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-21 10:47:33 +03:00
ZhangTao1596
ac82899e85 [#1088] writecache: Fix small flushed when persist
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2022-01-19 12:55:33 +03:00
Pavel Karpy
7c02a2e251 [#1087] *: Adopt SDK changes
- Update `neofs-sdk-go`:
v0.0.0-20211230072947-1fe37df88f80 => v0.0.0-20220113123743-7f3162110659

- Add client interface that duplicates SDK's client behaviour and new
`MultiAddressClient` interface that has method that iterates over wrapped
clients.

- Also start using simple client mode that does not require parsing statuses
outside the SDK library.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-01-14 17:29:03 +03:00
Evgenii Stratonikov
486d5c2e86 [#1060] writecache: compress big object if needed
Small objects use `blobstor.Put`, so no changes are required.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-13 14:20:36 +03:00
Evgenii Stratonikov
0d969d7a06 [#1060] blobstor: allow to disable compression based on content-type
For some data compression makes little sense, as it is already compressed.
This commit allows to leave such data unchanged based on `Content-Type`
attribute. Currently exact, prefix and suffix matching are supported.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-13 14:20:36 +03:00
Alex Vanin
0f1eb743af [#1077] service/object: Don't override status of payload overtake
Reverse payload overtake triggers direct payload overtake that
sets status and error. We should not override that.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-01-13 10:30:18 +03:00
Evgenii Stratonikov
5828f43e52 [#851] util/rand: use single random source
It is much more convenient to skip source creation.
Also fix some bugs:
1. `cryptoSource.Int63()` now returns number in [0, 1<<63) as required
   by `rand.Source` interface.
2. Replace `cryptoSource.Uint63()` with `cryptoSource.Uint64` to allow
   generate uint64 numbers directly (see rand.Source64 docs).

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-12 19:29:27 +03:00
Evgenii Stratonikov
c35cdb3684 [#1041] autocomplete: fetch output stream from provided command
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-12 18:33:48 +03:00
Evgenii Stratonikov
c01acba3ce [#1041] autocomplete: beautify help message
Indent sections corresponding to different shells for faster navigation
and remove `#` at the line beginning.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-12 18:33:48 +03:00
Evgenii Stratonikov
efabffbfd0 [#1041] cmd/*: factor out autocomplet command generation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-01-12 18:33:48 +03:00
Leonard Lyubich
fe29080ebb [#1025] ir: Read IR list from role management contract w/o mainchain
If mainchain is disabled in IR config then the node should read inner
ring list via role management contract.

Use `NeoFSAlphabetList` method of morph client as IR lister if
`withoutMainNet` flag is set in IR app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-12 18:30:36 +03:00
Leonard Lyubich
0b0c61f4a5 [#1063] node/object: Do not skip empty search responses in splitter
In previous implementation `Search` method of transport splitter skipped
responses with empty ID list.

Replace while-loop with do-while one in `TransportSplitter.Search`
method implementation in order to send responses with empty result too.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-01-11 12:02:54 +03:00
ZhangTao1596
6830d54a3c [#1069] morph/balance: Fix transfer to
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2022-01-11 12:02:11 +03:00
Pavel Karpy
97fd330fae [#1059] control: Fix ListShards wrapper name
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
77808c7b41 [#1059] control: Provide shard's real mode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
375394dc99 [#1059] shard: Add shard mode to shard Info
Provide shard mode information via `DumpInfo()`. Delete atomic field from
Shard structure since it duplicates new field.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
88bb8e2df9 [#1059] control: Add SetShardMode implementation to ctrl svc
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
fa78a37787 [#1059] control: Add unit test for SetShardMode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
29c69f37eb [#1059] control: Add SetShardMode RPC wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
25665cb0fd [#1059] control: Add marshalling and setters to SetShardMode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
4533784e11 [#1059] control: Recompile proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
41dd385b7b [#1059] control: Add SetShardMode rpc to proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-30 14:14:48 +03:00
Pavel Karpy
93bd6be743 [#1057] shard: Do not allow memory change operations in "read-only"
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-29 18:06:15 +03:00
Pavel Karpy
4f756bf121 [#1057] shard: Make shard to have only two mode
Shard's mode was not used in the Node, so added only two modes whose roles
are clear. More modes will be added in the future.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-29 18:06:15 +03:00
Evgenii Stratonikov
e96eb3e00b [#1054] neofs-node: add epoch metric
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-28 15:34:20 +03:00
Evgenii Stratonikov
e1137aa09f [#1054] innerring: add epoch metric
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-28 15:34:20 +03:00
Pavel Karpy
876b0c53de [#1048] control: Rename Blobstore to Blobstor
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
f5a9735e1c [#1048] cli: Add shard list operation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
284188f8f9 [#1048] node: Fill shard's info with its components' infos
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
0e5410603e [#1048] control: Add ListShards implementation to ctrl svc
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
4e989e7133 [#1048] control: Add unit test for ListShards
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
f88a12eaa7 [#1048] control: Add ListShards RPC wrappers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
a6b3e16975 [#1048] control: Add marshalling and setters to ListShards
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
a42b3d37f6 [#1048] control: Recompile proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Pavel Karpy
20f11c88ca [#1048] control: Add ListShards rpc to proto files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-28 13:21:44 +03:00
Alex Vanin
5d46035ae8 [#1052] Tidy INFO logs
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-27 14:28:01 +03:00
Alex Vanin
e610d1ea5f [#1045] Provide sanity check of session token in container service
Without sanity check, container service provides successful response,
even though such request will never be approved by Alphabet nodes.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-20 14:51:07 +03:00
Alex Vanin
bca7cf9470 [#1047] policer: Check context before job selection
When application is being terminated, replicator routine
might be on the object picking phase. Storage is terminated
asynchronously, thus `Select()` may return corresponding
error. If we don't process `context.Done()` in this case,
then application freezes on shutdown.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-12-20 14:42:57 +03:00
Pavel Karpy
c49e53ba9d [#1031] morph: Add maxConnPerHost option
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 18:55:53 +03:00
Pavel Karpy
df0d76dc19 [#1032] node/container/router: Do not check nil route points
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 16:02:29 +03:00
Pavel Karpy
fd9514f90e [#1032] node/container/router: Fix interface description
After #872 `NextStage` method description becomes outdated.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-16 16:02:29 +03:00
ZhangTao1596
1be0c42726 [#1022] writecache: Fix db counter decrement and flushed big object
Signed-off-by: ZhangTao1596 <zhangtao@ngd.neo.org>
2021-12-15 14:23:49 +03:00
Leonard Lyubich
fc42844fbf [#1028] writecache: Evict flushed big objects
Make `flushBigObjects` routine to call `evictObjects` which was
successfully written to `BlobStor`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-14 13:49:47 +03:00
Leonard Lyubich
d246e5193b [#1028] writecache: Mark big objects as flushed after write
Make `flushBigObjects` routine to mark objects which are written to
`BlobStor`. This prevents already flushed objects from being written on
the next iterator tick.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-14 13:49:47 +03:00
Leonard Lyubich
90fd883e32 [#915] blobovnicza: Use mean bucket size multiplier instead of min
For fullness estimation of `Blobovnicza` we use number of object stored
in each size bucket. In previous implementation we multiplied the number
by the difference in bucket boundaries. This expression rather
estimated the minimum volume (and for the smallest bucket, the maximum)
of objects in the bucket.

Multiply number of objects by mean bucket size.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-08 14:24:49 +03:00
Leonard Lyubich
2920c5203b [#915] blobovnicza: Re-use Bolt transaction in syncFullnessCounter
Make `syncFullnessCounter` to accept `bbolt.Tx` argument of Bolt
transaction within which counter should be synchronized. Pass
corresponding transaction during `Init`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-08 14:24:49 +03:00
Leonard Lyubich
feb0a65efb [#1008] ir/container: Customize fee for named container registration
In notary disabled environment, approval of container creation with nice
name attribute takes much more additional GAS than other operations
(due to NNS invocation).

Morph library changes:
  * add the ability to specify per-op fees using `StaticClient` options;
  * add the ability to customize fee for `Put` operation with named
    container in container morph client.

Inner Ring changes:
  * add `fee.named_container_register` config value which specifies
    additional GAS fee for the approvals of the named container
    registrations;
  * pass the config value to `WithCustomFeeForNamedPut` option of
    container morph client.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-08 13:57:01 +03:00
Leonard Lyubich
f9bcb6f5e3 [#1007] metabase: Copy BoltDB value returned by IsSmall
According to BoltDB documentation bucket `value is only valid for the
life of the transaction`.

Make `DB.IsSmall` copy value slice in order to prevent potential memory
corruptions (e.g. `runtime.stringtobyteslice` cast).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-07 18:59:42 +03:00
Leonard Lyubich
3d5eb2a396 [#1008] ir/netmap: Call Register method on peer approval
After https://github.com/nspcc-dev/neofs-contract/issues/154 alphabet
nodes should call `Register` method for approval of the notary
notifications spawned by `AddPeer` method.

Call `register` method for peer approval in Netmap processor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-07 17:15:44 +03:00
Evgenii Stratonikov
02be6c83a6 morph/client: update morph container wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2021-12-02 14:16:23 +03:00
Leonard Lyubich
973e50ad72 [#979] ir/subnet: Register parser and handler of RemoveNode event
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 19:27:16 +03:00
Leonard Lyubich
6f41cd86ed [#979] Upgrade SDK NeoFS Go dependency
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
3550ed9fe4 [#979] ir/subnet: Fix notification event names
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
98e0792b08 [#979] adm/subnet: Add commands to add/remove nodes
Implement `ManageNodes` operation on morph subnet client.
 Add `node add` and `node remove` commands to `subnet` section.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
ef57ba3fec [#979] ir/subnet: Fix notary event name of subnet creation
Notary event name equals to the name of the method which throws the
event.

Define name const of notary subnet creation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
ca2c6ad77f [#979] morph/client: Return zero values from CalculateNonceAndVUB
Make `CalculateNonceAndVUB` to return zero values if notary isn't
enabled.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
4b8ca75274 [#979] adm: Fix problems after testing
Use persistent flags on parent command in order to inherit flags in
sub-commands. Turn on notary mode of morph client in `subnet` command of
admin utility for notary environments.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
069a174129 [#979] morph/client: Get proxy contract address from NNS
Make `Client.EnableNotarySupport` method to call `NNSContractAddress`
for proxy contract if it is not specified in corresponding option.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Leonard Lyubich
e8f8e58e90 [#979] adm: Add subnet-related commands
Add `subnet` command which contains all subnet-related commands. Add
sub-commands:
  * `create` for creation;
  * `remove` for removal;
  * `get` for reading;
  * `admin` for admin management;
  * `client` for client management.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-12-01 17:31:04 +03:00
Pavel Karpy
1cee1b8f93 [#1002] ir: Add subnet check to the container Put process
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-01 16:26:15 +03:00
Pavel Karpy
2245bf85d8 [#1002] morph: Add UserAllowed method to subnetwork client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-01 16:26:15 +03:00
Pavel Karpy
5b8f195563 [#1000] ir/nodeValidators: Ignore zero subnet validation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-12-01 12:27:35 +03:00
Pavel Karpy
acf7331034 [#990] object: Fix linter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 17:18:08 +03:00
Pavel Karpy
8988ac2729 [#990] ir/nodeValidators: Add subnet entrance node Validation on AddPeer call
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 17:18:08 +03:00
Pavel Karpy
a079a8f727 [#990] nodeValidators: Add subnet entrance validator
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 17:18:08 +03:00
Pavel Karpy
b6db699b6e [#990] morph/client: Add NodeAllowed subnet method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 17:18:08 +03:00
Leonard Lyubich
b27c72c02a [#973] morph/subnet: Fix docs and comments of notification events
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-28 16:15:18 +03:00
Leonard Lyubich
52fc5bde6e [#973] ir/subnet: Remove alphabet re-signature of Delete request
Subnet contract doesn't work with alphabet signatures.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-28 16:15:18 +03:00
Leonard Lyubich
41eaa1e246 [#973] ir: Listen and process Put/Delete events of Subnet contract
Define notification events, implement parsers. Add morph client of
Subnet contract. Listen, verify and approve events in Inner Ring app.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-28 16:15:18 +03:00
Pavel Karpy
214c2bd0cb [#987] ir: Add RemoveNode parser and handler
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 14:38:44 +03:00
Pavel Karpy
81dc17718e [#987] ir: Add parsing subnet script hash from NNS contract
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 14:38:44 +03:00
Pavel Karpy
0c6cdd0afd [#987] morph/event: Add RemoveNode event and its parser
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2021-11-28 14:38:44 +03:00
Alex Vanin
011d0f605b [#965] replicator: Make HandleTask function public
Continues replication executed in separate pool of goroutines,
so there is no need in worker to handle replication tasks
asynchronously.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-26 15:39:38 +03:00
Alex Vanin
a74a402a7d [#965] policer: Implement continuous replication
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-26 15:39:38 +03:00
Alex Vanin
20f0b29a6e [#975] morph/event: Update notary preparator tests
Now includes testcases for notary
requests with 3 and 4 witnesses.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-26 15:37:17 +03:00
Alex Vanin
a0ff5b1bf8 [#975] morph/client: Use 4 witnesses in notary request
Add invoker witness as the third witness. Required for
netmap methods checks.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-26 15:37:17 +03:00
Alex Vanin
6f23dbfefe [#971] innerring: Use VUB pointer for initial committee vote
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-25 18:09:41 +03:00
Leonard Lyubich
7f5fb130c0 [#961] *: Support NeoFS API status returns
Upgrade NeoFS API Go library to version with status returns. Make all API
clients to pull out and return errors from failed statuses. Make signature
service to respond with status if client version supports it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-24 09:26:40 +03:00
Leonard Lyubich
44d0d453ef [#961] client: Clarify error returns in docs
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2021-11-24 09:26:40 +03:00
Alex Vanin
1f911830a5 [#948] engine: Define EndOfListing error in component packages
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
c80c83b0b8 [#948] metabase: Simplify cursor structure
Use bbolt bucket keys to seek offset in root bucket
and container buckets.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
468caa83d9 [#948] Adopt sdk-go client
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
08bdd0d561 [#948] engine: Fix comments of object listing with cursor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
aa9ce8a853 [#948] engine: Define cursor for object listing as a type
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
164cd10af8 [#948] engine: Hide shard IDs in ListWithCursor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
0f6d8f6eea [#948] metabase: Use seek for faster cursor listing
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
8d471c7e36 [#948] core/object: Move listing objects error to core
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
c02c7bee5b [#948] engine: Add ListWithCursor method
Interface for shard.ListWithCursor invocations.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00
Alex Vanin
5b6be7bc1c [#948] engine: Fix test file path
Before: `<t.Name()>.<id>.blobstor`
After:  `<t.Name()>/<id>.blobstor`
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2021-11-19 16:57:19 +03:00