fyrchik/frostfs-node
1
0
Fork 0
forked from TrueCloudLab/frostfs-node
Code Issues Pull requests Projects Releases Packages Wiki Activity
3019 commits 18 branches 26 tags 27 MiB
Commit graph

2001 commits

Author SHA1 Message Date
Evgenii Stratonikov
735931c842 [#1481] pilorama: Fix TreeApply 
Current implementation prevents invalid operations to become valid at
some later point (consider adding a child to the non-existent parent and
then adding the parent). This seems to diverge from the paper algorithm
and complicates implementation. Make it simpler.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
5ffbeb76e6 [#1456] services/tree: Wait some time before reconnecting after failure 
In case node is down or failing for some reason, we can expect `Dial` to
fail. In case we actively try to replicate and `Dial` always takes 2
seconds, replication-related channels quickly become full. That affects
latency of all other write operations.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
39f47f61c6 [#1445] services/tree: Cache the list of container nodes 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
a2edfec0c3 [#1446] services/tree: Use grpc.WithInsecure only for nodes without TLS 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad3038d16d [#1444] pilorama: Fix TreeMove in bbolt backend 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
33d8fb187a [#1427] services/tree: Parallelize replicator 
Before this commit the replication channel was quickly filled under
heavy load. This lead to the continuously increasing latency for all
write operations. Now it looks better.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8027b7bb6b [#1444] pilorama: Optimize internal encoding/decoding 
```
name                      old time/op    new time/op    delta
ApplySequential/bbolt-8     55.5µs ± 4%    55.5µs ± 3%     ~     (p=1.000 n=10+7)
ApplyReorderLast/bbolt-8     108µs ± 6%     112µs ± 8%     ~     (p=0.077 n=9+9)

name                      old alloc/op   new alloc/op   delta
ApplySequential/bbolt-8     28.8kB ± 3%    27.7kB ± 6%   -3.79%  (p=0.005 n=10+10)
ApplyReorderLast/bbolt-8    41.4kB ± 5%    38.9kB ± 5%   -6.19%  (p=0.001 n=10+9)

name                      old allocs/op  new allocs/op  delta
ApplySequential/bbolt-8        262 ± 2%       235 ±10%  -10.41%  (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8       684 ± 6%       616 ± 7%  -10.04%  (p=0.000 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
b04f712773 [#1446] services/tree: Cache connections to the container nodes 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
4437cd7113 [#1442] pilorama: Generate timestamp based on node position in the container 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3caa982283 [#1442] services/tree: Fix log message for failed Apply 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
01d2e06a9b [#1401] services/tree: Marshal public key once 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
1f5a650b05 [#1401] services/tree: Retransmit queries to container nodes 
Also fix a bug with replicator using the multiaddress instead of
<host>:<port> format expected by gRPC library.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3312924b82 [#1431] pilorama: Use Batch for write transactions 
Helps a lot in case of concurrent request flow.

```
name                      old time/op    new time/op    delta
ApplySequential/bbolt-8     78.0µs ± 9%    59.8µs ± 4%  -23.39%  (p=0.000 n=10+9)
ApplyReorderLast/bbolt-8     143µs ± 5%     113µs ±15%  -21.06%  (p=0.000 n=10+10)

name                      old alloc/op   new alloc/op   delta
ApplySequential/bbolt-8     56.9kB ± 8%    28.9kB ± 3%  -49.22%  (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8    87.3kB ± 3%    40.9kB ±10%  -53.16%  (p=0.000 n=10+10)

name                      old allocs/op  new allocs/op  delta
ApplySequential/bbolt-8        224 ±11%       262 ± 5%  +16.93%  (p=0.000 n=9+10)
ApplyReorderLast/bbolt-8       518 ± 4%       674 ±11%  +30.09%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f0a67f948d [#1431] pilorama: Cache attributes in the index 
Currently to find a node by path we iterate over all the children on
each level. This is far from optimal and scales badly with the number of
nodes on a single level. Thus we introduce "indexed attributes" for
which an additional information is stored and which can be use in
`*ByPath` operations. Currently this set only includes `FileName`
attribute but this may change in future.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f5d35571d0 [#1431] engine: Add benchmark for Select vs TreeGetByPath 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f4bc1c601a [#1329] services/tree: Synchronize from the last stored height 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
835170e452 [#1329] pilorama: Allow to benchmark all tree backends 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
536857ea5a [#1329] services/tree: Implement GetOpLog RPC 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
886baf3136 [#1426] services/tree: Do not replicate to a local node 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
7703dd5d7f [#1419] pilorama: Create new nodes in path if needed 
Consider a node `{FileName: "dir", Attribute: "xxx"}`. In case we add
a new node by path `["dir", "file.txt"]`, create a new intermediate node
with a single attribute.

`GetByPath` now also considers only nodes with a single attribute while building a path.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad48918a97 [#1406] pilorama: Return parent from TreeGetMeta 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Denis Kirillov
b30f14978d [#1404] services/tree: Fix nodeId in GetSubTree 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
45321b3f6d [#1328] services/tree: Fix grpc import path 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
16e3421825 [#1328] services/tree: Implement access control 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
94b9e13431 [#1328] services/tree: Fix proto field numbers 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8844d9b2db [#1344] pilorama: Document errors for Get* methods 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
d8ad68d613 [#1344] engine: Log errors in Tree* operations 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
910db42748 [#1344] pilorama: Use require.ErrorIs in tests 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
aea855e8f3 [#1326] services/tree: Implement GetSubTree RPC 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
62154da17c [#1324] services/tree: Implement Object Tree Service 
Object Tree Service allows changing trees assotiated with
the container in runtime.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
46f4ce2773 [#1324] engine: Implement Forest interface for storage engine 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8cf71b7f1c [#1324] local_object_storage: Implement tree service backend 
In this commit we implement algorithm for CRDT trees from
https://martin.klepmann.com/papers/move-op.pdf

Each tree is identified by the ID of a container it belongs to
and the tree name itself. Essentially, it is a sequence of operations
which should be applied in chronological order to get a usual tree
representation.

There are 2 backends for now: bbolt database and in-memory.
In-memory backend is here for debugging and will eventually act
as a memory-cache for the on-disk database.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
11c9df6f00 [#1611] shard: Print shard ID in component logs 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-19 15:57:53 +03:00
Pavel Karpy
9f7a22e2aa [#1561] object: Return OUT_OF_RANGE status 
Replace `ErrRangeOutOfBounds` error from `pkg/core/object` package with
`ObjectOutOfRange` from `apistatus` package. That error is returned by
storage node's server as NeoFS API statuses.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-19 13:29:45 +03:00
Evgenii Stratonikov
4a8dfd0ce4 [#1606] Remove deprecated functions 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-18 12:41:06 +03:00
Evgenii Stratonikov
6cc180391e [#1606] go.mod: Update neo-go 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-18 12:41:06 +03:00
Pavel Karpy
cb0bb7207c [#1461] shard: Add a separate ErrLockObjectRemoval 
Do not return `meta.ErrLockObjectRemoval` from shard's methods, add shard's
own error for that.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
558cc1193a [#1461] engine: Clarify force removal 
Document force removal behaviour in all the Storage engine parts.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
51afcc1182 [#1461] engine, policer: Force remove objects w/o container 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
18ec5d7c8e [#1461] meta: Return error on lock object removal 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
b5c56d459a [#1461] engine: Add force lock removal tests 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
63c00e785d [#1461] shard: Fix option naming 
`WitDeletedLockCallback` => `WithDeletedLockCallback`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
5122be34e7 [#1461] shard: Add lock tests 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
eaf96bccf7 [#1461] meta: Add lock tests 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
c6cf8e5c0b [#1461] engine: Fix LOCK test 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
9c5ef3bab8 [#1461] node: Allow force LOCK removal 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
25826bd96d [#1461] node: Do not accept lock object w/o expiration 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Pavel Karpy
fed9e6679d [#1461] node: Unlock locked object on its lock removal 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-18 11:42:25 +03:00
Evgenii Stratonikov
72c044e2eb [#1599] engine: Parallelize shard initialization 
Shard is intended to be used as a separate failure domain,
which usually resides on a separate disk. Thus, sequential
initialization is bound by IO and this change speeds up thing a bit.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-15 13:44:54 +03:00
Pavel Karpy
a421344727 [#1605] node: Stream forwarder requests 
Do not hold objects in memory in the forwarded Get/GetRange requests.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-15 11:16:35 +03:00
Pavel Karpy
89ebc278b8 [#1596] *: Remove .gitkeep files 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-14 15:10:57 +03:00
Evgenii Stratonikov
c4b86cf1f1 [#1592] Remove debug builds 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-14 10:39:51 +03:00
Pavel Karpy
4a316ceae9 [#1365] morph: Do not return errors if config key is missing 
Return default values instead of casting errors in `HomomorphicHashDisabled`
method.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-13 17:10:06 +03:00
Pavel Karpy
89118e9da0 [#1365] node: Calculate object homomorphic hash flexibly 
Do not calculate and do not write homomorphic hash for containers that were
configured to store objects without hash.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-13 17:10:06 +03:00
Pavel Karpy
ae4740f99f [#1365] ir: Check homomorphic hash flexibly in audit 
Do not perform that check if it was turned off for the container being
checked.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-13 17:10:06 +03:00
Pavel Karpy
8a61d33c65 [#1365] ir: Check homomorphic hash setting on ContainerPut 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-13 17:10:06 +03:00
Pavel Karpy
fa9c910648 [#1365] morph: Add HomomorphicHashDisabled config getter 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-13 17:10:06 +03:00
Pavel Karpy
ae86d2907c [#1365] cli: Calculate homomorphic hash flexibly 
Do not use homomorphic hash in storage group for containers that have
`homomorphic_hashing_disabled` set to `true`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-13 17:10:06 +03:00
Pavel Karpy
1e034c8d48 [#1532] node: Support signature status codes 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-12 18:22:59 +03:00
Pavel Karpy
5d5d21e3d0 [#1490] node: Require SG members to be unique 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-12 17:35:46 +03:00
Evgenii Stratonikov
0504c3e0c6 [#1266] object/acl: Check bearer token container ID 
If the container ID is not nil and not equal to the container ID in the
request, consider bearer token invalid.

See also nspcc-dev/neofs-api#207.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-12 12:25:02 +03:00
Pavel Karpy
1658242e00 [#1590] node: Smart memory allocation in GetRange 
Allocate memory only if a node chosen as the forwarded request receiver
has responded with a successful status.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-08 12:04:03 +03:00
Pavel Karpy
fed5e76e7f [#1590] node: Check status of forwarded requests 
After fixing version fields in forwarded requests, a node does not check
statuses since errors are not covered by direct call error checks.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-08 12:04:03 +03:00
Leonard Lyubich
fdf62e8562 [#1586] Upgrade NeoFS SDK Go to rc#5 
Error checkers now support wrapped errors so there is no need to
explicitly unwrap errors in `Policer`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-07-07 14:23:41 +03:00
Leonard Lyubich
9a11a75b77 [#1570] Upgrade NeoFS SDK Go with changed reputation API 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-07-06 18:21:24 +03:00
Pavel Karpy
9a6da336db [#1581] node: Do not lose API version on forwarding 
Forwarded requests contained zero version in their meta header. It did not
allow responding with API statuses (`v0.0` version considered to be older
than `v2.11`) to the forwarding node and, therefore, did not allow analyzing
responses.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-06 17:06:24 +03:00
Pavel Karpy
c8506b247e [#1582] *: Fix linter warnings 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-06 11:56:01 +03:00
Pavel Karpy
a153e040cb [#1402] ir: Do not use map for combining SG ID and SG object 
Also, add corresponding structure for that.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-06 09:56:26 +03:00
Pavel Karpy
ec07fda97b [#1402] pkg: Move SGSource to the core directory 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-06 09:56:26 +03:00
Pavel Karpy
27304455bf [#1402] ir: Filter expired SGs in the audit process 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-06 09:56:26 +03:00
Pavel Karpy
7864959d0c [#1402] ir: Move GetSG interface 
`auditor` does not need to request SG: processor will fetch that info before
audit context initialization.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-06 09:56:26 +03:00
Pavel Karpy
6370c2e160 [#1402] ir: Make ClientCache not depend on audit.Task 
That allows using `ClientCache` for storage group searching before task
context is initialized. Also, that makes it more general purpose.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-06 09:56:26 +03:00
Pavel Karpy
5f658c499c [#1402] ir: Collect SG from the object 
Do not use `Marshal()` with object's payload. Use `ReadFromObject` func from
SDK instead. That allows checking both attributes and SG body's expiration
epoch.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-06 09:56:26 +03:00
Leonard Lyubich
c165d1a9b5 [#1556] Upgrade NeoFS SDK Go with changed container API 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-07-05 11:26:06 +03:00
Evgenii Stratonikov
f699e82ea7 [#1560] morph/client: Remove customFees type 
It is private, is a simple map and there is no complex logic to be wrapped in methods.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-04 09:29:01 +03:00
Evgenii Stratonikov
0ccea802e9 [#1560] morph/client: Perform RPC switch and restore in one step 
Otherwise we could switch infinitely if subscription restore has failed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-04 09:29:01 +03:00
Evgenii Stratonikov
ee27c66952 [#1555] util/gendoc: Remove trailing dot from the help message 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-28 13:46:02 +03:00
Leonard Lyubich
305dd7598f [#1533] acl: Upgrade NeoFS SDK Go with refactored basic ACL 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-25 13:57:21 +03:00
Pavel Karpy
e812d78672 [#1537] node: Remove "trust" from non-reputation logs 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-23 10:06:36 +03:00
Leonard Lyubich
b67974a8d3 [#xxx] Upgrade NeoFS SDK Go with changed container sessions 
After recent changes in NeoFS SDK Go library session tokens aren't
embedded into `container.Container` and `eacl.Table` structures.

Group value, session token and signature in a structure for container
and eACL.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-22 16:38:57 +03:00
Evgenii Stratonikov
633b4e7d2d [#1483] metabase: Add VERSION.md 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
6f243a2a76 [#1483] metabase: Store version 
The main problem is to distinguish the case of initial initialization
and update from version 0. We can't do this at `Open`, because of
`resync_metabase` flag. Thus, the following approach was taken:
1. During `Open` check whether the metabase was initialized.
2. Check for the version in `Init` or write the new one if the metabase
   is new.
3. Update version in `Reset`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
7df50297cd [#1520] shard: Ignore errors on metabase refill 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-21 17:25:28 +03:00
Evgenii Stratonikov
78ea450c25 [#1502] shard: Process locks on metabase refill 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-21 11:07:26 +03:00
Evgenii Stratonikov
972ca83e23 [#1524] writecache: Add some bolt parameters to the configuration 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
07e06249d5 [#1524] metabase: Add some bolt parameters to the configuration 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
0408acc40e services/control: Autogenerate stable marshalers 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-20 16:52:58 +03:00
Leonard Lyubich
13e74fce8a [#1513] ir/settlement: Do not allocate intermediate slice of nodes 
After recent changes `buildContainer` method returns two-dimensional
slice of `NodeInfo` so there is no need to flatten it to build slice of
`common.NodeInfo`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 15:53:18 +03:00
Leonard Lyubich
c408a6a0db [#1513] morph/netmap: Use constant states in unit tests 
Use values of the node state enumeration from Netmap contract instead of
numeric literals.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 15:53:18 +03:00
Leonard Lyubich
820acebb7d [#1513] morph/netmap: Return result by value from net config reader 
Make `ReadNetworkConfiguration` method to return `NetworkConfiguration`
by value in order to follow storage engine improvements and prevent heap
escaping.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 15:53:18 +03:00
Leonard Lyubich
808ba87e82 [#1513] morph/netmap: Use node state constants from Netmap contract 
`Netmap` contract exports enumeration of the node states.

Replace using literals and constants from NeoFS API Go V2 with the
values provided by contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 15:53:18 +03:00
Leonard Lyubich
21d2f8f861 [#1513] Upgrade NeoFS SDK Go with changed netmap package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 15:53:18 +03:00
Evgenii Stratonikov
81f925d5a0 [#1516] metabase: Optimize ListWithCursor for long listings 
Cache buckets outside of the main loop and allocate memory for the
resulting offset only once.

```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       6.45µs ±14%    5.79µs ±11%  -10.24%  (p=0.002 n=10+10)
ListWithCursor/10_items-8     20.9µs ±17%    17.3µs ± 9%  -17.27%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     153µs ±12%     131µs ± 9%  -14.63%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       2.31kB ± 0%    1.91kB ± 0%  -17.46%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     6.94kB ± 0%    5.50kB ± 0%  -20.78%  (p=0.000 n=8+8)
ListWithCursor/100_items-8    53.3kB ± 0%    41.5kB ± 0%  -22.18%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         40.0 ± 0%      34.0 ± 0%  -15.00%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        121 ± 0%       100 ± 0%  -17.36%  (p=0.000 n=10+10)
ListWithCursor/100_items-8       930 ± 0%       758 ± 0%  -18.49%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
a93373fe71 [#1516] metabase: Cache graveyard buckets in ListWithCursor 
```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       6.40µs ±13%    6.45µs ±14%     ~     (p=0.739 n=10+10)
ListWithCursor/10_items-8     30.9µs ±21%    20.9µs ±17%  -32.49%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     274µs ±27%     153µs ±12%  -44.09%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       2.26kB ± 0%    2.31kB ± 0%   +2.46%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     10.8kB ± 0%     6.9kB ± 0%  -36.07%  (p=0.000 n=8+8)
ListWithCursor/100_items-8    96.8kB ± 0%    53.3kB ± 0%  -44.98%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         39.0 ± 0%      40.0 ± 0%   +2.56%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        192 ± 0%       121 ± 0%  -36.98%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     1.72k ± 0%     0.93k ± 0%  -45.93%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>

name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       5.23µs ±19%    5.26µs ±15%     ~     (p=0.853 n=10+10)
ListWithCursor/10_items-8     27.2µs ±15%    18.0µs ±19%  -33.80%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     250µs ±13%     139µs ±15%  -44.27%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       1.99kB ± 0%    2.04kB ± 0%   +2.82%  (p=0.000 n=8+8)
ListWithCursor/10_items-8     10.3kB ± 0%     6.4kB ± 0%  -37.83%  (p=0.000 n=8+10)
ListWithCursor/100_items-8    93.9kB ± 0%    50.4kB ± 0%  -46.37%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         35.0 ± 0%      36.0 ± 0%   +2.86%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        184 ± 0%       113 ± 0%  -38.59%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     1.67k ± 0%     0.88k ± 0%  -47.29%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
af4db8a73b [#1516] metabase: Cache address key and do not decode address twice 
```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       10.6µs ± 1%     6.4µs ±13%  -39.62%  (p=0.000 n=7+10)
ListWithCursor/10_items-8     75.3µs ± 2%    30.9µs ±21%  -58.97%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     726µs ± 2%     274µs ±27%  -62.28%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       3.19kB ± 0%    2.26kB ± 0%  -29.21%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     20.7kB ± 0%    10.8kB ± 0%  -47.68%  (p=0.000 n=10+8)
ListWithCursor/100_items-8     196kB ± 0%      97kB ± 0%  -50.65%  (p=0.000 n=7+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         55.0 ± 0%      39.0 ± 0%  -29.09%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        346 ± 0%       192 ± 0%  -44.51%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     3.25k ± 0%     1.72k ± 0%  -47.13%  (p=0.000 n=9+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
504f45e9ee [#1516] metabase: Add benchmark for ListWithCursor 
```
name                        time/op
ListWithCursor/1_item-8     10.6µs ± 1%
ListWithCursor/10_items-8   75.3µs ± 2%
ListWithCursor/100_items-8   726µs ± 2%

name                        alloc/op
ListWithCursor/1_item-8     3.19kB ± 0%
ListWithCursor/10_items-8   20.7kB ± 0%
ListWithCursor/100_items-8   196kB ± 0%

name                        allocs/op
ListWithCursor/1_item-8       55.0 ± 0%
ListWithCursor/10_items-8      346 ± 0%
ListWithCursor/100_items-8   3.25k ± 0%
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 20:49:41 +03:00
Pavel Karpy
36f4929e52 [#1507] node: Do not handle object concurrently by the policer 
Cache object that are being processed. That prevents concurrent
object handling when there is a few number of objects and object handling
takes more time that the policer needs for starting that object handling one
more time.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-15 20:43:32 +03:00
Pavel Karpy
256165045b [#1508] node: Do not replicate object twice 
If placement contains two vectors with intersecting nodes it was possible to
send the object to the nodes twice.
Also optimizes requests: do not ask about storing the object twice from the
same node.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-15 20:33:04 +03:00
Pavel Karpy
df8a3807fe [#1508] node: Remove unused replicator code 
The node does not support asynchronous object replication anymore, so it
does not need to have replicator worker, channel and `AddTask` function.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-15 20:33:04 +03:00
Leonard Lyubich
2e4a1cb6df [#1518] Upgrade NeoFS SDK Go with changed subnet package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-15 17:02:49 +03:00
Evgenii Stratonikov
f602d05b0a [#1494] *: Fix linter warnings 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
bbf8b8e74d [#1494] services/object: Do not ignore bearer token decode errors 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
795d1e0789 [#1494] go.mod: Update neofs-sdk-go 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
9269ed344d [#1510] placement: Cache container nodes 
In case we have lots of objects in a single container,
`GetContainerNodes` invoked indirectly by a policer can be seen in
pprof.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-14 20:17:43 +03:00
Pavel Karpy
8c77387982 [#1509] morph: Log WS client closing error 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-10 12:53:44 +03:00
Evgenii Stratonikov
38772e1a2e writecache: provide timeout to bbolt.Open 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-09 14:19:47 +03:00
Evgenii Stratonikov
91ed0d20ff [#1417] go.mod: Update neo-go to v0.99.0 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-08 10:27:49 +03:00
Leonard Lyubich
fa009db140 [#1464] ir/container: Fix verifying the operations within sessions 
In previous implementation `verifySignature` method of container
processor worked incorrectly for operations without a key and with
session: processor tried to verify signature with one of the bound owner
keys instead of session one.

Use `VerifySessionDataSignature` method to check the signature if
session is used. Refactor `verifySignature` a bit with session check
highlighting for readability.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-07 23:26:30 +03:00
Leonard Lyubich
0937513c14 [#1485] ir/container: Accept eACL only if extension is allowed 
In order to extend container ACL `F` bit must be set in basic ACL.

Make `Container` contract processor to deny eACL tables bound to
non-extendable containers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-07 16:34:15 +03:00
Evgenii Stratonikov
af7d15cc1a [#1487] innerring: Log errors in processors 
Log errors for network operations. The only places where we are not
interested in errors are `Submit` in pool and unmarshaling.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-07 14:11:58 +03:00
Pavel Karpy
010253a97a [#1460] blobovnicza: Do not use pointers as the results 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
0e4a1beecf [#1460] blobstor: Do not use pointers as the results 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
08bf8a68f1 [#1460] engine: Do not use pointers as the results 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
7b6363f4c6 [#1460] shard: Do not use pointers as the results 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
9b2932609b [#1460] meta: Do not use pointers as the results 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
a580429996 [#1460] meta: Add a benchmark on Get operation 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
b0c7b7851a [#1418] blobstor: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
14366bbd89 [#1418] engine: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
5f57db6bf8 [#1418] shard: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Pavel Karpy
6d4d920bad [#1418] meta: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-06 18:03:12 +03:00
Leonard Lyubich
72708296cc Upgrade NeoFS SDK Go to v1.0.0-rc.4 and NeoFS API Go to v2.12.2 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-03 17:17:50 +03:00
Leonard Lyubich
0e28902b0f [#1471] eacl: Fix incorrect request denial with incomplete object header 
Node shouldn't perform eACL verification during GET/HEAD request
processing until full object header is received. Otherwise, for some
eACL tables request may be falsely rejected.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-03 14:42:54 +03:00
Leonard Lyubich
c89035d544 [#1471] eacl: Add testcase with incomplete list of object headers 
Scenario:
 * HEAD request of some object
 * 1st eACL record allows op for objects with specific user attribute
 * 2nd eACL record forbids op by object ID
 * node doesn't store the requested object locally

With this scenario node shouldn't deny request.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-03 14:42:54 +03:00
Leonard Lyubich
f9504c1cba [#1471] eacl: Don't process object headers twice in response processing 
It is redundant to process object headers in responses w/o object field
since result will be the same.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-03 14:42:54 +03:00
Leonard Lyubich
8b94cb22d3 [#1471] eacl: Prevent aborting on local storage failure 
Request processing should not be interrupted in case of local storage
failure since error case in normal for relay nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-03 14:42:54 +03:00
Leonard Lyubich
a97ea2aa21 [#1471] eacl: Add testcase with local storage failure 
ACL service should not deny request on local storage failure since in
this case relay nodes won't be able to continue the operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-03 14:42:54 +03:00
Pavel Karpy
fdd54b0a03 [#1441] ir: Do not validate subnet removal 
It is useless process since subnet owner is able to delete subnet without an
Alphabet approval. The Alphabet should only validate netmap state after
removal:
1. Update nodes' attributes if they were included in the deleted subnet;
2. Remove nodes without any subnet entrance.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 14:36:58 +03:00
Evgenii Stratonikov
feef9a98f7 [#1462] writecache: remove useless logs 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
2ae7c94cd6 [#1462] *: Remove log.With invocations 
`log.With` is suitable during initialization, but in other places it induces
some overhead, even when branches with logging are not taken.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
43e776dfb1 [#1462] writecache: Persist objects in batches 
`Update` becomes a botleneck under a heavy load.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
6ad87e7959 [#1462] writecache: Fill flush marks during startup 
Some of the objects are already flushed, don't do it twice.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
f2a7503964 [#1462] fstree: Allow to fetch file content lazily 
If we should process address based on some condition, there is no need
to read file content in memory.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
54d4503701 [#1462] writecache: Rename flushWorkersCount 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
0148209168 [#1462] writecache: Use Timer instead of Ticker in flush 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
5073a37930 [#1433] metabase: Optimize Select 
For some filters we can scan only a subset of keys instead of checking
each key.

```
name                       old time/op  new time/op  delta
Select/string_equal-8      49.3µs ± 4%  11.0µs ± 4%  -77.68%  (p=0.000 n=10+10)
Select/string_not_equal-8  7.01ms ± 5%  7.06ms ±10%     ~     (p=0.971 n=10+10)
Select/common_prefix-8      118µs ± 6%    79µs ± 5%  -33.04%  (p=0.000 n=10+9)
Select/unknown-8           21.3µs ± 4%   3.2µs ± 4%  -84.88%  (p=0.000 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
5c48588c64 [#1433] metabase: Add more tests for Select 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
ea76c989ca [#1433] metabase: Add a benchmark for Select 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
b602fff01f [#1463] blobovnicza: Remove pointers from struct members 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 13:42:54 +03:00
Evgenii Stratonikov
d9f0ac8909 [#1463] blobovnicza: Determine db size via os.Stat 
Currently we use `(*bbolt.Bucket).Stats().KeyN` for estimating database
size. However, it iterates over all pages in bucket and thus heavily
depends on the bucket size. This commit replaces initial size estimation
with a single `os.Stat` call.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-03 13:42:54 +03:00
Pavel Karpy
021aa97965 [#1418] blobovnicza: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Pavel Karpy
281befec67 [#1418] blobstor: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Pavel Karpy
babd382ba5 [#1418] engine: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Pavel Karpy
6e752f36dc [#1418] shard: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Pavel Karpy
e265ce2d52 [#1418] meta: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Pavel Karpy
1c100fb4b0 [#1418] writecache: Do not use pointers as parameters 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-03 07:35:17 +03:00
Evgenii Stratonikov
d84d52924a [#1467] pkg/util: Allow to load default account from a wallet 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-02 16:13:20 +03:00
Leonard Lyubich
ff576d125d [#1454] container: Fix reading session token from request origin 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-01 17:41:45 +03:00
Leonard Lyubich
1c30414a6c [#1454] Upgrade NeoFS SDK Go module with new IDs 
Core changes:
 * avoid package-colliding variable naming
 * avoid using pointers to IDs where unnecessary
 * avoid using `idSDK` import alias pattern
 * use `EncodeToString` for protocol string calculation and `String` for
  printing

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-01 17:41:45 +03:00
Leonard Lyubich
cc6209e8a0 [#1449] object/delete: Fix NPE in collectChain 
`head.previous` method returns (nil, nil) if object has no previous
neighbor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-30 17:59:16 +03:00
Leonard Lyubich
96cdc04705 [#1449] policer: Unwrap status HEAD response 
Helper function `client.IsErrObjectNotFound` doesn't support error
unwrapping, so we need to do it on caller side.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-30 17:59:16 +03:00
Leonard Lyubich
e96ea4635c [#1449] policer: Fix selection of new storage candidates 
`Policer` should pass list of selected candidates into `WithNodes`
method of `replicator.Task`. In previous implementation `processNodes`
method passed an opposite list: failed nodes and/or the local one.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-30 17:59:16 +03:00
Evgenii Stratonikov
855cbf5a3a [#1159] services/container: Remove ContainerWithContext struct 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-05-30 15:40:22 +03:00
Leonard Lyubich
2bcc0051ab [#1423] session: Get session issuer from token structure 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-27 15:45:33 +03:00
Leonard Lyubich
2dec3a0a93 [#1423] ir/container: Fix error message related to container removal 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-27 15:45:33 +03:00
Leonard Lyubich
883f2f1ac6 [#1423] local storage: Activate skipped unit tests after fix 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-27 15:45:33 +03:00
Leonard Lyubich
4c8ec20e32 [#1423] session: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-27 15:45:33 +03:00
Pavel Karpy
57c5fccc8c [#1428] node/acl: Make OID optional 
Not all the NeoFS requests must contain OID in their bodies (or must NOT
contain them at all). Do not pass object address in helper functions, pass
CID and OID separately instead.
Also, fixed NPE in the ACL service: updated SDK library brought errors
when working with `Put` and `Search` requests without OID fields.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-05-25 12:11:03 +03:00
Pavel Karpy
d69eb2aaf3 [#1428] node/acl: Make container ID as required param 
Change pointer to value in request information since requests could not
exist without container ID.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-05-25 12:11:03 +03:00
Evgenii Stratonikov
c57114def3 [#1422] morph/client: Fetch group key properly 
NNS contract `resolve` method returns an array.
`neofs-adm` is already up to date.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-05-23 19:24:46 +03:00
Leonard Lyubich
86552cf3ae engine: make corruptSubDir to ignore blobovnicza dir 
There is a dir for Blobovnicza tree in BlobStor, so we need to ignore
it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-23 15:33:14 +03:00
Leonard Lyubich
bb25ecbd15 [#1400] owner: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-23 15:33:14 +03:00
Leonard Lyubich
f8ac4632f8 [#1335] policer: Prevent potential object loss 
In previous implementation `Policer` considered local object copy as
redundant on processing single placement vector.

Make `Policer` to call redundant copy callback after full placement
processing. Also fix 404 error parsing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-23 15:24:23 +03:00
Pavel Karpy
9f62d25b50 [#1224] ir: Delete nodes that do not have subnets 
Also, update their subnet entrance on subnet removal events.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-05-23 08:16:24 +03:00
Pavel Karpy
b3e1e90c01 [#1224] ir: Listen to subnet removal events 
IR must listen to subnet removal notifications always (regardless of notary
mode).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-05-23 08:16:24 +03:00
Pavel Karpy
71f18ba9ec [#1224] morph: Call Alphabet methods flexibly 
Call `UpdateStateIR` and `AddPeerIR` method instead of `UpdateState` and
`AddPeer` if calling client is configured as Alphabet in notary enabled
environment.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-05-23 08:16:24 +03:00
Pavel Karpy
a129fc98da [#1224] morph: Add configuration getters 
Includes `WithNotary` and `IsAlpha` methods to the `StaticClient`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-05-23 08:16:24 +03:00
Pavel Karpy
f0c9a68ad3 [#1224] morph: Delete wrong comment 
Current implementation does not use MurmurHash3.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-05-23 08:16:24 +03:00
Evgenii Stratonikov
df48ddcb3d Revert "[#1141] morph/client: Allow to use more integer types as arguments" 
This reverts commit 9349f422fd9a29ec529ef0648bbfabd1f9ff7651.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-05-20 16:45:15 +03:00
Evgenii Stratonikov
4761857fb3 [#1396] cmd: Allow to autogenerate documentation for all commands 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-05-20 13:35:22 +03:00
Pavel Karpy
4be5dce848 [#1405] ir: Fix settlement log message 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-05-20 12:40:12 +03:00
Evgenii Stratonikov
3f2889a4e9 [#1392] eacl/v2: Use EncodeToString where possible 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
63cdc16088 [#1386] eacl/v2: Do not allocate intermediate slices for XHeaders 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
915f87959e [#1386] eacl/v2: Allocate []eaclSDK.Header slice immediately 
There is no point in having intermediate `[]sessionSDK.XHeader`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
b24589b62d [#1386] eacl/v2: Use raw structs where possible 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
3f4475f97b [#1386] services/object: Fail eACL check if the request is invalid 
Parse all headers beforehand and reject invalid requests.
Another approach would be to remember the error and check
it after `CalculateAction`, which is a bit faster.
The rule of thumb here is "first validate, then use".

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-05-20 12:04:10 +03:00
Pavel Karpy
f99a0498da [#1403] util: Add details to errors 
Make error messages more descriptive when parsing LOCODE from string errors
appear.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-05-20 10:53:27 +03:00
Leonard Lyubich
aeb9884218 [#1389] crypto: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-19 10:12:14 +03:00
Leonard Lyubich
5e50ddd7f5 [#1389] storagegroup: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-19 10:12:14 +03:00
Evgenii Stratonikov
0f376a5d83 [#1388] services/control: Extend tests for ShardInfo marshaling 
It is nice to have different paths for different components and also
check that the information returned is different for different shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-05-17 13:48:07 +03:00
Leonard Lyubich
f15e6e888f [#1377] oid, cid: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-16 15:33:22 +03:00
Evgenii Stratonikov
f65898a354 [#1141] morph/client: Use boolean type directly 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-05-16 15:09:35 +03:00
Evgenii Stratonikov
230a5cd037 [#1141] morph/client: Remove unneeded argument conversions 
Client supports `util.Uint160` directly, no need to convert it to bytes.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-05-16 15:09:35 +03:00
Evgenii Stratonikov
1e8391d216 [#1141] morph/client: Allow to use more integer types as arguments 
Also, use `*big.Int` as integer value, see nspcc-dev/neo-go#2413.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-05-16 15:09:35 +03:00
Leonard Lyubich
3a188bb2e5 [#1371] bearer: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-12 15:16:17 +03:00
Leonard Lyubich
ae92074272 [#1369] checksum: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-12 09:48:31 +03:00
Leonard Lyubich
088df0e2a9 [#1369] version: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-12 09:48:31 +03:00
Leonard Lyubich
7a57d7b076 [#1369] audit: Upgrade SDK package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-12 09:48:31 +03:00
Elizaveta Chichindaeva
cc7a723d77 [#1320] English Check 
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
 2022-05-11 10:40:02 +03:00
Leonard Lyubich
cd545f0160 [#1351] cli: Fix connection scheme loss during endpoint parsing 
In previous implementation NeoFS CLI app used `network.Address.HostAddr`
as a server URI, which caused scheme loss since host address doesn't
contain it.

Rename `HostAddr` to `URIAddr` and make it to return URI address with
`grpcs` scheme if TLS is enabled. Make `TLSEnabled` unexported since it
was used to provide default `tls.Config` only (it is used by default in
SDK).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-05 10:43:27 +03:00
Evgenii Stratonikov
a4769d8624 *: use require.ErrorIs where possible 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-04-29 16:47:57 +03:00
Evgenii Stratonikov
3c39e6df11 [#1319] blobstor: Compress big objects in a streaming fashion 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-04-29 16:46:40 +03:00
Evgenii Stratonikov
1219ff89d4 [#1319] services/object_manager: Fix error message 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-04-29 16:46:40 +03:00
Evgenii Stratonikov
057d53459b [#1319] services/object: Remove Write method from local and remote targets 
They work with prepared objects only.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-04-29 16:46:40 +03:00
Evgenii Stratonikov
4ea03c01b5 [#1319] services/object: Store payload directly in distributedTarget 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-04-29 16:46:40 +03:00
Pavel Karpy
9da7df4d42 [#1318] metabase: Init metabase in test 
`Init()` creates buckets. It prevents NPE in tests.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
486ec8a2e5 [#1318] metabase: Reuse comparing func from require 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
1caeb0b648 [#1318] metabase: Add missing error check in tests 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
6f2723b47e [#1318] node: Fix linter 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
2aec5736e0 [#1318] node/gc: Fix tombstones handling 
Handle only expired tombstones.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00
Pavel Karpy
24ca8ca5c7 [#1318] object: Change default ttl 
This is need to reuse object service without passing any API v2 structs.
Also, fix comment.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-04-29 16:38:52 +03:00