TrueCloudLab/frostfs-node
19
1
Fork 27
Code Issues 91 Pull requests 7 Projects Releases 1 Activity Actions
2803 commits 8 branches 83 tags 53 MiB
Commit graph

2803 commits

This branch
This branch
All branches
Author SHA1 Message Date
Pavel Karpy
cfca88fc5c [#1402] cli: object <-> SG conversion via SDK 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-06 09:56:26 +03:00
Pavel Karpy
bc5882fc89 [#1402] cli: Add cid flag constant 
Also, move common flag constants in `root` file of the `storagegroup`
package.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-07-06 09:56:26 +03:00
Evgenii Stratonikov
6f2363cf31 [#1438] README.md: add build instructions 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-05 14:47:12 +03:00
Leonard Lyubich
c165d1a9b5 [#1556] Upgrade NeoFS SDK Go with changed container API 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-07-05 11:26:06 +03:00
Evgenii Stratonikov
f699e82ea7 [#1560] morph/client: Remove customFees type 
It is private, is a simple map and there is no complex logic to be wrapped in methods.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-04 09:29:01 +03:00
Evgenii Stratonikov
0ccea802e9 [#1560] morph/client: Perform RPC switch and restore in one step 
Otherwise we could switch infinitely if subscription restore has failed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-07-04 09:29:01 +03:00
Evgenii Stratonikov
6358f4d746 [#1053] config: Remove notification endpoints 
Missed this in #1170.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-30 10:51:57 +03:00
Evgenii Stratonikov
ee27c66952 [#1555] util/gendoc: Remove trailing dot from the help message 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-28 13:46:02 +03:00
Evgenii Stratonikov
ad81b6c90a [#1555] neofs-adm: Remove trailing dots from help messages 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-28 13:46:02 +03:00
Evgenii Stratonikov
b41658db04 [#1555] neofs-adm: Allow to remove nodes from the netmap 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-28 13:46:02 +03:00
Evgenii Stratonikov
32fd3692bf [#1555] neofs-adm: Fix cached group signer 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-28 13:46:02 +03:00
Evgenii Stratonikov
5c2ef79080 [#1547] .github: Assign issues to project when labeled 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-27 10:11:43 +03:00
Leonard Lyubich
305dd7598f [#1533] acl: Upgrade NeoFS SDK Go with refactored basic ACL 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-25 13:57:21 +03:00
Evgenii Stratonikov
b13dca8052 [#1545] .github: Auto assign new issues to the project 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-24 20:27:45 +03:00
Evgenii Stratonikov
451343c751 [#1539] neofs-adm: Retrieve storage wallet passwords from config 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-24 13:40:21 +03:00
Pavel Karpy
fc7940a5ea [#1544] cli: Init bearer flags in list-objects cmd 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-24 13:40:07 +03:00
Pavel Karpy
bb52372108 [#1544] go.mod: Update SDK 
The updated version supports reading network configuration uint64 values
that consist of less than 8 bytes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-24 13:40:07 +03:00
Evgenii Stratonikov
9b3e1bd27b [#1307] neofs-adm: Add tests for morph init command 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-24 10:33:34 +03:00
Evgenii Stratonikov
189507dc89 [#1541] neofs-cli: allow to print shards list output in JSON 
Eventually more parameters will be supported (#1390) and after blobstor
configuration refactoring the output will certainly change. Implement
the simplest approach now.

Marshaling the result directly results in too ugly names and they cannot
be easily customized. Marshaling the results via `jsonpb` is better but
is not that flexible in terms of what we want to output.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-24 10:33:22 +03:00
Evgenii Stratonikov
9816d59ec0 [#1323] neofs-cli: Reuse JSON flag for multiple commands 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-24 10:33:22 +03:00
Pavel Karpy
e812d78672 [#1537] node: Remove "trust" from non-reputation logs 
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-23 10:06:36 +03:00
Leonard Lyubich
b67974a8d3 [#xxx] Upgrade NeoFS SDK Go with changed container sessions 
After recent changes in NeoFS SDK Go library session tokens aren't
embedded into `container.Container` and `eacl.Table` structures.

Group value, session token and signature in a structure for container
and eACL.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-22 16:38:57 +03:00
Evgenii Stratonikov
633b4e7d2d [#1483] metabase: Add VERSION.md 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
6f243a2a76 [#1483] metabase: Store version 
The main problem is to distinguish the case of initial initialization
and update from version 0. We can't do this at `Open`, because of
`resync_metabase` flag. Thus, the following approach was taken:
1. During `Open` check whether the metabase was initialized.
2. Check for the version in `Init` or write the new one if the metabase
   is new.
3. Update version in `Reset`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
7df50297cd [#1520] shard: Ignore errors on metabase refill 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-21 17:25:28 +03:00
Evgenii Stratonikov
78ea450c25 [#1502] shard: Process locks on metabase refill 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-21 11:07:26 +03:00
Evgenii Stratonikov
972ca83e23 [#1524] writecache: Add some bolt parameters to the configuration 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
07e06249d5 [#1524] metabase: Add some bolt parameters to the configuration 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
d6043abc24 [#1524] neofs-node: Reuse boltdb config for different components 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
0408acc40e services/control: Autogenerate stable marshalers 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-20 16:52:58 +03:00
Leonard Lyubich
d2d1694113 [#1513] cli/storagegroup: Fix linter remarks 
`staticcheck` linter scolds on potential nil-dereference since it
doesn't know `common.ExitOnErr` definitely exists. Explicit `return`
fixes linter's remark and prevents potential bugs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 15:53:18 +03:00
Leonard Lyubich
13e74fce8a [#1513] ir/settlement: Do not allocate intermediate slice of nodes 
After recent changes `buildContainer` method returns two-dimensional
slice of `NodeInfo` so there is no need to flatten it to build slice of
`common.NodeInfo`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 15:53:18 +03:00
Leonard Lyubich
c408a6a0db [#1513] morph/netmap: Use constant states in unit tests 
Use values of the node state enumeration from Netmap contract instead of
numeric literals.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 15:53:18 +03:00
Leonard Lyubich
eb7b8bf7c4 [#1513] node: Encode node's public key into binary format once 
Avoid per-iteration key encoding due to the immutability of the key.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 15:53:18 +03:00
Leonard Lyubich
4574ba646d [#1513] cli/storagegroup: Fix parsing container and SG flags 
Read `id` flag with the storage group ID. Prevent NPE-panic if flag is
missing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 15:53:18 +03:00
Leonard Lyubich
820acebb7d [#1513] morph/netmap: Return result by value from net config reader 
Make `ReadNetworkConfiguration` method to return `NetworkConfiguration`
by value in order to follow storage engine improvements and prevent heap
escaping.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 15:53:18 +03:00
Leonard Lyubich
808ba87e82 [#1513] morph/netmap: Use node state constants from Netmap contract 
`Netmap` contract exports enumeration of the node states.

Replace using literals and constants from NeoFS API Go V2 with the
values provided by contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 15:53:18 +03:00
Leonard Lyubich
21d2f8f861 [#1513] Upgrade NeoFS SDK Go with changed netmap package 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 15:53:18 +03:00
Evgenii Stratonikov
24b4c1ecf4 [#1484] neofs-cli: Print error on a separate line from a progress bar 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-16 16:38:33 +03:00
Evgenii Stratonikov
e42b7f6a65 [#1484] neofs-cli: Print progress bar after the header has been sent 
Make it similar to `object get`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-16 16:38:33 +03:00
Evgenii Stratonikov
b68778ea90 [#1380] neofs-cli: move container command to a separate package 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-16 09:27:07 +03:00
Evgenii Stratonikov
fd48b96082 [#1384] neofs-cli: move storagegroup command to a separate package 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-16 09:27:07 +03:00
Evgenii Stratonikov
a219e3a667 [#1383] neofs-cli: move object command to a separate package 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-16 09:27:07 +03:00
Evgenii Stratonikov
12bc5607f7 [#1382] neofs-cli: move netmap command to a separate package 
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-16 09:27:07 +03:00
Evgenii Stratonikov
81f925d5a0 [#1516] metabase: Optimize ListWithCursor for long listings 
Cache buckets outside of the main loop and allocate memory for the
resulting offset only once.

```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       6.45µs ±14%    5.79µs ±11%  -10.24%  (p=0.002 n=10+10)
ListWithCursor/10_items-8     20.9µs ±17%    17.3µs ± 9%  -17.27%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     153µs ±12%     131µs ± 9%  -14.63%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       2.31kB ± 0%    1.91kB ± 0%  -17.46%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     6.94kB ± 0%    5.50kB ± 0%  -20.78%  (p=0.000 n=8+8)
ListWithCursor/100_items-8    53.3kB ± 0%    41.5kB ± 0%  -22.18%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         40.0 ± 0%      34.0 ± 0%  -15.00%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        121 ± 0%       100 ± 0%  -17.36%  (p=0.000 n=10+10)
ListWithCursor/100_items-8       930 ± 0%       758 ± 0%  -18.49%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
a93373fe71 [#1516] metabase: Cache graveyard buckets in ListWithCursor 
```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       6.40µs ±13%    6.45µs ±14%     ~     (p=0.739 n=10+10)
ListWithCursor/10_items-8     30.9µs ±21%    20.9µs ±17%  -32.49%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     274µs ±27%     153µs ±12%  -44.09%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       2.26kB ± 0%    2.31kB ± 0%   +2.46%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     10.8kB ± 0%     6.9kB ± 0%  -36.07%  (p=0.000 n=8+8)
ListWithCursor/100_items-8    96.8kB ± 0%    53.3kB ± 0%  -44.98%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         39.0 ± 0%      40.0 ± 0%   +2.56%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        192 ± 0%       121 ± 0%  -36.98%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     1.72k ± 0%     0.93k ± 0%  -45.93%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>

name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       5.23µs ±19%    5.26µs ±15%     ~     (p=0.853 n=10+10)
ListWithCursor/10_items-8     27.2µs ±15%    18.0µs ±19%  -33.80%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     250µs ±13%     139µs ±15%  -44.27%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       1.99kB ± 0%    2.04kB ± 0%   +2.82%  (p=0.000 n=8+8)
ListWithCursor/10_items-8     10.3kB ± 0%     6.4kB ± 0%  -37.83%  (p=0.000 n=8+10)
ListWithCursor/100_items-8    93.9kB ± 0%    50.4kB ± 0%  -46.37%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         35.0 ± 0%      36.0 ± 0%   +2.86%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        184 ± 0%       113 ± 0%  -38.59%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     1.67k ± 0%     0.88k ± 0%  -47.29%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
af4db8a73b [#1516] metabase: Cache address key and do not decode address twice 
```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       10.6µs ± 1%     6.4µs ±13%  -39.62%  (p=0.000 n=7+10)
ListWithCursor/10_items-8     75.3µs ± 2%    30.9µs ±21%  -58.97%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     726µs ± 2%     274µs ±27%  -62.28%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       3.19kB ± 0%    2.26kB ± 0%  -29.21%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     20.7kB ± 0%    10.8kB ± 0%  -47.68%  (p=0.000 n=10+8)
ListWithCursor/100_items-8     196kB ± 0%      97kB ± 0%  -50.65%  (p=0.000 n=7+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         55.0 ± 0%      39.0 ± 0%  -29.09%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        346 ± 0%       192 ± 0%  -44.51%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     3.25k ± 0%     1.72k ± 0%  -47.13%  (p=0.000 n=9+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
504f45e9ee [#1516] metabase: Add benchmark for ListWithCursor 
```
name                        time/op
ListWithCursor/1_item-8     10.6µs ± 1%
ListWithCursor/10_items-8   75.3µs ± 2%
ListWithCursor/100_items-8   726µs ± 2%

name                        alloc/op
ListWithCursor/1_item-8     3.19kB ± 0%
ListWithCursor/10_items-8   20.7kB ± 0%
ListWithCursor/100_items-8   196kB ± 0%

name                        allocs/op
ListWithCursor/1_item-8       55.0 ± 0%
ListWithCursor/10_items-8      346 ± 0%
ListWithCursor/100_items-8   3.25k ± 0%
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
 2022-06-15 20:49:41 +03:00
Pavel Karpy
36f4929e52 [#1507] node: Do not handle object concurrently by the policer 
Cache object that are being processed. That prevents concurrent
object handling when there is a few number of objects and object handling
takes more time that the policer needs for starting that object handling one
more time.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-15 20:43:32 +03:00
Pavel Karpy
256165045b [#1508] node: Do not replicate object twice 
If placement contains two vectors with intersecting nodes it was possible to
send the object to the nodes twice.
Also optimizes requests: do not ask about storing the object twice from the
same node.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
 2022-06-15 20:33:04 +03:00