Compare commits

...

1550 commits

Author SHA1 Message Date
a0d51090a4 [#482] Enable staticcheck in foregjo actions
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-07 10:12:49 +03:00
033eaf77e1 [#496] node: Fix linter importas
Standardize the alias of the
import frostfs-sdk-go/object as objectSDK.

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-07-06 15:36:41 +03:00
4bbe9cc936 [#496] .golangci.yml: Add importas linter
Add importas linter

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-07-06 15:35:23 +03:00
6eefe9747e treesvc: Do not provide credentials unless TLS is used
4f413fe86e was perfect, except it did the opposite of what we needed.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-06 08:04:28 +00:00
f354b8a270 [#473] Add tests for rem values of traverser
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-07-05 20:13:23 +03:00
c6df6c84ae [#473] Do not modify traverse plan when [0] copies number vector is provided
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-07-05 19:54:43 +03:00
b520a3049e [#483] cli: Fix object put cmd with flag --prepare-locally
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-04 18:58:26 +03:00
4c248d573e [#483] cli: Allow to split object on the client side
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-03 18:31:46 +03:00
90e9a85acc [#483] Update dependencies
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-03 15:44:59 +03:00
8d16d95376 [#484] morph: Add expired tx logging
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-03 08:03:25 +00:00
26acf5689e [#92] Ensure policer objects cannot be worked on concurrently
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
3223402c90 [#92] Embed policer's objectsInWork mutex
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
f9730f090d [#92] Refactor policer and add some unit tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
0c5b025788 [#470] grpc: Increase message limits
For send message limit set to 2GiB, but there are custom
GET/GET RANGE limiters.
For receive message limit set to 256 MiB, but actual chunk size
will be managed by client.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-30 16:32:15 +03:00
f437ab8f15 [#197] object: Make Delete method return correct status
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-06-30 12:58:56 +00:00
d01c064674 [#455] Fix non-informative message when parse node attributes
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-28 15:02:29 +00:00
d0ab552a90 [#478] *: Fix funlen linter warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 15:01:49 +00:00
33d9ebbe7f [#478] .golangci.yml: Increase timeout
Make it enough for CI.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 15:01:49 +00:00
f91cfb36e6 [#478] .forgejo: Add build/test workflows
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 15:01:49 +00:00
8a4e250dae [#468] *: replace outdated TODO crypto-related links
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
4f413fe86e [#1] treesvc: Properly check for secure transport
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
cab51c8cbe [#1] metabase: Rename blindlyProcess()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
73a71a71b0 [#1] node: Use a proper validation of a substorage type
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
f4c71cea65 [#1] *: Replace outdated FIXME/TODO links
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
64e1383df6 [#1] Makefile: Remove image-storage-testnet target
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
13b53258b4 [#1] go.mod: Remove retract version
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
0c866f62c5 [#473] placement: Fix backwards compatibility for copies_number
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-27 11:13:16 +03:00
43d263c3d5 [#428] linter: Fix unkeyed assignment
Thanks to gopls.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 13:29:39 +00:00
cac4ed93d6 [#428] engine: Add low_mem config parameter
Concurrent initialization in case of the metabase resync leads to
high memory consumption and potential OOM.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 13:29:39 +00:00
71a63b8e9c [#439] node: Update SDK-Go version
This fixed version: null header for complex object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 07:56:32 +00:00
4bf345225c [#447] pilorama: Use named constant for the key size
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
b4ce0b0412 [#447] pilorama: Do not undo log for create ops
```
goos: linux
goarch: amd64
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
         │     old      │                 new                  │
         │    sec/op    │    sec/op     vs base                │
Create-8   36.48µ ± 11%   30.34µ ± 14%  -16.84% (p=0.000 n=10)

         │     old      │                 new                  │
         │     B/op     │     B/op      vs base                │
Create-8   43.01Ki ± 4%   37.78Ki ± 5%  -12.15% (p=0.000 n=10)

         │    old     │                new                 │
         │ allocs/op  │ allocs/op   vs base                │
Create-8   166.0 ± 3%   146.0 ± 3%  -12.05% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
dd3874eff1 [#447] pilorama: Add benchmark for create ops
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
72fedff7ad [#426] cli: Add object nodes command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 07:41:34 +00:00
ab489265b3 [#449] cli: Allow to exec tree get-by-path without bearer token
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-23 14:44:19 +03:00
71889234b7 [#449] tree: Allow reading requests signed by keys from allow list
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-23 14:44:19 +03:00
40eae22109 [#460] services/util: Remove CreateRequestStreamer
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:08:04 +03:00
a64dc9ad70 [#460] services/util: Remove HandleServerStreamRequest
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:08:01 +03:00
167a67f0b8 [#460] services/util: Remove HandleUnaryRequest
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:07:56 +03:00
785d81a68a [#460] services/object: Reduce distibutedTarget memory footprint
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:07:50 +03:00
4d48377cec [#459] blobovniczatree: Fix get error
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 16:44:53 +03:00
03aa210145 [#373] metrics: Move labels to consts
To unify label naming all lable keys and other consts are moved to
one file.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
b5d9f4a285 [#373] metrics: Add pilorama metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
e89fa110c7 [#373] metrics: Add metabase metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
af608da952 [#373] metrics: Add blobovnizca metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
56f320dd85 [#373] metrics: Add blobstor metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
16a142cd0c [#373] metrics: Add FSTree metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
d8ecc69d00 [#373] local storage: Pass parent ID
This is required to add shard ID as metric label.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
d5aaec1107 [#373] pilorama: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
059e9e88a2 [#373] metabase: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
f54cc0b607 [#373] blobstor: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
8318d90ad0 [#373] blobovniczatree: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
3ae3c8dfdb [#373] fstree: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
a8526d45e9 [#373] blobovnizca: Add missed/fix tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
028d4a8058 [#373] blobovnicza: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
01a0c97760 [#453] engine: Set Disabled mode to deleted shard
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-20 12:04:07 +03:00
50caa388b0 [#303] ir: Use pub key when validate container deletion
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-20 08:02:48 +00:00
69df0d21c2 [#446] engine: Move to read-only on blobstor errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:53:32 +03:00
fe01781811 [#446] los: Wrap SSD errors in a separate type
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:53:31 +03:00
20b84f183a [#446] engine: Simplify logs for shard mode change
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:51:29 +03:00
69b788a90b [#424] metrics: Rename mode to mode_info
To follow best practice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 18:39:49 +03:00
26b305f82b [#424] morph: Fix cache metrics
Use separate morph cache metrics for node and IR

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 18:26:19 +03:00
4449006862 [#424] metrics: Use mode value as metric value for shard
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 18:26:19 +03:00
847732605c [#424] metrics: Rename counter to total
Suffix total should be used for a unit-less accumulating count.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
c348ae35b0 [#424] metrics: Drop embedded metrics
It was not obvious where metrics are used.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
1b364d8cf4 [#424] metrics: Refactor engine metrics
Use histogram vector to measure request duration.
Fix naming like in Prometheus best practice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
c8023a9c8d [#424] morphcache: Use labels for method duration
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
85deb12f4d [#424] writecache: Drop metrics when close
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:52:40 +03:00
07f155ac77 [#424] metrics: Use labels for writecache methods and operations
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:52:40 +03:00
71bbeddb64 [#424] metrics: Drop unused arg
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:51:40 +03:00
fb8fee0c8e [#442] Use strconv.FormatBool instead of untyped conversion
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-14 11:08:36 +00:00
344d6b2ae1 [#xx] Fix invalid log metric namespace identifier
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-13 15:36:34 +03:00
4887f489a1 [#17] Add morph client metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-13 14:06:04 +03:00
90e9247b69 [#371] ir: Add morph cache metrics
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-06-13 10:10:13 +00:00
4f83ab0fb4 [#409] node: Do not sent initial bootstrap under maintenance
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:02:36 +00:00
e68384d4e3 [#409] node: Fetch last bootstrap info on startup
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:02:36 +00:00
898f0686b1 [#409] node: Log maintenance state on startup
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:02:36 +00:00
957a43a124 [#266] services/tree: Add sync check
Do not accept requests until initial sync is finished.
`Apply` is deliberately left out -- we don't want to miss anything new.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
e69a1e8482 [#266] services/tree: Return operation log up to some height
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
2541d319de [#266] pilorama: Allow to get current tree height
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
0c40d98f7a [#375] Add log metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-13 09:52:45 +00:00
83d600ed77 [#424] node: Update api-go version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 07:19:55 +00:00
0400153b7d [#424] object: Do not store large slices in pool
Dynamically growing an unbounded buffers can cause a large
amount of memory to be pinned and never be freed.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 07:19:55 +00:00
41ab4d070e [#423] *: Use hrw.StringHash() where possible
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-13 07:18:25 +00:00
cd92d8a9e7 [#423] go.mod: Update sdk-go and hrw
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-13 07:18:25 +00:00
9bc1a25c07 [#434] metrics: Fix writecache duration
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 07:18:01 +00:00
508e2064eb [#438] cli: Drop tracing debug print
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-09 14:12:25 +03:00
5b75432ca2 [#431] metrics: Add missed label
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-07 13:51:36 +00:00
263c6fdc50 [#372] node: Add metrics for the error counter in the engine
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-07 13:04:47 +00:00
189a367ef2 [#390] frostfs-cli: Pass bearer token to Tree srv
* Add --bearer flag for "tree" subcommand

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-06-07 11:08:34 +03:00
a770b89fd8 [#422] adm: Fix ValidatorCount problems after neo-go update
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 14:48:51 +03:00
f8c1e0639d [#422] cli: Provide context to NetmapSnapshot()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 14:48:47 +03:00
f1f56ef871 [#406] cli: Add --trace flag
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 11:44:04 +00:00
96c9843591 [#406] cmd/common: Execute PersistentPostRun() on errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 11:44:04 +00:00
9562123c49 [#406] cli: Pass context to internal client
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 11:44:04 +00:00
63473d0806 [#417] go.mod: Update neo-go
Update to master, because after the API update notary signer is being
incorrectly formed.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 11:43:48 +00:00
2dd3fc8b7e [#xx] cli: Add policy-playground command
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-02 09:23:40 +00:00
55c28fd5f4 [#412] cache: Pass DialOptions to create new conn
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
dcdfb6ed41 [#412] node: Use observability interceptors
Use metrics and tracing interceptors.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
c09144ecf1 [#412] node: Replace metrics package
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
74578052f9 [#412] node: Replace tracing package
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
7e9a1f394a [#412] node: Update deps
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
f7c4c07453 [#xx] Create innerring metrics before metrics usage
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-01 10:07:48 +03:00
4476a1dbaf [#413] morph/client: Fix govet warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-31 15:38:17 +03:00
dbf41391b5 [#401] engine: Extend evacuation logs
Add operation-tag to logger.
Log evacuation results.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 13:24:30 +03:00
3220c4df9f [#376] metrics: Add GC metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 10:22:12 +00:00
faca861451 [#411] Remove unnecessary pointers for sync objects
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-31 10:19:14 +00:00
f934abed8f [#402] doc: Update evacuation docs
Add estimated time left output to examples.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 10:18:44 +00:00
f64322576a [#402] cli: Add estimated evacuation time left
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 10:18:44 +00:00
ebcc8afbee [#374] Add inner-ring event metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-31 10:11:48 +00:00
8dcd06c587 [#394] node: Use Context in Blobovniczas.Iterate()
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-31 10:09:18 +00:00
a3e30062df [#325] node: Introduce unsafe_disable param to disable policer
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-05-31 10:09:05 +00:00
365a7ca0f4 [#366] node: Stop GC once termination signal received
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-29 09:35:08 +03:00
802168c0c6 [#364] node: Stop flushing big object when termination signal received
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-26 16:46:58 +03:00
9c54a24101 [#364] Fix trailing whitespace
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-26 16:46:58 +03:00
271a56c2ab [#395] metrics: Drop redundant metrics
HistogramVec already has labeled counter.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 13:39:37 +00:00
bc34fee6a7 [#370] Add tree service metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-26 13:39:12 +00:00
f2e5dead7e [#398] pilorama: Disallow applying same operations
1. In redo() we save the old state.
2. If we do redo() for the same operation twice, the old state will be
   overritten with the new one.
3. This in turn affects undo() and subsequent isAncestor() check.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-26 12:14:29 +00:00
5983617069 [#393] shard: Create tombstone source when reload
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 12:14:02 +00:00
20a489bdb5 [#393] gc: Use defer to mark handler done
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 12:14:02 +00:00
9119199f6e [#397] cli: Fix evacuation method names
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 11:28:30 +03:00
2613351008 [#387] gc: Cancel GC is change mode requested
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-25 09:38:16 +03:00
a1823423cf [#312] node: Add WC metrics info to CHANGELOG
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
2ce43935f9 [#312] metrics: Add writecache metrcis
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
d212d908b5 [#312] wc: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
4503a61997 [#312] wc: Delete unused Iterate method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
53a1b15693 [#338] ir: Drop notaryless code from governance
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
0ab589dd52 [#338] ir: Always enable notary on sidechain
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
81718afb39 [#338] ir: Drop named named put fee
Named put fee value used only when notary disabled.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
656fd7f376 [#338] ir: Drop notaryless code from netmap
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
fb708b3a2d [#338] ir: Drop container notaryless code
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
03ab0ca30f [#338] adm: Drop notaryless code
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
cf8531ccd7 [#381] go.mod: Update api-go and sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-23 08:18:01 +03:00
4b768fd115 [#381] *: Move to sync/atomic
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-23 08:18:01 +03:00
ff570847a4 [#381] go.mod: Update bbolt
Adopt new `ForEachBucket` function where possible.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-22 11:49:14 +03:00
7eb8fa6350 [#381] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-22 11:49:14 +03:00
731bf5d0ee [#380] node: go version up
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-22 08:48:14 +00:00
e3ad3c2965 [#283] cli: Move control healthcheck command under control ir
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-22 08:13:52 +00:00
35c9b6b26d [#314] writecache: remove objects right after they are flushed
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-19 09:37:05 +00:00
Pavel Karpy
bf79d06f03 [#314] writecache: Do not lose small objects on disk errors
Do return error if an object could not been stored on WC's disk.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-19 09:37:05 +00:00
Pavel Karpy
9e56592be3 [#314] writecache: Simplify background workers naming
Also, drop not used arg.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-19 09:37:05 +00:00
483fac03d6 [#329] docs: Add shard evacuation description
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
f7c0b50d70 [#329] cli: Add async evacuate commands
Add start, stop evacuate and evacuate status commands.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
e4889e06ba [#329] node: Make evacuate async
Now it's possible to run evacuate shard in async.
Also only one evacuate process can be in progress.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
100b1b5128 [#329] node: Add async evacuate proto methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
0fc494637f [#367] node: Disable unit tests cache
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-18 15:14:55 +00:00
aabcd8d3e4 [#362] node: Cancel ctx in the same routine as for signal watcher
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-18 09:28:10 +03:00
df9e099fa7 [#352] Add explicit max number of alphabet nodes
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-17 15:01:26 +00:00
13a7a90101 [#355] Increase tree svc client cache size to test hypotheses
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-17 14:11:46 +03:00
6f47c75e43 [#125] ir: Set extra wallets on SIGHUP
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-16 12:47:43 +00:00
0624820909 [#336] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-16 12:45:13 +00:00
869fcbf591 [#332] gc: Fix expired complex object deletion
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-16 12:44:57 +00:00
ab07bad33d [#332] gc: Add complex object unit test
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-16 12:44:57 +00:00
8da6530f41 [#351] cli: Support copies number parameter in object put
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-16 08:22:56 +00:00
079b28fa0f [#341] Register candidates in separate transactions
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-15 12:46:12 +00:00
d4d921dcaf [#346] adm: fix race in wallet generation
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-15 11:50:20 +03:00
429a87e83b [#39] ir: Use defer for wg.Done()
Not important, but `exitOnErr` can alter control flow, let's be
explicit.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 17:49:47 +03:00
Pavel Karpy
f604d6bbdc [#39] ir: Add optional profilers
Includes `block` and `mutex` profiles configuration.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 17:49:47 +03:00
f989bc52be [#39] ir: Do not store config keys in httpComponent
Pprof will have specific options, it seems wrong to have them in a
generic struct.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 17:49:46 +03:00
61776033c2 [#39] ir: Do not reload services if they are disabled
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 15:49:04 +03:00
Pavel Karpy
14c35d776e [#39] node: Add optional profilers
Include settings for block and mutex profilers.
They are disabled by default, as in Go runtime itself.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 15:49:04 +03:00
a6ee7a3087 [#324] Add replicator metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-12 11:00:17 +00:00
6055b18362 [#342] morph: Remove unused toStackParameter()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 13:24:49 +03:00
02c02974b3 [#309] locode: Parallelize DB generation
For v0.4.0 release:
Before:
```
Executed in  571.64 secs    fish           external
   usr time  283.07 secs  744.00 micros  283.07 secs
   sys time    8.41 secs  179.00 micros    8.41 secs
```

After:
```
Executed in   54.23 secs    fish           external
   usr time  418.65 secs    1.01 millis  418.65 secs
   sys time    0.61 secs    0.25 millis    0.60 secs
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:59:13 +00:00
147ae8728a [#309] go.mod: Update paulmach/orb to v0.9.1
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:59:13 +00:00
c62025c836 [#321] metabase/test: execute tests in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
c8c5f14e2e [#321] adm/test: Check wallet correctness in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
fe4082799a [#321] adm: Create multisig accounts in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
945454f60c [#321] engine/test: Execute tests in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
4578d00619 [#321] shard/test: Execute tests in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
d35e4c389f [#321] shard/test: Parallelize TestWriteCacheObjectLoss
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
969bfb603f [#321] shard/test: Parallelize TestShard_List
```
go test -count=1 -run TestShard_List -race .
Before: 2.492s
After:  0.109s
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
47b0ec33c3 [#337] netmap: Remove unused events
Done in TrueCloudLab/frostfs-contract#16.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
b480df4985 [#337] container: Remove unused events
Done in TrueCloudLab/frostfs-contract#16.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
Anna Shaleva
bcdb0f330d [#337] morph: Completely remove fallbackTime from client cfg
It's unused and not needed, default fallback lifetime is set by Notary
actor.

Signed-off-by: Anna Shaleva <anna@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
Anna Shaleva
ddcc156ecc [#337] morph: Use Notary Actor for notary requests
Signed-off-by: Anna Shaleva <anna@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
35fdf6f315 [#337] morph: Move subscription logic to subscriber
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
Roman Khimov
a5f118a987 [#337] subscriber: Drop unused UnsubscribeForNotification
It's not really needed, closing the connection works fine when exiting and
normally the app doesn't need to unsubscribe at all.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
800eb5e983 [#125] ir: Reconfigure pprof and metrics on SIGHUP
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-11 12:38:34 +03:00
a181c9e434 [#332] gc: Add additional logging
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 17:34:40 +03:00
90799497d3 [#114] Add remove-node IR control command
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-10 14:31:44 +00:00
ea10abb42a [#296] morph: Add parser unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 09:40:26 +03:00
1309622b20 [#296] morph: Add listener unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 09:40:26 +03:00
b2ffd7df53 [#291] object: Use PayloadSizeLimiter from SDK
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-05 19:07:06 +03:00
35ea207df6 [#291] object: Split validating target in two
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-05 19:07:06 +03:00
Pavel Karpy
ee58b390bb [#221] node: Allow using vector copies_number
Also, take into account that value in general (it was not used before at
all).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-05 16:07:13 +00:00
d02950ad63 [#330] morph: Fix linter issue
Revive became smarter after go 1.20.4 upgrade.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-05 18:02:38 +03:00
973af12854 [#327] tests: replace os.MkdirTemp with t.TempDir
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-05 11:24:51 +00:00
cedd07bbc8 [#304] morph: Iterate endpoints when create ws client in constructor
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-05 10:07:30 +03:00
Pavel Karpy
479c5a65e1 [#322] node: Fix tree svc panic
If a connection has not been established earlier, it stores `nil` in LRU
cache. Cache eviction tries to close every connection (even a `nil` one) and
panics but not crash the app because we are using pools.
That ugly bug also leads to a deadlock where `Unlock` is not called via
`defer` func (and that is the way I found it).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-04 19:48:51 +03:00
2f6757c828 [#311] ir: Fix data race in unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-04 16:59:15 +03:00
872fe90c40 [#317] pre-commit: Add go-mod-tidy hook
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-04 13:58:19 +00:00
d1661ae7dc [#308] Remove downloadContractsFromGithub from the frostfs-adm
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-04 14:08:42 +03:00
5f1af84587 [#297] go.mod: Update compress to v1.16.5
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 16:28:29 +03:00
a1b4ba9980 [#209] compression: Do not store uncompressible data
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 16:28:29 +03:00
529d0bc710 [#302] tree: Drop unused ctx
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 16:54:06 +03:00
eca5c210dd [#299] evacuate: Add context cancel checks
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 15:55:30 +03:00
b939e4e5c5 [#299] ir: Drop unused structs
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 14:05:35 +03:00
235fe84ea3 [#298] innerring: Fix broken tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-02 11:52:29 +03:00
d00b1c0d29 [#280] ir: Add netmap processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
fb5dcc15d2 [#280] ir: Add governance processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
31b4da225a [#280] ir: Add frostfs processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
5010b35466 [#280] ir: Add container processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
686f01bce5 [#280] ir: Add balance processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
e89fa7f69f [#280] ir: Add alphabet processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
53693071de [#280] ir: Add block timer unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
e70f808dc3 [#280] ir: Add parser unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
2dbe382b5f [#280] ir: Add state unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
500611f3c8 [#280] ir: Add indexer tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
3b64dffda2 [#280] ir: Add fee config tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
22d47376a6 [#277] getsvc: Refactor errors
Move errors to separate files.
Use zap.Error for error logging.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
45438e7b06 [#277] getsvc: Rename and reorder code
Rename execCtx to request.
Move code to appropriate files.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
1440450606 [#277] getsvc: Drop cyclic struct dependency
Drop cyclic dependency between execCtx and Service.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
591c4e7d50 [#277] getsvc: Move headOnly to request params
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
265d2326a0 [#277] getsvc: Extract remote storage
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
30e1b62b67 [#277] getsvc: Fix service deps
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
8fc082b688 [#277] getsvc: Do not return status error
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
3bac5a485d [#248] config: Remove audit-related parameters
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
f368ccbdf0 [#248] morph: Remove audit client
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
b2bc1fccbc [#248] logs: Remove unused messages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
a9c4ba62c3 [#248] metabase: Remove storage group bucket
Backwards compatible change, so no version increase.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
586f5986bc [#248] core: Remove storage group
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
f1ea8fec93 [#248] object_manager: Remove storage group
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
7f49f07255 [#248] services: Remove audit service
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
8879c6ea4a [#248] innerring: Remove audit and settlement code
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
8b2aae73c6 [#248] cli: Remove storagegroup commands
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
f73ac6e02d [#290] control: Use generics for response wrappers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 10:57:24 +03:00
ff25521204 [#270] Add IR epoch tick control call
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-04-28 07:57:00 +00:00
58f1ba4b51 [#288] pilorama: Add missing operation in log
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
daa26f6e9b [#288] pilorama/test: Check operation order for TreeGetByPath()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
291f9e809a [#288] pilorama: Remove getMeta() wrapper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
0045f1bcd4 [#288] pilorama: Use more descriptive names for memory tree
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
f856ad7480 [#288] pilorama: Remove childMap from memory forest
Memory forest is here to check the correctness of boltdb optimized
implementation. Let's keep it simple.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
ada081dfd5 [#19] node: Make policier read shards concurrently
* Introduce ListWithMultiCursor that simultaneously reads objects
  from different shards

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-27 11:11:47 +03:00
1f4061c0e2 [#285] blobonicza: Optimize upperPowerOfTwo()
The real reason is this:
```
pkg/local_object_storage/blobovnicza/sizes.go:36:69
    revive empty-block: this block is empty, you can remove it
```

Didn't want to make this function longer or to add `nolint`, thus this
change. To justify:
```
UpperBound/size=1-8          0.4924n ± 1%   0.2472n ± 2%  -49.80% (p=0.000 n=10)
UpperBound/size=1023-8       0.4936n ± 3%   0.2442n ± 1%  -50.52% (p=0.000 n=10)
UpperBound/size=66560-8      0.8201n ± 2%   0.2436n ± 1%  -70.29% (p=0.000 n=10)
UpperBound/size=41943040-8   6.6900n ± 5%   0.2432n ± 0%  -96.36% (p=0.000 n=10)
geomean                       1.075n        0.2446n       -77.24%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:11 +03:00
dfe4ada838 [#285] lint: Resolve revive/if-return
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:08 +03:00
f07e2d4812 [#285] lint: Fix revive/unused-parameter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:04 +03:00
57718bd6b4 [#285] *: Update golangci-lint to v1.52.2
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:04 +03:00
14f83b8aa9 [#125] ir: Change log level on SIGHUP
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:54:19 +03:00
563780057d [#125] ir: Use internal/common/config for reading config
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
ef222e2487 [#125] cmd: Refactor internal/common/viper
Add `opts.WithViper`, set opts struct as private.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
e61aec4a7d [#125] node: Remove unused config.Prm
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
eb7be82e87 [#125] node: Avoid panic when reading config
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
d390f093e0 [#125] node: Move viper creation to internal/common/config
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
b2123bfd1a [#125] node: Add new option WithEnvPrefix
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
ee01275d25 [#125] node: Remove redundant env from config/internal
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
9d01029733 [#166] node: Parallelize background tree service sync by batching
* Merge operations

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-26 10:17:56 +00:00
299b24b974 [#166] node: Parallelize background tree service sync by batching
* Concurrently dispatch TreeApply operations for batching in forest

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-26 10:17:56 +00:00
700f39c3f8 [#229] Update CHANGELOG.md
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-26 10:23:33 +03:00
Denis Kirillov
dce5924a89 [#229] services/tree: Use bearer owner as signer
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-26 10:23:33 +03:00
89530534a1 [#229] service/tree: Disable container owner check in tree service
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-04-26 10:23:33 +03:00
c04f6c5e59 [#229] acl: Allow Impersonate
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-04-26 10:23:33 +03:00
04be9415d9 [#231] node: Fix race condition in TTL cache
Use key locker to lock by key.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-25 09:54:12 +03:00
ddbc9e255f [#231] node: Invalidate container cache on PutSuccess event
For example: frostfs-cli creates container and makes polling
GetContainer requests. These requests go through container cache,
so not found error stores in container cache.
So container cache can contain not found error when PutSuccess event received.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-25 09:42:02 +03:00
c70306b324 [#164] scripts: Add metrics description exporter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00
6fef2726b8 [#164] metrics: Allow to export metrics description
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00
4ade5339da [#164] metrics: Rename metrics.go -> node.go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00
015d62425b [#164] metrics: Fill local registry explicitly
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00
Pavel Karpy
59822f7fb4 [#1248] node: Do not update cache twice
Do not request morph values on morph cache misses concurrently.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-21 09:45:53 +00:00
dc2e9d70c7 [#271] go.mod: Update prometheus to v1.15.0
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-20 19:30:26 +03:00
Pavel Karpy
09938a9841 Revert "[#262] meta: Do not return old expired objects"
This reverts commit 3d23b087

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 19:22:55 +03:00
Pavel Karpy
e9461686b8 [#274] wc: Resolve possible deadlock
If operation with WC are _fast enough_ (e.g. `Init` failed and `Close` is
called immediately) there is a race and a deadlock that do not allow finish
(and start, in fact) an initialization routine because of taken `modeMtx`
and also do not allow finish `Close` call because of awaiting initialization
finish. So do stop initialization _before_ any mutex is taken.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 19:22:54 +03:00
Pavel Karpy
6b6f33ed71 [#274] wc: Make wait groups work more explicit
Do not run routine that calls `wg.Done()` inside, it is hard to read.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 19:22:54 +03:00
Pavel Karpy
4f5f832137 [#268] notary_preparator: Actualize notary requests parsing
After 75d7891ca1
`neo-go` does claim that an empty invocation script is the only way to
fill missing signature for unsigned notary requests. The new notary actor
does it that way and, therefore, breaks notary request parsing by the
Alphabet because of skipping any request that is not filled with a dummy (64
zeros) invocation script. Support both way. The "Dummy" approach will be
dropped later.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 10:29:34 +03:00
6c90bb87f1 [#118] node: add ctx for unit tests for blobstor
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-19 18:09:33 +03:00
Pavel Karpy
3d23b08773 [#262] meta: Do not return old expired objects
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-19 13:37:38 +00:00
13c8afcb02 [#118] node: add unit concurrent tests for blobstor
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-19 10:22:50 +00:00
Pavel Karpy
20cd080323 [#255] write-cache: Fix init race condition
Do not use WC's internals in the initialization routines without mode
protection. WC should be able to change its mode even if the initialization
is not finished yet.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-19 09:29:07 +00:00
3d43b0f7f9 [#265] node: Fix after SDK & API-Go version up
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-18 12:09:19 +03:00
a358255c1b [#265] node: Drop unused
Resolve unused linter.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-18 12:09:16 +03:00
b447ff99aa [#265] node: Up SDK and API-Go versions
Fix tracing panic.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-18 12:09:12 +03:00
7b981bfe97 [#249] logs: Drop unused consts
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
f07d4158f5 [#249] node: Drop subnet from IR and morph
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
d757d881d0 [#249] node: Drop subnet from config
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
05c870f39a [#249] cli: Drop subnet support
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
160147b05d [#249] adm: Drop subnet support
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
Pavel Karpy
262c9c2b93 [#256] blobovniczaTree: Make Exists test stable
Corrupt and request _the same_ file.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 16:46:24 +03:00
0b42a00a60 [#254] innerring: Remove unused TimersHandlers() method from processors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 15:34:15 +03:00
8466894fdf [#250] control: remove DumpShard and RestoreShard RPC
We have `Evacuate` with a cleaner interface.
Also, remove them from CLI and engine.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 12:28:49 +00:00
Pavel Karpy
070154d506 [#247] client: Drop reputation related RPCs
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:54:28 +03:00
Pavel Karpy
beabed788c [#247] network_config: Drop reputation
Drop the code that was expected to work with global reputation network
parameters.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:53:22 +03:00
Pavel Karpy
b453bb754c [#247] logs: Drop reputation log messages
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:44:14 +03:00
Pavel Karpy
8799138fcb [#247] morph: Drop reputation contract
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:44:14 +03:00
Pavel Karpy
960e3c219e [#247] config, doc: Drop reputation references
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:44:14 +03:00
Pavel Karpy
560f73ab7e [#247] node, ir: Drop reputation related code
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:44:14 +03:00
6121b541b5 [#242] treesvc: Add tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
d62c6e4ce6 [#242] node: Add tracing spans
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
200fc8b882 [#242] put: Pass context to relay function
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
995db117d0 [#238] node: Read cfg from dir even if cfg file not set
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 10:19:10 +00:00
8d2f443868 [#238] Fix linter error
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 10:19:10 +00:00
41eb3129ae [#139] Refactor blobovnicza exist test to not use chmod
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-04-14 12:16:14 +03:00
adcfce39cf [#246] .gitattributes: Do not show diff for go.sum
When we update dependencies it can be rather big. However it is
generated automatically with `go mod tidy`, no need to review.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 06:02:04 +00:00
299b6a6938 [#100] adm: Use netmap constants from pkg
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 05:12:50 +00:00
0c6aeaaf18 [#100] adm: Take net settings into account during netmap contract update
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 05:12:50 +00:00
4496999e52 [#100] Fix CHANGELOG
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 05:12:50 +00:00
cffcc7745e [#240] logs: Factor out common service log messages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 05:06:09 +00:00
0e31c12e63 [#240] logs: Move log messages to constants
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 05:06:09 +00:00
Roman Khimov
d29b13454f [#239] morph/client: Simplify code interacting with magic numbers
It can't be uint64 in fact, but this error is buried deeply in the NetworkInfo
API structure, so we're not touching MagicNumber() for now.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
d686ab49e8 [#202] adm: Remove deprecated RPC client methods
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
Roman Khimov
f41ad9d419 [#239] morph/client: Recreate actor/wrappers in SetGroupSignerScope
That's the reason #2230 and #2263 were not detected earlier, we actually had
Global scope being used before reconnection to RPC node.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
Roman Khimov
be4df989e5 [#239] morph/client: Deduplicate signers in Client a bit
One signer in the cfg is enough.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
Roman Khimov
96b38f7e86 [#239] morph/client: Add CalledByEntry into the "grouped" scope
Fixes #2230, fixes #2263. CustomGroups are nice while we're only calling NeoFS
contracts, but it doesn't work at all for standard ones like GAS or Notary.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
2f1beddfd3 [#202] adm: Remove deprecated warnings in tests
`VerifyBlocks` is now `SkipBlockVerification` and is false by default.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
01c0c90a86 [#113] cli: add "name" option for "get container" command
* Make get container command filter out the container by attribute name

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-12 18:25:37 +00:00
7d39fecc6a Release v0.36.0
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-12 16:57:02 +03:00
04727ce1d6 Revert "[#135] signature: Add tracing"
This reverts commit 5778980252.
2023-04-12 16:57:02 +03:00
08769f413f Revert "[#135] acl: Add tracing spans"
This reverts commit b2ca730547.
2023-04-12 16:54:13 +03:00
5d2affa5cd testutil: Fix linter warning
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-12 11:00:02 +03:00
5778980252 [#135] signature: Add tracing
Add tracing to verify request and sign response.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
b2ca730547 [#135] acl: Add tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
0920d848d0 [#135] get-object: Add tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
5af9f58469 [#135] tracing: Add tracing to node gRPC endpoints
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
72565a91ef [#135] node: Update api-go version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
c4865783fc [#236] blobstor/test: Prefill storage in parallel in read benchmark
`blobovniczatree` takes a really long time to prefill, because each
batch takes at least 10ms, so for 10k iterations we have at least 100s of
prefill.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:11 +03:00
6ad5c38225 [#236] testutil: Use random object id in RandObjGenerator
Before this commit it was like this:
```
BenchmarkSubstorageWritePerf/memstore-rand10-8            227425              4859 ns/op
BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8     --- FAIL: BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
BenchmarkSubstorageWritePerf/fstree-rand10-8            --- FAIL: BenchmarkSubstorageWritePerf/fstree-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
c85a0bc866 [#236] blobstor/test: Reduce test descriptions
I tried to add 4 more tests and suddenly, it became harder to navigate in
code. Move directory creation in a common function.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
6bf11f7cca [#230] CHANGELOG.md: Remove older entries
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 10:07:27 +00:00
2c07f831c7 [#223] node: Refactor cache usage
Drop excess type args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
93eba19a8e [#223] objectsvc: Refactor split-tree traverse
Resolve funlen & gocognit linters for traverseSplitChain method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
2ed9fd3f94 [#223] objectsvc: Refactor request parameters
Resolve containedctx linter for commonPrm.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
ccf8463e69 [#223] controlsvc: Drop unnecessary nolint
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
ae86cda58c [#223] sg: Refactor storage group parameters
Resolve containedctx linter for SearchSGPrm and GetSGPrm  structs.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
6016d78a45 [#223] core: Refactor object format validator
Resolve funlen linter for FormatValidator.ValidateContent method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 13:56:09 +03:00
02831d427b [#213] metrics: Refactor object metrics
Resolve funlen linter for newObjectServiceMetrics function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 09:43:24 +03:00
38ae71cc7d [#213] metrics: Refactor engine metrics
Resolve funlen linter for newEngineMetrics function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 09:43:24 +03:00
b689027d57 [#191] cli: Add control shards doctor command
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
0b9622c418 [#191] control: Add Doctor RPC
Doctor RPC performs complex operations on the storage engine.
Currently only duplicate removal is supported.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
dbc3811ff4 [#191] engine: Allow to remove redundant object copies
RemoveDuplicates() removes all duplicate object copies stored on
multiple shards. All shards are processed and the command tries to leave
a copy on the best shard according to HRW.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
cb172e73a6 [#228] node: Use uber atomic package instead standard
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 15:37:27 +00:00
c236b54a65 [#212] reputationsvc: Resolve funlen linter
Resolve funlen linter for Calculator.iterateDaughter method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
7ebbfa3358 [#212] reputationsvc: Resolve linters and rename
Resolved containedctx linters.
Renamed context structs and interfaces to more understandble names.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
469e8a6e59 [#212] reputationsvc: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
e2f13d03d7 [#222] auditsvc: Refactor PoR audit
Resolve funlen linter for Context.checkStorageGroupPoR method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
e8d340287f [#222] auditsvc: Refactor audit task
Resolve containedctx linter. Cancel task by listen cancel.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
3dbff0a478 [#222] auditsvc: Resolve containedctx linter
Resolve containedctx linter for commonCommunicatorPrm type.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
fe87735073 [#219] morph: Refactor notary preparator
Resolve funlen linter for Preparator.Prepare method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
d07e40d6fe [#219] morph: Refactor moprh event listener
Resolve funlen and gocognit linters for listener.listenLoop method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
775179f823 [#219] morph: Refactor notary invoke
Resolve funlen linter for Client.notaryInvoke method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
e815b19101 [#219] morph: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
56282edf02 [#166] node: Parallelize background tree service sync
* Run sync task for nodes in parallel within errgroup worker pool

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 04:43:32 +00:00
9027695371 [#203] node: Add staticcheck target and pre-commit
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
4ec69cbbf8 [#203] control: Fix test
Fix shard compare loop.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
f32f61df87 [#203] pilorama: Refactor tests
Do not pass 0 as channel capacity.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
8908798f59 [#203] node: Resolve unused vars
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
bab11492ad [#203] node: Resolve never used errors
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
e21c5bea21 [#203] cli: Fix error message
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
68a2f36636 [#203] morph: Mark depracated methods
Skip staticcheck for depracated methods. Will be fixed soon.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
9e2df4b7c7 [#203] node: Fix double imports
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
ab891517de [#116] node: Fix bug with extra generated files in TestReload
* Create testNewEngine in engineWithShards without default opts

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-05 18:49:34 +03:00
c58ab0c369 [#193] getsvc: Reduce private key requests
Get private key only once for request forwaring.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
89924071cd [#193] getsvc: Edit request forwarder signature
Pass context to forwarder direct, without closure.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
6c7b708a98 [#193] getsvc: Refactor get range params creation
Resolve funlen linter for toRangePrm function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
b0786d2e5c [#193] getsvc: Refactor get params creation
Resolve funlen linter for toPrm function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
f889893216 [#193] getsvc: Refactor head param creation
Resolve funlen linter for toHeadPrm method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
91ead04fa4 [#193] getsvc: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
1bf21dbb47 [#193] getsvc: Resolve context linters
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
206458c841 [#217] containersvc: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:00 +00:00
279261ace3 [#217] containersvc: Refactor route passing
Resolve containedctx for routeCtx.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:00 +00:00
6f7b6a8813 [#116] node: Improve shard/engine construction in tests
* Introduce testEngineWrapper that can be constructed with different options

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-05 14:36:40 +00:00
d6486d172e [#210] policier: Refactor nodes processing
Resolve funlen linter for processNodes method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
080be5cfcd [#210] policier: Refactor object placement
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
23575e1ac0 [#210] policier: Resolve contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
9098d0eec0 [#211] engine: Unify shard mode checks for tree operations
All operations must ensure the shard is not in a degraded mode.
Write operations must also ensure the shard is not in a read-only mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-05 11:10:39 +00:00
760af6b912 [#211] fstree: Consider ENOSPC a logical error
We already do this for file writing, however directory creation can also
fail.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-05 11:10:39 +00:00
d85703a963 [#208] searchsvc: Refactor request forwarding
Resolve funlen & gocognit linters for toPrm method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 16:24:55 +03:00
0b38419fbf [#208] searchsvc: Resolve context linters
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 16:24:50 +03:00
5f2a1531fe [#208] deletesvc: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 16:24:40 +03:00
4941926c9d [#207] aclsvc: Drop outdated tag
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:14 +00:00
585415fa92 [#207] aclsvc: Refactor send checker
Resolve funlen linter for putStreamBasicChecker.Send method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:14 +00:00
9ef790f782 [#207] aclsvc: Refactor object headers read
Resolve funlen linter for readObjectHeaders method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:13 +00:00
cd33a57f44 [#207] aclsvc: Refactor EACL check
Resolve funlen linter for CheckEACL method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:13 +00:00
1f1aed87be [#188] metabase: Refactor object inhume
Resolve funlen linter for db.Inhume method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
5a66db80c5 [#188] engine: Refactor shard evacuation
Resolve funlen and gocognit linter for StorageEngine.Evacuate method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
456bc097f7 [#188] engine: Refactor get range from engine
Resolve funlen linter for StorageEngine.getRange method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
3010ca2649 [#188] engine: Refactor get object from engine
Resolve funlen linter for StorageEngine.get method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
0739c36a3b [#188] metabase: Refactor object put to metabase
Resolve funlen linter for db.put method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
8273a3dfb2 [#188] blobstor: Refactor blobstor test
Resolve funlen linter for TestIterate function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
594b5821ed [#188] blobstor: Refactor put data to blobovniczas
Resolve funlen linter for Blobovniczas.Put method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
ee7468daa7 [#205] innerring: Provide alphabetState param to epochTimer
Fix NPE, introduced in f09ee27a.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-04 10:47:31 +00:00
49cc23e03c [#175] adm: pipeline container iteration
Do not accumulate everything in memory.
Also, CLI should be responsive.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-04 07:26:23 +00:00
e85e5382e4 [#175] adm: list containers using containersOf
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-04 07:26:23 +00:00
8e5a0dcf27 [#204] gc: Fix GC handlers start
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 06:48:27 +00:00
0948a280fa [#195] morph: use blocking unlimited pool for notifications
With non-blocking pool restricted by 10 in capacity, the probability of
dropping events is unexpectedly big. Notifications are an essential part of the FrostFS,
we should not drop anything, especially new epochs.
```
Mar 31 07:07:03 vedi neofs-ir[19164]: 2023-03-31T07:07:03.901Z        debug        subscriber/subscriber.go:154        new notification event from sidechain        {"name": "NewEpoch"}
Mar 31 07:07:03 vedi neofs-ir[19164]: 2023-03-31T07:07:03.901Z        warn        event/listener.go:248        listener worker pool drained        {"chain": "morph", "capacity": 10}
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-03 15:58:16 +00:00
ece6c820e7 [#199] putsvc: Refactor streamer initialization
Resolve funlen linter for initTarget method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
27bdddc48f [#199] putsvc: Refactor put object
Resolve containedctx linter for streamer and remote target

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
cecea8053a [#199] putsvc: Refactor streamer pool
Resolve staticcheck linter for putBytesPool

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
14d894178e [#199] putsvc: Refactor placement iterator
Resolve funlen linter for iteratePlacement method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
Pavel Karpy
a69c6d1ec9 [#2272] morph: Do not subscribe to events without listening
It led to a neo-go dead-lock in the `subscriber` component. Subscribing to
notifications is the same RPC as any others, so it could also be blocked
forever if no async listening (reading the notification channel) routine
exists. If a number of subscriptions is big enough (or a caller is lucky
enough) subscribing loop might have not finished subscribing before the
first notification is received and then: subscribing RPC is blocked by
received notification (non)handling and listening notifications routine is
blocked by not finished subscription loop.
That commit starts listening notification channel _before_ any subscription
actions.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-03 08:42:41 +00:00
Pavel Karpy
2bdf7126b8 [#181] Update CHANGELOG
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:54:05 +03:00
Pavel Karpy
aa92f977ef [#181] ir: Do not pay for audit by non-alphabet nodes
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:53:37 +03:00
Pavel Karpy
f09ee27af9 [#181] ir: Do not process container estimations by non-alphabet nodes
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:53:09 +03:00
Pavel Karpy
db5321309d [#181] ir: Do not sync the Alphabet by non-alphabet nodes
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:52:16 +03:00
Pavel Karpy
44d5412e10 [#181] ir: Do not deposit notary GAS by non-alphabet nodes
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:51:12 +03:00
ed28ce24cd [#168] node: Refactor reputation service
Resolve funlen linter for initReputationService function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
1f929fdd57 [#168] node: Refactor netmap service init
Resolve funlen linter for initNetmapService function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
dd572825c7 [#168] node: Refactor object service init
Resolve funlen linter for initObjectService function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
56161d39b4 [#168] node: Refactor container services
Resolve containedctx for remoteLoadAnnounceWriter struct

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
28dc9e2190 [#168] node: Refactor container service init
Resolve funlen linter for initContainerService function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
dcd39f8fdd [#168] node: Refactor shard opts initialization
Resolve funlen linter for shardOpts method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
c94372e6f9 [#168] node: Refactor config initialization
Resolve funlen linter for initCfg function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
3bbb516528 [#168] node: Refactor node config read
Resolve funlen linter for readConfig method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
a7c79c773a [#168] node: Refactor node config
Resolve containedctx linter for cfg

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
8426d25f4b [#185] ir: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
c1cbe6ff2d [#185] ir: Refactor signature verification
Resolve funlen linter for verifySignature method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
aeb4bbc51e [#185] ir: Refactor alphabet update
Resolve funlen linter for processAlphabetSync method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
30c18d46cc [#185] ir: Refactor audit
Resolve funlen linter for processStartAudit method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
6be2688fb4 [#185] ir: Refactor gas emit
Resolve funlen linter for processEmit method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
4d160bd4ab [#185] ir: Refactor ir service creation
Resolve funlen linter for New function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
c8a6978563 [#185] ir: Refactor ir start
Resolve funlen linter for Server.Start method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
85fb9e77c4 [#185] ir: Resolve funlen linter
Resolve funlen linter for GlagoliticLetter.String method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
7be5a0fd79 [#189] node: Fix bug with sync/atomic package
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-03-30 14:50:43 +00:00
ab32067152 [#183] gc: Fix drop expired locked complex objects
Do not delete bucket keys during iteration

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
9f0bce5c15 [#183] gc: Fix drop expired locked simple objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
341fe1688f [#139] test: Add test storage implementation
This aims to reduce the usage of chmod hackery to induce or simulate
OS-related failures.

Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-29 14:28:49 +00:00
Pavel Karpy
e843e7f090 [#184] ir: Delete unused config reader param
Not used since notary environments do endless and non-configurable
deposits.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
Pavel Karpy
ba58a77f8c [#184] node, ir: Do not wait for already made notary deposits
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
Pavel Karpy
3646723ae3 [#184] morph: Make deposit logs INFO
We have already had and solved plenty of deposit issues and notary balance
is a really important thing. Deserves to be INFO even before the huge logs
severity refactor, happens on an app start only.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
Pavel Karpy
97e201993b [#184] *: Unify done contexts handling
If `ctx.Done()`, return `ctx.Err()` in every function that returns an error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
221203beeb [#180] node: Refactor panics in unit test
* Replace panics in unit tests by require.NoError and t.Fatalf

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-03-29 12:39:07 +03:00
91717d4b98 [#176] morph: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 18:49:55 +03:00
382ecae96a [#172] Use ContainersOf() for container list fetching
Previously we were limited by ~2048 containers because of
neo-go VM limits.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 14:32:55 +00:00
9e54646248 [#172] client/container: Support listing containers with containersOf
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 14:32:55 +00:00
bf7d80f44b [#172] morph/client: Support iterators via neo-go session API
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 14:32:55 +00:00
5bf1ec348f [#161] adm: Refactor storage-config command
Resolve funlen linter for storageConfig function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
93cb9e3a94 [#161] morph: Refactor invokeNotary method
Resolve funlen linter

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
512b72591a [#161] adm: Refactor subnet commands
Resolve funlen linter for init method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
e1b99dacad [#161] adm: Refactor netmap add/remove commands
Resolve funlen linter for manageSubnetAdmins function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
92f8810970 [#161] adm: Refactor commands initialization
Resolve funlen linter for init function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
b0fefcb21f [#161] adm: Refactor deposit-notary command
Resolve funlen linter for depositNotary function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
d62b11f5e7 [#161] adm: Refactor init context creation
Resolve funlen linter for newInitializeContext function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
ecbc211016 [#161] adm: Refactor update-contracts command
Resolve funlen linter for updateContracts method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
5f7d70c59c [#161] adm: Refactor deploy command
Resolve funlen linter for deployContractCmd function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
9534ade716 [#161] adm: Refactor restore-containers command
Resolve funlen linter for restoreContainers function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
f2e880465e [#165] ir: Refactor config default init
Resolve funlen linter for defaultConfiguration function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
04b3d9d068 [#165] cli: Drop redundant nolint comment
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
1d5f2dd681 [#165] cli: Refactor collectObjectRelatives
Resolve funlen linter

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
c2cf708e0e [#165] cli: Refactor put command
Resolve funlen linter for putObject method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
c78e9cc857 [#165] cli: Refactor get command
Resolve funlen linter for getObject method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
Pavel Karpy
3c7ed21f74 [#141] Add big object's parts to a lock object's body
That will prevent part/link object from being removed by both an external
`DELETE` call and the object expiration procedure.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-27 07:28:32 +00:00
a5ece7889d [#174] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 07:24:58 +00:00
Pavel Karpy
f3ff9fd251 [#73] morph: Rename vars that collide with package names
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-24 09:42:30 +00:00
Pavel Karpy
533e9f8b75 [#59] morph: Adopt updated neo-go client API for subs
It does not use deprecated methods anymore but also adds more code that
removes. Future refactor that will affect more components will optimize
usage of the updated API.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-24 09:42:30 +00:00
9ffa0d8fea [#171] Remove unit tests from pipeline
Unit tests are included as pre-commit hook. They are
triggered by the change of .go files.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-24 10:50:11 +03:00
d857ffeb2e [#171] Syncrhonize pre-commit settings across FrostFS repos
This change allows to use `[#xx]` placeholders for issue number.
This change adds go unit test run if .go files were changed.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-24 10:43:55 +03:00
aarifullin
34329d67ff [#86] node: Fix unit test and linter errors
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 12:42:58 +03:00
9808dec591 [#86] node: Move testing utils to one package
Move testing utils from tests in local_object_storage package to
unified testutil package

Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 08:19:15 +00:00
342e571d89 [#159] Add handle __SYSTEM__ sys attributes
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-22 17:35:20 +03:00
f111704ceb [#158] Fix gitlint run in pre-commit under CI
Wrong stage was used.
It still may run gitlint against only latest commit in PR.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-22 08:58:34 +00:00
Pavel Karpy
da8da1c63a [#98] fstree: Do not fail iteration over just removed files
A directory is read and files are saved to a local variable. The iteration
over such files may lead to a non-existing files reading due to a normal SN
operation cycle and, therefore, may lead to a returning the OS error to a
caller. Skip just removed (or lost) files as the golang std library does in
similar situations:
5f1a0320b9/src/os/dir_unix.go (L128-L133).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-22 07:26:27 +00:00
49234b915e [#155] search-service: Fix search with ST
Search should return only objects allowed in static session

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-22 07:21:09 +00:00
1637a3edce [#155] search-service: Add search with ST test
In case of session token (ST) with object IDs search should
return only objects allowed in static session

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-22 07:21:09 +00:00
cbc2efb1d6 [#156] object/get: Make toHeadPrm() pass gocognit
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
484ac502ca [#156] frostfs-adm: Make dumpBalances() pass linter checks
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
8014fdb21a [#156] metabase: Make freePotentialLocks() pass linter checks
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
fb13902db9 [#156] shard: Make refillMetabase() pass linter checks
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
3f6b962349 [#156] services/tree: Pass context to replicationWorker()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
5368c4207a [#156] services/tree: Split syncLoop() in functions
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
47e8c5bf23 [#156] pilorama: Remove CIDDescriptor from TreeApply()
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
   only used in controlled environments.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
ec2c5d45b4 Add to Possible Solutions
Signed-off-by: Liza <e.chichindaeva@yadro.com>
2023-03-21 15:41:20 +03:00
7eb9e88f8f Add bug label; Delte label addition
Signed-off-by: Liza <e.chichindaeva@yadro.com>
2023-03-21 15:09:22 +03:00
9aeea0b974 [#153] ci: Minor pipeline fixes
- We can skip full pre-commit run
- On a very slow agent golangci run may take up to 10 minutes

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-21 09:33:45 +00:00
Pavel Karpy
9a4f40626c [#128] IR: Do not try to emit GAS to nobody
Fix sending GAS to an empty extra wallets receivers list. Also, send GAS to
extra wallets even if netmap is empty.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-21 08:59:46 +00:00
7a31988a36 [#145] docs: Add expired object collector params
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
5059dcc19d [#145] shard-gc: Delete expired objects after locks
GC deletes expired locks and objects sequentially. Expired locks and
objects are now being deleted concurrently in batches. Added a config
parameter that controls the number of concurrent workers and batch size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
6c4a1699ef [#145] shard-gc: Expired locked unit test
Added unit test that verifies that GC deletes expired
locked objects in one epoch.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
Pavel Karpy
9cd8f7cea0 [#152] IR: Process empty basic incomes
If network is not configured for basic income earnings, do not distribute
GAS by the Alphabet nodes.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-21 06:59:33 +00:00
44b86bac5a [#148] linter: Add contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
481a1ca6f3 [#148] linter: Add gocognit linter
Code with high cognitive complexity is hard intuitively to understand

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
97c36ed3ec [#148] linter: Add funlen linter
Long functions are hard to understand and source of errors

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
cc8ff015b4 [#148] linter: Add containedctx linter
Context has to be passed as an argument: https://pkg.go.dev/context

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:52:39 +03:00
2dc86058c3 [#148] memstore: Drop space line
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:52:39 +03:00
573d920821 [#149] Use custom image and kludges for node
Until #139 is fixed, we can't use root inside Docker container running
CI, but Woodpecker CI can't run non-root containers until they fix
https://github.com/woodpecker-ci/woodpecker/issues/1077, hence we use
temporary kludges with custom image and manual permissions in pipelines.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-20 18:37:37 +03:00
d64fb887ff [#149] Reorder pre-commit hooks
Minor changes to see what fails first

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-20 18:37:34 +03:00
392be818e5 [#149] Fix gitlint regex to match our policy
In our policy we mark commits not having a PR/Issue yet with a `[#XX]`
reference to be replaced after PR creation.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-20 18:37:27 +03:00
db3ccd2762 [#128] innerring: Add GAS pouring mechanism for a configurable list of wallets
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-03-20 12:50:05 +00:00
Pavel Karpy
abd21f8099 [#136] Revert "[#2260] services/object: Do not assemble object with TTL=1"
This reverts commit 2567f8020e. It assumes
that assembling logic could break some failover scenarios if request
forwarding is done. However, it also breaks requesting big objects via a
non-container node with TTL=2. Failover has been rechecked without that
commit and no problems were found. Any (if found) other bugs related to
the forwarding and object assembling must be solved more carefully.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-20 10:52:18 +03:00
Pavel Karpy
10c419adf0 [#67] node: Fix infinite recursion in SE's wrapper
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-17 18:47:59 +03:00
b70caa216b [#144] Don't copy cache inside Docker environment
May make docker builds not so clean.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-17 10:36:50 +03:00
Pavel Karpy
64bde68fb9 [#67] node: Accept expired locked objects
Allow replication of any (expired too) locked object. Information about
object locking is considered to be presented on the _container nodes_.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-16 16:22:19 +03:00
Pavel Karpy
f006f3b342 [#67] node: Make engine's IsLocked public
It will allow reusing that method in expiration checks.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-16 16:20:45 +03:00
22be532cbd object/put: Persist session token till the end of a session
Previously a token could've expired in the middle of an object.PUT
stream, leading to upload being interrupted. This is bad, because user
doesn't always now what is the right values for the session token
lifetime. More than that, setting it to a very high value will
eventually blow up the session token database.

In this commit we read the session token once and reuse it for the whole
stream duration.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-16 06:45:50 +00:00
724debfdcd [#81] node: Add basic read/write benchmarks for substorages
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-15 16:37:04 +00:00
b1c165a93b [#83] Fix shellcheck pre-commit action
Original shellcheck action requires Docker to run and it's not always
available, especially inside Docker containers. Replacing it with python
wrapper to simplify usage with Docker-based CI systems like Drone and
WoodpeckerCI.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-15 12:24:41 +00:00
ac0a278a05 [#85] get-service: Drop unused assemble flag
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
b8e93d4c08 [#85] get-service: Use assembler to assemble LOB
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
07de839f18 [#85] get-service: Fix corrupted chain logic
Should return an error in case of a broken LOB reference chain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
2886b1581b [#85] get-service: Add unit tests
Add unit tests to cover all assemble statements

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
8b9e40a848 [#85] get-service: Add assembler
Extract assemble logic to assembler

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
b4582239bf [#130] adm: Fix adding of pub key for group.frostfs at init step
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-13 15:05:53 +03:00
4e244686cf [#83] Makefile fixes for pre-commit
Add make targets to simplify pre-commit setup for individual developers.

Signed-off-by: Stanislav Bogatyrev <realloc@realloc.spb.ru>
2023-03-13 11:39:18 +00:00
6cd806f998 [#82] services/tree: Save last synchronized height in a persistent storage
Remember the last synchronized height and use it after service restart.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 11:25:44 +00:00
3e6fd4c611 [#82] pilorama: Allow to store last sync height
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 11:25:44 +00:00
5ae4446280 [#50] ir: Add Health status
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-13 11:24:42 +00:00
5890cd4d7d [#50] ir: Fix config property name for prometheus
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-13 11:24:42 +00:00
365adb4ebd [#133] .github: Restore logo.svg
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 10:39:12 +03:00
bce5827f64 [#83] pre-commit: Add shellcheck hook
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:30 +00:00
05471d3827 [#83] util/autocomplete: Fix deprecated warning
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:30 +00:00
8226d49376 [#83] pre-commit: Add gitlint hook
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:30 +00:00
0893689c6a [#83] pre-commit: Add golangci-lint hook
Skip deprecated warning for now, adopting new neo-go API will be done in
another task.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:29 +00:00
a4931ea4c7 [#83] .github: Remove CODEOWNERS and actions
Issue templates are still supported by Gitea:
https://docs.gitea.io/en-us/issue-pull-request-templates/ .

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:29 +00:00
861e9ab59a [#83] pre-commit: Add initial configuration
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:29 +00:00
Leonard Lyubich
24a540caa8 [#132] cli/util: Fix basic ACL rendering
In previous implementation pretty-printer of basic ACL in NeoFS CLI had
mistakes:
 * F-bit was set to `Extendable()` property instead of its inversion
 * B-bits were set to `acl.RoleInnerRing` rights

Make `PrettyPrintTableBACL` to correctly render mentioned bits.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2023-03-10 14:55:45 +03:00
6226c3ba86 [#129] policer: Use safer defaults
If `processNodes` exits earlier for some reason, `needLocalCopy` could
be false.
See https://github.com/nspcc-dev/neofs-node/issues/2267

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-10 10:59:15 +00:00
f2250a316f [#129] tree: Do not remove tree if the netmap is empty
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-10 10:59:15 +00:00
9929dcf50b [#126] adm: Exclude group.frostfs key from output of the dump-hashes
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-09 18:29:36 +03:00
7486c02bbc [#88] adm: Fix method nnsResolveKey
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-09 15:41:28 +03:00
Pavel Karpy
f1f3c80dbf [#32] node: Init write-cache asynchronously
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-09 11:07:33 +00:00
Pavel Karpy
381e363a8b [#32] node: Always close general components after testing
It will prevent test fails with `-race` flag on components that have
background processes and make some actions on test framework.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-09 11:07:33 +00:00
20de74a505 Rename package name
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
79ba34714a [#79] cli: Fix panic when setting domain for container
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-01 11:11:11 +03:00
6d4f48f37a [TrueCloudLab#78] .github: Fix CODEOWNERS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-28 16:44:50 +03:00
e9f3c24229 [#65] Use strings.Cut instead of strings.Split* where possible
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-28 13:39:14 +03:00
88e3868f47 [#37] cli: Add nns-name and nns-zone for container create
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-28 13:37:23 +03:00
6925fb4c59 [TrueCloudLab/hrw#2] node: Use typed HRW methods
Update HRW lib and use typed HRW methods to sort shards and nodes

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-28 13:36:25 +03:00
c3a7039801 [TrueCloudLab/hrw#2] node: Optimize shard hash
Compute shard hash only once

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-28 13:36:25 +03:00
a1ab25b33e [#72] .github: Fix CODEOWNERS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-22 16:29:58 +03:00
73bb590cb1 [#64] node: Use pool_size_local and separate pool for local puts
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-22 13:43:19 +03:00
7eaf159a8b [#63] adm: Fix contract wallet creation
Create contract wallet only by init and update-config command.

Close #63

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-22 10:08:37 +03:00
cb5468abb8 [#66] node: Replace interface{} with any
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-21 16:47:07 +03:00
3d873237d5 [#44] Update Changelog
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
633c5a35de [#44] adm: Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
5f06232d34 [#44] cli: Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
bed5a36235 [#44] ir: Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
87e69b9349 [#44] node: Support multiple configs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-21 10:00:28 +03:00
Pavel Karpy
337049b2ce [#56] node: Allow reading expired locked object
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-21 09:56:57 +03:00
Pavel Karpy
3beef10f89 [#61] node: Do not fetch missing objects
If an object is missing in a `meta`, shard should not look for it in
a `blobstor`.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 14:47:38 +03:00
5303736acd [#62] Fix CHANGELOG.md
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-02-20 14:25:21 +03:00
22f3c7d080 [#1868] Reload config for pprof and metrics on SIGHUP
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-20 13:53:27 +03:00
2b755ddb12 [#2260] node: Use a separate client cache for PUT service
Currently, under a mixed load one failed PUT can lead to closing
connection for all concurrent GETs. For PUT it does no harm: we have
many other nodes to choose from. For GET we are limited by `REP N`
factor, so in case of failover we can close the connection with the only
node posessing an object, which leads to failing the whole operation.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
0b61a3c961 [#2260] network/cache: Ignore clients only on Dial errors
The problem is that accidental timeout errors can make us to ignore
other nodes for some time. The primary purpose of the whole ignore
mechanism is not to degrade in case of failover. For this case,
closing connection and limiting the amount of dials is enough.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
bf1e59bb83 [#2260] network/cache: Ignore context cancelled errors
Timeouts on client side should node affect inter-node communication.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
2567f8020e [#2260] services/object: Do not assemble object with TTL=1
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
d1d123d180 [#2234] writecache: Fix possible panic in initFlushMarks
In case we have many small objects in the write-cache, `indices` should
not be reused between iterations.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
315141dc2c [#2252] fstree: Allow concurrent writes
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
b422ac9f94 [#2164] node: Fix multi-client error reporting
Missing `ReportError` method did not allow casing multi-client interface to
`errorReporter` interface and dropping broken connections.
`replicationClient` embeds that interface, and it is widely used across
node's code. Embedded interface does not allow casting its parent structure
to `errorReporter` and breaks multi client error reporting logic.
Multi-client scheme is extremely hard to maintain, it makes unpredictable
casts and does not allow tracking code flow, so it will be refactored in the
future anyway.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
95ee905861 [#2244] node: Fix subscriptions lock
Subscribing without async listening could lead to a dead-lock in the
`neo-go` client.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
07ec51ea60 [#2244] node: Add object address to WC's operations
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
dbbbef9ddb [#2244] node: Update expired storage ID by WC
Previously, node could get an "infinite" small object: it could be expired
and thus could not be flushed (update its storage ID) to metabase => could
not be marked as flushed => node never removes such object and repeat all
the cycle one more time. If object exists and is not marked with GC (meta
returns `ErrObjectIsExpired`, not `ObjectNotFound` and not
`ObjectAlreadyRemoved`), its ID is safe to update _in the same_ bbolt
transaction.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
351fdd9fa2 [#2246] node: Allow to configure tombsone lifetime
Currently, DELETE service sets tombstone expiration epoch to
`current epoch + 5`. This works less than ideal in private networks
where an epoch can be e.g. 10 minutes. In this case, after a node is
unavailable for more than 1 hour, already deleted objects have a chance
to reappear.

After this commit tombstone lifetime can be configured.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
6fd88a036f [#2241] metrics: Fix request count metrics names
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
2272c55c4d [#2238] engine: Add test for component initialization failures
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
5cb2c5ae62 [#2238] engine: Add test for component initialization failures
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
59748b7ae8 [#2238] neofs-node: Gracefully handle shard initialization errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
427fe276f2 [#2238] shard: Try closing all components
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
c53903ccd0 [#2238] engine: Make Open and Init similar
1. Both could initialize shards in parallel.
2. Both should close shards after an error.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
e0309e398c [#2239] writecache: Fix possible deadlock
LRU `Peek`/`Contains` take LRU mutex _inside_ of a `View` transaction.
`View` transaction itself takes `mmapLock` [1], which is lifted after tx
finishes (in `tx.Commit()` -> `tx.close()` -> `tx.db.removeTx`)

When we evict items from LRU cache mutex order is different:
first we take LRU mutex and then execute `Batch` which _does_ take
`mmapLock` in case we need to remap. Thus the deadlock.

[1] 8f4a7e1f92/db.go (L708)

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
58367e4df6 [#2232] pilorama: Merge in-queue batches
To achieve high performance we must choose proper values for both
batch size and delay. For user operations we want to set low delay.
However it would prevent tree synchronization operations to form big
enough batches. For these operations, batching gives the most benefit
not only in terms of on-CPU execution cost, but also by speeding up
transaction persist (`fsync`).
In this commit we try merging batches that are already
_triggered_, but not yet _started to execute_. This way we can still
query batches for execution after the provided delay while also allowing
multiple formed batches to execute faster.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
236c4af615 [#2224] adm: Use native neo-go sessions in dump-hashes
If we had lots of domains in one zone, `dump-hashes` for all others
can miss some domains, because we need to restrict ourselves with _some_
number.
In this commit we use neo-go sessions by default, with a proper
failback to in-script iterator unwrapping.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
Pavel Karpy
40822adb51 [#2213] node: Do not return object expired object
"Object is expired" means that object is presented in `meta` but it is not
`ObjectNotFound` error. Previous implementation made `shard` search for an
object without `meta` which was an error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-20 13:53:27 +03:00
Roman Khimov
ad93d4db7c CHANGELOG: add more fancy glyphs
How could you forget adding it?

Signed-off-by: Roman Khimov <roman@nspcc.ru>
2023-02-20 13:53:27 +03:00
Roman Khimov
a6f071d66f CHANGELOG: fix whitespacing errors
Signed-off-by: Roman Khimov <roman@nspcc.ru>
2023-02-20 13:53:27 +03:00
9afe86ba3e [#2212] morph: Fix subscription restoration
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-20 13:53:27 +03:00
c43b2dbac9 [#1465] Add log entry for morph components shutdown action
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-17 12:13:00 +03:00
85cf1f47ac [#1465] node: Prevent process from killing by systemd when shutting down
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-17 12:13:00 +03:00
362f24953a [#47] shard: Switch container size metric from physical to logical capacity
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-02-17 12:03:42 +03:00
Pavel Karpy
901d62567d [#57] node: Broadcast link objects
It boosts object assembling by an _average_ container node.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-17 11:58:27 +03:00
Aleksey Pastukhov
269a4e9b50 [#53] Fix dirty in version
Signed-off-by: Aleksey Pastukhov <a.pastukhov@yadro.com>
2023-02-16 11:10:43 +03:00
3e5bc394b5 [#48] adm: Add initialize test for 1 node
Single node is used in dev-env, worth testing.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-16 09:07:18 +03:00
1d3669232e [#48] adm: Allow using nonzero magic with local client
neo-go actor API uses `getVersion` call which returned incorrect magic.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-16 09:07:18 +03:00
204cd3a11c [#31] fstree: Optimize treePath
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
dee4498c1e [#31] fstree: Do not check for a file existence twice
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
abbecf49d6 [#31] fstree: Speedup string-to-address conversion
```
name                  old time/op    new time/op    delta
_addressFromString-8    1.25µs ±30%    1.02µs ± 6%  -18.49%  (p=0.000 n=9+9)

name                  old alloc/op   new alloc/op   delta
_addressFromString-8      352B ± 0%      256B ± 0%  -27.27%  (p=0.000 n=9+10)

name                  old allocs/op  new allocs/op  delta
_addressFromString-8      6.00 ± 0%      4.00 ± 0%  -33.33%  (p=0.000
n=10+10)
```

Also, assure compiler that `s` doesn't escape:
Before this commit:
```
./fstree.go:74:24: leaking param: s
./fstree.go:90:6: moved to heap: addr
```

After this commit:
```
./fstree.go:74:24: s does not escape
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-10 12:49:31 +03:00
ab21d90cfb [#1794] shard: Add increasing case for the payload size metric
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-02-09 13:30:23 +03:00
5ffa826897 [#1] Update Changelog
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
cb016d53a6 [#1] Fix comments and error messages
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
c761a95eef [#1] Fix project name in control service
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
f825cfac78 [#1] Fix project name in comments here and there
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
c6645ef775 [#1] Documentation rebranding
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
488eece25f [#1] Fix Debian package references to old upstream
Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
1cedd446bb [#1] Fix sample configs
- Update sample configs to match unit tests
- Remove Docker container for N3 testnet
  Will return with updates when FrostFS enters N3

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
1858f11146 [#1] Fix viper env prefix in cli tools
Changing env prefix and corresponding example config files.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-02-06 17:41:14 +03:00
Pavel Karpy
73bc1b0b68 [#38] node: Fix linter warnings
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-02-06 17:27:54 +03:00
515c60bdf4 [#1889] adm: Add command morph netmap-candidates
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-06 17:26:34 +03:00
ee24815748 [#1889] Move flag --config in cmd/frostfs-adm/internal/commonflags/flags.go
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-06 17:26:34 +03:00
2b09564355 [#1889] Move netmap.go and exit.go from cli to cmd/internal/common
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-06 17:26:34 +03:00
5a9d6a09d8 [#8] cli: Set flag mode required for control shards set-mode
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-06 16:45:03 +03:00
2540598779 [#36] Packaging: allow any version
Also updated changelog

Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2023-01-31 16:33:32 +03:00
5d64a354cb [#1] Fix naming in FrostFS Adm help output
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-01-31 11:24:50 +03:00
d7e9e2ef9e [#1] Fix naming in FrostFS Lens help output
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-01-31 11:24:50 +03:00
d31d8c5335 [#1] Fix naming in FrostFS CLI help output
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-01-31 11:24:50 +03:00
406ff1360f [#1] Fix version output for all compiled binaries
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-01-31 11:24:50 +03:00
Pavel Karpy
89a0266f5e [#1794] metrics: Track physical object capacity per shard
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-01-26 20:06:28 +03:00
Evgenii Stratonikov
9513f163aa [#2116] metrics: Track physical object capacity in the container
Currently we track based on `PayloadSize`, because it is already stored
in the metabase and it is easier to calculate without slowing down the
whole system.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-01-26 20:06:28 +03:00
d65a95a2c6 [#28] pilorama: Remove LogMove struct
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
46c62be7e8 [#28] Fix linter issues
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
072a7d61ab [#2176] neofs-node: Do not cache full container list
We rarely need to list all containers: as one example
we need it for tree service synchronization once per epoch.
Given that cache TTL has the order of block time it makes no sense
to cache the list of all containers.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
8078af3424 [#2176] neofs-node: Do not invalidate old cache items
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
e5a40b90b6 [#2176] neofs-node: Peek during cache invalidation
`Get` needs write mutex and makes our item move to top.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
93eb72ef44 [#2176] neofs-node: Remove unused field from morphContainerWriter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
8ee590794f [#1962] cli: common.PrintVerbose prints via cobra.Command.Printf
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
cff4184cd3 [#2128] Add doc for extended headers
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
c72576e72f [#2208] engine: Log time-consuming shard operations
Currently the only way to tell whether `evacuate/set-mode` is finished
is to set a very big timeout and _hope_ that the operation will finish.
In this commit we add INFO logs for such operations which should
simplify the life of an administrator.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
87f0e3ea25 [#2208] fstree: Rename file after write
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
792319a044 [#2208] fstree: Remove file if there was an error during write
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
67c97c6804 [#2210] services/tree: Drop messages not in queue
Currently, under high load clients are blocked on channel send
and the number of goroutines can increase indefinitely.
In this commit we drop replication messages if send/recv queue is full
and rely on a background synchronization.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
25d5995cef [#2210] pilorama: Allocate bucket name outside of batches
1. Reduce allocations inside transactions.
2. Do not encode container ID to string: it allocates a lot and takes more
space.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
165a600624 [#2210] pilorama: Reduce the amount of keys per node
Under high load we are limited by the _amount_ of keys we need to update
in a single transaction. In this commit we try storing all state
with a single key.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
64a5294b27 [#2200] shard: Do not fetch big objects from blobovniczas
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
91757329ae [#2200] shard: Fix blobstor obj fetching
In the previous implementation any non-nil error that preceded object
fetching from blobstor led to iterating over every storage (in other words,
no storage ID information was taken into account). Now storage ID is
skipped only if metabase (storage ID source) returns any error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
cf1a91a758 [#2206] blobovnicza: Use Latin letters in the code
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
c33ad3c474 [#2164] node: Use reconnect_interval from config
Not always the default one.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
0d8366f475 [#2207] object/acl: Return status error for expired session token
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
6451f019d2 [#2203] shard: Do not panic in Close after unsuccessful Init
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
6efa93be0a [#1621] services/tree: Return Apply result asyncronously
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Evgenii Stratonikov
ac81c70c09 [#1621] pilorama: Batch related operations
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
9009612a82 [#2198] blobovniczatree: Properly handle concurrent active blobovnicza update
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
cedbd380f2 [#2197] pilorama: Close database in degraded mode
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
Pavel Karpy
1d21b1e3e8 [#1978] node: Do not drop clients on split errors
After the reconnection interval feature there was an bug related to the big
objects collecting: split error is returned from a client directly, not
via API status and was considered as a connection error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
554b85411f [#2190] services/object: Log service error with INFO level
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
b0ad1b9ed2 [#2193] pilorama: Use do in TreeMove
It should be similar to a `TreeAddByPath`. `applyOperation` is used for
`Apply` when the operation can be inserted in the middle of a log.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
ba393e3e91 [#2188] engine: Fix panic during setting shard mode
Under load changing shard mode can lead to it being removed from the
list during some other PUT.
```
Dec 28 07:01:26 az neofs-node[364505]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 28 07:01:26 az neofs-node[364505]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xc9fbb1]
Dec 28 07:01:26 az neofs-node[364505]: goroutine 11791912 [running]:
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).putToShard(0xc000435490, {0xc0003f7a28?, 0xc0001192c0?}, 0x2, {0x0, 0x>
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:91 +0x1b1
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).put.func1(0xc000435490?, {0xc0003f7a28?, 0xc0001192c0?})
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:71 +0x19c
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).iterateOverSortedShards(0x1?, {{0x62, 0x23, 0xfe, 0x60, 0x67, 0xd5, 0x>
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/shards.go:225 +0xc8
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).put(0xc000435490, {0x1?})
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:66 +0x2a9
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Put.func1()
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:43 +0x2a
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).execIfNotBlocked(0x8?, 0x38?)
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/control.go:147 +0xcf
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.(*StorageEngine).Put(0xc4df775a80?, {0x0?})
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:42 +0x65
Dec 28 07:01:26 az neofs-node[364505]: github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine.Put(0xc06d928b80?, 0xc06b1b8dc8?)
Dec 28 07:01:26 az neofs-node[364505]:         github.com/nspcc-dev/neofs-node/pkg/local_object_storage/engine/put.go:158 +0x19
Dec 28 07:01:26 az neofs-node[364505]: main.engineWithoutNotifications.Put({0x20301b?}, 0x20301b?)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:47 +03:00
118783c4cf [#26] .github: Fix CODEOWNERS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-25 15:31:23 +03:00
3d57f4c961 [#2179] test: Fix test TestEvacuateNetwork/multiple_shards,_evacuate_many
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-01-24 13:37:49 +03:00
d4d2a8c865 [#11] Rename NeoFS to FrostFS in docs
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
3d1d2ee7b1 [#11] Regenerate proto files
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
054bc4a727 [#11] Rename contract-related NeoFS occurences
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
9cb4b4cc17 [#11] Rename neofsid contract to frostfsid
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
023396e6a4 [#11] adm: Remove NeoFS mentions
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
19d180b510 [#11] Rename neofs contract to frostfs
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-01-12 08:55:47 +03:00
e355442532 [#10] .docker: Update go version to 1.18
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-01-10 12:27:25 +03:00
cc71e6902a [#7] Rebranding leftovers
- logo update
- docker image build fixes
- README and other texts updates

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-01-09 11:41:51 +03:00
fdb0affc31 [#5] frost-node: Used generic cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
9936b112b8 [#5] blobstor: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
f0be0befc5 [#5] services/object_manager: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
1b3374ac7f [#5] services/tree: User generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
d0a0432a51 [#5] innerring: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
4155c1bdff [#5] writecache: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
8f61cc1dcc [#5] policer: Use generic LRU client
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
55b403e0ee [#5] morph/client: Use generic LRU cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
d8fb9c85eb [#5] go.mod: Bump supported go version to 1.18
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-31 23:04:06 +03:00
6f8dd816fb [#2189] go.mod: Update nats-io
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
26560b6b8d [#2184] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
0272218eb9 [#2184] compression: Properly calculate upper bound
If the data is not compressible allocating `len(data)` will lead to a
slice reallocation. For a compressible data the results for small size
are flaky and we allocate a bit more. However, it feels right to use a
provided function if we need to pick any size at all.

```
name                                                           old time/op    new time/op    delta
Compression/size=128/zeroed_slice-8                              2.23µs ±12%    2.06µs ± 6%   -7.35%  (p=0.009 n=10+10)
Compression/size=128/not_so_random_slice_(block_=_123)-8         19.0µs ±10%    15.8µs ±16%  -17.09%  (p=0.000 n=9+10)
Compression/size=128/random_slice-8                              17.6µs ±15%    16.1µs ±16%     ~     (p=0.075 n=10+10)
Compression/size=1024/zeroed_slice-8                             3.05µs ±11%    2.84µs ±10%     ~     (p=0.089 n=10+10)
Compression/size=1024/not_so_random_slice_(block_=_123)-8        18.1µs ± 6%    18.2µs ±12%     ~     (p=0.971 n=10+10)
Compression/size=1024/random_slice-8                             48.6µs ± 6%    45.6µs ± 5%   -6.07%  (p=0.006 n=10+9)
Compression/size=32768/zeroed_slice-8                            26.8µs ± 3%    28.7µs ± 8%   +7.23%  (p=0.001 n=10+10)
Compression/size=32768/not_so_random_slice_(block_=_123)-8       44.3µs ± 8%    43.7µs ±13%     ~     (p=0.762 n=8+10)
Compression/size=32768/random_slice-8                            97.3µs ±32%    68.9µs ±15%  -29.13%  (p=0.000 n=10+10)
Compression/size=33554432/zeroed_slice-8                         29.8ms ± 9%    30.3ms ±17%     ~     (p=1.000 n=9+9)
Compression/size=33554432/not_so_random_slice_(block_=_123)-8    33.1ms ±14%    30.3ms ±11%   -8.61%  (p=0.043 n=10+10)
Compression/size=33554432/random_slice-8                         41.7ms ± 3%    30.1ms ± 8%  -27.72%  (p=0.000 n=9+10)

name                                                           old alloc/op   new alloc/op   delta
Compression/size=128/zeroed_slice-8                                128B ± 0%      144B ± 0%  +12.50%  (p=0.000 n=10+10)
Compression/size=128/not_so_random_slice_(block_=_123)-8           384B ± 0%      144B ± 0%  -62.50%  (p=0.000 n=10+10)
Compression/size=128/random_slice-8                                384B ± 0%      144B ± 0%  -62.50%  (p=0.000 n=10+10)
Compression/size=1024/zeroed_slice-8                             1.02kB ± 0%    1.15kB ± 0%  +12.50%  (p=0.000 n=10+10)
Compression/size=1024/not_so_random_slice_(block_=_123)-8        1.02kB ± 0%    1.15kB ± 0%  +12.50%  (p=0.000 n=10+10)
Compression/size=1024/random_slice-8                             2.56kB ± 0%    1.15kB ± 0%  -55.00%  (p=0.000 n=10+10)
Compression/size=32768/zeroed_slice-8                            32.8kB ± 0%    41.0kB ± 0%  +25.00%  (p=0.000 n=10+10)
Compression/size=32768/not_so_random_slice_(block_=_123)-8       32.8kB ± 0%    41.0kB ± 0%  +25.00%  (p=0.000 n=10+10)
Compression/size=32768/random_slice-8                            81.9kB ± 0%    41.0kB ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=33554432/zeroed_slice-8                         33.6MB ± 0%    33.6MB ± 0%   +0.02%  (p=0.000 n=9+9)
Compression/size=33554432/not_so_random_slice_(block_=_123)-8    33.6MB ± 0%    33.6MB ± 0%   +0.02%  (p=0.000 n=8+10)
Compression/size=33554432/random_slice-8                         75.5MB ± 0%    33.6MB ± 0%  -55.55%  (p=0.000 n=10+10)

name                                                           old allocs/op  new allocs/op  delta
Compression/size=128/zeroed_slice-8                                1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=128/not_so_random_slice_(block_=_123)-8           2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=128/random_slice-8                                2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=1024/zeroed_slice-8                               1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=1024/not_so_random_slice_(block_=_123)-8          1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=1024/random_slice-8                               2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=32768/zeroed_slice-8                              1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=32768/not_so_random_slice_(block_=_123)-8         1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=32768/random_slice-8                              2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Compression/size=33554432/zeroed_slice-8                           1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=33554432/not_so_random_slice_(block_=_123)-8      1.00 ± 0%      1.00 ± 0%     ~     (all equal)
Compression/size=33554432/random_slice-8                           2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
874be99076 [#2184] go.mod: Update neo-go to v0.100.1
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
71c64ae253 Release v0.35.0
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
Mikhail Petrov
cfdb53a788 [#614] Add logo for dark theme
Signed-off-by: Mikhail Petrov <mike@nspcc.ru>
2022-12-30 11:07:35 +03:00
a2fe912d1a [#2164] neofs-node: Use a separate client cache for replicator
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
741482c26f [#2164] neofs-node: Allow to set reconnect interval
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
6f5edac730 [#2164] network/cache: Do not reconnect to failed clients immediately
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
f3caf6acfe [#2164] network/cache: Separate mutex for addr
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
0ace28e43d [#2175] blobovniczatree: Close all non-active blobovniczas
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
c1cf418956 [#2175] blobovniczatree: Make function parameters more descriptive
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
b4e90cdf51 [#2165] pilorama: Optimize TreeApply when used for synchronization
Because synchronization _most likely_ will have apply already existing
operations, it is much faster to check their presence in a read
transaction. However, always doing this will degrade the perfomance
for normal `Apply`. And, let's be honest, it is already not good.
Thus we add a separate parameter which specifies whether this logic is
enabled.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
f9fcd85363 [#2165] services/tree: Remember starting height for the synchronization
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
06137dbf8e [#2165] services/tree: Do not export synchronizeAllTrees
It is used only privately.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
c299b98afe [#2165] services/tree: Parallelize synchronization
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
de9957e076 [#2165] services/tree: Always synchronize all containers
In case of split-brain we must synchronize everything.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
d3054e577a [#2165] services/tree: Allow to set custom synchronization interval
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
6a4e5e6f0a [#2144] node: Try node's private key if dynamic token fetching failed
`GETRANGEHASH` request spawns `GETRANGE` requests if an object could not be
found locally. If the original request contains session, it can be static
and, therefore, fetching session key can not be performed successfully.
As the best effort a node could request object's range with its own key.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
86a4fba571 [#2144] node: Clarify KeyStorage.GetKey method
Actualize the doc, fix API status error return.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
b7a99c757a [#2097] cli: Clarify help for --expire-at parameter for commands object lock/put and bearer create
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
e406036629 [#2166] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
21717262ec [#2016] shard: Check meta first on Get
`meta` should prevent returning removed objects (`GCMark` and `TS` relations
are `meta` abstractions).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
74ec71446f [#2167] shard: Do not use write-cache by default in Head
Both `meta` and `write-cache` are expected to have a fast underlying disk,
so it does not seem like an optimisation. Moreover, `write-cache`'s `Head`
is a `Get` with payload cutting, it _must_ use more memory for no reason
(`meta` was created for such requests). Also, `write-cache` does not allow
performing any "meta" relations checks (such as locking, tombstoning).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
1608fd1c07 [#2167] write-cache: Add "write-cache" to its logs
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
eea2892109 [#1956] node: Lock shard's mode on its methods switch
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
33d279a3f2 [#2152] cli: Do not search for LOCK objects when delete container when session provided
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
04b5ec759b [#2139] object/put: Use sync.Pool for temporary payloads
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
9e0decd12d [#2162] services/tree: Close connection after the syncronization
There was a goroutine leak here.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
e1c3bdbfa6 [#1621] pilorama: Remove Timestamp field from nodeInfo
It is already present in `Meta`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
1044adbe94 [#1621] pilorama: Improve memory allocation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
2539d466a6 [#1621] pilorama: Seek after cursor invalidation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
e9ba8931f8 [#1621] pilorama: Simplify bucket creation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Evgenii Stratonikov
fe7ddfdc6a [#1621] pilorama: Compare memory forests properly
Node children are not sorted and could occur in any order.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-30 11:07:35 +03:00
Pavel Karpy
306609030a [#2159] node: Add tree replication timeout configuration
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
edb1428248 [#2022] Add metric readonly to get shards mode
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-30 11:07:35 +03:00
e5c304536b [#2161] pilorama: Do not apply already existing operations
Speeds up synchronization a bit.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
3bb5a320d7 [#2154] services/tree: Do not log an error when synchronizing container of 1 node
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
387d1e2977 [#2127] services/tree: Randomize node order for synchronization
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
21c58c92a9 [#2145] meta: Do allow force inhuming a locked object
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
a30310b2ca [#2145] cli: Do not print "rpc error:" twice
That error part is already included in GRPC's library.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
b207dc424f [#2158] policer: Reduce default cache size
We use cache to avoid policing the same object multiple times in a short
time span (< 30 seconds). If we have 200_000 objects in a blobstor, it is a bit useless
-- if it takes 1 second to process an object and we have `replicator.pool_size: 20`
in config, the next iteration will happen in 10_000 second which is much
larger than 30 second. However we still consume a lot of memory, so it
makes sense to use saner default.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
0244f2e5ce [#2156] core/netmap: Change node addresses iteration order
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Leonard Lyubich
a68ff67ed8 [#2106] cli: Verify container owner in container delete command
In NeoFS containers can be removed on behalf of its owner only. To
improve user experience, there is a need to add ownership check to the
removal command of the NeoFS CLI.

Check container ownership in `container delete` command `Run` function.
The check can be skipped by `--force` option.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
73ef5b18c7 [#2092] go.mod: Update SDK
Includes a fix that adds error messages' text, it affects incomplete object
put error message.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
b413094704 [#2095] node: Fix collecting child objects
Stop child objects collection if the last returned object (the most "left"
object in the collected chain) starts exactly from the `GETRANGE`'s `from`
value.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-30 11:07:35 +03:00
Pavel Karpy
350eecfa13 [#2095] node: Do not allow GETRANGE requests with zero length
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-30 11:07:35 +03:00
Pavel Karpy
d6196c3971 [#2095] cli: Do not panic on object range
Also, includes range parsing error messages enhancement.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-30 11:07:35 +03:00
0b78af467e [#2140] engine: Fix error handling in TreeMove
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2022-12-30 11:07:35 +03:00
Pavel Karpy
923f84722a Move to frostfs-node
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2022-12-28 15:04:29 +03:00
Evgenii Stratonikov
42554a9298 [#2068] writecache: Remove deleted objects from the writecache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
4a49ea0855 [#2068] writecache: Allow to open FSTree in read-only mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
857d2dc3f5 [#2068] writecache: Optimize initial flush existence checking
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
63f604e948 [#2068] blobstor: Allow to provide storage ID in Exists
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
Evgenii Stratonikov
6ad2b5d5b8 [#2068] blobovnicza: Add Exists method
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-02 11:52:05 +03:00
9eccf3bbf5 [#1486] Update changelog
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
b0eeb0dfcf [#1486] node: Use endless notary deposit for side chain
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
ba74244d1b [#1486] ir: Use endless notary deposit for side chain
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
b6a40241f8 [#1486] morph: Add method for endless notary deposit
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-02 11:43:33 +03:00
Pavel Karpy
d54022eacc [#2047] node: Do not send chunk twice on request forwarding
That could happen if a node forwards request to a node that closed the
connection during the original object stream.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-12-02 11:27:48 +03:00
Evgenii Stratonikov
bd25db5d4a [#1984] metrics: Use separate metrics for success/failed requests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-01 14:18:10 +03:00
Evgenii Stratonikov
e21c472dc7 [#1984] services/object: Increase put_req_count after the request is processed
As it is specified in metrics description.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-12-01 14:18:10 +03:00
afabd6be91 [#2038] neofs-cli: Check the sufficiency of the number of nodes in the selector for replicas
Perform this check on container creation.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-12-01 11:32:27 +03:00
Pavel Karpy
50d28b72c3 [#2075] node, ir: Log notary state with info
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
0f41c09207 [#2075] node: Do not make notary requests on shutdown
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
b8c30b88f6 [#2075] morph: Do not do notary deposit with zero balance
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
c01d4ecb50 [#2080] morph: Close morph clients
Could be related to "websocket users limit reached" on the `neo-go` server
side when an SN/IR is rebooting repeatedly.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 19:26:33 +03:00
Pavel Karpy
3d0768a1d3 [#2061] node: Unify meta.Get benchmarks
Make them get exactly one (different) object per a bench iteration.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 18:29:14 +03:00
Pavel Karpy
bc905f169d [#2061] meta: Add parallel bench for Get
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 18:29:14 +03:00
Pavel Karpy
761e82fecd [#2079] cli: Do not panic in object hash
Sign RPC requests with the provided key.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 16:58:52 +03:00
Pavel Karpy
960a2d0629 [#2081] ir: Set default key in IR's SDK clients
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 16:58:52 +03:00
8d15c14be6 [#2078] adm: Pack parameters for setPrice invocation
Contract arguments have to be packed.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-11-30 16:58:52 +03:00
Pavel Karpy
51963abce7 [#1972] node: Fix errors comments in the Put service
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-30 16:58:52 +03:00
Evgenii Stratonikov
7335a52f29 [#1732] pilorama: Improve logical error handling
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:58:52 +03:00
Evgenii Stratonikov
ae7b473768 [#2064] blobovniczatree: Remove index too big log
There is no need to log about a situation which is expected.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:53:18 +03:00
8a77b4638a [#2012] Add commands neofs-cli acl basic/extended print to show ACL table in human readable format
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-30 16:45:37 +03:00
Evgenii Stratonikov
59db66cdb6 [#2091] neofs-adm: Sign blocks properly in tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
6882887bdd [#2091] neofs-adm: Do not query hashes via network
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
c1ea6fd854 [#2091] morph/client: Use notary.FakeContractAccount wrapper
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
b93be8869b [#2091] morph/client: Simplify code
1. Replace `mn` function with a `sigCount`.
2. Use `notary.FakeMultisigAccount` for account creation.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
cfefebd5b3 [#2091] neofs-adm: Do not use deprecated methods
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
Evgenii Stratonikov
6af52c46d8 [#2091] morph/client: Do not use deprecated methods
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-30 16:45:13 +03:00
51e886dd67 [#2090] neofs-cli: Remove --header from object get
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-30 11:46:02 +03:00
79130f781e [#2089] neofs-cli: Remove -g option from neofs-cli control ... and neofs-cli container create commands
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-30 09:14:50 +03:00
Evgenii Stratonikov
dd76ceadf1 [#2060] go.mod: Retract v1.* versions
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-29 09:29:42 +03:00
Pavel Karpy
ca3596dd05 [#2104] cli: OID signature in output
OID signature should always be present in an object; it does not relate
to the object split.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-29 08:52:09 +03:00
Evgenii Stratonikov
816c74d185 [#2075] morph/client: Ignore error if a transaction already exists
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
71853348b9 [#2063] morph/client: Support new hash format in morph nns client
Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-11-19 11:01:04 +03:00
Pavel Karpy
ed4351aab0 [#2074] write-cache: Do not flush same object twice
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
dd225906a0 [#2074] write-cache: Remove unused variables
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
9f7ac6bf9f [#2069] innerring: Do not panic in Head
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
98e6dc5ce8 [#409] debian: Refactor storage service paths
Separate User data and Service data:
 - /var/lib/neofs/storage for service persistence
 - /srv/neofs for user data

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
660c38d07e [#2062] services/policer: Use a proper key for object cache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
b1025bdb42 [#2057] meta: Fail write operations in R/O mode
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fdeea1dfac [#2057] meta: Fix concurrent mode changes
Includes:
1. mode change read lock operation in every exported method that r/w the
underlying database;
2. returning `ErrDegradedMode` logical error if any exported method is
called in degraded (without a metabase) mode.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
3d6defd3e8 [#2057] meta: Do not lock the whole meta on GET
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fa231b8c56 [#2057] blobstor: Block operations on a mode change
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
1779664644 [#2058] services/policer: Fix panic in shardPolicyWorker
```
2022/11/15 08:40:56 worker exits from a panic: runtime error: index out of range [0] with length 0
2022/11/15 08:40:56 worker exits from panic: goroutine 1188 [running]:
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:58 +0x10c
panic({0x1042b60, 0xc0015ae018})
	runtime/panic.go:1038 +0x215
github.com/nspcc-dev/neofs-node/pkg/services/policer.(*Policer).shardPolicyWorker.func1()
	github.com/nspcc-dev/neofs-node/pkg/services/policer/process.go:65 +0x366
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
	github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x68
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
b673d9e472 [#2053] engine: Do not switch mode because of logical errors
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
9a20498f34 [#1940] Removing all trees by container ID if tree ID is empty in pilorama.Forest.TreeDrop
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
426fe97990 [#2026] neofs-adm: Make contract update idempotent
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
634792077e [#1502] node: Store lock object on every container node
Includes extending listing methods in the Storage Engine with object types.
It allows tuning replication/policer algorithms: container nodes do
not remove `LOCK` objects as redundant and try to fulfill `LOCK` placement
on the ohter container nodes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fe09cd9c70 [#1502] core: Add AddressWithType
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
3b61cb4f49 [#1502] engine: Check all shards for LOCK'ing before inhuming
It allows keeping all the locked objects safe after metabase
resynchronization. Currently, all `LOCK` objects are broadcast to all nodes
in a container, it guarantees `LOCK` object presence in a regular situation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
34e8d2ba56 [#1502] shard: Add IsLocked method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
9a039ba582 [#1502] meta: Add IsLocked method
It gets an object and returns its locking status.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
7ef0303e13 [#2003] neofs-node: Allow to configure replicator pool size
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
659011c143 [#2048] network/cache: Optimize ClientCache
1. Remove a layer of indirection for mutex, `ClientCache` is already
   used by pointer.
2. Fix duplication of a `AllowExternal` field.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
597ed18269 [#2048] neofs-node: Use a separate client cache for client operations
Background workers can prevent user operations to complete because of
locking in cache.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
d5a14041e0 [#2040] node: Do not attach tokens in the assembly process
A container node is expected to have full "get" access to assemble the
object.
A non-container node is expected to forward any request to a container node.
Any token is expected to be issued for an original request sender not for a
node so any new request is invalid by design with that token.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
fd61bdadcb [#2040] node: Attach original meta to the spawned requests
Do not lose meta information of the original requests: cache session and
bearer tokens of the original request b/w a new generated ones. Middle
request wrappers should not contain any meta information, since it is
useless (e.g. ACL service checks only the original tokens).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
d65604ad30 [#1985] blobstor: Allow to report multiple errors to caller
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
f2d7e65e39 [#2035] engine: Allow moving to degraded from background workers
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
c85bea15ef [#1978] cli: Add children to the static session on DELETE
If an external session is provided and is not opened by CLI itself, add
children objects to it too. It fixes "not found" errors when removing a big
object with a predefined session (from a file).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
f48b1de54b [#2029] cli: Fix panic caused by flag redefinition
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
09a59fef56 [#2029] cli: Allow attaching static session to object hash
All the other object commands already have it.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
481b48b942 [#2028] node: Check session token's NBF and IAT
ACL service did not check "Not Valid Before" and "Issued At" claims.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
aadd2ad050 [#2028] node: Do not wrap malformed request errors
After presenting request statuses on the API level, all the errors are
unwrapped before sending to the caller side. It led to a losing invalid
request's context.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
2522d924b9 [#2037] services/object: Fix concurrent map writes in traverser
```
fatal error: concurrent map writes

goroutine 4337 [running]:
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*traversal).submitProcessed(...)
        github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:78
github.com/nspcc-dev/neofs-node/pkg/services/object/put.(*distributedTarget).iteratePlacement.func1()
        github.com/nspcc-dev/neofs-node/pkg/services/object/put/distributed.go:198 +0x265
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:68 +0x97
created by github.com/panjf2000/ants/v2.(*goWorker).run
        github.com/panjf2000/ants/v2@v2.4.0/worker.go:48 +0x65
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
37f813604f [#2000] cli: Provide a bearer token to spawned HEAD by DELETE
If a `neofs-cli object delete` operation is performing using a bearer token,
add it to the new `HEAD` requests that collects children OIDs.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
b0e94b6a6b [#1906] writecache: Do not require read-only mode in Flush
It was needed before we started to flush during transition to
`degraded` mode. Now it is confusing.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
aa478f1def [#2024] services/object: Unify status errors
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
3875fef542 [#2024] services/object: Cover corner cases for children OutOfRange
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
b9fb0d6050 [#1972] node: Fix object format unit tests
Includes:
1. Unused func removal;
2. Err check of the `Sign` method.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
aab398f4f5 [#1972] node: Do not save objects if node not in a container
Do not use node's local storage if it is clear that an object will be
removed anyway as a redundant. It requires moving the changing local storage
logic from the validation step to the local target implementation.
It allows performing any relations checks (e.g. object locking) only if a
node is considered as a valid container member and is expected to store
(stored previously) all the helper objects (e.g. `LOCK`, `TOMBSTONE`, etc).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
a77392e9ce [#1972] cli: Fix lifetime flag in the lock command
That part of the code was refactored incorrectly.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Pavel Karpy
2849e465f9 [#1699] meta: Do not return SplitInfoError on Delete
It is not an error: removing virtual object is expected and should be just
skipped. Getting a virtual object with `raw` flag is considered as an
impossible action, all the virtual objects removals will be handled via
their children's removals implicitly.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
a3e7365cbd [#1732] pilorama: Fill parent mark correctly
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
134f2ba02e [#1732] pilorama: Fix backwards log insertion
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
2ef38cfbc4 [#1996] engine: Ignore pilorama.ErrTreeNotFound for write operations
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
a455ec18c3 [#2007] services/object: Allocate memory on-demand in GET_RANGE
For big objects we want to get OutOfRange error before all the memory is
allocated.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
ff5526038d [#2007] services/object: Fix comment
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-19 11:01:04 +03:00
Evgenii Stratonikov
d8d3588e1b [#1996] engine: Always select proper shard for a tree
Currently there is a possibility for modifying operations to fail
because of I/O errors and a new tree to be created on another shard.
This commit adds existence check for modifying operations.
Read operations remain as they are, not to slow things.
`TreeDrop` is an exception, because this is a tree removal and trying
multiple shards is not an unwanted behaviour.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-03 15:29:23 +03:00
Pavel Karpy
08efe6eb11 [#2010] github: Run CI on support branches
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-02 14:29:17 +03:00
Evgenii Stratonikov
777fd32d4f [#1818] writecache: Increase error counter on background errors
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
bffb0f894c [#1818] writecache: Update storage ID during flush
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
5cf75404dc [#1818] metabase: Add UpdateStorageID operation
By default writecache puts the whole object to update storage ID.
This logic comes from the times when we needed to put objects
in the metabase by the writecache itself. Now this is done by the
blobstor at unmarshaling objects during flush only to update storage ID
is an overkill.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
b64b14eb54 [#1818] writecache: Reuse FSTree flushing code between flushes
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
Evgenii Stratonikov
a56927e3d4 [#1818] writecache: Remove unused variable
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-02 14:24:02 +03:00
bb52857b2b [#1338] neofs-cli: Add support to store/restore/delete binary objects
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-01 15:30:00 +03:00
Pavel Karpy
1f82c583e3 [#1971] cli: Unify CID and OID flags provision
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-01 15:07:19 +03:00
Pavel Karpy
7daa57d4d2 [#1991] cli: Refine container placement description
Not to confuse a user by mixing a replication vector number with its copy
number.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-11-01 15:00:00 +03:00
51e3810285 [#1689] Add new command morph list-containers in neofs-adm
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-11-01 09:43:02 +03:00
Evgenii Stratonikov
98a152256b [#1992] writecache: Allow to open in NOSYNC mode
Applicable only to FSTree as we cannot handle corrupted databases
properly yet.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Evgenii Stratonikov
b6930f2219 [#1992] neofs-node: Allow to open fstree in NOSYNC mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Evgenii Stratonikov
148da5fdbb [#1994] docs: Update storage node configuration
Reflect the reality after a not so recent refactoring.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Evgenii Stratonikov
f564430b90 [#1992] fstree: Allow working in SYNC mode
Make O_SYNC the default and allow to opt-out explicitly.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-11-01 09:42:26 +03:00
Pavel Karpy
694d888219 Release v0.34.0 - Marado (마라도, 馬羅島)
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-31 15:07:53 +03:00
Pavel Karpy
d739e06289 [#1980] make: Update linter version in docker command
To be in sync with GitHub actions.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-31 15:07:53 +03:00
Pavel Karpy
cd2faf29a9 [#1980] go.mod: Update neofs-sdk to rc7
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-31 15:07:53 +03:00
Evgenii Stratonikov
34501685b7 [#1969] local_object_storage: Move ErrObjectIsExpired to another package
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
56de2f1363 [#1969] local_object_storage: Simplify logic error construction
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Evgenii Stratonikov
fcdbf5e509 [#1969] local_object_storage: Add a type for logical errors
All logic errors are wrapped in `logicerr.Logical` type and do not
affect shard error counter.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-31 11:41:24 +03:00
Leonard Lyubich
98034005f1 [#1900] node: Fix loosing the "maintenance" status
In previous implementation node lost maintenance status after successful
switching to it. For example, after some period of time node sent
bootstrap requests with the "online" state instead of "maintenance".

Make `startMaintenance` method to set maintenance status in the
`networkState`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:59 +03:00
Leonard Lyubich
db92e96e40 [#1900] node: Refactor bootstrap methods
In previous implementation bootstrapping state was chosen according to
bool flag which was not convenient.

Create separate method to boostrap with "online" and the current state.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:59 +03:00
Leonard Lyubich
e8c5f03c30 [#1905] shard: Don't log read-only errors of write-cache
There is no need to log `writecache.ErrReadOnly` errors in `Delete`
method of the `Shard`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:45 +03:00
Leonard Lyubich
b1fa084756 [#1905] shard: Decrease severity level of write-cache failure logs
In previous implementation `Shard.Delete` logged writecache's removal
failures in `error` level. There is a need to decrease severity of these
log records since they aren't critical and don't require individual
review.

Change level of the message to `info`.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 18:30:45 +03:00
Leonard Lyubich
98ac525272 [#1978] cli/object: Gather all related object in delete session
Object removal session should reflect all objects related to the
removing one.

Make `OpenSessionViaClient` to gather the split members of the original
object in order to spread the session to them.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-28 13:49:41 +03:00
Stanislav Bogatyrev
7653a1f626 [#409] Fix package building doc typos
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
43fe156f7c [#409] Fix more lintian warnings
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
c640374d98 [#409] Return OS release suffix to package version
Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2022-10-28 12:58:32 +03:00
1e0b1919ed [#409] Fix lintian warnings
Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
bbfc1d9263 [#409] Fix neofs-locode-db dependency
And update CHANGELOG.md =)

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
e1a085ffd5 [#409] Remove docker target
It would not work this way for non-root users. Let's postpone.

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
Stanislav Bogatyrev
ff67e903ca [#409] Add default package changelog
Make it work with simple `dpkg-buildpackage`

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-10-28 12:58:32 +03:00
92e9782c44 [#409] Debian packaging
Debian package building, including:

 - users creation if not exist;
 - dirs creation and ownership/ACLs set;
 - man/doc/examples installation;
 - shell completion generation;
 - locode DB binary download

Using:
	make debpackage

Signed-off-by: Dmitriy Zabolotskiy <d.zabolotskiy@yadro.com>
2022-10-28 12:58:32 +03:00
Evgenii Stratonikov
846ff515e6 [#1812] policer: Do not remove copies if there are maintenance nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 19:13:17 +03:00
Evgenii Stratonikov
280e56f4bb [#1893] neofs-node: Do not fail unless all gRPC endpoints are unavailable
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:48:38 +03:00
Evgenii Stratonikov
1e6588e761 [#1974] shard: Do not panic in degraded mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00
Evgenii Stratonikov
7395ab8ef7 [#1944] metabase: Assume static buckets are created on Init
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00
Evgenii Stratonikov
e4bc3d0e9d [#1944] metabase: Recreate static buckets instead of resetting
From the `Bucket.ForEach` doc:
```
The provided function must not modify the bucket; this will result in undefined behavior.
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 12:41:12 +03:00
Leonard Lyubich
3c6daa2995 [#1883] cli/shards: Exclude degraded-read-write mode from help
The mode is not expected to be used by the user. It is supported only
for developers.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-26 08:24:40 +03:00
Leonard Lyubich
eb206d6e59 [#1883] config/example: Actualize desc of shard modes
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-26 08:24:40 +03:00
Leonard Lyubich
c4a2a283ae [#1883] cli/shards: Make supporting new modes easier
Add new shard modes as a map entry to automatically parse them in
`set-mode` command. The change also automatically adds new modes to help
message.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-26 08:24:40 +03:00
Evgenii Stratonikov
3b939d190c [#1957] engine: Move shard to read-only if cannot move to degraded
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-26 08:20:53 +03:00
Leonard Lyubich
7f54462e60 [#1920] cli: Print used RPC timeout in verbose exec
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-26 08:20:36 +03:00
Pavel Karpy
9a0824766c [#1713] node: Add Neo RPC switch interval to the config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
0631b38da1 [#1713] ir: Add Neo RPC switch interval to the config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
d240e2dc87 [#1713] morph: Allow switch interval configuration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
d09ec2e3fe [#1615] node/config: Remove unused default constants
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
17f7d0a2ee [#1615] morph: Switch to a more prioritized RPC node
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Pavel Karpy
7c0aa69d11 [#1615] morph: Fix the switch comment message
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-25 17:38:48 +03:00
Evgenii Stratonikov
713fdab177 [#1907] shard: Return from Close after GC has stopped
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-25 11:54:45 +03:00
Evgenii Stratonikov
4fe1aad30f [#1966] neofs-adm: Fix signature creation for multiactor scenario
`SignTx` from neo-go appended verification script only if `Scripts`
array was not completely filled.

```
can't persist transactions: transaction b9cd02fb81721c7316c908687301eeaf8ef384e381476632c24a77bdf9ffacd5 failed to verify: witness #1: unknown verification contract
```

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-25 11:22:00 +03:00
Evgenii Stratonikov
2dcf8d51be [#1966] neofs-adm: Set VerifyBlocks: true and fix NetworkFee calculation
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-25 11:22:00 +03:00
Pavel Karpy
41f93dcc1d [#1953] blobstor: Log any object removal operation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:52:18 +03:00
Pavel Karpy
942cc38146 [#1953] engine: Add refactored storage helpers
After the refactor there are new storage characteristics: a type and
a general storage id (that could be stringified).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:52:18 +03:00
Pavel Karpy
0371d15b2f [#1953] shard: Fix debug log messages
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:52:18 +03:00
Pavel Karpy
f8180447a1 [#1938] meta: Make version error messages more descriptive
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-24 21:46:18 +03:00
Evgenii Stratonikov
f396ffcc0f [#1959] CHANGELOG.md: Update neofs-adm remarks
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
6c9423cfb0 [#1959] neofs-adm: Add tests for more commands
Also, fix `restore-containers`, but it is hard to test it now.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
7c3ac7150d [#1959] neofs-adm: Remove newTempDir function
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
2404a267b9 [#1959] neofs-adm: Move testdata generation to a separate function
Later we could make it a separate tests an run neofs-adm tests on CI.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
6c21e2cc28 [#1959] neofs-adm: Fix set-config for >4 nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
c87c3d3721 [#1959] neofs-adm: Refactor tests
Make it easier to test both common deployment scenarios.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
02bea52f40 [#1959] neofs-adm: Allow to test all commands with a local dump
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
8648b102db [#1959] neofs-adm: Dump all blocks in file
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
23622ed283 [#1959] neofs-adm: Check transaction validity in tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
942155be6e [#1959] neofs-adm: Fix double multisig
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
2484c1d35e [#1959] neofs-adm: Verify blocks and transactions in tests
Also update to the actual contracts version.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 21:43:23 +03:00
Evgenii Stratonikov
926830bb9c [#1961] network/cache: Do not return nil client
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 17:56:34 +03:00
Leonard Lyubich
a4573d5026 [#1933] cli: Document sessions
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 17:45:22 +03:00
Leonard Lyubich
7b418c36b4 [#1930] services/session: Log calling Create RPC
There is a need to check if session is opened during system
testing/debug.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 17:45:22 +03:00
Leonard Lyubich
8bba490c30 [#1933] cli: Support binary sessions
There is a need to support NeoFS-binary sessions along with JSON ones in
NeoFS CLI.

Provide generic `common.ReadBinaryOrJSON` functions which tries to
decode NeoFS-binary structure and falls back to JSON format. Use this
function in all places with token reading.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 17:45:22 +03:00
Evgenii Stratonikov
04b67f3ba5 [#1949] neofs-adm: Fix epoch tick for >4 node committee
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:36:08 +03:00
Evgenii Stratonikov
8796807040 [#1936] network/cache: Handle dial errors
After an SDK update `Dial` can return an error.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:53 +03:00
Evgenii Stratonikov
a189eca5d5 [#1908] docs: Describe SIGHUP behaviour
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
1beafea0b5 [#1869] shard: Add logs for SetMode operations on reload
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
87be4f1629 [#1869] shard: Restore shard mode on failed reloads
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
c785e11b20 [#1869] shard: Allow to reload metabase on SIGHUP
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Evgenii Stratonikov
f769fc83fc [#1869] shard: Embed gcCfg as raw struct
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-24 13:28:26 +03:00
Leonard Lyubich
60e9de8d63 [#1916] control: Check maintenance allowance on Control server
In previous implementation turning to maintenance mode using NeoFS CLI
required NeoFS API endpoint. This was not convenient from the user
perspective. It's worth to move networks settings' check to the server
side.

Add `force_maintenance` field to `SetNetmapStatusRequest.Body` message
of Control API. Add `force` flag to `neofs-cli control set-status`
command which sets corresponding field in the requests body if status is
`maintenance`. Force flag is ignored for any other status.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 09:20:24 +04:00
Leonard Lyubich
c50603494b [#1916] cli/control: Pre-check maintenance mode in set-status
Node's maintenance state can be set only if the network allows it.

Make `neofs-cli control set-status` to read current network settings and
check is maintenance state is allowed. Fail the execution if the mode is
not allowed.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-24 09:20:24 +04:00
Pavel Karpy
e1be0180f6 [#1329] tree: Sync trees when a node first time appears in a container
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-21 18:47:56 +03:00
Leonard Lyubich
217b030d20 [#1946] cli/internal: Fix test by wallet non-closing
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-21 15:21:59 +03:00
Leonard Lyubich
810087d8b9 [#1946] Bump neo-go module to v0.99.4
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-21 15:21:59 +03:00
Evgenii Stratonikov
de055f27c7 [#1942] neofs-adm: Fix multisig checks in tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-21 10:16:51 +03:00
Pavel Karpy
7d456cb4d0 [#1902] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
49c38d642d [#1902] engine: Search for the tree IDs in every shard
Iterate over every shard and search for the container's trees. Final result
is a concatenation of shards' results. It is considered that one fixed tree
is placed on one fixed shard but the different trees of a fixed container
could be placed on different shards.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
aa37078570 [#1902] node: Sync all the trees on bootstrap
Do not limit synchronization by "system" and "version" tree IDs.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
1766ca2039 [#1902] tree: Allow synchronize all the container trees
Add `SynchronizeAllTrees` method of the Tree service. It allows fetching
tree IDs and sync all of them. Share common logic b/w the new method and
the `SynchronizeTree`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
1805c6606d [#1332] cli: Add tree list
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
6d4beea187 [#1902] tree: Extend grpc service with ListTrees method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
24e9e3f3bf [#1902] engine, shard: Implement TreeList method
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
2e199c7ab1 [#1902] shard: Fix pilorama disabled err message
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
19850ef157 [#1902] pilorama: Add TreeList method
To both `bolt` and `memory` forests; extend `Forest` interface.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Pavel Karpy
3aa9938b8f [#1902] Update protoc to v3.21.7
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-20 16:17:57 +03:00
Evgenii Stratonikov
9ec01bb9c1 [#1931] control: Allow to clear errors in SetShardMode RPC
It hasn't been working since the initial implementation 7fb15fa1d0.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-20 15:51:31 +03:00
Evgenii Stratonikov
a95fad833e [#1909] docs: Add shard modes description
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-20 08:53:58 +03:00
Evgenii Stratonikov
0d14ef69f0 [#1922] neofs-node: Allow to go online after maintenance
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-19 10:14:56 +03:00
Evgenii Stratonikov
1980ed968a [#1926] neofs-node: Use writecache.max_object_size from the config
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 16:44:28 +03:00
Evgenii Stratonikov
13e818e2ca [#1910] .github: Update golangci-lint version
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
0d65888005 [#1910] .golangci.yml: Add predeclared linker
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
1cb892c579 [#1910] .golangci.yml: Add misspell linker
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
9e46799cd4 [#1910] .golangci.yml: Add more useful linkers
Steal them from the neo-go repo.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
d772e35aba [#1910] .golangci.yml: Add godot linker
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
347912ea0b [#1910] .golangci.yml: Move unused to a list of default linters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 15:08:26 +03:00
Evgenii Stratonikov
0a1530afa0 [#1917] neofs-cli: Support timeout in control commands
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-18 10:58:10 +03:00
Pavel Karpy
4c63be6629 [#1329] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
bc67f77d86 [#1329] tree: Fix sync applying
Add the node position in a container and the container size to the CID
descriptor that is passed to the `TreeApply`. Previously, `checkValid` does
not allow any log operarations.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
c16a31ef3e [#1329] node: Set undefined netmap status by default
It allows general usage of `controlNetmapStatus` getter: does not matter of
handler (`NewEpoch` handler of initial setter).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
f4a3fa2977 [#1329] tree: Sync tree on startup
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
f5f560d903 [#1329] tree: Do not sync trees that are from external containers
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:53:34 +03:00
Pavel Karpy
ea9a5690cb [#1918] Fix v0.33.0 CHANGELOG record
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 20:42:32 +03:00
Evgenii Stratonikov
e2970bf892 Release v0.33.0
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-17 19:09:42 +03:00
Evgenii Stratonikov
b40dd10b7a [#1913] go.mod: Update neofs-api-go to v2.14.0
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-17 19:09:42 +03:00
Evgenii Stratonikov
f524c812c1 [#1913] go.mod: Update neo-contract to v0.16.0
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-17 19:09:42 +03:00
Leonard Lyubich
d96139b201 [#1900] node: Do not override maintenance state on bootstrap
In previous implementation node always sent bootstrap requests with
"online" network state. This provoked switching to online of the nodes
under maintenance.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-17 18:59:22 +03:00
Pavel Karpy
f76516a883 [#1860] meta: Fix 0,1 -> 2+ version migration
In the 2nd version, there was a database format change: buckets have changed
their keys, so it becomes impossible to check the version in the 1 -> 2+
migrations because of different buckets that store info about the version.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-17 13:42:05 +03:00
Leonard Lyubich
6eb82c4cbf [#1890] docs: Describe maintenance mode's behavior
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-17 13:40:56 +03:00
Evgenii Stratonikov
8145e79d7f [#1903] neofs-adm: Expand update parameters in MakeCall
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-17 13:37:48 +03:00
anastasia prasolova
bcf3df354b Add CODEOWNERS file
Signed-off-by: anastasia prasolova <anastasia@nspcc.ru>
2022-10-16 21:09:07 +03:00
Leonard Lyubich
1406d096a2 [#1680] service/object: Fail all operations in maintenance mode
Storage node should not provide NeoFS Object API service when it is
under maintenance.

Declare `Common` service that unifies behavior of all object operations.
The implementation pre-checks if node is under maintenance and returns
`apistatus.NodeUnderMaintenance` if so. Use `Common` service as a first
logical processor in object service pipeline.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-14 13:54:32 +04:00
Leonard Lyubich
05420173cc [#1894] services/object: Ignore unrelated sessions in client
In some scenarios original session can be unrelated to the objects which
are read internally by the node. For example, node requests child
objects when removing the parent one.

Tune internal NeoFS API client used by node's Object API server to
ignore unrelated sessions in `GetObject` / `HeadObject` / `PayloadRange`
ops.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-14 12:37:31 +03:00
Evgenii Stratonikov
2050a623ac [#1896] neofs-node: Set MAINTENANCE status on update
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-14 11:46:19 +03:00
Pavel Karpy
ade19077ee [#1332] Update Changelog
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
beb1998ed1 [#1332] cli: Print missing nodes message verbosely
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
6e4f7180fe [#1332] cli: Implement tree add-by-path command
It is `TreeService.AddNodeByPath` method implementation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
3d8349d7f8 [#1332] cli: Implement tree get-by-path command
It is `TreeService.GetNodeByPath` method implementation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
2c02e66939 [#1332] cli: Implement tree add command
It is `TreeService.Add` method implementation.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Pavel Karpy
13c4a9f4b8 [#1332] tree: Make SignMessage public
It will allow reusing signing routine in other components
(e.g. `neofs-cli`).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 20:01:48 +03:00
Evgenii Stratonikov
80d3c7f9d6 [#1892] neofs-adm: Support MaintenanceModeAllowed network setting
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-13 17:45:47 +03:00
Evgenii Stratonikov
f2793060c5 [#1892] neofs-cli: Display maintenance status in netmap nodeinfo
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-13 17:45:47 +03:00
Pavel Karpy
31c623636d [#1863] node: Fix shard id in the object counter metrics
If shard ID is stored in metabase (it is not the first time boot), read it,
set it, use it (not a generated one) in the metrics writer.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-13 13:06:41 +03:00
8714fc42b5 [#1765] Use hex format to print storage node ID
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-13 12:55:21 +03:00
Pavel Karpy
5a2daadd37 [#1770] Update Changelog
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Pavel Karpy
b6806ea6b9 [#1770] node: Support logger config rereading
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Pavel Karpy
8c75cb1dad [#1770] node: Validate logger config section
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Pavel Karpy
d7c7022bbd [#1770] logger: Support runtime level reloading
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Pavel Karpy
f037022a7a [#1770] logger: Refactor Logger component
Make it store its internal `zap.Logger`'s level. Also, make all the
components to accept internal `logger.Logger` instead of `zap.Logger`; it
will simplify future refactor.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 18:11:05 +03:00
Evgenii Stratonikov
4baf00aa21 [#1884] services/object: Fallback to GET in GET_RANGE
Current spec allows denying GET_RANGE requests from other storage nodes.
However, GET should always be allowed and it is enough to perform
GET_RANGE locally

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 17:05:51 +03:00
Leonard Lyubich
dde4d4df2a [#1878] services/object: Fix child check in GET
In previous implementation `ObjectService.Get` RPC handler failed with
`parent address in child object differs` while assembling the "big"
object. This was caused by the child check which required parent
reference to be set in all child objects. The check was impracticable
because not all elements of the split-chain have a link to the parent.

Make `execCtx.isChild` to return `true` if parameterized object has no
parent header in its own header.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-12 16:56:37 +03:00
Leonard Lyubich
9a006ac14f [#1865] cli/object: Do not open remote sessions in reading ops
It does not make sense to open remote sessions with the storage node in
`get`, `head`, `search`, `range` and `hash` sub-commands of `neofs-cli
object` command.

Do not use NeoFS API `SessionService` in mentioned commands. Decode
object session from JSON file specified `--session` flag. Perform some
sanity checks instantly on CLI side.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-12 13:22:19 +03:00
Evgenii Stratonikov
49eab6318c [#1867] control: Fix degraded-read-only mode parsing
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
Evgenii Stratonikov
c0199dee93 [#1867] services/control: Interpret empty list of IDs as all shards
In neofs-cli the flag is still required, but `all` can be used to
process all shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
Evgenii Stratonikov
b632260995 [#1867] neofs-cli: Support multiple shard IDs in control subcommands
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
Evgenii Stratonikov
19c0a74e94 [#1867] services/control: Allow to provide multiple shard IDs to some commands
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 11:20:48 +03:00
74d2f2c8d3 [#1854] cli: Unify help messages
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-12 09:53:11 +03:00
Pavel Karpy
9e2edfedc6 [#1833] node: Update local node info on nil updates
If the contract returns a netmap that does not contain the node, update
local `NodeInfo`. It fixes `neofs-cli netmap nodeinfo` command that printed
"state: online" previously.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-12 09:52:51 +03:00
Evgenii Stratonikov
c5fdb7bedf [#1876] .github: Add an action monitoring config updates
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-12 09:33:28 +03:00
Leonard Lyubich
050ad2762c [#1680] replicator: Consider NODE_UNDER_MAINTENANCE as OK
Node response with `NODE_UNDER_MAINTENANCE` status signals that the node
was switched to maintenance mode. There is a delay between the actual
switch and the reflection in the network map of up to one epoch. To
speed up the reaction to the maintenance, it is required to recognize
such node responses in the Policer.

Make `Policer.processNodes` to exclude elements with shortage decreasing
on `NODE_UNDER_MAINTENANCE` status response.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-11 12:54:27 +03:00
Leonard Lyubich
e99e25b52f [#1680] replicator: Consider nodes under maintenance as OK
Nodes under maintenance SHOULD not respond to object requests. Based on
this, storage node's Policer SHOULD consider such nodes as problem ones.
However, to prevent spam with the new replicas, on the contrary, Policer
should consider them normal.

Make `Policer.processNodes` to exclude elements if `IsMaintenance()`
with shortage decreasing.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-11 12:54:27 +03:00
Leonard Lyubich
df5d7bf729 [#1680] replicator: Work with netmap.NodeInfo in TaskResult
Make `replicator.TaskResult` to accept `netmap.NodeInfo` type instead of
uint64 in order to clarify the meaning and prevent passing the random
numbers.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-11 12:54:27 +03:00
Leonard Lyubich
e6f8904040 [#1680] policer: Refactor tracking the processed nodes
Add clear methods with docs. Use the methods instead of direct map
and bool instructions.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-11 12:54:27 +03:00
feaa9eace7 [#1768] Add healthcheck method to the Tree service
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-11 12:53:45 +03:00
Leonard Lyubich
807c0a1321 [#1859] services/object: Do not session check relation in PUT
It doesn't make sense to check object relation in session check of
`ObjectService.Put` RPC which has been spawned by `ObjectService.Delete`
with session. Session issuer can't predict identifier of the tombstone
object to be created.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-10 20:09:47 +03:00
Evgenii Stratonikov
2d43892fc9 [#1840] neofs-node: Use blobstor paths to identify shard
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
4b005d3178 [#1840] blobstor: Return info about all components
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
6d7ffefec5 [#1840] blobstor/test: Add tests for auxiliary functions
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
328691c94f [#1840] blobstor: Return Path from components
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Evgenii Stratonikov
9b241e4a17 [#1840] neofs-node: Allow to use mode: disabled in config
Currently, when removing shard special care must be taken with respect
to shard numbering. `mode: disabled` allows to leave shard configuration
in place while also ignoring it during initialization. This makes
disk replacement much more convenient.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-10 11:14:55 +03:00
Pavel Karpy
1360273fec [#1856] node: Fix config rereading
Config was reread from the file only once in two SIGHUPs.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-10 10:35:52 +03:00
Evgenii Stratonikov
b2aa9947c2 [#1829] engine: Delete split objects properly
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 16:35:46 +03:00
5e493b7f1c [#1704] Add command container nodes to output list of nodes for container, grouped by replica (#1704)
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-07 14:58:01 +03:00
Evgenii Stratonikov
6557f5d249 [#1839] engine: Handle Inhume errors properly
If shard is in read-only or degraded mode, there is no need to increase
error counter.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 14:25:52 +03:00
Evgenii Stratonikov
7356ee91ff [#1837] services/object: Optimize uniqueIDWriter
Avoid encoding object ID to string.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 13:29:15 +03:00
Evgenii Stratonikov
2ac42b70ce [#1826] services/object: Parallelize object search
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 13:29:15 +03:00
Evgenii Stratonikov
90bfe0bad9 [#1826] neofs-cli: Add --timeout flag
Allow to specify it everywhere `--rpc-endpoint` flag is present.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 13:29:15 +03:00
Evgenii Stratonikov
0cb8e7f6f1 [#1837] services/object: Fix log messages
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 13:29:15 +03:00
Evgenii Stratonikov
ca8dc872b2 [#1846] neofs-node: Make morph.cache_ttl equal to block time by default
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 10:34:55 +03:00
Leonard Lyubich
e54b52ec03 [#1420] object/acl: Fix correlation of object session to request
In previous implementation of `neofs-node` app object session was not
checked for substitution of the object related to it. Also, for access
checks, the session object was substituted instead of the one from the
request. This, on the one hand, made it possible to inherit the session
from the parent object for authorization for certain actions. On the
other hand, it covered the mentioned object substitution, which is a
critical vulnerability.

Next changes are applied to processing of all Object service requests:
 - check if object session relates to the requested object
 - use requested object in access checks.

Disclosed problem of object context inheritance will be solved within

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-07 10:34:38 +03:00
Evgenii Stratonikov
5834f9807e [#1847] services/policer: Provide container ID in logs
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-07 09:58:16 +03:00
Evgenii Stratonikov
c20c2e3b39 [#1845] morph/client: Use 0 as OfflineState in update peer
As the documentation prescribes.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-06 12:26:25 +03:00
Pavel Karpy
01ddb3f8e6 [#1770] node: Refactor application configuration
Split all the fields in `cfg` structure on:
1. `applicationConfiguration`;
2. `internals`; // shared entities for an application work, such as
`context.Context`
3. `shared`; // holder for the shared entities b/w;
4. `cfgXXX`; // configuration for internal services.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-05 14:25:33 +04:00
Leonard Lyubich
713aea06fa [#1681] node: Block only Object service ops under maintenance
In previous implementation node blocked any operation of local object
storage in maintenance mode. There is a need to perform some storage
operations like data evacuation or restoration.

Do not call block storage engine in maintenance mode. Make all Object
service operations to return `apistatus.NodeUnderMaintenance` error from
each local op.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
082602b668 [#1680] Update changelog
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
a7668618c9 [#1680] cli/netmap: Support MAINTENANCE mode
Make `netmap snapshot` command to print `MAINTENANCE` state of the nodes
with `IsMaintenance()` flag set.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
eb1fba5182 [#1680] morph/netmap: Adopt to recent contract changes
After recent Netmap contract changes all read methods which return
network map (either candidates or snapshots) encode node descriptors
into same structure.

Decode `netmap.Node` contract-side structure from the call results.
Replace node state with the value from the `netmap.Node.State` field.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
d6c01199c8 [#1680] node: Return MAINTENANCE status in corresponding mode
Make storage node to return `NODE_UNDER_MAINTENANCE` status
error on each local object operation if the node is in `MAINTENANCE`
mode.

Pass `apistatus.NodeUnderMaintenance` to `StorageEngine.BlockExecution`
during `ControlService.SetNetmapStatus` RPC processing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
9fcc80ea38 [#1680] node/control: Send request to switch to MAINTENANCE state
After recent changes `MAINTENANCE` state is reflected in the Sidechain.
Storage node should switch its state to "maintenance" during serving the
`ControlService.SetNetmapStatus` RPC with correspoding status in the
request.

Call `UpdatePeerState` operation of Netmap contract's client in
`control.NodeState` provider on Storage node app side. The op is
executed if `BlockExecution` on local object storage is succeeded.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
6a2ec21641 [#1681] ir/netmap: Check MAINTENANCE state in UpdateState
Storage node can be requested to be switched into `MAINTENANCE` state.
Inner Ring should accept such requests only if network configuration
allows it.

Make `Processor` of Netmap contract's notifications to depend on
`state.NetworkSettings`. Make `Processor.processUpdatePeer` to call
`MaintenanceModeAllowed` if notification event relates to `MAINTENANCE`
mode`. Share singe `state.NetworkSettings` provider in Inner Ring
application.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
bdb8243a5a [#1681] morph/netmap: Support MAINTENANCE state notification
After recent changes Netmap contract can send `UpdateState` notification
event with `MAINTENANCE` node's state. There is a need to provide
functionality to work with the status.

Provide `UpdatePeer.Maintenance` method. Support new state in
`ParseUpdatePeer` and `ParseUpdatePeerNotary` functions.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
0903b2af93 [#1680] morph/netmap: Allow to set MAINTENANCE state
After recent changes in NeoFS protocol storage nodes can be in
`MAINTENANCE` state. There is a need to support this state in
`UpdateState` operation.

Add `UpdatePeerPrm.SetMaintenance` method which makes node to be
switched into `MAINTENANCE` mode after the `UpdatePeerState` operation.

New functionality is going to be used in Storage node application for
Control API serving.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
6b99f2a0df [#1680] ir/netmap: Correlate node's state with network settings
Inner Ring should allow registering of storage nodes with `MAINTENANCE`
state in the NeoFS network only if its configuration allows this status.

Make `networkSettings.MaintenanceModeAllowed` to call
`MaintenanceModeAllowed` method of underlying Netmap contract's client
in order to assert state allowance.

From now nodes will be accepted to the network with `MAINTENANCE` state
only with the appropriate network configuration.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
876e132d22 [#1680] cli/netmap: Print MaintenanceModeAllowed config
After recent changes network configuration provided by NeoFS storage
nodes contains `MaintenanceModeAllowed` flag. There is
a need to support this value in NeoFS CLI application.

Print `MaintenanceModeAllowed` flag in `netmap netinfo` command.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
7a3a827d35 [#1680] node/netmap: Support MaintenanceModeAllowed config in RPC
After recent changes network configuration stored in the Netmap contract
of the NeoFS Sidechain contains `MaintenanceModeAllowed` flag. There is
a need to support this value in Storage node application.

Make `NetmapService.NetworkInfo` RPC server of the storage node to set
`MaintenanceModeAllowed` flag according to corresponding value in the
Sidechain.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
0826310b2a [#1681] morph/netmap: Support MaintenanceModeAllowed config
After recent changes in the NeoFS API protocol network configuration
contains `MaintenanceModeAllowed` boolean flag. There is a need to
support the config value in all NeoFS applications.

Provide `Client.MaintenanceModeAllowed` method which read the config
from the Sidechain. Extend `NetworkConfiguration` structure with
`MaintenanceModeAllowed` field.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
d2d4191868 [#1680] morph/netmap: Support HomomorphicHashingDisabled config
`NetworkConfiguration` represents NeoFS network configuration stored in
the Sidechain. In previous implementation the configuration missed flag
of disabled homomorphic hashing.

Add `NetworkConfiguration.HomomorphicHashingDisabled` boolean field.
Decode the field in `Client.ReadNetworkConfiguration` method. Print this
value in `netmap netinfo` command of NeoFS CLI.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
e0194dbde5 [#1681] morph/netmap: Refactor reading the boolean configurations
`readBoolConfig` method is going to be reused for reading other
configuration values. All boolean settings are `false` by default, so it
makes sense to return default value on missing key directly from
`readBoolConfig`.

Handle `ErrConfigNotFound` case in `readBoolConfig` method. Change
`HomomorphicHashDisabled` method to call `readBoolConfig` directly.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
f64ae55806 [#1681] ir/netmap: Require MAINTENANCE mode to be allowed by network
There is a need to prevent limitless abuse of MAINTENANCE status of the
storage nodes. To do this, configuration of the NeoFS network is going
to be extended with the flag which allows the state. Until this is done,
it makes sense to prepare a site for this in the code.

Define `state.NetworkSettings` interface as an abstraction of global
network configuration within the `state` package. Make
`NetMapCandidateValidator` to depend on `NetworkSettings` and provide
corresponding field setter. Change `VerifyAndUpdate` method's behavior
to return an error for candidates with MAINTENANCE state if this state
is disallowed by the network configuration. Provide `NetworkSettings`
from the wrapper over Netmap contract's client on Inner Ring application
side. The provider is implemented to statically disallow MAINTENANCE
mode in order to save previous behavior.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
42fb40e841 [#1680] morph/netmap: Pre-refactor processing of node states
New network status of storage nodes is going to be introduced. To
simplify the addition, it would be useful to prepare the code for this.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
4c94faac67 [#1797] node/control: Verify states in SetNetmapStatus RPC server
In previous implementation storage node interpreted all status values
sent in `SetNetmapStatus` RPC as `OFFLINE` except `ONLINE` and
`MAINTENANCE`. This could lead to incorrect processing of new values,
and also didn't allow detection of problems with sending garbage values.

Make implementation of `NodeState` interface used by Control API server
to deny requests with statuses other than protocol-declared enum.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
Leonard Lyubich
8858840751 [#1796] ir/netmap: Allow to call AddPeer with ONLINE state only
In previous implementation Inner Ring allowed storage nodes with any
state to register in the network. According to the current design, only
nodes with ONLINE state are allowed to enter the network map.

Create new `state` sub-package of `nodevalidation` package of Inner Ring
application. Define `state.NetMapCandidateValidator` type and provide
`NodeValidator` interface required by the Inner Ring's processor of
`Netmap` contract's notification events. Embed new validator into the
one used by the Inner Ring application.

From now all `AddPeer` notifications with node state other than `ONLINE`
will be denied.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-10-05 11:41:49 +03:00
8bf82d738b [#1704] cli: Add force option to the command container create
Validate policy before container creation

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-10-05 09:14:13 +03:00
Pavel Karpy
4eb0ed11f8 [#1809] node: Do not boot up if metabase is outdated
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 12:32:10 +03:00
Evgenii Stratonikov
8b3b16fe62 [#1825] writecache: Flush cache when moving to the DEGRADED mode
Degraded mode allows us to operate without an SSD,
thus writecache should be unavailable in this mode.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 12:13:09 +03:00
Evgenii Stratonikov
236414df49 [#1817] network: Allow to use network addresses from the iterator
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 12:12:18 +03:00
Evgenii Stratonikov
76893bdc50 [#1817] go.mod: Update neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 12:12:18 +03:00
Evgenii Stratonikov
2e3ef817f4 [#1819] engine: Increase error counter for PUT errors
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
Evgenii Stratonikov
b89e71fa78 [#1819] common: Add ErrNoSpace
Add a common error for this case because it is not an error
which should increase error counter. Single error simplifies checks on
the call-site.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
Evgenii Stratonikov
af56574849 [#1819] engine: Fix error counter in Inhume
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-10-04 10:11:52 +03:00
Pavel Karpy
1e35c12cc1 [#1770] node: Validate config before apply it
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
8ebe95747e [#1770] node: Do not lock on shard's Close call
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
ab3ef7110e [#1770] node: Reread config files on SIGHUP
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
6fc3268ebf [#1770] node: Support configuration reread on SIGHUP
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
887afeaddb [#1770] engine: Do not lock on shard init
Init can take a lot of time. Because the mutex is taken, all new operations
are blocked.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
fbd5bc8c38 [#1770] engine: Support configuration reload
Currently, it only supports changing the compound of the shards.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
91b56ad3e8 [#1770] node: Read storage config in a separate struct
It will allow rereading config values and will simplify distinguishing them
from the custom values in the `cfg` structure.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
5c69e19016 [#1770] fstree: Depth parameter int -> uint64
Negative values have no sense. On the other hand it differs from the
blobovnicza's configuration and prevents unification.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
4aa4694152 [#1770] node: Do not init local storage in reading config routine
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
2d7166f8d0 [#1770] shard: Move NewEpoch event routing on SE level
It will allow dynamic shard management. Closing a shard does not allow
removing event handlers.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
Pavel Karpy
9374823950 [#1770] node: Do not create meta dir on config read
It is created in `Open` anyway.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-10-04 10:08:55 +03:00
76cfcc242c [#1820] neofs-adm: Add wallet-address flag in refill command
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-10-03 10:11:38 +03:00
Evgenii Stratonikov
1edc048870 [#1697] services/object: Return proper error if session token is missing
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-30 16:13:01 +03:00
5284ac53f9 [#1821] neofs-cli: Fix description of cli commands
1. nodeinfo - updated description of node.
2. eacl - fixed misprint.

Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-09-28 16:09:24 +03:00
Evgenii Stratonikov
0a411908ee [#1806] writecache: Allow to ignore read errors during flush
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
f2045c10d7 [#1806] shard: Check each component mode when setting mode
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
1c62f1b2c4 [#1806] neofs-cli: Add control flush-cache command
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
51b8f26a31 [#1806] services/control: Allow to flush write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
3d882e9f47 [#1806] engine: Allow to flush write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
0b4c867ef1 [#1806] writecache: Allow to start flush manually
Allow user to initiate flushing objects from a writecache.
We need this in 2 cases:
1. During writecache storage schema update, it should be flushed with
   the old version of node and started clean with a new one.
2. During SSD replacement, to avoid data loss.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
55148404ee [#1806] writecache: Use interfaces for blobstor and metabase
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Evgenii Stratonikov
36ab1a2472 [#1731] neofs-cli: Make --endpoint flag optional
Allow to provide it using config file.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-28 09:28:01 +03:00
Leonard Lyubich
93742d37b7 [#1793] node/netmap: Change interface of the latest network map reader
Replace `ProcessCurrentNetMap` method of `NodeState` interface with
`ReadCurrentNetMap` one with two changes:
 * Replace network map type from NeoFS SDK package with the
   protocol-generated message. This replaces all the business logic to
   the application layer.
 * Support error return. This allows to cover problem node states.

Return an error from `NodeState.ReadCurrentNetMap` method implemeted
through `atomic.Value` if `Store` method has not been called yet.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-27 09:03:17 +03:00
Leonard Lyubich
485a5418d2 [#1793] node: Serve NetmapService.NetmapSnapshot RPC
There is no more need to serve the same request on Control API.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-27 09:03:17 +03:00
Evgenii Stratonikov
59de20fbba [#1764] neofs-node: Allow to check configuration
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-27 08:57:20 +03:00
Evgenii Stratonikov
0fb5c51ac9 [#1764] neofs-node: Validate config before usage
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-27 08:57:20 +03:00
Evgenii Stratonikov
6f45cc81fc [#1764] neofs-node: Use constants for storage types
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-27 08:57:20 +03:00
Evgenii Stratonikov
9113793688 [#1764] neofs-node: Allow to return error from IterateShards
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-27 08:57:20 +03:00
Evgenii Stratonikov
898689ec14 [#1731] services/replicator: Unify Task interface with other parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-24 13:47:48 +03:00
Evgenii Stratonikov
4e043a801c [#1731] services/control: Replicate object over network in EvacuateShard RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-24 13:47:48 +03:00
Evgenii Stratonikov
a49137349b [#1731] engine: Allow to use user handler for evacuated objects
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-24 13:47:48 +03:00
Pavel Karpy
93ae3f0b19 [#1808] .github: Add changelog workflow
It checks whether the CHANGELOG.md was updated in a just-created PR.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-23 10:20:25 +04:00
Evgenii Stratonikov
7cc68cf4d4 [#1810] blobovniczatree: Add a test for blobovnicva usage
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-23 10:19:26 +04:00
c7f85994e5 [nspcc-dev#1692] cli: Remove --generate-key option in neofs-cli container delete
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-09-20 07:19:46 +04:00
bb02913c39 [nspcc-dev#1128] cli: Remove WIF and NEP2 support in --wallet argument
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2022-09-19 14:21:37 +03:00
Evgenii Stratonikov
3df98ce7ba [#1731] engine: Return the amount of actually moved objects in Evacuate
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
8fc88487db [#1731] neofs-cli: Add control shards evacuate command
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
091d7d30f6 [#1731] services/control: Allow to evacuate data from shard
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
a51b76056e [#1731] engine: Add Evacuate command
Make it possible to move all data from 1 shard to other shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
7377979e12 [#1731] engine: Move single shard PUT to a separate function
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-19 11:33:52 +03:00
Evgenii Stratonikov
4208f7c0cf [#1741] neofs-adm: Use ValidUntilBlock to wait for tx persist
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 14:27:16 +03:00
Evgenii Stratonikov
5321f8ef9c [#1786] engine: Unify parameter setters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov
b064fb24d8 [#1616] engine: Do not use batches in delete
Use a simple loop at the callsite. This way we remove as much as we can.
Also, `Delete` metrics is more meaningful now.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov
cda8f9df2e [#1786] services/control: Remove WithDeletedObjectHandler option
Use storage engine directly instead. It is already provided in the
options.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:48 +03:00
Evgenii Stratonikov
ae1dab29bc [#1482] metabase: Encode database keys in binary
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-15 10:28:26 +03:00
Leonard Lyubich
d6fef68a62 Release v0.32.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-14 17:52:49 +04:00
Leonard Lyubich
d895a272ad docs: Improve release instructions
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-14 17:52:49 +04:00
Leonard Lyubich
ae655b74f0 *: Exec make fmts
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-14 17:52:49 +04:00
Evgenii Stratonikov
4b8b575ca4 [#1782] go.mod: Update neofs-sdk-go
Add fixes for key parameter in `object.Get` operation.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-14 13:18:19 +04:00
Pavel Karpy
532c41ca05 [#1658] meta: Do not check object expiration in counters
Do not perform operations that produce unused results in `syncCounter`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
431e331373 [#1658] shard: Update metric counters
Use meta's operation results to change the metrics. Support typed object
counters.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
ad47e2a985 [#1658] meta: Add logic counter
- Meta now supports (and requires) inc/dec labeled counters
- The new logic counter is incremented on `Put` operations and is
decremented on `Inhume` and `Delete` operations that are performed on
_stored_ objects only
- Allow force counters sync. "Force" mode should be used on metabase resync
and should not be used on a regular meta start

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
d872862710 [#1658] meta: Force counters resync process
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
beb7f2a048 [#1634] node: Change default epoch in tests
Do not treat objects with expiration as expired by default.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
edef26a4fd [#1658] engine: Update metrics interfaces
It now supports typed object counter metrics.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Pavel Karpy
175e2b9fa0 [#1658] metrics: Add object type label to the counters
It will allow to use labeled object counters, e.g.: "phy" for all
the physically stored objects, "logic" for only available objects. Also, it
will allow to add typed (regular, TS, SG, LOCK) counters if needed.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 21:32:37 +04:00
Leonard Lyubich
a789159e5b [#1189] config/example: Mention persistent_sessions.path default
In current implementation storage node app uses in-memory session
storage if `persistent_sessions.path` value is empty/missing in the
config.

Clarify default behavior in `config/example/node.yaml`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-13 21:29:04 +04:00
Leonard Lyubich
d25b7e177b [#1189] node/config: Fix PersistentSessionsConfig.Path method's docs
Method never returns `PersistentStatePathDefault` value, and this is
correct.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-13 21:29:04 +04:00
Pavel Karpy
4083a62679 [#1628] tree: Pre-allocate errors in ACL checks
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
5adf089c1d [#1628] tree: Log unacceptable bearer attachment
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
2ffcd02ac3 [#1628] tree: Document ACL checks in tree service
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
8d0906c6ab [#1628] tree: Skip eACL filters for tree requests
Do not call `CalculateAction` for the eACL checks since it requires object
headers that are meaningless in the tree context.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
876e014b5d [#1628] tree: Make ACL checks the same way as for object requests
1. Do not require a request to be signed by the container owner if a
bearer token is missing
2. Do not check the system role since public requests are not expected to
be signed by IR or a container node (unlike the object requests)

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
4f18893d9b [#1628] node: Move common EACLSource interface to core pkg
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-13 10:33:50 +03:00
Pavel Karpy
4afb928ab6 [#1674] *: Expire entities after the expiration epoch
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-12 19:56:59 +04:00
Evgenii Stratonikov
f1572a674b [#1775] go.mod: Update neofs-sdk-go
Make use of fixes for the placement policy parser.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 19:10:49 +04:00
Evgenii Stratonikov
d25ec52459 [#1778] neofs-node: Panic if config type is invalid
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 17:06:00 +03:00
Evgenii Stratonikov
60b2930417 [#1778] neofs-node: Fix blobovnicza config type
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 17:06:00 +03:00
Evgenii Stratonikov
9da5d784cb [#1630] neofs-node: Remove trees on container removal event
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Evgenii Stratonikov
87bd49563e [#1761] pilorama: Use batch size of 1 for tests
Improve tests speed by a lot and use more iterations in `ApplyRandom` test.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Evgenii Stratonikov
a2bb3a2a96 [#1630] pilorama: Support dropping trees
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-12 09:54:15 +03:00
Leonard Lyubich
699b534416 [#1763] node/netmap: Write log message about parsed NewEpoch event
There is a need to have the ability to track NeoFS timeline on storage
nodes. Epochs tick on notifications receipt, so the most obvious way to
know about received epochs is logging the events.

Wrap `morphEvent.ParseNewEpoch` event parser into function which writes
log message about new epoch number.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-12 09:53:05 +04:00
Leonard Lyubich
3a206b5d7a [#1763] morph/event: Write log message about new notification event
Sometimes it is useful to track sidechain events processed by the node.
For example, we need some easy way to look up for log messages about new
epoch notifications.

Make `subscriber.Subscriber` to log names of all events received from
the sidechain notification channel.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-12 09:53:05 +04:00
Leonard Lyubich
8e8265d5ea [#1707] blobovnicza/get: Iterate over all buckets regardless of config
Re-configuration of Blobovnicza's object size limit must not affect
already stored objects. In previous implementation `Blobovnicza` didn't
see objects stored in buckets which became too big after size limit
re-configuration. For example, lets consider 1st configuration with 64KB
size limit for stored objects. Lets assume that we stored object of 64KB
size, and re-configured `Blobovnicza` with 32KB limit. After reboot
object should be still available, but actually it isn't. This is caused
by `Get` operation algorithm which iterates over configured size ranges
only, and doesn't process any other existing size bucket. By the way,
increasing of the object size limit didn't lead to the problem even in
previous implementation.

Make `Blobovnicza.Get` method to iterate over all size buckets
regardless of current configuration. This covers the described scenario.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-08 12:24:28 +04:00
Pavel Karpy
8e6e89aca3 [#1711] amd: Cache committee actor in the init context
Also simplify the code using `invoker.Invoker`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Pavel Karpy
60aa53651b [#1711] adm: Append witnesses only if they are missing
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Pavel Karpy
1e57565f6c [#1711] adm: Implement local client's methods
Implement methods that `actor.Actor` requires:
1. `InvokeFunction` -- wrapper over `InvokeScript`
2. `GetVersion` returns default struct
3. `CalculateNetworkFee` copied and simplified from Neo-go server side.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Pavel Karpy
4bd8608b37 [#1711] *: Simplify code using neo-go actors
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-08 10:09:44 +03:00
Evgenii Stratonikov
4354359aed [#1746] network: Set timeout for streaming operations
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 16:00:51 +04:00
Evgenii Stratonikov
0140ac354b [#1759] services/tree: Make logs more descriptive
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 12:57:33 +04:00
Evgenii Stratonikov
74c861342e [#1753] services/tree: Add parent ID to the GetNodeByPath response
Currently, you need to use `GetSubTree` to get parent, which seems an
overkill.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 10:52:37 +03:00
Evgenii Stratonikov
8a47f9ba12 [#1753] services/tree: Remove MaxGetSubTreeCount constant
It was used only in `GetOpLog` and ignored on the server side.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 10:52:37 +03:00
Evgenii Stratonikov
9840936a4f [#1753] services/tree: Do not restrict depth in GetSubTree
Previously, the depth was restricted because with BFS the amount of
nodes we have in memory blows up exponentially. With DFS is is linear,
so we can process trees of arbitrary depth.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 10:52:37 +03:00
c4c2840b52 [#1632] neofs-adm: Update autogenerated storage node config
Replace `diable_cache` with `cache_ttl` value.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-09-07 10:34:46 +03:00
944665dd5d [#1632] neofs-node: Update docs
Replace `diable_cache` with `cache_ttl` value

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-09-07 10:34:46 +03:00
Evgenii Stratonikov
e4dcc4d6a9 [#1749] neofs-adm: Register contract hashes in NNS in 2 formats
NNS proposal describes string N3 address format, however we must also
have hex-string for backwards compatibility.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 09:58:05 +03:00
Evgenii Stratonikov
fe0c6db67d [#1749] neofs-adm: Delete NNS records before updating
In case we already have the record and it is invalid, we should
overwrite it instead of having several conflicting records.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 09:58:05 +03:00
Evgenii Stratonikov
0818d8d43a [#1751] .github: Add label checklist to templates
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-07 09:57:50 +03:00
Pavel Karpy
fa18100489 [#1714] lens: Add open*COMPONENT* funcs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 18:09:18 +04:00
Pavel Karpy
01d7c007aa [#1714] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 18:09:18 +04:00
Pavel Karpy
2132b78aba [#1714] lens: Add meta subcommand
Includes:
1. `inspect`
2. `list-garbage`
3. `list-graveyard`

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 18:09:18 +04:00
Pavel Karpy
adff08ad02 [#1714] lens: Separate commands
Separate `inspect` and `list` on `write-cache` and `blobovnicza`
subcommands.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 18:09:18 +04:00
Pavel Karpy
387953c1e0 [#1756] make: Fix commit abbrev length in version
Make it uniform across all our repos.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-06 10:53:38 +03:00
Evgenii Stratonikov
fef4f6d155 [#1748] neofs-adm: Allow to dump hashes from a custom zone
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
d4ac5bdb97 [#1748] neofs-adm: Allow to parse hashes in multiple formats
NEO NNS proposal uses addresses. We should eventually use the same,
but must stay compatible now.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
4ccacb89e8 [#1748] neofs-adm: Set custom contract hashes as N3 address
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
11e6f03aa2 [#1748] neofs-adm: Refactor NNS price handling
Make `nnsRegisterDomainScript` simpler.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
9290a7e1fe [#1748] neofs-adm: Remove addRecord from nnsRegisterDomainScript
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Evgenii Stratonikov
eb5e1121c5 [#1748] neofs-adm: Move dump-hashes command to a separate file
I have just lost 3 minutes trying to find it.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 21:31:32 +03:00
Leonard Lyubich
e26e70ffcf [#1652] cli/container: Mention new flag in the CHANGELOG
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 16:22:16 +04:00
Leonard Lyubich
33143b18a9 [#1652] cli/container: Invert pre-check flag of set-eacl command
Flag `--pre-check` of `set-eacl` command found to be in demand in most
cases. based on this, it makes sense to add its action to the default
behavior.

Pre-check container extensibility by default. Rename flag to
`--no-precheck` and invert its action.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 16:22:16 +04:00
Leonard Lyubich
46b815c863 [#1652] cli/container: Move eaclPathFrom flag var into the struct
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 16:22:16 +04:00
Leonard Lyubich
491a908af1 [#1652] cli/container: Pre-check ACL extensibility in set-eacl
Container ACL in NeoFS can be extended only for container in which the
corresponding option is enabled. In previous implementation command
`set-eacl` could hang up on modifying eACL of the non-existent or
non-extendable container. To improve UX, there is a need to pre-check
the availability of `SETEACL` operation.

Add boolean `precheck` flag to `set-eacl` cmd which reads the container
before the actual transaction formation. If flag is set, command fails
on non-extendable container ACL.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 16:22:16 +04:00
Pavel Karpy
17059b34ea [#1743] morph: Fix non-notary calls
Some methods add "IR" suffix to its names in notary enabled envs
because of contract logic. It was broken due to incorrect notary state
reading (tryNotary != notary is enabled).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 12:11:57 +03:00
Evgenii Stratonikov
e6cf0b3d42 [#1745] writecache: Set flush parameters based on max size
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
c04126f35e [#1745] writecache: Remove IsErrNotFound
We specify the error in the doc-comment, and it is the same for all our
components.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
9e41e85295 [#1745] writecache: Actualize docs
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
e9c6ee2623 [#1745] writecache: Rename constants for default values
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
177e8e01b1 [#1745] neofs-node: Remove memcache_capacity from the configuration
It is unused since ddaed283e9 .

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
bda084f331 [#1745] writecache: Simplify object counters
Remove unused option and additional pointers to db/fstree.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Evgenii Stratonikov
20abdaeed4 [#1745] writecache: Set flush mark only on success
Set flush mark in the inside the flush worker because writing to the blobstor
can fail. Because each evicted object must be deleted, it is reasonable
to do this in the evict callback.

The evict callback is protected by LRU mutex and thus potentially interferes
with `Get` and `Iterate` methods. This problem will be addressed in the
future.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-09-02 11:59:53 +03:00
Pavel Karpy
82839cb1c9 [#1711] morph: Add nolint for AddNetworkFee
Waits for `v0.99.3` neo-go unreleased yet.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Pavel Karpy
62b293b7ab [#1711] adm: Add error context
To make all the calls in the command to be in the same form.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Pavel Karpy
a99474c40e [#1711] adm: Drop deprecated neo-go calls
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Pavel Karpy
6d277a57aa [#1711] morph: Simplify code using neo-go actors
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Pavel Karpy
4582b8f0d4 [#1711] morph: Drop deprecated neo-go calls
Does not include notary actor and does not drop the deprecated
`AddNetworkFee`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-09-02 10:41:18 +03:00
Leonard Lyubich
ae6ebccd9c [#1632] node/container: Don't add new container to missed cache record
If container listing cache on node's side is missing (for particular
owner), then updating it as a reaction to successful container creation
leads to potentially invalid cache value for a period of time equivalent
to cache TTL.

Immediately return from `ttlContainerLister.update` method if owner's
container list isn't cached.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
fa3124fc33 [#1632] node: Configure TTL of Sidechain caches
From now cache TTL can be parameterized in the `neofs-node` app using
`cache_ttl` config key. `disable_cache` value is no longer supported.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
5ce8315cd8 [#1632] node/container: Update LIST cache on successful writes
In previous implementation storage node responded with the outdated
container list after successful creation/removal up until cache
invalidation due to TTL. In order to decrease the probability of
outdated responses node should update its cache on event receipts.

Implement `ttlContainerLister.update` method which actualizes cached
list of the owner's containers. Make node to call `update` method
on `PutSuccess`/`DeleteSuccess` notifications from the `Container`
contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
d15a7d8d3d [#1632] node/container: Update GET cache on successful removal
In previous implementation storage node responded with the removed
container up until cache invalidation due to TTL. In order to avoid
false-positive responses node should update its cache on `DeleteSuccess`
events.

Make node to call `handleRemoval` method of the container cache which
leads to subsequent `apistatus.ErrContainerNotFound` errors.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
a529149e6f [#1632] node: Cache Sidechain errors along with the values
In previous implementation failed requests to the Sidechain weren't
cached. It makes sense to cache errors along with the values in order to
decrease potential load spikes onto Sidechain nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Leonard Lyubich
7c1babb7d6 [#1632] node: Subscribe on the successful container creations/removals
There is a need to sync container-related caching mechanism with the
actual Sidechain changes. To do this, node should be able to listen
incoming notifications about container ops.

Define `PutSuccess` / `DeleteSuccess` notification event's parsers.
Subscribe to these events in node app. As initial implementation node
will log event receipts. Later handling is going to be practically
complicated.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-09-02 11:29:55 +04:00
Evgenii Stratonikov
d8a00c365a [#1691] blobovniczatree: Fix active blobovnicza caching
Maintain an invariant that any blobovnicza is present either
in `opened` or in `active` map. Otherwise, the logic becomes too
complicate because it is not obvious when we should close the blobovnicza.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-31 19:11:48 +03:00
Evgenii Stratonikov
dfac4e1c0b [#1691] blobovniczatree: Do not fill cache during initialization
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-31 19:11:48 +03:00
Evgenii Stratonikov
df88afeef3 [#1738] neofs-adm: Clean tx hash list after awaiting
Do not query application log for the same transaction more than once.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 16:53:00 +03:00
Evgenii Stratonikov
ae52d53609 [#1698] pilorama: Add a test for the empty FileName
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 16:52:51 +03:00
Evgenii Stratonikov
180e5e938f [#1735] go.mod: Update API and SDK
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 12:27:46 +03:00
Evgenii Stratonikov
78bf17c0b0 [#1686] blobstor/*: Remove unneeded tests
The blobovniczatree test also checks the amount of objects we can put,
so leave it here.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
b9a2055e1c [#1686] blobstor/*: Return ErrReadOnly for modifying operations
This check should occur on the shard level, but because
blobstor components expose `Open(readOnly bool)` interface,
it is reasonable to expect an error here.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
6c2d3b020f [#1686] fstree: Return proper error for concurrent Delete
If the file doesn't exist, return `apistatus.ObjectNotFound`.
First check is still there as a shortcut.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
0b95a21701 [#1686] blobstor: Add generic tests
This tests check that each blobstor component behaves similarly when
same methods are being used. It is intended to serve as a specification
for all future components.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
b2d4cc556e [#1686] blobovnicza: Remove IterateObjects
It is used only once, makes sense to inline.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
5f95498364 [#1686] blobstor: Add generic tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
9792e6a4ef [#1686] metabase: Add generic tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
482a7e7f2f [#1686] pilorama: Add generic tests
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
b8b9d25f9d [#1686] local_object_storage: Add generic tests
Use them for writecache as a simple example.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Evgenii Stratonikov
bc20851cd1 [#1686] shard: Open all components in case metabase has failed
All components must be opened first, before any other operation is
performed.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-30 10:02:43 +03:00
Pavel Karpy
c7c1c257e1 [#1658] node: Init object counter on meta's Init
Includes:
1. Renaming counter key to distinguish logical and physical objects
2. Version update dropping since changes could be done in a compatible way

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
cd6f8e051a [#1658] Update CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
9c7b3ce799 [#1658] node: Read metrics from meta on startup
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
745f72fff0 [#1658] node: Support object counter metrics
Increment shard's object counter on successful `Put` calls and decrement on
`Delete`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
067ab44e0b [#1658] metrics: Add object counter metric
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Pavel Karpy
626766db08 [#1658] meta: Add object counter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-25 19:20:33 +03:00
Evgenii Stratonikov
61f0d85834 [#1727] neofs-adm: Allow to register custom TLD for contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-25 12:31:00 +03:00
Evgenii Stratonikov
ddbe129cb2 [#1729] neofs-adm: Change default domain expiration time
It is a noop for current contract versions, but will matter in the next
release.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-25 12:31:00 +03:00
Evgenii Stratonikov
5827a926c9 [#1729] go.mod: Update neofs-contract to v0.15.5
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-25 12:31:00 +03:00
Evgenii Stratonikov
19585c63ed [#1730] go.mod: Update neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-24 17:17:42 +03:00
e68c80df0a [#1728] Don't describe default value twice
Cobra will mention default value automatically.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-08-24 16:44:44 +03:00
31b6af5667 [#1728] Update neofs-adm morph deploy command description
Add more details about expected file names in contract dir.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-08-24 16:44:44 +03:00
Pavel Karpy
b54f34d7df [#1683] adm: Do not deploy custom contract with a group
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-24 13:52:55 +03:00
Pavel Karpy
bbd6b9780f [#1683] adm: Minimize registration price
Register NNS domain in one TX:
1. Set minimal (`1`) registration price;
2. Register domain;
3. Return registration price back.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-24 13:52:55 +03:00
Evgenii Stratonikov
d57c57010f [#1726] neofs-cli: Truncate file before writing the object
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-23 18:07:12 +03:00
Evgenii Stratonikov
40f1cf36e1 [#1719] neofs-cli: Save object with proper permissions
`curl` uses 0644, for example.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-23 18:07:12 +03:00
Evgenii Stratonikov
871fae8321 [#1705] go.mod: Update neo-go to v0.99.2
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-23 11:59:23 +03:00
Evgenii Stratonikov
e4370c4129 [#1702] neofs-adm: Set HighPriority attribute for committee transactions
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-23 11:59:23 +03:00
Pavel Karpy
0720d96c9d [#1687] go.mod: Update neofs-sdk-go
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
d40ed00438 [#1687] go.mod: Update neofs-api-go
It contains the signature fix for go `v1.19`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
56792d5eaf [#1687] .github: Update actions to the latest version
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
99903e1aba [#1687] Support v1.19 Golang
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
30341f2192 [#1687] *: Perform go fmt using go v1.19
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 18:59:57 +03:00
Pavel Karpy
37ab26bfa9 [#1296] cli: "ID" -> "OID"
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-22 14:29:50 +04:00
Evgenii Stratonikov
e8c6dce466 [#1523] blobstor: Rename compression.CConfig to Config
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
0042c1d4a6 [#1584] blobovniczatree: move public methods to different files
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
26b4a258e0 [#1523] neofs-node: Refactor configuration
1. Move compression parameters to the `shard` section.
2. Allow to use multiple sub-storage components in the blobstor.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
13cdbde2e2 [#1523] blobstor: Remove unused types
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
0c9e4e6a35 [#1523] blobstor: Unify request dispatch logic
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
266458fe5c [#1523] blobstor: Unify fstree and blobovnicza interfaces
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
5aa3defc67 [#1523] writecache: Remove cloneBytes
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
ddaed283e9 [#1523] writecache: Simplify logic
1. Remove in-memory cache. It doesn't persist objects and if we want
   more speed, `NoSync` option can be used for the bolt DB.
2. Put to the metabase in a synchronous fashion. This considerably
   simplifies overall logic and plays nicely with the metabase bolt DB
   batch settings.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
4176b2a1bc [#1523] local_object_storage: Unify parameters for the Iterate operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
73f8bb3e5f [#1523] shard: Store generic storage ID in metabase
Allow to extend blobstor with more storage sub-systems. Currently
objects stored in the FSTree have empty byte slice descriptor and object
from blobovnicza tree have the same id as earlier. Each such change in
the identifier formation should be accompanied with metabase version
increase.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
9eb018672c [#1523] blobstor: Unify parameters for Put operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
7d6df543d7 [#1523] blobstor: Export single Get, GetRange and Delete methods
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
ca15083a50 [#1523] local_object_storage: Unify parameters for the Exists operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
d75d030a90 [#1523] local_object_storage: Unify parameters for the Delete operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
36c88f0dc8 [#1523] local_object_storage: Unify parameters for the GetRange operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
d9a2f280c9 [#1523] local_object_storage: Unify parameters for the Get operation
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Evgenii Stratonikov
b621f5983a [#1523] local_object_storage: Move blobovnicza tree to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-08-22 13:14:19 +03:00
Pavel Karpy
5139dc9864 [#1706] cli: Do not duplicate payload on SG put
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-19 09:49:47 +04:00
Evgenii Stratonikov
4e361e2ab4 [#1647] writecache: Open DB in sync mode
Close #1647.

Initially, `Sync: false` was provided because we can already lose
objects cached in memory. However, future changes in writecache will
remove inmemory cache and speed up it via other means.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-16 15:27:18 +04:00
Evgenii Stratonikov
89cba2cc7b [#1684] .github: Update golangci-lint version
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-15 10:57:31 +03:00
Evgenii Stratonikov
f8b106ac85 [#1684] *: Fix linter warnings
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-15 10:57:31 +03:00
Leonard Lyubich
67af4c89a8 [#1649] cli: Refactor --owner flag definition in container list cmd
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-08-15 10:18:51 +03:00
Leonard Lyubich
7a26b2a57a [#1649] cli: Add option to print attributes in list
Define `--with-attr` flag of `container list` which makes the command to
request and print user attributes for each found element.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-08-15 10:18:51 +03:00
Leonard Lyubich
2740bf7ee4 [#1649] cli: Add option to print attributes in list-objects
Define `--with-attr` flag of `container list-objects` which makes the
command to request and print user attributes for each object from the
container.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-08-15 10:18:51 +03:00
e07921fc46 Use only v-prefixed tags for version calculation
This allows us to use any custom tags along with tags that denote releases

Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2022-08-10 10:34:07 +03:00
Pavel Karpy
da2975a2f9 [#1664] write-cache: Fix panic on Delete operation
If an object is found in the Write-cache and is placed at the end of
the in-memory cache, the memory counter update operation tries to
dereference the index that is out of the sliced array. Moreover, even if
panic does not appear, the counter is updated with the wrong value.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-09 20:18:26 +03:00
a.y.volkov
81684b6f04 [#1665] neofs-lens: Use stdout as default output
Signed-off-by: a.y.volkov <a.y.volkov@yadro.com>
2022-08-09 13:23:32 +03:00
Pavel Karpy
cd71de69a0 Release v0.31.0
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:04:33 +03:00
Pavel Karpy
5ff1df285b Release v0.30.2
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:04:33 +03:00
Evgenii Stratonikov
058538768b [#1640] Release v0.30.1
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:04:33 +03:00
Pavel Karpy
a5da665e69 [#1656] meta: Fix MatchCommonPrefix operation
Return all the objects on the empty common prefix search without search
optimizations that breaks boltDB logic.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:32:00 +03:00
Pavel Karpy
156ba85326 [#1634] node: Do not return expired objects
If an object has not been marked for removal by the GC in the current epoch
yet but has already expired, respond with `ErrObjectNotFound` api status.
Also, optimize shard iteration: a node must stop any iteration if the object
 is found but gonna be removed soon.
All the checks are performed by the Metabase.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
Pavel Karpy
9aba0ba512 [#1634] meta: Add epoch state
It allows performing expiration checks on the stored objects.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:31:49 +03:00
Pavel Karpy
a97dee008c [#1648] morph: Change endpoint priority order
The lowest value means the highest priority.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:11:24 +03:00
Pavel Karpy
2467be0117 [#1648] morph: Init endpoints before first client creation
Sort the endpoint by their priority before the first WS client creation to
start the morph client with the highest priority endpoint.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 16:11:24 +03:00
Pavel Karpy
1c1b5043fc [#1653] morph: Handle chain notifications via pool
Handling notification in a synchronous manner may lead to a blocking state
if a handler uses neo-go client.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:06:53 +04:00
Pavel Karpy
8f44335925 [#1651] cli: Fix CID setting in eACL creation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 17:04:42 +04:00
Evgenii Stratonikov
9a5f9d6f0e [#1643] innerring: Exit if we cannot bind to the control endpoint
Return listen errors in a synchronous fashion.
Another solution would be to use buffered channel, but this is not
scalable: for each new similar runner we would need to extend the
buffer.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-08-04 15:41:41 +03:00
Pavel Karpy
0a60524a9c [#1654] ir, node: Drop deprecated profiler and metrics config sections
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 15:38:35 +03:00
Pavel Karpy
713cfa5610 [#1655] cli: Do not force specifying session lifetime
We have the default value which is also printed in the help messages but any
call that does not specify that flag leads to an error.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-04 15:37:50 +03:00
Pavel Karpy
c2918fce3a [#1645] node: Support EACL_NOT_FOUND status
Remove internal `ErrEACLNotFound` error.
Also, update `neofs-api-go` and `neofs-sdk-go` libraries.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-01 20:45:36 +03:00
Pavel Karpy
8c8ae56960 [#1645] go.mod: Update dependencies
- `neofs-api-go` to `v2.13.1`;
- `neofs-sdk-go` to `7a99cc916c8e`.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-08-01 20:45:36 +03:00
Evgenii Stratonikov
8ffc2fdf5e [#1642] engine: Do not increase error counter if the pilorama is disabled
After a4adb79db new logical error could be returned. Do not increase
error counter in this case.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-30 18:39:22 +03:00
Evgenii Stratonikov
4558f30575 [#1644] services/tree: Do not use deprecated gRPC options
Get rid of `grpc.WithInsecure`.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-30 18:39:08 +03:00
Evgenii Stratonikov
8d0884e74f [#1644] storagelog: Fix doc comment
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-30 18:39:08 +03:00
Evgenii Stratonikov
7ecf54c6df [#1631] neofs-adm: Set group for updated alphabet contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:09:48 +03:00
Evgenii Stratonikov
d3eac7626d [#1631] neofs-adm: Set group for a proper contract
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:09:48 +03:00
Evgenii Stratonikov
5efcd34680 [#1639] go.mod: Update neofs-sdk-go to v1.0.0-rc.6
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:07:53 +03:00
Evgenii Stratonikov
5d02613c33 [#1633] neofs-adm: Allow to set custom NNS zone for deployed contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
6146df4998 [#1633] neofs-adm: Allow to provide arguments to deployed contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
dd30703a6b [#1633] neofs-adm: Allow to update arbitrary contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
fc06b6e89a [#1633] neofs-adm: Allow to deploy arbitrary contracts
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
38558a3238 [#1633] neofs-adm: Start chain only on happy path
If the error is returned, it will not be closed properly.

Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
642d7328ab [#1633] neofs-adm: Fix dump generation in init
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 12:02:40 +03:00
Evgenii Stratonikov
4abe5a7245 shard: add more checks for GetRange parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 11:22:04 +03:00
Evgenii Stratonikov
72586f17d4 shard: fix GetRange for objects stored in the write-cache
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-29 11:22:04 +03:00
Evgenii Stratonikov
07465849a4 [#1637] go.mod: Update neo-go to v0.99.1
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-28 20:11:45 +03:00
Alex Vanin
71fd86f220 [#1636] ir: Listen balance contract in balance processor
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-07-28 18:47:20 +03:00
Pavel Karpy
8162b27264 [#1627] node: Add tree service to the config framework
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-27 12:15:56 +03:00
Pavel Karpy
a0fae0443f [#1627] node: Add missing tree service to the config files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-27 12:15:56 +03:00
Evgenii Stratonikov
54fe7667fb [#1625] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <evgeniy@morphbits.ru>
2022-07-25 19:00:07 +03:00
Evgenii Stratonikov
2a1c5557f2 [#1624] Release v0.30.0
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-25 16:56:02 +03:00
Evgenii Stratonikov
10c855efef [#1624] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-25 16:56:02 +03:00
Evgenii Stratonikov
91361c4fe2 [#1624] CHANGELOG.md: Fix v0.29.0 release name
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-25 16:56:02 +03:00
Evgenii Stratonikov
25a0f539ea [#1624] config: Fix tree section example
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-25 16:56:02 +03:00
Pavel Karpy
589a54805d [#1618] node: Use OID/CID from the request in eACL checks
Also, try to fetch object header info from the local storage to find as much
object info as possible for the requests which do not assume returning
object header as a response.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-25 09:41:11 +03:00
Evgenii Stratonikov
6049022f7e [#1612] neofs-cli: Remove RunE functions from commands
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-22 16:18:09 +03:00
Evgenii Stratonikov
a52e7c2c99 [#1612] neofs-cli: Unify expiration flags
Use `expire-at` everywhere expiration epoch is expected.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-22 16:18:09 +03:00
Evgenii Stratonikov
175e9da3a7 [#1612] go.mod: Update cobra dependency
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-22 16:18:09 +03:00
Evgenii Stratonikov
7e717e80ef [#1623] services/tree: Fix error message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-22 15:13:03 +03:00
Pavel Karpy
2f4e465cf4 [#1620] node: Remove debug attribute printing
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-22 13:50:36 +03:00
Pavel Karpy
ee9312dcfb [#1619] example: Fix IR RPC
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-22 12:37:25 +03:00
Pavel Karpy
fb1fac02e9 [#1619] ir: Fix RPC endpoint iteration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-22 12:37:25 +03:00
Evgenii Stratonikov
1691364653 [#1559] local_object_storage: Fix tests and some data races
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
50e28f22f9 [#1559] shard: Change Degraded mode string representation
It is a flag, but is a `degraded-read-write` mode for a user.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Leonard Lyubich
fabe717d32 [#1549] shard: Turn to read-only mode on metabase failure
If metabase can't be opened in the default mode, try opening shard
first in `ReadOnly` mode and then in `DegradedReadOnly`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
4944490ffb [#1559] local_object_storage: Move shard to the DegradedReadOnly mode
`Degraded` mode can be set by the administrator if needed.
Modifying operations in this mode can lead node into an inconsistent state
because metabase checks such as lock checking are not performed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
3df62769c0 [#1559] local_object_storage: Allow to set mode for all components
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
1e786233bf [#1559] local_object_storage: Provide readOnly flag to Open
We should be able to reopen storage in readonly in runtime.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Leonard Lyubich
e38b0aa4ba [#1549] engine: Disable shard on blobovnicza init failure
There is a need to support working w/o shard if it has problems with
blobovnicza tree.

Make `BlobStor.Init` to return new `ErrInitBlobovniczas` error. Remove
shard from storage engine's shard set if it returned this error from
`Init` call. So if some of the shards (but not all) return this error,
the node will be able to continue working without them.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
40a56c6b42 [#1559] engine: Do not count logical errors as storage ones
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
c8911d72d0 [#1559] shard: Do not consult metabase in a degraded mode
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
339864b720 [#1559] local_object_storage: Move shard.Mode to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
d8ba954aff [#1559] shard: Use Set prefix for parameter setting
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
7b882b26d8 [#1559] shard: Remove public functions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
34d319fed2 [#1559] metabase: Use Set prefix for parameter setting
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
f58234aa2f [#1559] metabase: Remove public functions
Reduce public interface of this package. Later each result will contain
an additional status, so it makes more sense to use the same functions
and result processing everywhere.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 17:56:06 +03:00
Evgenii Stratonikov
30c7925b3c [#1609] morph/client: Retry connecting to failed endpoint during the switch
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 16:08:42 +03:00
Evgenii Stratonikov
7410827db8 [#1609] config: Allow to prioritize N3 endpoints
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 16:08:42 +03:00
Evgenii Stratonikov
aed83d1660 [#1609] config: Remove mainchain section
It is deprecated and unused, there is no need to have it in the
documentation.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 16:08:42 +03:00
Pavel Karpy
a4ed91f9b5 [#1614] node: Update health metric on its change
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-21 16:07:47 +03:00
Pavel Karpy
581a9901c9 [#1614] metrics: Add health metrics
Also, rename metrics structure since it collects not only storage metrics
now.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-21 16:07:47 +03:00
Evgenii Stratonikov
2455b72844 [#1607] CHANGELOG.md: add tree service info
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
b549cc314c [#1607] services/tree: allow to customize some parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
2334e94fdb [#1607] neofs-node: Initialize storage before other services
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
a4adb79db7 [#1607] pilorama: Enable tree service explicitly
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
d62723f038 [#1505] pilorama: Provide timeout to bbolt.Open
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
26041f18bf [#1505] pilorama: Allow to customize database parameters
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f80e52fbea [#1334] services/tree: Document *.proto files
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3904b0e017 [#1333] neofs-cli: Add control synchronize-tree command
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
bfdd68dcb3 [#1333] services/control: Allow to synchronize local trees
Do not check that a node indeed belongs to the container, because the
synchronization will fail in this case anyway.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
0dc1a4e336 [#1333] neofs-node: Initialize tree service before the control one
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
c7b10598f9 [#1333] engine: Increase error counter for pilorama errors
1. Modifying operations are not expected to fail, unless the shard is
   read-only.
2. `Get*` operations should increase error counter too, unless the
   error is `ErrTreeNotFound`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
5e843a73f9 [#1333] services/control: Return pilorama info in ListShards RPC
Do not return backend type from the service for now, because memory
backend is expected to vanish.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8f4ee1aded [#1333] local_object_storage: Support ReadOnly mode in pilorama
The tricky part here is the engine itself: we stop iteration on
`ErrReadOnly` because it is better to synchronize the shard later than
to have partial trees stored in 2 shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
735931c842 [#1481] pilorama: Fix TreeApply
Current implementation prevents invalid operations to become valid at
some later point (consider adding a child to the non-existent parent and
then adding the parent). This seems to diverge from the paper algorithm
and complicates implementation. Make it simpler.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
5ffbeb76e6 [#1456] services/tree: Wait some time before reconnecting after failure
In case node is down or failing for some reason, we can expect `Dial` to
fail. In case we actively try to replicate and `Dial` always takes 2
seconds, replication-related channels quickly become full. That affects
latency of all other write operations.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
39f47f61c6 [#1445] services/tree: Cache the list of container nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
a2edfec0c3 [#1446] services/tree: Use grpc.WithInsecure only for nodes without TLS
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad3038d16d [#1444] pilorama: Fix TreeMove in bbolt backend
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
33d8fb187a [#1427] services/tree: Parallelize replicator
Before this commit the replication channel was quickly filled under
heavy load. This lead to the continuously increasing latency for all
write operations. Now it looks better.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8027b7bb6b [#1444] pilorama: Optimize internal encoding/decoding
```
name                      old time/op    new time/op    delta
ApplySequential/bbolt-8     55.5µs ± 4%    55.5µs ± 3%     ~     (p=1.000 n=10+7)
ApplyReorderLast/bbolt-8     108µs ± 6%     112µs ± 8%     ~     (p=0.077 n=9+9)

name                      old alloc/op   new alloc/op   delta
ApplySequential/bbolt-8     28.8kB ± 3%    27.7kB ± 6%   -3.79%  (p=0.005 n=10+10)
ApplyReorderLast/bbolt-8    41.4kB ± 5%    38.9kB ± 5%   -6.19%  (p=0.001 n=10+9)

name                      old allocs/op  new allocs/op  delta
ApplySequential/bbolt-8        262 ± 2%       235 ±10%  -10.41%  (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8       684 ± 6%       616 ± 7%  -10.04%  (p=0.000 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
b04f712773 [#1446] services/tree: Cache connections to the container nodes
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
4437cd7113 [#1442] pilorama: Generate timestamp based on node position in the container
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3caa982283 [#1442] services/tree: Fix log message for failed Apply
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
01d2e06a9b [#1401] services/tree: Marshal public key once
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
1f5a650b05 [#1401] services/tree: Retransmit queries to container nodes
Also fix a bug with replicator using the multiaddress instead of
<host>:<port> format expected by gRPC library.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
3312924b82 [#1431] pilorama: Use Batch for write transactions
Helps a lot in case of concurrent request flow.

```
name                      old time/op    new time/op    delta
ApplySequential/bbolt-8     78.0µs ± 9%    59.8µs ± 4%  -23.39%  (p=0.000 n=10+9)
ApplyReorderLast/bbolt-8     143µs ± 5%     113µs ±15%  -21.06%  (p=0.000 n=10+10)

name                      old alloc/op   new alloc/op   delta
ApplySequential/bbolt-8     56.9kB ± 8%    28.9kB ± 3%  -49.22%  (p=0.000 n=10+10)
ApplyReorderLast/bbolt-8    87.3kB ± 3%    40.9kB ±10%  -53.16%  (p=0.000 n=10+10)

name                      old allocs/op  new allocs/op  delta
ApplySequential/bbolt-8        224 ±11%       262 ± 5%  +16.93%  (p=0.000 n=9+10)
ApplyReorderLast/bbolt-8       518 ± 4%       674 ±11%  +30.09%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f0a67f948d [#1431] pilorama: Cache attributes in the index
Currently to find a node by path we iterate over all the children on
each level. This is far from optimal and scales badly with the number of
nodes on a single level. Thus we introduce "indexed attributes" for
which an additional information is stored and which can be use in
`*ByPath` operations. Currently this set only includes `FileName`
attribute but this may change in future.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f5d35571d0 [#1431] engine: Add benchmark for Select vs TreeGetByPath
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
f4bc1c601a [#1329] services/tree: Synchronize from the last stored height
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
835170e452 [#1329] pilorama: Allow to benchmark all tree backends
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
536857ea5a [#1329] services/tree: Implement GetOpLog RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
886baf3136 [#1426] services/tree: Do not replicate to a local node
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
7703dd5d7f [#1419] pilorama: Create new nodes in path if needed
Consider a node `{FileName: "dir", Attribute: "xxx"}`. In case we add
a new node by path `["dir", "file.txt"]`, create a new intermediate node
with a single attribute.

`GetByPath` now also considers only nodes with a single attribute while building a path.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
ad48918a97 [#1406] pilorama: Return parent from TreeGetMeta
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Denis Kirillov
b30f14978d [#1404] services/tree: Fix nodeId in GetSubTree
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
45321b3f6d [#1328] services/tree: Fix grpc import path
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
16e3421825 [#1328] services/tree: Implement access control
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
94b9e13431 [#1328] services/tree: Fix proto field numbers
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
36e24f7f78 [#1342] neofs-node: Use the default endpoint for tree service
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8844d9b2db [#1344] pilorama: Document errors for Get* methods
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
d8ad68d613 [#1344] engine: Log errors in Tree* operations
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
910db42748 [#1344] pilorama: Use require.ErrorIs in tests
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
aea855e8f3 [#1326] services/tree: Implement GetSubTree RPC
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
62154da17c [#1324] services/tree: Implement Object Tree Service
Object Tree Service allows changing trees assotiated with
the container in runtime.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
46f4ce2773 [#1324] engine: Implement Forest interface for storage engine
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
8cf71b7f1c [#1324] local_object_storage: Implement tree service backend
In this commit we implement algorithm for CRDT trees from
https://martin.klepmann.com/papers/move-op.pdf

Each tree is identified by the ID of a container it belongs to
and the tree name itself. Essentially, it is a sequence of operations
which should be applied in chronological order to get a usual tree
representation.

There are 2 backends for now: bbolt database and in-memory.
In-memory backend is here for debugging and will eventually act
as a memory-cache for the on-disk database.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-21 15:08:24 +03:00
Evgenii Stratonikov
198beae703 [#1613] neofs-cli: Print lock object ID
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 17:15:15 +03:00
Evgenii Stratonikov
5c8d725447 [#1610] neofs-adm: Fix generated storage config
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 16:19:41 +03:00
Evgenii Stratonikov
11c9df6f00 [#1611] shard: Print shard ID in component logs
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 15:57:53 +03:00
Pavel Karpy
9f7a22e2aa [#1561] object: Return OUT_OF_RANGE status
Replace `ErrRangeOutOfBounds` error from `pkg/core/object` package with
`ObjectOutOfRange` from `apistatus` package. That error is returned by
storage node's server as NeoFS API statuses.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-19 13:29:45 +03:00
Evgenii Stratonikov
ccba07fe19 [#1608] neofs-adm: Register candidates in a single transaction
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 13:28:17 +03:00
Evgenii Stratonikov
59d08b8dae [#1608] neofs-adm: Print info about already performed steps
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 13:28:17 +03:00
Evgenii Stratonikov
5e1636e50c [#1604] docs: Describe storage node configuration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 11:12:52 +03:00
Evgenii Stratonikov
eeef83a74c [#1604] config: Change yaml example a bit
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-19 11:12:52 +03:00
Evgenii Stratonikov
e5c0338f8c [#1602] neofs-ir: Fix defaults for pprof and prometheus
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 19:41:22 +03:00
Evgenii Stratonikov
3a0b06d5da [#1602] config: Rename metrics and profiler sections
Depracate old names and remove them in the next release.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 19:41:22 +03:00
Evgenii Stratonikov
ac46d1a11f [#1602] config: Enable metrics and profiler services with a flag
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 19:41:22 +03:00
Evgenii Stratonikov
12dc5c3395 [#1563] config: Replace 127.0.0.1 with localhost
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 19:41:22 +03:00
Evgenii Stratonikov
4a8dfd0ce4 [#1606] Remove deprecated functions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 12:41:06 +03:00
Evgenii Stratonikov
6cc180391e [#1606] go.mod: Update neo-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-18 12:41:06 +03:00
Pavel Karpy
cb0bb7207c [#1461] shard: Add a separate ErrLockObjectRemoval
Do not return `meta.ErrLockObjectRemoval` from shard's methods, add shard's
own error for that.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
558cc1193a [#1461] engine: Clarify force removal
Document force removal behaviour in all the Storage engine parts.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
b1e54dec9f [#1461] cli: Add context to GetCurrentEpoch helper
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
51afcc1182 [#1461] engine, policer: Force remove objects w/o container
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
18ec5d7c8e [#1461] meta: Return error on lock object removal
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
b5c56d459a [#1461] engine: Add force lock removal tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
63c00e785d [#1461] shard: Fix option naming
`WitDeletedLockCallback` => `WithDeletedLockCallback`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
5122be34e7 [#1461] shard: Add lock tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
eaf96bccf7 [#1461] meta: Add lock tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
c6cf8e5c0b [#1461] engine: Fix LOCK test
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
9c5ef3bab8 [#1461] node: Allow force LOCK removal
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
25826bd96d [#1461] node: Do not accept lock object w/o expiration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
de31bb27b6 [#1461] cli: Require LOCK object to be expired
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
595fc60d72 [#1461] cli: Reorganize common functions
Make `ParseEpoch` and `GetCurrentEpoch` funcs public and move the first one
to the `common` package.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Pavel Karpy
fed9e6679d [#1461] node: Unlock locked object on its lock removal
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-18 11:42:25 +03:00
Evgenii Stratonikov
72c044e2eb [#1599] engine: Parallelize shard initialization
Shard is intended to be used as a separate failure domain,
which usually resides on a separate disk. Thus, sequential
initialization is bound by IO and this change speeds up thing a bit.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-15 13:44:54 +03:00
Evgenii Stratonikov
6eb5260562 [#1500] neofs-cli: Check if container contains LOCK objects before removal
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-15 13:25:56 +03:00
Pavel Karpy
a421344727 [#1605] node: Stream forwarder requests
Do not hold objects in memory in the forwarded Get/GetRange requests.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-15 11:16:35 +03:00
Pavel Karpy
89ebc278b8 [#1596] *: Remove .gitkeep files
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-14 15:10:57 +03:00
Evgenii Stratonikov
c4b86cf1f1 [#1592] Remove debug builds
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-14 10:39:51 +03:00
Evgenii Stratonikov
b8508585a5 [#1592] cmd: Ignore error when fetching a defined flag
There is an error only if the flag is not defined, such errors should be
caught during debugging.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-14 10:39:51 +03:00
Evgenii Stratonikov
f7d5045876 [#1592] neofs-lens: Add --version flag
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-14 10:39:51 +03:00
Evgenii Stratonikov
5569ff82ef [#1587] Do not print build time in version
This makes our build more reproducible.
Also print `Component` and `GoVersion`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-14 10:39:51 +03:00
Pavel Karpy
b3272e7cf1 [#1367] adm: Support homomorphic hashing config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
4a316ceae9 [#1365] morph: Do not return errors if config key is missing
Return default values instead of casting errors in `HomomorphicHashDisabled`
method.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
9bd1951ca4 [#1365] adm: Add homomorphic hash disabling option
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
89118e9da0 [#1365] node: Calculate object homomorphic hash flexibly
Do not calculate and do not write homomorphic hash for containers that were
configured to store objects without hash.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
ae4740f99f [#1365] ir: Check homomorphic hash flexibly in audit
Do not perform that check if it was turned off for the container being
checked.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
8a61d33c65 [#1365] ir: Check homomorphic hash setting on ContainerPut
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
fa9c910648 [#1365] morph: Add HomomorphicHashDisabled config getter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
ae86d2907c [#1365] cli: Calculate homomorphic hash flexibly
Do not use homomorphic hash in storage group for containers that have
`homomorphic_hashing_disabled` set to `true`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
68e583f143 [#1365] cli: Sync container with network config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Pavel Karpy
49ba1ef1b6 [#1365] cli: Add SyncContainerSettings func to internal client
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-13 17:10:06 +03:00
Evgenii Stratonikov
1119014e53 [#1579] config: Move storage.default to shard.default
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-13 12:31:42 +03:00
Evgenii Stratonikov
0ab168a00a [#1564] .github: Remove assign-to-project action
It doesn't work now and needs to be done differently.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-13 11:35:48 +03:00
Evgenii Stratonikov
f4a3fc5f11 [#1594] neofs-adm: Support better completion for set-policy
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-13 11:00:35 +03:00
Pavel Karpy
466b9b4273 [#1593] adm: Support changing network config
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 21:03:43 +03:00
Pavel Karpy
1e034c8d48 [#1532] node: Support signature status codes
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 18:22:59 +03:00
Pavel Karpy
dac4a838fa [#1490] Update the CHANGELOG
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 17:35:46 +03:00
Pavel Karpy
96a24b5721 [#1490] cli: Require SG members to be unique
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 17:35:46 +03:00
Pavel Karpy
5d5d21e3d0 [#1490] node: Require SG members to be unique
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-12 17:35:46 +03:00
Evgenii Stratonikov
0504c3e0c6 [#1266] object/acl: Check bearer token container ID
If the container ID is not nil and not equal to the container ID in the
request, consider bearer token invalid.

See also nspcc-dev/neofs-api#207.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-12 12:25:02 +03:00
Leonard Lyubich
3a2c025843 Release v0.29.0
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-08 12:32:36 +03:00
Pavel Karpy
1658242e00 [#1590] node: Smart memory allocation in GetRange
Allocate memory only if a node chosen as the forwarded request receiver
has responded with a successful status.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-08 12:04:03 +03:00
Pavel Karpy
fed5e76e7f [#1590] node: Check status of forwarded requests
After fixing version fields in forwarded requests, a node does not check
statuses since errors are not covered by direct call error checks.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-08 12:04:03 +03:00
Leonard Lyubich
fdf62e8562 [#1586] Upgrade NeoFS SDK Go to rc#5
Error checkers now support wrapped errors so there is no need to
explicitly unwrap errors in `Policer`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-07 14:23:41 +03:00
Leonard Lyubich
9a11a75b77 [#1570] Upgrade NeoFS SDK Go with changed reputation API
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-06 18:21:24 +03:00
Pavel Karpy
9a6da336db [#1581] node: Do not lose API version on forwarding
Forwarded requests contained zero version in their meta header. It did not
allow responding with API statuses (`v0.0` version considered to be older
than `v2.11`) to the forwarding node and, therefore, did not allow analyzing
responses.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 17:06:24 +03:00
Pavel Karpy
c8506b247e [#1582] *: Fix linter warnings
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 11:56:01 +03:00
Stanislav Bogatyrev
df0d352305 [#1578] Reorder .gitignore
Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-07-06 10:00:19 +03:00
Stanislav Bogatyrev
930b51b57d [#1578] Minor Makefile fixes
- Add VERSION file to make it work without .git folder
- Remove obsolete GO111MODULE flag
- Move linter version to variable
- clean .cache on `make clean`
- Move help target to a separate help.mk file

Signed-off-by: Stanislav Bogatyrev <stanislav@nspcc.ru>
2022-07-06 10:00:19 +03:00
Pavel Karpy
a153e040cb [#1402] ir: Do not use map for combining SG ID and SG object
Also, add corresponding structure for that.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
ec07fda97b [#1402] pkg: Move SGSource to the core directory
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
27304455bf [#1402] ir: Filter expired SGs in the audit process
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
7864959d0c [#1402] ir: Move GetSG interface
`auditor` does not need to request SG: processor will fetch that info before
audit context initialization.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
6370c2e160 [#1402] ir: Make ClientCache not depend on audit.Task
That allows using `ClientCache` for storage group searching before task
context is initialized. Also, that makes it more general purpose.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
5f658c499c [#1402] ir: Collect SG from the object
Do not use `Marshal()` with object's payload. Use `ReadFromObject` func from
SDK instead. That allows checking both attributes and SG body's expiration
epoch.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
12d41918e6 [#1402] cli: Support storage group expiration
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
cfca88fc5c [#1402] cli: object <-> SG conversion via SDK
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Pavel Karpy
bc5882fc89 [#1402] cli: Add cid flag constant
Also, move common flag constants in `root` file of the `storagegroup`
package.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-07-06 09:56:26 +03:00
Evgenii Stratonikov
6f2363cf31 [#1438] README.md: add build instructions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-05 14:47:12 +03:00
Leonard Lyubich
c165d1a9b5 [#1556] Upgrade NeoFS SDK Go with changed container API
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-07-05 11:26:06 +03:00
Evgenii Stratonikov
f699e82ea7 [#1560] morph/client: Remove customFees type
It is private, is a simple map and there is no complex logic to be wrapped in methods.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-04 09:29:01 +03:00
Evgenii Stratonikov
0ccea802e9 [#1560] morph/client: Perform RPC switch and restore in one step
Otherwise we could switch infinitely if subscription restore has failed.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-07-04 09:29:01 +03:00
Evgenii Stratonikov
6358f4d746 [#1053] config: Remove notification endpoints
Missed this in #1170.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-30 10:51:57 +03:00
Evgenii Stratonikov
ee27c66952 [#1555] util/gendoc: Remove trailing dot from the help message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-28 13:46:02 +03:00
Evgenii Stratonikov
ad81b6c90a [#1555] neofs-adm: Remove trailing dots from help messages
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-28 13:46:02 +03:00
Evgenii Stratonikov
b41658db04 [#1555] neofs-adm: Allow to remove nodes from the netmap
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-28 13:46:02 +03:00
Evgenii Stratonikov
32fd3692bf [#1555] neofs-adm: Fix cached group signer
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-28 13:46:02 +03:00
Evgenii Stratonikov
5c2ef79080 [#1547] .github: Assign issues to project when labeled
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-27 10:11:43 +03:00
Leonard Lyubich
305dd7598f [#1533] acl: Upgrade NeoFS SDK Go with refactored basic ACL
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-25 13:57:21 +03:00
Evgenii Stratonikov
b13dca8052 [#1545] .github: Auto assign new issues to the project
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-24 20:27:45 +03:00
Evgenii Stratonikov
451343c751 [#1539] neofs-adm: Retrieve storage wallet passwords from config
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-24 13:40:21 +03:00
Pavel Karpy
fc7940a5ea [#1544] cli: Init bearer flags in list-objects cmd
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-24 13:40:07 +03:00
Pavel Karpy
bb52372108 [#1544] go.mod: Update SDK
The updated version supports reading network configuration uint64 values
that consist of less than 8 bytes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-24 13:40:07 +03:00
Evgenii Stratonikov
9b3e1bd27b [#1307] neofs-adm: Add tests for morph init command
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-24 10:33:34 +03:00
Evgenii Stratonikov
189507dc89 [#1541] neofs-cli: allow to print shards list output in JSON
Eventually more parameters will be supported (#1390) and after blobstor
configuration refactoring the output will certainly change. Implement
the simplest approach now.

Marshaling the result directly results in too ugly names and they cannot
be easily customized. Marshaling the results via `jsonpb` is better but
is not that flexible in terms of what we want to output.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-24 10:33:22 +03:00
Evgenii Stratonikov
9816d59ec0 [#1323] neofs-cli: Reuse JSON flag for multiple commands
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-24 10:33:22 +03:00
Pavel Karpy
e812d78672 [#1537] node: Remove "trust" from non-reputation logs
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-23 10:06:36 +03:00
Leonard Lyubich
b67974a8d3 [#xxx] Upgrade NeoFS SDK Go with changed container sessions
After recent changes in NeoFS SDK Go library session tokens aren't
embedded into `container.Container` and `eacl.Table` structures.

Group value, session token and signature in a structure for container
and eACL.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-22 16:38:57 +03:00
Evgenii Stratonikov
633b4e7d2d [#1483] metabase: Add VERSION.md
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
6f243a2a76 [#1483] metabase: Store version
The main problem is to distinguish the case of initial initialization
and update from version 0. We can't do this at `Open`, because of
`resync_metabase` flag. Thus, the following approach was taken:
1. During `Open` check whether the metabase was initialized.
2. Check for the version in `Init` or write the new one if the metabase
   is new.
3. Update version in `Reset`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:48:28 +03:00
Evgenii Stratonikov
7df50297cd [#1520] shard: Ignore errors on metabase refill
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 17:25:28 +03:00
Evgenii Stratonikov
78ea450c25 [#1502] shard: Process locks on metabase refill
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-21 11:07:26 +03:00
Evgenii Stratonikov
972ca83e23 [#1524] writecache: Add some bolt parameters to the configuration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
07e06249d5 [#1524] metabase: Add some bolt parameters to the configuration
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
d6043abc24 [#1524] neofs-node: Reuse boltdb config for different components
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 17:04:35 +03:00
Evgenii Stratonikov
0408acc40e services/control: Autogenerate stable marshalers
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-20 16:52:58 +03:00
Leonard Lyubich
d2d1694113 [#1513] cli/storagegroup: Fix linter remarks
`staticcheck` linter scolds on potential nil-dereference since it
doesn't know `common.ExitOnErr` definitely exists. Explicit `return`
fixes linter's remark and prevents potential bugs.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
13e74fce8a [#1513] ir/settlement: Do not allocate intermediate slice of nodes
After recent changes `buildContainer` method returns two-dimensional
slice of `NodeInfo` so there is no need to flatten it to build slice of
`common.NodeInfo`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
c408a6a0db [#1513] morph/netmap: Use constant states in unit tests
Use values of the node state enumeration from Netmap contract instead of
numeric literals.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
eb7b8bf7c4 [#1513] node: Encode node's public key into binary format once
Avoid per-iteration key encoding due to the immutability of the key.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
4574ba646d [#1513] cli/storagegroup: Fix parsing container and SG flags
Read `id` flag with the storage group ID. Prevent NPE-panic if flag is
missing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
820acebb7d [#1513] morph/netmap: Return result by value from net config reader
Make `ReadNetworkConfiguration` method to return `NetworkConfiguration`
by value in order to follow storage engine improvements and prevent heap
escaping.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
808ba87e82 [#1513] morph/netmap: Use node state constants from Netmap contract
`Netmap` contract exports enumeration of the node states.

Replace using literals and constants from NeoFS API Go V2 with the
values provided by contract.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Leonard Lyubich
21d2f8f861 [#1513] Upgrade NeoFS SDK Go with changed netmap package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-17 15:53:18 +03:00
Evgenii Stratonikov
24b4c1ecf4 [#1484] neofs-cli: Print error on a separate line from a progress bar
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-16 16:38:33 +03:00
Evgenii Stratonikov
e42b7f6a65 [#1484] neofs-cli: Print progress bar after the header has been sent
Make it similar to `object get`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-16 16:38:33 +03:00
Evgenii Stratonikov
b68778ea90 [#1380] neofs-cli: move container command to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-16 09:27:07 +03:00
Evgenii Stratonikov
fd48b96082 [#1384] neofs-cli: move storagegroup command to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-16 09:27:07 +03:00
Evgenii Stratonikov
a219e3a667 [#1383] neofs-cli: move object command to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-16 09:27:07 +03:00
Evgenii Stratonikov
12bc5607f7 [#1382] neofs-cli: move netmap command to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-16 09:27:07 +03:00
Evgenii Stratonikov
81f925d5a0 [#1516] metabase: Optimize ListWithCursor for long listings
Cache buckets outside of the main loop and allocate memory for the
resulting offset only once.

```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       6.45µs ±14%    5.79µs ±11%  -10.24%  (p=0.002 n=10+10)
ListWithCursor/10_items-8     20.9µs ±17%    17.3µs ± 9%  -17.27%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     153µs ±12%     131µs ± 9%  -14.63%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       2.31kB ± 0%    1.91kB ± 0%  -17.46%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     6.94kB ± 0%    5.50kB ± 0%  -20.78%  (p=0.000 n=8+8)
ListWithCursor/100_items-8    53.3kB ± 0%    41.5kB ± 0%  -22.18%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         40.0 ± 0%      34.0 ± 0%  -15.00%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        121 ± 0%       100 ± 0%  -17.36%  (p=0.000 n=10+10)
ListWithCursor/100_items-8       930 ± 0%       758 ± 0%  -18.49%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
a93373fe71 [#1516] metabase: Cache graveyard buckets in ListWithCursor
```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       6.40µs ±13%    6.45µs ±14%     ~     (p=0.739 n=10+10)
ListWithCursor/10_items-8     30.9µs ±21%    20.9µs ±17%  -32.49%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     274µs ±27%     153µs ±12%  -44.09%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       2.26kB ± 0%    2.31kB ± 0%   +2.46%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     10.8kB ± 0%     6.9kB ± 0%  -36.07%  (p=0.000 n=8+8)
ListWithCursor/100_items-8    96.8kB ± 0%    53.3kB ± 0%  -44.98%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         39.0 ± 0%      40.0 ± 0%   +2.56%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        192 ± 0%       121 ± 0%  -36.98%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     1.72k ± 0%     0.93k ± 0%  -45.93%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>

name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       5.23µs ±19%    5.26µs ±15%     ~     (p=0.853 n=10+10)
ListWithCursor/10_items-8     27.2µs ±15%    18.0µs ±19%  -33.80%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     250µs ±13%     139µs ±15%  -44.27%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       1.99kB ± 0%    2.04kB ± 0%   +2.82%  (p=0.000 n=8+8)
ListWithCursor/10_items-8     10.3kB ± 0%     6.4kB ± 0%  -37.83%  (p=0.000 n=8+10)
ListWithCursor/100_items-8    93.9kB ± 0%    50.4kB ± 0%  -46.37%  (p=0.000 n=10+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         35.0 ± 0%      36.0 ± 0%   +2.86%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        184 ± 0%       113 ± 0%  -38.59%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     1.67k ± 0%     0.88k ± 0%  -47.29%  (p=0.000 n=10+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
af4db8a73b [#1516] metabase: Cache address key and do not decode address twice
```
name                        old time/op    new time/op    delta
ListWithCursor/1_item-8       10.6µs ± 1%     6.4µs ±13%  -39.62%  (p=0.000 n=7+10)
ListWithCursor/10_items-8     75.3µs ± 2%    30.9µs ±21%  -58.97%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     726µs ± 2%     274µs ±27%  -62.28%  (p=0.000 n=10+10)

name                        old alloc/op   new alloc/op   delta
ListWithCursor/1_item-8       3.19kB ± 0%    2.26kB ± 0%  -29.21%  (p=0.000 n=10+10)
ListWithCursor/10_items-8     20.7kB ± 0%    10.8kB ± 0%  -47.68%  (p=0.000 n=10+8)
ListWithCursor/100_items-8     196kB ± 0%      97kB ± 0%  -50.65%  (p=0.000 n=7+10)

name                        old allocs/op  new allocs/op  delta
ListWithCursor/1_item-8         55.0 ± 0%      39.0 ± 0%  -29.09%  (p=0.000 n=10+10)
ListWithCursor/10_items-8        346 ± 0%       192 ± 0%  -44.51%  (p=0.000 n=10+10)
ListWithCursor/100_items-8     3.25k ± 0%     1.72k ± 0%  -47.13%  (p=0.000 n=9+10)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Evgenii Stratonikov
504f45e9ee [#1516] metabase: Add benchmark for ListWithCursor
```
name                        time/op
ListWithCursor/1_item-8     10.6µs ± 1%
ListWithCursor/10_items-8   75.3µs ± 2%
ListWithCursor/100_items-8   726µs ± 2%

name                        alloc/op
ListWithCursor/1_item-8     3.19kB ± 0%
ListWithCursor/10_items-8   20.7kB ± 0%
ListWithCursor/100_items-8   196kB ± 0%

name                        allocs/op
ListWithCursor/1_item-8       55.0 ± 0%
ListWithCursor/10_items-8      346 ± 0%
ListWithCursor/100_items-8   3.25k ± 0%
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 20:49:41 +03:00
Pavel Karpy
36f4929e52 [#1507] node: Do not handle object concurrently by the policer
Cache object that are being processed. That prevents concurrent
object handling when there is a few number of objects and object handling
takes more time that the policer needs for starting that object handling one
more time.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-15 20:43:32 +03:00
Pavel Karpy
256165045b [#1508] node: Do not replicate object twice
If placement contains two vectors with intersecting nodes it was possible to
send the object to the nodes twice.
Also optimizes requests: do not ask about storing the object twice from the
same node.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-15 20:33:04 +03:00
Pavel Karpy
df8a3807fe [#1508] node: Remove unused replicator code
The node does not support asynchronous object replication anymore, so it
does not need to have replicator worker, channel and `AddTask` function.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-15 20:33:04 +03:00
Leonard Lyubich
2e4a1cb6df [#1518] Upgrade NeoFS SDK Go with changed subnet package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-15 17:02:49 +03:00
Evgenii Stratonikov
f602d05b0a [#1494] *: Fix linter warnings
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
bbf8b8e74d [#1494] services/object: Do not ignore bearer token decode errors
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
795d1e0789 [#1494] go.mod: Update neofs-sdk-go
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 12:26:10 +03:00
Evgenii Stratonikov
aab6094a7c [#1515] neofs-node: Cache max object size
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-15 11:14:37 +03:00
Evgenii Stratonikov
9269ed344d [#1510] placement: Cache container nodes
In case we have lots of objects in a single container,
`GetContainerNodes` invoked indirectly by a policer can be seen in
pprof.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-14 20:17:43 +03:00
Evgenii Stratonikov
0dad0114f0 [#1511] CHANGELOG.md: Fill unreleased section
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-10 16:21:07 +03:00
Pavel Karpy
8c77387982 [#1509] morph: Log WS client closing error
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-10 12:53:44 +03:00
Pavel Karpy
1708cc0cc7 Fix changelog
It contained duplicated `Fixed`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-09 16:17:30 +03:00
Evgenii Stratonikov
3db4526437 [#1498] neofs-adm: Save wallets as pretty JSON
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-09 16:16:46 +03:00
Evgenii Stratonikov
9e430abf83 [#1498] go.mod: Update neo-go to master
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-09 16:16:46 +03:00
Evgenii Stratonikov
38772e1a2e writecache: provide timeout to bbolt.Open
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-09 14:19:47 +03:00
Pavel Karpy
68a0112193 Release v0.28.3
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-08 14:21:58 +03:00
Pavel Karpy
9f2dd441c3 [#1466] Update changelog
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-08 11:06:19 +03:00
Pavel Karpy
4ed3063212 [#1466] adm: Remove notary flag in for subnet command
It is possible to deduce that information from the network.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-08 11:06:19 +03:00
Pavel Karpy
815e02f185 [#1466] adm: Do not use morph client in adm
Using morph client leads to requiring WebSocket endpoints.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-08 11:06:19 +03:00
Evgenii Stratonikov
60789dd475 [#1489] neofs-adm: Include alphabet contracts to the group
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-08 11:01:16 +03:00
Evgenii Stratonikov
91ed0d20ff [#1417] go.mod: Update neo-go to v0.99.0
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-08 10:27:49 +03:00
Leonard Lyubich
fa009db140 [#1464] ir/container: Fix verifying the operations within sessions
In previous implementation `verifySignature` method of container
processor worked incorrectly for operations without a key and with
session: processor tried to verify signature with one of the bound owner
keys instead of session one.

Use `VerifySessionDataSignature` method to check the signature if
session is used. Refactor `verifySignature` a bit with session check
highlighting for readability.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-07 23:26:30 +03:00
Leonard Lyubich
f6d121a4e4 [#1485] Upgrade NeoFS SDK Go module
New version includes the fix of the session signing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-07 16:34:15 +03:00
Leonard Lyubich
0937513c14 [#1485] ir/container: Accept eACL only if extension is allowed
In order to extend container ACL `F` bit must be set in basic ACL.

Make `Container` contract processor to deny eACL tables bound to
non-extendable containers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-07 16:34:15 +03:00
Evgenii Stratonikov
af7d15cc1a [#1487] innerring: Log errors in processors
Log errors for network operations. The only places where we are not
interested in errors are `Submit` in pool and unmarshaling.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-07 14:11:58 +03:00
Pavel Karpy
79d72a6713 [#1460] Update the changelog
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
010253a97a [#1460] blobovnicza: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
0e4a1beecf [#1460] blobstor: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
08bf8a68f1 [#1460] engine: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
7b6363f4c6 [#1460] shard: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
9b2932609b [#1460] meta: Do not use pointers as the results
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
a580429996 [#1460] meta: Add a benchmark on Get operation
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
b0c7b7851a [#1418] blobstor: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
14366bbd89 [#1418] engine: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
5f57db6bf8 [#1418] shard: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Pavel Karpy
6d4d920bad [#1418] meta: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-06 18:03:12 +03:00
Evgenii Stratonikov
a46f585fb3 [#1074] neofs-cli: Inline some of the functions
They are quite small, however will produce problems during subsequent refactoring.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-06 13:54:29 +03:00
Evgenii Stratonikov
9efec21d34 [#1074] neofs-cli: Move session preparation to modules/session package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-06 13:54:29 +03:00
Evgenii Stratonikov
a2bcb3e0ce [#1074] neofs-cli: Move common API flags to a separate package
TTL and XHeader flags are reused between multiple commands.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-06 13:54:29 +03:00
Evgenii Stratonikov
736e09a70d [#1074] neofs-cli: Move PrettyPrintUnixTime to the common package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-06 13:54:29 +03:00
Evgenii Stratonikov
b432558aaa Release v0.28.2
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 17:38:55 +03:00
Leonard Lyubich
72708296cc Upgrade NeoFS SDK Go to v1.0.0-rc.4 and NeoFS API Go to v2.12.2
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 17:17:50 +03:00
Evgenii Stratonikov
c41d9c3fbe [#1474] neofs-node/config: Remove "shard_num" from the "grpc" section
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:44:06 +03:00
Evgenii Stratonikov
73a88c2965 [#1474] neofs-node/config: Remove num from the "grpc" section
It isn't needed and can be replaced with additional checks during the
iteration.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:44:06 +03:00
Leonard Lyubich
0e28902b0f [#1471] eacl: Fix incorrect request denial with incomplete object header
Node shouldn't perform eACL verification during GET/HEAD request
processing until full object header is received. Otherwise, for some
eACL tables request may be falsely rejected.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Leonard Lyubich
c89035d544 [#1471] eacl: Add testcase with incomplete list of object headers
Scenario:
 * HEAD request of some object
 * 1st eACL record allows op for objects with specific user attribute
 * 2nd eACL record forbids op by object ID
 * node doesn't store the requested object locally

With this scenario node shouldn't deny request.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Leonard Lyubich
f9504c1cba [#1471] eacl: Don't process object headers twice in response processing
It is redundant to process object headers in responses w/o object field
since result will be the same.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Leonard Lyubich
8b94cb22d3 [#1471] eacl: Prevent aborting on local storage failure
Request processing should not be interrupted in case of local storage
failure since error case in normal for relay nodes.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Leonard Lyubich
a97ea2aa21 [#1471] eacl: Add testcase with local storage failure
ACL service should not deny request on local storage failure since in
this case relay nodes won't be able to continue the operation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-03 14:42:54 +03:00
Pavel Karpy
fdd54b0a03 [#1441] ir: Do not validate subnet removal
It is useless process since subnet owner is able to delete subnet without an
Alphabet approval. The Alphabet should only validate netmap state after
removal:
1. Update nodes' attributes if they were included in the deleted subnet;
2. Remove nodes without any subnet entrance.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 14:36:58 +03:00
Evgenii Stratonikov
feef9a98f7 [#1462] writecache: remove useless logs
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
2ae7c94cd6 [#1462] *: Remove log.With invocations
`log.With` is suitable during initialization, but in other places it induces
some overhead, even when branches with logging are not taken.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
43e776dfb1 [#1462] writecache: Persist objects in batches
`Update` becomes a botleneck under a heavy load.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
6ad87e7959 [#1462] writecache: Fill flush marks during startup
Some of the objects are already flushed, don't do it twice.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
f2a7503964 [#1462] fstree: Allow to fetch file content lazily
If we should process address based on some condition, there is no need
to read file content in memory.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
54d4503701 [#1462] writecache: Rename flushWorkersCount
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
0148209168 [#1462] writecache: Use Timer instead of Ticker in flush
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:30:00 +03:00
Evgenii Stratonikov
5073a37930 [#1433] metabase: Optimize Select
For some filters we can scan only a subset of keys instead of checking
each key.

```
name                       old time/op  new time/op  delta
Select/string_equal-8      49.3µs ± 4%  11.0µs ± 4%  -77.68%  (p=0.000 n=10+10)
Select/string_not_equal-8  7.01ms ± 5%  7.06ms ±10%     ~     (p=0.971 n=10+10)
Select/common_prefix-8      118µs ± 6%    79µs ± 5%  -33.04%  (p=0.000 n=10+9)
Select/unknown-8           21.3µs ± 4%   3.2µs ± 4%  -84.88%  (p=0.000 n=10+9)
```

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
5c48588c64 [#1433] metabase: Add more tests for Select
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
ea76c989ca [#1433] metabase: Add a benchmark for Select
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 14:25:07 +03:00
Evgenii Stratonikov
b602fff01f [#1463] blobovnicza: Remove pointers from struct members
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 13:42:54 +03:00
Evgenii Stratonikov
d9f0ac8909 [#1463] blobovnicza: Determine db size via os.Stat
Currently we use `(*bbolt.Bucket).Stats().KeyN` for estimating database
size. However, it iterates over all pages in bucket and thus heavily
depends on the bucket size. This commit replaces initial size estimation
with a single `os.Stat` call.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-03 13:42:54 +03:00
Pavel Karpy
021aa97965 [#1418] blobovnicza: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
281befec67 [#1418] blobstor: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
babd382ba5 [#1418] engine: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
6e752f36dc [#1418] shard: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
e265ce2d52 [#1418] meta: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
1c100fb4b0 [#1418] writecache: Do not use pointers as parameters
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-03 07:35:17 +03:00
Pavel Karpy
da3ae202f0 [#1436] node: Log service initialization/boot up
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-06-02 16:16:39 +03:00
Evgenii Stratonikov
d84d52924a [#1467] pkg/util: Allow to load default account from a wallet
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-06-02 16:13:20 +03:00
Leonard Lyubich
2da8396a9f [#1454] cli/util: Support container sessions in sign session-token
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Leonard Lyubich
ff576d125d [#1454] container: Fix reading session token from request origin
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Leonard Lyubich
1c30414a6c [#1454] Upgrade NeoFS SDK Go module with new IDs
Core changes:
 * avoid package-colliding variable naming
 * avoid using pointers to IDs where unnecessary
 * avoid using `idSDK` import alias pattern
 * use `EncodeToString` for protocol string calculation and `String` for
  printing

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-06-01 17:41:45 +03:00
Leonard Lyubich
cc6209e8a0 [#1449] object/delete: Fix NPE in collectChain
`head.previous` method returns (nil, nil) if object has no previous
neighbor.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-30 17:59:16 +03:00
Leonard Lyubich
96cdc04705 [#1449] policer: Unwrap status HEAD response
Helper function `client.IsErrObjectNotFound` doesn't support error
unwrapping, so we need to do it on caller side.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-30 17:59:16 +03:00
Leonard Lyubich
e96ea4635c [#1449] policer: Fix selection of new storage candidates
`Policer` should pass list of selected candidates into `WithNodes`
method of `replicator.Task`. In previous implementation `processNodes`
method passed an opposite list: failed nodes and/or the local one.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-30 17:59:16 +03:00
Evgenii Stratonikov
855cbf5a3a [#1159] services/container: Remove ContainerWithContext struct
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-30 15:40:22 +03:00
Leonard Lyubich
2bcc0051ab [#1423] session: Get session issuer from token structure
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Leonard Lyubich
2dec3a0a93 [#1423] ir/container: Fix error message related to container removal
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Leonard Lyubich
883f2f1ac6 [#1423] local storage: Activate skipped unit tests after fix
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Leonard Lyubich
4c8ec20e32 [#1423] session: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-27 15:45:33 +03:00
Evgenii Stratonikov
dda56f1319 [#1437] neofs-cli: Move util command to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-26 12:03:39 +03:00
Pavel Karpy
786c920fb2 [#1430] go.mod: Update neo-go
v0.98.2 => v0.98.3

That patch includes fixing panic when using WS client with a closed
connection.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-25 13:54:58 +03:00
Pavel Karpy
57c5fccc8c [#1428] node/acl: Make OID optional
Not all the NeoFS requests must contain OID in their bodies (or must NOT
contain them at all). Do not pass object address in helper functions, pass
CID and OID separately instead.
Also, fixed NPE in the ACL service: updated SDK library brought errors
when working with `Put` and `Search` requests without OID fields.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-25 12:11:03 +03:00
Pavel Karpy
d69eb2aaf3 [#1428] node/acl: Make container ID as required param
Change pointer to value in request information since requests could not
exist without container ID.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-25 12:11:03 +03:00
Evgenii Stratonikov
6cb9c13c5e [#1424] neofs-cli: Fail immediately if a client can't be created
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-25 09:46:41 +03:00
Evgenii Stratonikov
295ec3700a [#1424] neofs-cli: Fail immediately if a key can't be fetched
If the key can't be fetched, an error is always returned, so it makes
sense to fail the whole command inside of a `key.Get*()`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-25 09:46:41 +03:00
Evgenii Stratonikov
d9c5ca5e77 [#1381] neofs-cli: Move control command to a separate module
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-25 09:46:41 +03:00
Evgenii Stratonikov
71d823f192 [#1381] neofs-cli: Move JSON and verbose printing to a common package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-25 09:46:41 +03:00
Evgenii Stratonikov
c57114def3 [#1422] morph/client: Fetch group key properly
NNS contract `resolve` method returns an array.
`neofs-adm` is already up to date.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-23 19:24:46 +03:00
Leonard Lyubich
86552cf3ae engine: make corruptSubDir to ignore blobovnicza dir
There is a dir for Blobovnicza tree in BlobStor, so we need to ignore
it.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:33:14 +03:00
Leonard Lyubich
bb25ecbd15 [#1400] owner: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:33:14 +03:00
Leonard Lyubich
f8ac4632f8 [#1335] policer: Prevent potential object loss
In previous implementation `Policer` considered local object copy as
redundant on processing single placement vector.

Make `Policer` to call redundant copy callback after full placement
processing. Also fix 404 error parsing.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-23 15:24:23 +03:00
Pavel Karpy
9f62d25b50 [#1224] ir: Delete nodes that do not have subnets
Also, update their subnet entrance on subnet removal events.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Pavel Karpy
b3e1e90c01 [#1224] ir: Listen to subnet removal events
IR must listen to subnet removal notifications always (regardless of notary
mode).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Pavel Karpy
71f18ba9ec [#1224] morph: Call Alphabet methods flexibly
Call `UpdateStateIR` and `AddPeerIR` method instead of `UpdateState` and
`AddPeer` if calling client is configured as Alphabet in notary enabled
environment.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Pavel Karpy
a129fc98da [#1224] morph: Add configuration getters
Includes `WithNotary` and `IsAlpha` methods to the `StaticClient`.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Pavel Karpy
f0c9a68ad3 [#1224] morph: Delete wrong comment
Current implementation does not use MurmurHash3.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-23 08:16:24 +03:00
Evgenii Stratonikov
df48ddcb3d Revert "[#1141] morph/client: Allow to use more integer types as arguments"
This reverts commit 9349f422fd9a29ec529ef0648bbfabd1f9ff7651.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 16:45:15 +03:00
Evgenii Stratonikov
4761857fb3 [#1396] cmd: Allow to autogenerate documentation for all commands
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:35:22 +03:00
Evgenii Stratonikov
c6325fdc91 [#1379] neofs-cli: Move accounting command to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:20:34 +03:00
Evgenii Stratonikov
56f33436dd [#1379] neofs-cli: Move exitOnErr to internal package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:20:34 +03:00
Evgenii Stratonikov
094534e31a [#1379] neofs-cli: Add key.GetOrGenerate helper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:20:34 +03:00
Evgenii Stratonikov
cd46a7478e [#1379] neofs-cli: Bind key-related arguments to viper
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:20:34 +03:00
Evgenii Stratonikov
a198a4858c [#1379] neofs-cli: Move client creation to internal/client
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:20:34 +03:00
Evgenii Stratonikov
cbc4ca800d [#1379] neofs-cli: Move common flags to a separate package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 13:20:34 +03:00
Pavel Karpy
4be5dce848 [#1405] ir: Fix settlement log message
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-20 12:40:12 +03:00
Pavel Karpy
b11f35a5c7 [#1403] CHANGELOG: Update changelog
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-20 12:40:12 +03:00
Evgenii Stratonikov
3f2889a4e9 [#1392] eacl/v2: Use EncodeToString where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
63cdc16088 [#1386] eacl/v2: Do not allocate intermediate slices for XHeaders
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
915f87959e [#1386] eacl/v2: Allocate []eaclSDK.Header slice immediately
There is no point in having intermediate `[]sessionSDK.XHeader`.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
b24589b62d [#1386] eacl/v2: Use raw structs where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Evgenii Stratonikov
3f4475f97b [#1386] services/object: Fail eACL check if the request is invalid
Parse all headers beforehand and reject invalid requests.
Another approach would be to remember the error and check
it after `CalculateAction`, which is a bit faster.
The rule of thumb here is "first validate, then use".

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-20 12:04:10 +03:00
Pavel Karpy
f99a0498da [#1403] util: Add details to errors
Make error messages more descriptive when parsing LOCODE from string errors
appear.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-05-20 10:53:27 +03:00
Leonard Lyubich
aeb9884218 [#1389] crypto: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-19 10:12:14 +03:00
Leonard Lyubich
5e50ddd7f5 [#1389] storagegroup: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-19 10:12:14 +03:00
Evgenii Stratonikov
0f376a5d83 [#1388] services/control: Extend tests for ShardInfo marshaling
It is nice to have different paths for different components and also
check that the information returned is different for different shards.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-17 13:48:07 +03:00
Leonard Lyubich
f15e6e888f [#1377] oid, cid: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-16 15:33:22 +03:00
Evgenii Stratonikov
f65898a354 [#1141] morph/client: Use boolean type directly
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:09:35 +03:00
Evgenii Stratonikov
230a5cd037 [#1141] morph/client: Remove unneeded argument conversions
Client supports `util.Uint160` directly, no need to convert it to bytes.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:09:35 +03:00
Evgenii Stratonikov
1e8391d216 [#1141] morph/client: Allow to use more integer types as arguments
Also, use `*big.Int` as integer value, see nspcc-dev/neo-go#2413.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:09:35 +03:00
Evgenii Stratonikov
1c7195666c [#722] neofs-adm: Unify pasword retrieval functions
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:07:31 +03:00
Evgenii Stratonikov
b685af487f [#722] neofs-adm: Replace path with filepath package
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:07:31 +03:00
Evgenii Stratonikov
5bd6624eb6 [#722] neofs-adm: Provide contract state to getContractDeployParameters
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:07:31 +03:00
Evgenii Stratonikov
efb6545bfe [#722] neofs-adm: Allow to initialize local dump
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:07:31 +03:00
Evgenii Stratonikov
c55950bd70 [#722] neofs-adm: Check returned tx hash
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:07:31 +03:00
Evgenii Stratonikov
37f9d083fb [#722] neofs-adm: Hide N3 client behind an interface
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-16 15:07:31 +03:00
Evgenii Stratonikov
57200e18cd [#1375] neofs-adm: Do not update NNS group if the key is the same
If the group key is already set, do not send any transactions.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-13 12:31:46 +03:00
Leonard Lyubich
3a188bb2e5 [#1371] bearer: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 15:16:17 +03:00
Leonard Lyubich
ae92074272 [#1369] checksum: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Leonard Lyubich
088df0e2a9 [#1369] version: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Leonard Lyubich
7a57d7b076 [#1369] audit: Upgrade SDK package
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-12 09:48:31 +03:00
Evgenii Stratonikov
875f0e79a2 [#1346] neofs-adm: do not ask password twice
When a contract wallet is created, there is no need to decrypt
newly-created accounts.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-05-11 12:01:11 +03:00
tasselsd
379c70cc90 [#1363] Use network package to parse and verify the announced address.
Signed-off-by: tasselsd <tasselsd@outlook.com>
2022-05-11 11:22:34 +03:00
Alex Vanin
8f1a0fb6e7 [#1358] Update CHANGELOG.md
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-11 11:15:54 +03:00
Alex Vanin
75b6f854e1 [#1358] example: Update README.md
Mention Inner ring and CLI applications.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-11 11:15:54 +03:00
Alex Vanin
5988653fc6 [#1358] example: Drop comments in ir.env file
Field descriptions are available in ir.yaml file.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-11 11:15:54 +03:00
Alex Vanin
d6be5d4087 [#1358] example: Add configs for inner ring application
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-05-11 11:15:54 +03:00
Elizaveta Chichindaeva
cc7a723d77 [#1320] English Check
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
2022-05-11 10:40:02 +03:00
Leonard Lyubich
d99800ee93 Release v0.28.1
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-05 11:55:35 +03:00
Leonard Lyubich
cd545f0160 [#1351] cli: Fix connection scheme loss during endpoint parsing
In previous implementation NeoFS CLI app used `network.Address.HostAddr`
as a server URI, which caused scheme loss since host address doesn't
contain it.

Rename `HostAddr` to `URIAddr` and make it to return URI address with
`grpcs` scheme if TLS is enabled. Make `TLSEnabled` unexported since it
was used to provide default `tls.Config` only (it is used by default in
SDK).

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-05-05 10:43:27 +03:00
Leonard Lyubich
3a44010180 Release v0.28.0 - Heuksando (흑산도, 黑山島)
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-29 19:05:02 +03:00
Evgenii Stratonikov
a4769d8624 *: use require.ErrorIs where possible
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:47:57 +03:00
Evgenii Stratonikov
3c39e6df11 [#1319] blobstor: Compress big objects in a streaming fashion
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov
1219ff89d4 [#1319] services/object_manager: Fix error message
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov
057d53459b [#1319] services/object: Remove Write method from local and remote targets
They work with prepared objects only.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Evgenii Stratonikov
4ea03c01b5 [#1319] services/object: Store payload directly in distributedTarget
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-29 16:46:40 +03:00
Pavel Karpy
9da7df4d42 [#1318] metabase: Init metabase in test
`Init()` creates buckets. It prevents NPE in tests.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
486ec8a2e5 [#1318] metabase: Reuse comparing func from require
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
1caeb0b648 [#1318] metabase: Add missing error check in tests
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
6f2723b47e [#1318] node: Fix linter
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
2aec5736e0 [#1318] node/gc: Fix tombstones handling
Handle only expired tombstones.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
a275a71a87 [#1318] node: Use new tombstone handling
Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
24ca8ca5c7 [#1318] object: Change default ttl
This is need to reuse object service without passing any API v2 structs.
Also, fix comment.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
7799f8e4cf [#1318] engine: Change tombstone clear process
- Delete objects physically on tombstone's arrival;
- Store information about tombstones in the Graveyard;
- Clear Graveyard every epoch based on the information about TS in the
network.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
e4cfeec449 [#1318] service: Add tombstone checker service
The service fetches tombstones from the network via object service, every
request is handled in the following order:
1. checks local LRU cache;
2. checks local storage engine;
3. tries to find object in the placement nodes.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
2583f608e8 [#1318] metabase: Add DropGraves method
Method clean up Graveyard and should be called by the routine that handles
tombstone expiration.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
daab30c391 [#1318] metabase: Add iteration offset
Add offset element to the iterations over deleted objects (both the
Graveyard and the Garbage buckets).

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
fe8076e60a [#1318] metabase: Add immediate object deletion
Mark objects with GC immediately after a Tombstone is received.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Pavel Karpy
8107c8d1a9 [#1318] metabase: Separate buckets with TS and GC marks
It allows storing information about object in both ways at the same time:
1. Metabase should know if an object is covered by a tombstone (that is
not expired yet);
2. It should be possible to physically delete objects covered by a
tombstone immediately (mark with GC) but keep tombstone knowledge.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-29 16:38:52 +03:00
Evgenii Stratonikov
b74fb2b932 [#1316] neofs-cli: Print object ID in base-58
It was missed from dd9bd05ba.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:54:18 +03:00
Evgenii Stratonikov
5736b834c3 [#1286] services/object: Validate object when created with sesssion token
All fields set by a user should be verified.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:53:22 +03:00
Evgenii Stratonikov
c6a9c5cd8c [#1283] services/object: Disallow creating objects without a session token
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:53:22 +03:00
Evgenii Stratonikov
a254fd6bc8 [#1216] neofs-cli: Allow to use session token from file
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:47:43 +03:00
Evgenii Stratonikov
a95bdb1811 [#1216] neofs-cli: Allow to create and save session token
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:47:43 +03:00
Evgenii Stratonikov
8d79168129 [#1216] neofs-cli: Reuse key retrieving code between modules
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:47:43 +03:00
Evgenii Stratonikov
006d6e8b48 [#1216] neofs-cli: Reuse GetSDKClient between CLI modules
Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:47:43 +03:00
Evgenii Stratonikov
fd8c00400d [#1216] neofs-cli: Rename token subcommand to bearer
It works only with the bearer token.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-04-25 10:47:43 +03:00
Pavel Karpy
bd27837364 [#1321] node: Register GC event channel before shard init
Morph "NewEpoch" event handling was registered in a closure over
`addNewEpochNotificationHandler` func. That may lead to the data race:
if a shard was initialized before the event registration, everything works
as planned, but if registration was made earlier, it was not able to
include GC handlers since a shard has not called `eventChanInit` yet and,
therefore, it has not registered handler yet.

Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
2022-04-25 10:39:37 +03:00
Leonard Lyubich
d4569946c5 [#1313] container: Do not invalidate cache on Container.Delete
In previous implementation `Container.Delete` operation caused local
node's cache invalidation (container itself, eACL and listings). Any
subsequent `Container.Get` operation reversed invalidation. Given the
low latency sensitivity of deleting a container, there is no need to
touch the cache. With this approach, all pending deletion operations on
the node via the NeoFS API protocol will be delayed by the cache TTL.

Do not call cache invalidation ops in `morphContainerWriter.Delete`.
Remove no longer needed `InvalidateContainerListByCID` and
`InvalidateContainer` methods.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
2022-04-25 10:34:12 +03:00
1284 changed files with 71757 additions and 50906 deletions

View file

@ -1,19 +1,19 @@
FROM golang:1.17 as builder
FROM golang:1.20 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-adm
RUN make bin/frostfs-adm
# Executable image
FROM alpine AS neofs-adm
FROM alpine AS frostfs-adm
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /src/bin/neofs-adm /bin/neofs-adm
COPY --from=builder /src/bin/frostfs-adm /bin/frostfs-adm
CMD ["neofs-adm"]
CMD ["frostfs-adm"]

25
.docker/Dockerfile.ci Normal file
View file

@ -0,0 +1,25 @@
FROM golang:1.20
WORKDIR /tmp
# Install apt packages
RUN apt-get update && apt-get install --no-install-recommends -y \
pip \
&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
&& rm -rf /var/lib/apt/lists/*
# Dash → Bash
RUN echo "dash dash/sh boolean false" | debconf-set-selections
RUN DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
RUN useradd -u 1234 -d /home/ci -m ci
USER ci
ENV PATH="$PATH:/home/ci/.local/bin"
COPY .pre-commit-config.yaml .
RUN pip install "pre-commit==3.1.1" \
&& git init . \
&& pre-commit install-hooks \
&& rm -rf /tmp/*

View file

@ -1,19 +1,19 @@
FROM golang:1.17 as builder
FROM golang:1.20 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-cli
RUN make bin/frostfs-cli
# Executable image
FROM alpine AS neofs-cli
FROM alpine AS frostfs-cli
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /src/bin/neofs-cli /bin/neofs-cli
COPY --from=builder /src/bin/frostfs-cli /bin/frostfs-cli
CMD ["neofs-cli"]
CMD ["frostfs-cli"]

View file

@ -3,6 +3,6 @@ RUN apk add --no-cache bash ca-certificates
WORKDIR /
COPY bin/neofs-adm /bin/neofs-adm
COPY bin/frostfs-adm /bin/frostfs-adm
CMD ["neofs-adm"]
CMD ["frostfs-adm"]

View file

@ -3,6 +3,6 @@ RUN apk add --no-cache bash ca-certificates
WORKDIR /
COPY bin/neofs-cli /bin/neofs-cli
COPY bin/frostfs-cli /bin/frostfs-cli
CMD ["neofs-cli"]
CMD ["frostfs-cli"]

View file

@ -3,6 +3,6 @@ RUN apk add --no-cache bash ca-certificates
WORKDIR /
COPY bin/neofs-ir /bin/neofs-ir
COPY bin/frostfs-ir /bin/frostfs-ir
CMD ["neofs-ir"]
CMD ["frostfs-ir"]

View file

@ -3,6 +3,6 @@ RUN apk add --no-cache bash ca-certificates
WORKDIR /
COPY bin/neofs-node /bin/neofs-node
COPY bin/frostfs-node /bin/frostfs-node
CMD ["neofs-node"]
CMD ["frostfs-node"]

View file

@ -1,18 +1,18 @@
FROM golang:1.17 as builder
FROM golang:1.20 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-ir
RUN make bin/frostfs-ir
# Executable image
FROM alpine AS neofs-ir
FROM alpine AS frostfs-ir
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/neofs-ir /bin/neofs-ir
COPY --from=builder /src/bin/frostfs-ir /bin/frostfs-ir
CMD ["neofs-ir"]
CMD ["frostfs-ir"]

View file

@ -1,18 +1,18 @@
FROM golang:1.17 as builder
FROM golang:1.20 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-node
RUN make bin/frostfs-node
# Executable image
FROM alpine AS neofs-node
FROM alpine AS frostfs-node
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/neofs-node /bin/neofs-node
COPY --from=builder /src/bin/frostfs-node /bin/frostfs-node
CMD ["neofs-node"]
CMD ["frostfs-node"]

View file

@ -1,19 +0,0 @@
FROM golang:1.17 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/neofs-node
# Executable image
FROM alpine AS neofs-node
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/neofs-node /bin/neofs-node
COPY --from=builder /src/config/testnet/config.yml /config.yml
CMD ["neofs-node", "--config", "/config.yml"]

View file

@ -5,4 +5,5 @@ docker-compose.yml
Dockerfile
temp
.dockerignore
docker
docker
.cache

View file

@ -0,0 +1,38 @@
name: Build
on: [pull_request]
jobs:
build:
name: Build Components
runs-on: ubuntu-latest
strategy:
matrix:
go_versions: [ '1.19', '1.20' ]
steps:
- uses: actions/checkout@v3
with:
# Allows to fetch all history for all branches and tags.
# Need this for proper versioning.
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '${{ matrix.go_versions }}'
- name: Build CLI
run: make bin/frostfs-cli
- name: Build NODE
run: make bin/frostfs-node
- name: Build IR
run: make bin/frostfs-ir
- name: Build ADM
run: make bin/frostfs-adm
- name: Build LENS
run: make bin/frostfs-lens

View file

@ -0,0 +1,72 @@
name: Tests and linters
on: [pull_request]
jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.20'
cache: true
- name: golangci-lint
uses: https://github.com/golangci/golangci-lint-action@v3
with:
version: latest
tests:
name: Tests
runs-on: ubuntu-latest
strategy:
matrix:
go_versions: [ '1.19', '1.20' ]
fail-fast: false
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '${{ matrix.go_versions }}'
cache: true
- name: Run tests
run: make test
tests-race:
name: Tests with -race
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.20'
cache: true
- name: Run tests
run: go test ./... -count=1 -race
staticcheck:
name: Staticcheck
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.20'
cache: true
- name: Install staticcheck
run: make staticcheck-install
- name: Run staticcheck
run: make staticcheck-run

1
.gitattributes vendored
View file

@ -1,2 +1,3 @@
/**/*.pb.go -diff -merge
/**/*.pb.go linguist-generated=true
/go.sum -diff

View file

@ -2,7 +2,7 @@
name: Bug report
about: Create a report to help us improve
title: ''
labels: community, triage
labels: community, triage, bug
assignees: ''
---
@ -18,8 +18,11 @@ assignees: ''
If suggesting a change/improvement, explain the difference from current behavior -->
## Possible Solution
<!-- Not obligatory, but suggest a fix/reason for the bug,
or ideas how to implement the addition or change -->
<!-- Not obligatory
If no reason/fix/additions for the bug can be suggested,
uncomment the following phrase:
No fix can be suggested by a QA engineer. Further solutions shall be up to developers. -->
## Steps to Reproduce (for bugs)
<!-- Provide a link to a live example, or an unambiguous set of steps

View file

@ -18,3 +18,11 @@ assignees: ''
## Additional context
<!-- Add any other context or screenshots about the feature request here. -->
## Don't forget to add labels!
- component label (`neofs-adm`, `neofs-storage`, ...)
- issue type (`enhancement`, `refactor`, ...)
- `goodfirstissue`, `helpwanted` if needed
- does this issue belong to an epic?
- priority (`P0`-`P4`) if already triaged
- quarter label (`202XQY`) if possible

197
.github/logo.svg vendored
View file

@ -1,129 +1,70 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
sodipodi:docname="logo_fs.svg"
inkscape:version="1.0 (4035a4fb49, 2020-05-01)"
id="svg57"
version="1.1"
viewBox="0 0 105 25"
height="25mm"
width="105mm">
<defs
id="defs51">
<clipPath
clipPathUnits="userSpaceOnUse"
id="clipPath434">
<path
d="M 0,0 H 1366 V 768 H 0 Z"
id="path432" />
</clipPath>
</defs>
<sodipodi:namedview
inkscape:window-maximized="0"
inkscape:window-y="0"
inkscape:window-x="130"
inkscape:window-height="1040"
inkscape:window-width="1274"
height="50mm"
units="mm"
showgrid="false"
inkscape:document-rotation="0"
inkscape:current-layer="layer1"
inkscape:document-units="mm"
inkscape:cy="344.49897"
inkscape:cx="468.64708"
inkscape:zoom="0.7"
inkscape:pageshadow="2"
inkscape:pageopacity="0.0"
borderopacity="1.0"
bordercolor="#666666"
pagecolor="#ffffff"
id="base" />
<metadata
id="metadata54">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title></dc:title>
</cc:Work>
</rdf:RDF>
</metadata>
<g
id="layer1"
inkscape:groupmode="layer"
inkscape:label="Layer 1">
<g
id="g424"
transform="matrix(0.35277777,0,0,-0.35277777,63.946468,10.194047)">
<path
d="m 0,0 v -8.093 h 12.287 v -3.94 H 0 V -24.067 H -4.534 V 3.898 H 15.677 V 0 Z"
style="fill:#00e396;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path426" />
</g>
<g
transform="matrix(0.35277777,0,0,-0.35277777,-315.43002,107.34005)"
id="g428">
<g
id="g430"
clip-path="url(#clipPath434)">
<g
id="g436"
transform="translate(1112.874,278.2981)">
<path
d="M 0,0 C 1.822,-0.932 3.354,-2.359 4.597,-4.28 L 1.165,-7.373 c -0.791,1.695 -1.779,2.924 -2.966,3.686 -1.186,0.763 -2.768,1.145 -4.745,1.145 -1.949,0 -3.461,-0.389 -4.534,-1.166 -1.074,-0.777 -1.61,-1.772 -1.61,-2.987 0,-1.13 0.523,-2.027 1.568,-2.69 1.045,-0.664 2.909,-1.236 5.593,-1.716 2.514,-0.452 4.512,-1.024 5.995,-1.716 1.483,-0.693 2.564,-1.554 3.242,-2.585 0.677,-1.031 1.016,-2.309 1.016,-3.834 0,-1.639 -0.466,-3.079 -1.398,-4.322 -0.932,-1.243 -2.239,-2.197 -3.919,-2.86 -1.681,-0.664 -3.623,-0.996 -5.826,-0.996 -5.678,0 -9.689,1.892 -12.033,5.678 l 3.178,3.178 c 0.903,-1.695 2.068,-2.939 3.495,-3.729 1.426,-0.791 3.199,-1.186 5.318,-1.186 2.005,0 3.58,0.345 4.724,1.038 1.144,0.692 1.716,1.674 1.716,2.945 0,1.017 -0.516,1.835 -1.547,2.457 -1.031,0.621 -2.832,1.172 -5.402,1.653 -2.571,0.479 -4.618,1.073 -6.143,1.779 -1.526,0.706 -2.635,1.582 -3.326,2.627 -0.693,1.045 -1.039,2.316 -1.039,3.813 0,1.582 0.438,3.023 1.314,4.322 0.875,1.299 2.14,2.33 3.792,3.093 1.653,0.763 3.58,1.144 5.783,1.144 C -4.018,1.398 -1.822,0.932 0,0"
style="fill:#00e396;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path438" />
</g>
<g
id="g440"
transform="translate(993.0239,277.5454)">
<path
d="m 0,0 c 2.054,-1.831 3.083,-4.465 3.083,-7.902 v -17.935 h -4.484 v 16.366 c 0,2.914 -0.626,5.024 -1.877,6.332 -1.253,1.308 -2.924,1.962 -5.016,1.962 -1.495,0 -2.896,-0.327 -4.204,-0.981 -1.308,-0.654 -2.381,-1.719 -3.222,-3.194 -0.841,-1.477 -1.261,-3.335 -1.261,-5.576 v -14.909 h -4.484 V 1.328 l 4.086,-1.674 0.118,-1.84 c 0.933,1.681 2.222,2.923 3.867,3.727 1.643,0.803 3.493,1.205 5.548,1.205 C -4.671,2.746 -2.055,1.83 0,0"
style="fill:#000033;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path442" />
</g>
<g
id="g444"
transform="translate(1027.9968,264.0386)">
<path
d="m 0,0 h -21.128 c 0.261,-2.84 1.205,-5.044 2.83,-6.613 1.625,-1.57 3.727,-2.355 6.305,-2.355 2.054,0 3.763,0.356 5.128,1.065 1.363,0.71 2.288,1.738 2.774,3.083 l 3.755,-1.961 c -1.121,-1.981 -2.616,-3.495 -4.484,-4.54 -1.868,-1.046 -4.259,-1.569 -7.173,-1.569 -4.223,0 -7.538,1.289 -9.948,3.867 -2.41,2.578 -3.615,6.146 -3.615,10.704 0,4.558 1.149,8.127 3.447,10.705 2.298,2.578 5.557,3.867 9.779,3.867 2.615,0 4.876,-0.58 6.782,-1.738 1.905,-1.158 3.343,-2.728 4.315,-4.707 C -0.262,7.827 0.224,5.605 0.224,3.139 0.224,2.092 0.149,1.046 0,0 m -18.298,10.144 c -1.513,-1.457 -2.438,-3.512 -2.775,-6.165 h 16.982 c -0.3,2.615 -1.159,4.661 -2.578,6.137 -1.42,1.476 -3.307,2.214 -5.661,2.214 -2.466,0 -4.455,-0.728 -5.968,-2.186"
style="fill:#000033;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path446" />
</g>
<g
id="g448"
transform="translate(1057.8818,276.4246)">
<path
d="m 0,0 c 2.41,-2.578 3.615,-6.147 3.615,-10.705 0,-4.558 -1.205,-8.126 -3.615,-10.704 -2.41,-2.578 -5.726,-3.867 -9.948,-3.867 -4.222,0 -7.537,1.289 -9.947,3.867 -2.41,2.578 -3.615,6.146 -3.615,10.704 0,4.558 1.205,8.127 3.615,10.705 2.41,2.578 5.725,3.867 9.947,3.867 C -5.726,3.867 -2.41,2.578 0,0 m -16.617,-2.858 c -1.607,-1.906 -2.41,-4.522 -2.41,-7.847 0,-3.326 0.803,-5.94 2.41,-7.846 1.607,-1.905 3.83,-2.858 6.669,-2.858 2.839,0 5.063,0.953 6.67,2.858 1.606,1.906 2.41,4.52 2.41,7.846 0,3.325 -0.804,5.941 -2.41,7.847 C -4.885,-0.953 -7.109,0 -9.948,0 c -2.839,0 -5.062,-0.953 -6.669,-2.858"
style="fill:#000033;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path450" />
</g>
</g>
</g>
<g
id="g452"
transform="matrix(0.35277777,0,0,-0.35277777,5.8329581,6.5590171)">
<path
d="m 0,0 0.001,-38.946 25.286,-9.076 V -8.753 L 52.626,1.321 27.815,10.207 Z"
style="fill:#00e599;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path454" />
</g>
<g
id="g456"
transform="matrix(0.35277777,0,0,-0.35277777,15.479008,10.041927)">
<path
d="M 0,0 V -21.306 L 25.293,-30.364 25.282,9.347 Z"
style="fill:#00b091;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path458" />
</g>
</g>
<?xml version="1.0" encoding="utf-8"?>
<!-- Generator: Adobe Illustrator 25.0.1, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
<svg version="1.1" id="Слой_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
viewBox="0 0 184.2 51.8" style="enable-background:new 0 0 184.2 51.8;" xml:space="preserve">
<style type="text/css">
.st0{display:none;}
.st1{display:inline;}
.st2{fill:#01E397;}
.st3{display:inline;fill:#010032;}
.st4{display:inline;fill:#00E599;}
.st5{display:inline;fill:#00AF92;}
.st6{fill:#00C3E5;}
</style>
<g id="Layer_2">
<g id="Layer_1-2" class="st0">
<g class="st1">
<path class="st2" d="M146.6,18.3v7.2h10.9V29h-10.9v10.7h-4V14.8h18v3.5H146.6z"/>
<path class="st2" d="M180,15.7c1.7,0.9,3,2.2,4,3.8l-3,2.7c-0.6-1.3-1.5-2.4-2.6-3.3c-1.3-0.7-2.8-1-4.3-1
c-1.4-0.1-2.8,0.3-4,1.1c-0.9,0.5-1.5,1.5-1.4,2.6c0,1,0.5,1.9,1.4,2.4c1.5,0.8,3.2,1.3,4.9,1.5c1.9,0.3,3.7,0.8,5.4,1.6
c1.2,0.5,2.2,1.3,2.9,2.3c0.6,1,1,2.2,0.9,3.4c0,1.4-0.5,2.7-1.3,3.8c-0.9,1.2-2.1,2.1-3.5,2.6c-1.7,0.6-3.4,0.9-5.2,0.8
c-5,0-8.6-1.6-10.7-5l2.9-2.8c0.7,1.4,1.8,2.5,3.1,3.3c1.5,0.7,3.1,1.1,4.7,1c1.5,0.1,2.9-0.2,4.2-0.9c0.9-0.5,1.5-1.5,1.5-2.6
c0-0.9-0.5-1.8-1.3-2.2c-1.5-0.7-3.1-1.2-4.8-1.5c-1.9-0.3-3.7-0.8-5.5-1.5c-1.2-0.5-2.2-1.4-3-2.4c-0.6-1-1-2.2-0.9-3.4
c0-1.4,0.4-2.7,1.2-3.8c0.8-1.2,2-2.2,3.3-2.8c1.6-0.7,3.4-1.1,5.2-1C176.1,14.3,178.2,14.8,180,15.7z"/>
</g>
<path class="st3" d="M73.3,16.3c1.9,1.9,2.9,4.5,2.7,7.1v15.9h-4V24.8c0-2.6-0.5-4.5-1.6-5.7c-1.2-1.2-2.8-1.8-4.5-1.7
c-1.3,0-2.5,0.3-3.7,0.8c-1.2,0.7-2.2,1.7-2.9,2.9c-0.8,1.5-1.1,3.2-1.1,4.9v13.3h-4V15.1l3.6,1.5v1.7c0.8-1.5,2.1-2.6,3.6-3.3
c1.5-0.8,3.2-1.2,4.9-1.1C68.9,13.8,71.3,14.7,73.3,16.3z"/>
<path class="st3" d="M104.4,28.3H85.6c0.1,2.2,1,4.3,2.5,5.9c1.5,1.4,3.5,2.2,5.6,2.1c1.6,0.1,3.2-0.2,4.6-0.9
c1.1-0.6,2-1.6,2.5-2.8l3.3,1.8c-0.9,1.7-2.3,3.1-4,4c-2,1-4.2,1.5-6.4,1.4c-3.7,0-6.7-1.1-8.8-3.4s-3.2-5.5-3.2-9.6s1-7.2,3-9.5
s5-3.4,8.7-3.4c2.1-0.1,4.2,0.5,6.1,1.5c1.6,1,3,2.5,3.8,4.2c0.9,1.8,1.3,3.9,1.3,5.9C104.6,26.4,104.6,27.4,104.4,28.3z
M88.1,19.3c-1.4,1.5-2.2,3.4-2.4,5.5h15.1c-0.2-2-1-3.9-2.3-5.5c-1.4-1.3-3.2-2-5.1-1.9C91.5,17.3,89.6,18,88.1,19.3z"/>
<path class="st3" d="M131,17.3c2.2,2.3,3.2,5.5,3.2,9.5s-1,7.3-3.2,9.6s-5.1,3.4-8.8,3.4s-6.7-1.1-8.9-3.4s-3.2-5.5-3.2-9.6
s1.1-7.2,3.2-9.5s5.1-3.4,8.9-3.4S128.9,15,131,17.3z M116.2,19.9c-1.5,2-2.2,4.4-2.1,6.9c-0.2,2.5,0.6,5,2.1,7
c1.5,1.7,3.7,2.7,6,2.6c2.3,0.1,4.4-0.9,5.9-2.6c1.5-2,2.3-4.5,2.1-7c0.1-2.5-0.6-4.9-2.1-6.9c-1.5-1.7-3.6-2.7-5.9-2.6
C119.9,17.2,117.7,18.2,116.2,19.9z"/>
<polygon class="st4" points="0,9.1 0,43.7 22.5,51.8 22.5,16.9 46.8,7.9 24.8,0 "/>
<polygon class="st5" points="24.3,17.9 24.3,36.8 46.8,44.9 46.8,9.6 "/>
</g>
<g>
<g>
<path class="st6" d="M41.6,17.5H28.2v6.9h10.4v3.3H28.2v10.2h-3.9V14.2h17.2V17.5z"/>
<path class="st6" d="M45.8,37.9v-18h3.3l0.4,3.2c0.5-1.2,1.2-2.1,2.1-2.7c0.9-0.6,2.1-0.9,3.5-0.9c0.4,0,0.7,0,1.1,0.1
c0.4,0.1,0.7,0.2,0.9,0.3l-0.5,3.4c-0.3-0.1-0.6-0.2-0.9-0.2C55.4,23,54.9,23,54.4,23c-0.7,0-1.5,0.2-2.2,0.6
c-0.7,0.4-1.3,1-1.8,1.8s-0.7,1.8-0.7,3v9.5H45.8z"/>
<path class="st6" d="M68.6,19.6c1.8,0,3.3,0.4,4.6,1.1c1.3,0.7,2.4,1.8,3.1,3.2s1.1,3.1,1.1,5c0,1.9-0.4,3.6-1.1,5
c-0.8,1.4-1.8,2.5-3.1,3.2c-1.3,0.7-2.9,1.1-4.6,1.1s-3.3-0.4-4.6-1.1c-1.3-0.7-2.4-1.8-3.2-3.2c-0.8-1.4-1.2-3.1-1.2-5
c0-1.9,0.4-3.6,1.2-5s1.8-2.5,3.2-3.2C65.3,19.9,66.8,19.6,68.6,19.6z M68.6,22.6c-1.1,0-2,0.2-2.8,0.7c-0.8,0.5-1.3,1.2-1.7,2.1
s-0.6,2.1-0.6,3.5c0,1.3,0.2,2.5,0.6,3.4s1,1.7,1.7,2.2s1.7,0.7,2.8,0.7c1.1,0,2-0.2,2.7-0.7c0.7-0.5,1.3-1.2,1.7-2.2
s0.6-2.1,0.6-3.4c0-1.4-0.2-2.5-0.6-3.5s-1-1.6-1.7-2.1C70.6,22.8,69.6,22.6,68.6,22.6z"/>
<path class="st6" d="M89.2,38.3c-1.8,0-3.4-0.3-4.9-1c-1.5-0.7-2.7-1.7-3.5-3l2.7-2.3c0.5,1,1.3,1.8,2.3,2.4
c1,0.6,2.2,0.9,3.6,0.9c1.1,0,2-0.2,2.6-0.6c0.6-0.4,1-0.9,1-1.6c0-0.5-0.2-0.9-0.5-1.2s-0.9-0.6-1.7-0.8l-3.8-0.8
c-1.9-0.4-3.3-1-4.1-1.9c-0.8-0.9-1.2-1.9-1.2-3.3c0-1,0.3-1.9,0.9-2.7c0.6-0.8,1.4-1.5,2.5-2s2.5-0.8,4-0.8c1.8,0,3.3,0.3,4.6,1
c1.3,0.6,2.2,1.5,2.9,2.7l-2.7,2.2c-0.5-1-1.1-1.7-2-2.1c-0.9-0.5-1.8-0.7-2.8-0.7c-0.8,0-1.4,0.1-2,0.3c-0.6,0.2-1,0.5-1.3,0.8
c-0.3,0.3-0.4,0.7-0.4,1.2c0,0.5,0.2,0.9,0.5,1.3s1,0.6,1.9,0.8l4.1,0.9c1.7,0.3,2.9,0.9,3.7,1.7c0.7,0.8,1.1,1.8,1.1,2.9
c0,1.2-0.3,2.2-0.9,3c-0.6,0.9-1.5,1.6-2.6,2C92.1,38.1,90.7,38.3,89.2,38.3z"/>
<path class="st6" d="M112.8,19.9v3H99.3v-3H112.8z M106.6,14.6v17.9c0,0.9,0.2,1.5,0.7,1.9c0.5,0.4,1.1,0.6,1.9,0.6
c0.6,0,1.2-0.1,1.7-0.3c0.5-0.2,0.9-0.5,1.3-0.8l0.9,2.8c-0.6,0.5-1.2,0.9-2,1.1c-0.8,0.3-1.7,0.4-2.7,0.4c-1,0-2-0.2-2.8-0.5
s-1.5-0.9-2-1.6c-0.5-0.8-0.7-1.7-0.8-3V15.7L106.6,14.6z"/>
<path d="M137.9,17.5h-13.3v6.9h10.4v3.3h-10.4v10.2h-3.9V14.2h17.2V17.5z"/>
<path d="M150.9,13.8c2.1,0,4,0.4,5.5,1.2c1.6,0.8,2.9,2,4,3.5l-2.6,2.5c-0.9-1.4-1.9-2.4-3.1-3c-1.1-0.6-2.5-0.9-4-0.9
c-1.2,0-2.1,0.2-2.8,0.5c-0.7,0.3-1.3,0.7-1.6,1.2c-0.3,0.5-0.5,1.1-0.5,1.7c0,0.7,0.3,1.4,0.8,1.9c0.5,0.6,1.5,1,2.9,1.3
l4.8,1.1c2.3,0.5,3.9,1.3,4.9,2.3c1,1,1.4,2.3,1.4,3.9c0,1.5-0.4,2.7-1.2,3.8c-0.8,1.1-1.9,1.9-3.3,2.5s-3.1,0.9-5,0.9
c-1.7,0-3.2-0.2-4.5-0.6c-1.3-0.4-2.5-1-3.5-1.8c-1-0.7-1.8-1.6-2.5-2.6l2.7-2.7c0.5,0.8,1.1,1.6,1.9,2.2
c0.8,0.7,1.7,1.2,2.7,1.5c1,0.4,2.2,0.5,3.4,0.5c1.1,0,2.1-0.1,2.9-0.4c0.8-0.3,1.4-0.7,1.8-1.2c0.4-0.5,0.6-1.1,0.6-1.9
c0-0.7-0.2-1.3-0.7-1.8c-0.5-0.5-1.3-0.9-2.6-1.2l-5.2-1.2c-1.4-0.3-2.6-0.8-3.6-1.3c-0.9-0.6-1.6-1.3-2.1-2.1s-0.7-1.8-0.7-2.8
c0-1.3,0.4-2.6,1.1-3.7c0.7-1.1,1.8-2,3.2-2.6C147.3,14.1,148.9,13.8,150.9,13.8z"/>
</g>
</g>
</g>
</svg>

Before

Width:  |  Height:  |  Size: 6.5 KiB

After

Width:  |  Height:  |  Size: 5.5 KiB

View file

@ -1,21 +0,0 @@
name: DCO check
on:
pull_request:
branches:
- master
jobs:
commits_check_job:
runs-on: ubuntu-latest
name: Commits Check
steps:
- name: Get PR Commits
id: 'get-pr-commits'
uses: tim-actions/get-pr-commits@master
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: DCO Check
uses: tim-actions/dco@master
with:
commits: ${{ steps.get-pr-commits.outputs.commits }}

View file

@ -1,57 +0,0 @@
name: neofs-node tests
on:
push:
branches:
- master
paths-ignore:
- '*.md'
pull_request:
branches:
- master
paths-ignore:
- '*.md'
jobs:
test:
runs-on: ubuntu-20.04
strategy:
matrix:
go: [ '1.17.x', '1.18.x' ]
steps:
- name: Setup go
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go }}
- name: Check out code
uses: actions/checkout@v2
- name: Cache go mod
uses: actions/cache@v2
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-go-${{ matrix.go }}-
- name: Run go test
run: go test -coverprofile=coverage.txt -covermode=atomic ./...
- name: Codecov
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
run: bash <(curl -s https://codecov.io/bash)
lint:
runs-on: ubuntu-20.04
steps:
- name: Check out code
uses: actions/checkout@v2
- name: golangci-lint
uses: golangci/golangci-lint-action@v2
with:
version: v1.45.2
args: --timeout=5m
only-new-issues: true

49
.gitignore vendored
View file

@ -1,12 +1,51 @@
# IDE
.idea
bin
release
.vscode
# Vendoring
vendor
# tempfiles
.DS_Store
*~
.cache
temp
tmp
# binary
bin/
release/
# coverage
coverage.txt
coverage.html
# testing
cmd/test
/plugins/
/vendor/
testfile
# misc
.neofs-cli.yml
.cache
# debhelpers
debian/*debhelper*
# logfiles
debian/*.log
# .substvars
debian/*.substvars
# .bash-completion
debian/*.bash-completion
# Install folders and files
debian/frostfs-cli/
debian/frostfs-ir/
debian/files
debian/frostfs-storage/
debian/changelog
man/
debs/

11
.gitlint Normal file
View file

@ -0,0 +1,11 @@
[general]
fail-without-commits=True
regex-style-search=True
contrib=CC1
[title-match-regex]
regex=^\[\#[0-9Xx]+\]\s
[ignore-by-title]
regex=^Release(.*)
ignore=title-match-regex

View file

@ -4,7 +4,7 @@
# options for analysis running
run:
# timeout for analysis, e.g. 30s, 5m, default is 1m
timeout: 5m
timeout: 20m
# include test files or not, default is true
tests: false
@ -24,6 +24,19 @@ linters-settings:
govet:
# report about shadowed variables
check-shadowing: false
staticcheck:
checks: ["all", "-SA1019"] # TODO Enable SA1019 after deprecated warning are fixed.
funlen:
lines: 80 # default 60
statements: 60 # default 40
gocognit:
min-complexity: 40 # default 30
importas:
no-unaliased: true
no-extra-aliases: false
alias:
pkg: git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object
alias: objectSDK
linters:
enable:
@ -34,16 +47,27 @@ linters:
# some default golangci-lint linters
- errcheck
- gosimple
- godot
- ineffassign
- staticcheck
- typecheck
- unused
# extra linters
- bidichk
- durationcheck
- exhaustive
- exportloopref
- gofmt
- whitespace
- goimports
- unused
- misspell
- predeclared
- reassign
- whitespace
- containedctx
- funlen
- gocognit
- contextcheck
- importas
disable-all: true
fast: false

51
.pre-commit-config.yaml Normal file
View file

@ -0,0 +1,51 @@
ci:
autofix_prs: false
repos:
- repo: https://github.com/jorisroovers/gitlint
rev: v0.19.1
hooks:
- id: gitlint
stages: [commit-msg]
- id: gitlint-ci
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: check-added-large-files
- id: check-case-conflict
- id: check-executables-have-shebangs
- id: check-shebang-scripts-are-executable
- id: check-merge-conflict
- id: check-json
- id: check-xml
- id: check-yaml
- id: trailing-whitespace
args: [--markdown-linebreak-ext=md]
- id: end-of-file-fixer
exclude: ".key$"
- repo: https://github.com/shellcheck-py/shellcheck-py
rev: v0.9.0.2
hooks:
- id: shellcheck
- repo: https://github.com/golangci/golangci-lint
rev: v1.52.2
hooks:
- id: golangci-lint
- repo: local
hooks:
- id: go-unit-tests
name: go unit tests
entry: make test
pass_filenames: false
types: [go]
language: system
- repo: https://github.com/TekWizely/pre-commit-golang
rev: v1.0.0-rc.1
hooks:
- id: go-staticcheck-repo-mod
- id: go-mod-tidy

View file

@ -0,0 +1,11 @@
pipeline:
# Kludge for non-root containers under WoodPecker
fix-ownership:
image: alpine:latest
commands: chown -R 1234:1234 .
pre-commit:
image: git.frostfs.info/truecloudlab/frostfs-ci:v0.36
commands:
- export HOME="$(getent passwd $(id -u) | cut '-d:' -f6)"
- pre-commit run --hook-stage manual

File diff suppressed because it is too large Load diff

View file

@ -3,8 +3,8 @@
First, thank you for contributing! We love and encourage pull requests from
everyone. Please follow the guidelines:
- Check the open [issues](https://github.com/nspcc-dev/neofs-node/issues) and
[pull requests](https://github.com/nspcc-dev/neofs-node/pulls) for existing
- Check the open [issues](https://github.com/TrueCloudLab/frostfs-node/issues) and
[pull requests](https://github.com/TrueCloudLab/frostfs-node/pulls) for existing
discussions.
- Open an issue first, to discuss a new feature or enhancement.
@ -23,23 +23,23 @@ everyone. Please follow the guidelines:
## Development Workflow
Start by forking the `neofs-node` repository, make changes in a branch and then
Start by forking the `frostfs-node` repository, make changes in a branch and then
send a pull request. We encourage pull requests to discuss code changes. Here
are the steps in details:
### Set up your GitHub Repository
Fork [NeoFS node upstream](https://github.com/nspcc-dev/neofs-node/fork) source
Fork [FrostFS node upstream](https://github.com/TrueCloudLab/frostfs-node/fork) source
repository to your own personal repository. Copy the URL of your fork (you will
need it for the `git clone` command below).
```sh
$ git clone https://github.com/nspcc-dev/neofs-node
$ git clone https://github.com/TrueCloudLab/frostfs-node
```
### Set up git remote as ``upstream``
```sh
$ cd neofs-node
$ git remote add upstream https://github.com/nspcc-dev/neofs-node
$ cd frostfs-node
$ git remote add upstream https://github.com/TrueCloudLab/frostfs-node
$ git fetch upstream
$ git merge upstream/master
...
@ -79,7 +79,7 @@ Description
```
```
$ git commit -am '[#123] Add some feature'
$ git commit -sam '[#123] Add some feature'
```
### Push to the branch
@ -106,7 +106,8 @@ contributors".
To sign your work, just add a line like this at the end of your commit message:
```
Signed-off-by: Samii Sakisaka <samii@nspcc.ru>
Signed-off-by: Samii Sakisaka <samii@ivunojikan.co.jp>
```
This can easily be done with the `--signoff` option to `git commit`.

View file

@ -1,5 +1,7 @@
# Credits
FrostFS continues the development of NeoFS.
Initial NeoFS research and development (2018-2020) was done by
[NeoSPCC](https://nspcc.ru) team.
@ -8,7 +10,7 @@ In alphabetical order:
- Alexey Vanin
- Anastasia Prasolova
- Anatoly Bogatyrev
- Evgeny Kulikov
- Evgeny Kulikov
- Evgeny Stratonikov
- Leonard Liubich
- Sergei Liubich

96
Makefile Normal file → Executable file
View file

@ -2,14 +2,13 @@
SHELL = bash
REPO ?= $(shell go list -m)
VERSION ?= $(shell git describe --tags --dirty --always)
BUILD ?= $(shell date -u --iso=seconds)
DEBUG ?= false
VERSION ?= $(shell git describe --tags --dirty --match "v*" --always --abbrev=8 2>/dev/null || cat VERSION 2>/dev/null || echo "develop")
HUB_IMAGE ?= nspccdev/neofs
HUB_IMAGE ?= truecloudlab/frostfs
HUB_TAG ?= "$(shell echo ${VERSION} | sed 's/^v//')"
GO_VERSION ?= 1.17
GO_VERSION ?= 1.19
LINT_VERSION ?= 1.52.2
ARCH = amd64
BIN = bin
@ -17,24 +16,31 @@ RELEASE = release
DIRS = $(BIN) $(RELEASE)
# List of binaries to build.
CMDS = $(notdir $(basename $(wildcard cmd/*)))
CMDS = $(notdir $(basename $(wildcard cmd/frostfs-*)))
BINS = $(addprefix $(BIN)/, $(CMDS))
.PHONY: help all images dep clean fmts fmt imports test lint docker/lint prepare-release
# .deb package versioning
OS_RELEASE = $(shell lsb_release -cs)
PKG_VERSION ?= $(shell echo $(VERSION) | sed "s/^v//" | \
sed -E "s/(.*)-(g[a-fA-F0-9]{6,8})(.*)/\1\3~\2/" | \
sed "s/-/~/")-${OS_RELEASE}
.PHONY: help all images dep clean fmts fmt imports test lint docker/lint
prepare-release debpackage pre-commit unpre-commit
# To build a specific binary, use it's name prefix with bin/ as a target
# For example `make bin/neofs-node` will build only storage node binary
# For example `make bin/frostfs-node` will build only storage node binary
# Just `make` will build all possible binaries
all: $(DIRS) $(BINS)
# help target
include help.mk
$(BINS): $(DIRS) dep
@echo "⇒ Build $@"
CGO_ENABLED=0 \
GO111MODULE=on \
go build -v -trimpath \
-ldflags "-X $(REPO)/misc.Version=$(VERSION) \
-X $(REPO)/misc.Build=$(BUILD) \
-X $(REPO)/misc.Debug=$(DEBUG)" \
-ldflags "-X $(REPO)/misc.Version=$(VERSION)" \
-o $@ ./cmd/$(notdir $@)
$(DIRS):
@ -44,7 +50,7 @@ $(DIRS):
# Prepare binaries and archives for release
.ONESHELL:
prepare-release: docker/all
@for file in `ls -1 $(BIN)/neofs-*`; do
@for file in `ls -1 $(BIN)/frostfs-*`; do
cp $$file $(RELEASE)/`basename $$file`-$(ARCH)
strip $(RELEASE)/`basename $$file`-$(ARCH)
tar -czf $(RELEASE)/`basename $$file`-$(ARCH).tar.gz $(RELEASE)/`basename $$file`-$(ARCH)
@ -54,32 +60,33 @@ prepare-release: docker/all
dep:
@printf "⇒ Download requirements: "
CGO_ENABLED=0 \
GO111MODULE=on \
go mod download && echo OK
@printf "⇒ Tidy requirements : "
CGO_ENABLED=0 \
GO111MODULE=on \
go mod tidy -v && echo OK
# Regenerate proto files:
protoc:
@GOPRIVATE=github.com/nspcc-dev go mod vendor
@GOPRIVATE=github.com/TrueCloudLab go mod vendor
# Install specific version for protobuf lib
@go list -f '{{.Path}}/...@{{.Version}}' -m github.com/golang/protobuf | xargs go install -v
@GOBIN=$(abspath $(BIN)) go install -mod=mod -v git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/util/protogen
# Protoc generate
@for f in `find . -type f -name '*.proto' -not -path './vendor/*'`; do \
echo "⇒ Processing $$f "; \
protoc \
--proto_path=.:./vendor:/usr/local/include \
--plugin=protoc-gen-go-frostfs=$(BIN)/protogen \
--go-frostfs_out=. --go-frostfs_opt=paths=source_relative \
--go_out=. --go_opt=paths=source_relative \
--go-grpc_opt=require_unimplemented_servers=false \
--go-grpc_out=. --go-grpc_opt=paths=source_relative $$f; \
done
rm -rf vendor
# Build NeoFS component's docker image
# Build FrostFS component's docker image
image-%:
@echo "⇒ Build NeoFS $* docker image "
@echo "⇒ Build FrostFS $* docker image "
@docker build \
--build-arg REPO=$(REPO) \
--build-arg VERSION=$(VERSION) \
@ -88,7 +95,7 @@ image-%:
-t $(HUB_IMAGE)-$*:$(HUB_TAG) .
# Build all Docker images
images: image-storage image-ir image-cli image-adm image-storage-testnet
images: image-storage image-ir image-cli image-adm
# Build dirty local Docker images
dirty-images: image-dirty-storage image-dirty-ir image-dirty-cli image-dirty-adm
@ -109,45 +116,68 @@ fmts: fmt imports
# Reformat code
fmt:
@echo "⇒ Processing gofmt check"
@GO111MODULE=on gofmt -s -w cmd/ pkg/ misc/
@gofmt -s -w cmd/ pkg/ misc/
# Reformat imports
imports:
@echo "⇒ Processing goimports check"
@GO111MODULE=on goimports -w cmd/ pkg/ misc/
@goimports -w cmd/ pkg/ misc/
# Run Unit Test with go test
test:
@echo "⇒ Running go test"
@GO111MODULE=on go test ./...
@go test ./... -count=1
pre-commit-run:
@pre-commit run -a --hook-stage manual
# Run linters
lint:
@golangci-lint --timeout=5m run
# Install staticcheck
staticcheck-install:
@go install honnef.co/go/tools/cmd/staticcheck@latest
# Run staticcheck
staticcheck-run:
@staticcheck ./...
# Run linters in Docker
docker/lint:
docker run --rm -t \
-v `pwd`:/src \
-u `stat -c "%u:%g" .` \
--env HOME=/src \
golangci/golangci-lint:v1.42.1 bash -c 'cd /src/ && make lint'
golangci/golangci-lint:v$(LINT_VERSION) bash -c 'cd /src/ && make lint'
# Activate pre-commit hooks
pre-commit:
pre-commit install -t pre-commit -t commit-msg
# Deactivate pre-commit hooks
unpre-commit:
pre-commit uninstall -t pre-commit -t commit-msg
# Print version
version:
@echo $(VERSION)
# Show this help prompt
help:
@echo ' Usage:'
@echo ''
@echo ' make <target>'
@echo ''
@echo ' Targets:'
@echo ''
@awk '/^#/{ comment = substr($$0,3) } comment && /^[a-zA-Z][a-zA-Z0-9_-]+ ?:/{ print " ", $$1, comment }' $(MAKEFILE_LIST) | column -t -s ':' | grep -v 'IGNORE' | sort -u
# Delete built artifacts
clean:
rm -rf vendor
rm -rf .cache
rm -rf $(BIN)
rm -rf $(RELEASE)
# Package for Debian
debpackage:
dch -b --package frostfs-node \
--controlmaint \
--newversion $(PKG_VERSION) \
--distribution $(OS_RELEASE) \
"Please see CHANGELOG.md for code changes for $(VERSION)"
dpkg-buildpackage --no-sign -b
debclean:
dh clean

View file

@ -1,50 +1,80 @@
<p align="center">
<img src="./.github/logo.svg" width="500px" alt="NeoFS">
<img src="./.github/logo.svg" width="500px" alt="FrostFS">
</p>
<p align="center">
<a href="https://fs.neo.org">NeoFS</a> is a decentralized distributed object storage integrated with the <a href="https://neo.org">NEO Blockchain</a>.
<a href="https://frostfs.info">FrostFS</a> is a decentralized distributed object storage integrated with the <a href="https://neo.org">NEO Blockchain</a>.
</p>
---
[![Report](https://goreportcard.com/badge/github.com/nspcc-dev/neofs-node)](https://goreportcard.com/report/github.com/nspcc-dev/neofs-node)
![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/nspcc-dev/neofs-node?sort=semver)
![License](https://img.shields.io/github/license/nspcc-dev/neofs-node.svg?style=popout)
[![Report](https://goreportcard.com/badge/github.com/TrueCloudLab/frostfs-node)](https://goreportcard.com/report/github.com/TrueCloudLab/frostfs-node)
![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/TrueCloudLab/frostfs-node?sort=semver)
![License](https://img.shields.io/github/license/TrueCloudLab/frostfs-node.svg?style=popout)
# Overview
NeoFS Nodes are organized in a peer-to-peer network that takes care of storing
FrostFS Nodes are organized in a peer-to-peer network that takes care of storing
and distributing user's data. Any Neo user may participate in the network and
get paid for providing storage resources to other users or store his data in
NeoFS and pay a competitive price for it.
get paid for providing storage resources to other users or store their data in
FrostFS and pay a competitive price for it.
Users can reliably store object data in the NeoFS network and have a transparent
Users can reliably store object data in the FrostFS network and have a transparent
data placement process due to a decentralized architecture and flexible storage
policies. Each node is responsible for executing the storage policies that the
users select for geographical location, reliability level, number of nodes, type
of disks, capacity, etc. Thus, NeoFS gives full control over data to users.
of disks, capacity, etc. Thus, FrostFS gives full control over data to users.
Deep [Neo Blockchain](https://neo.org) integration allows NeoFS to be used by
Deep [Neo Blockchain](https://neo.org) integration allows FrostFS to be used by
dApps directly from
[NeoVM](https://docs.neo.org/docs/en-us/basic/technology/neovm.html) on the
[Smart Contract](https://docs.neo.org/docs/en-us/intro/glossary.html)
code level. This way dApps are not limited to on-chain storage and can
manipulate large amounts of data without paying a prohibitive price.
NeoFS has a native [gRPC API](https://github.com/nspcc-dev/neofs-api) and has
FrostFS has a native [gRPC API](https://git.frostfs.info/TrueCloudLab/frostfs-api) and has
protocol gateways for popular protocols such as [AWS
S3](https://github.com/nspcc-dev/neofs-s3-gw),
[HTTP](https://github.com/nspcc-dev/neofs-http-gw),
S3](https://github.com/TrueCloudLab/frostfs-s3-gw),
[HTTP](https://github.com/TrueCloudLab/frostfs-http-gw),
[FUSE](https://wikipedia.org/wiki/Filesystem_in_Userspace) and
[sFTP](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol) allowing
developers to integrate applications without rewriting their code.
# Supported platforms
For now we only support GNU/Linux on amd64 CPUs with AVX/AVX2 instructions. More
Now, we only support GNU/Linux on amd64 CPUs with AVX/AVX2 instructions. More
platforms will be officially supported after release `1.0`.
The latest version of neofs-node works with neofs-contract
[v0.13.0](https://github.com/nspcc-dev/neofs-contract/releases/tag/v0.13.0).
The latest version of frostfs-node works with frostfs-contract
[v0.16.0](https://github.com/TrueCloudLab/frostfs-contract/releases/tag/v0.16.0).
# Building
To make all binaries you need Go 1.19+ and `make`:
```
make all
```
The resulting binaries will appear in `bin/` folder.
To make a specific binary use:
```
make bin/frostfs-<name>
```
See the list of all available commands in the `cmd` folder.
## Building with Docker
Building can also be performed in a container:
```
make docker/all # build all binaries
make docker/bin/frostfs-<name> # build a specific binary
```
## Docker images
To make docker images suitable for use in [frostfs-dev-env](https://github.com/TrueCloudLab/frostfs-dev-env/) use:
```
make images
```
# Contributing
@ -56,7 +86,7 @@ the feature/topic you are going to implement.
# Credits
NeoFS is maintained by [NeoSPCC](https://nspcc.ru) with the help and
FrostFS is maintained by [True Cloud Lab](https://github.com/TrueCloudLab/) with the help and
contributions from community members.
Please see [CREDITS](CREDITS.md) for details.

1
VERSION Normal file
View file

@ -0,0 +1 @@
v0.36.0

View file

@ -1,42 +1,41 @@
# NeoFS Admin Tool
# FrostFS Admin Tool
## Overview
Admin tool provides an easier way to deploy and maintain private installation
of NeoFS. Private installation provides a set of N3 consensus nodes, NeoFS
Alphabet, and Storage nodes. Admin tool generates consensus keys, initializes
side chain, and provides functions to update the network and register new
of FrostFS. Private installation provides a set of N3 consensus nodes, FrostFS
Alphabet, and Storage nodes. Admin tool generates consensus keys, initializes
the sidechain, and provides functions to update the network and register new
Storage nodes.
## Build
To build binary locally, use `make bin/neofs-adm` command.
To build binary locally, use `make bin/frostfs-adm` command.
For clean build inside a docker container use `make docker/bin/neofs-adm`.
For clean build inside a docker container, use `make docker/bin/frostfs-adm`.
Build docker image with `make image-adm`.
At NeoFS private install deployment, neofs-adm requires compiled NeoFS
contracts. Find them in the latest release of
[neofs-contract repository](https://github.com/nspcc-dev/neofs-contract/releases).
At FrostFS private install deployment, frostfs-adm requires compiled FrostFS
contracts. Find them in the latest release of
[frostfs-contract repository](https://git.frostfs.info/TrueCloudLab/frostfs-contract/releases).
## Commands
### Config
Config section provides `init` command that creates configuration file for the
Config section provides `init` command that creates a configuration file for
private installation deployment and updates. Config file is optional, all
parameters can be passed by arguments or read from standard input (wallet
parameters can be passed by arguments or read from standard input (wallet
passwords).
Config example:
```yaml
rpc-endpoint: https://address:port # side chain RPC node endpoint
rpc-endpoint: https://address:port # sidechain RPC node endpoint
alphabet-wallets: /path # path to consensus node / alphabet wallets storage
network:
max_object_size: 67108864 # max size of single NeoFS object, bytes
epoch_duration: 240 # duration of NeoFS epoch in blocks, consider block generation frequency in side chain
max_object_size: 67108864 # max size of a single FrostFS object, bytes
epoch_duration: 240 # duration of a FrostFS epoch in blocks, consider block generation frequency in the sidechain
basic_income_rate: 0 # basic income rate, for private consider 0
fee:
audit: 0 # network audit fee, for private installation consider 0
@ -58,41 +57,45 @@ credentials: # passwords for consensus node / alphabet wallets
#### Network deployment
- `generate-alphabet` generates a set of wallets for consensus and
Alphabet nodes.
- `generate-alphabet` generates a set of wallets for consensus and
Alphabet nodes.
- `init` initializes side chain by deploying smart contracts and
setting provided NeoFS network configuration.
- `init` initializes the sidechain by deploying smart contracts and
setting provided FrostFS network configuration.
- `generate-storage-wallet` generates wallet for the Storage node that
is ready for deployment. It also transfers a bit of side chain GAS, so this
wallet can be used for NeoFS bootstrap.
- `generate-storage-wallet` generates a wallet for the Storage node that
is ready for deployment. It also transfers a bit of sidechain GAS, so this
wallet can be used for FrostFS bootstrap.
#### Network maintenance
- `force-new-epoch` increments NeoFS epoch number and executes new epoch
handlers in NeoFS nodes.
- `set-config` add/update configuration values in the Netmap contract.
- `refill-gas` transfers side chain GAS to the specified wallet.
- `force-new-epoch` increments FrostFS epoch number and executes new epoch
handlers in FrostFS nodes.
- `refill-gas` transfers sidechain GAS to the specified wallet.
- `update-contracts` updates contracts to a new version.
#### Container migration
If the network has to be redeployed, these commands will migrate all container meta
If a network has to be redeployed, these commands will migrate all container meta
info. These commands **do not migrate actual objects**.
- `dump-containers` saves all containers and metadata registered in the container
contract to a file.
- `restore-containers` restores previously saved containers by their repeated registration in
container contract.
- `restore-containers` restores previously saved containers by their repeated registration in
the container contract.
- `list-containers` output all containers ids.
#### Network info
- `dump-config` prints NeoFS network configuration.
- `dump-config` prints FrostFS network configuration.
- `dump-hashes` prints NeoFS contract addresses stored in NNS.
- `dump-hashes` prints FrostFS contract addresses stored in NNS.
## Private network deployment

View file

@ -1,33 +1,34 @@
# Step-by-step private NeoFS deployment
# Step-by-step private FrostFS deployment
This is a short guide on how to deploy private NeoFS storage network on bare
This is a short guide on how to deploy a private FrostFS storage network on bare
metal without docker images. This guide does not cover details on how to start
consensus, Alphabet, or Storage nodes. This guide covers only `neofs-adm`
consensus, Alphabet, or Storage nodes. This guide covers only `frostfs-adm`
related configuration details.
## Prerequisites
To follow this guide you need:
- latest released version of [neo-go](https://github.com/nspcc-dev/neo-go/releases) (v0.97.2 at the moment),
- latest released version of [neofs-adm](https://github.com/nspcc-dev/neofs-node/releases) utility (v0.25.1 at the moment),
- latest released version of compiled [neofs-contract](https://github.com/nspcc-dev/neofs-contract/releases) (v0.11.0 at the moment).
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases) utility (v0.25.1 at the moment),
- latest released version of compiled [frostfs-contract](https://github.com/TrueCloudLab/frostfs-contract/releases) (v0.11.0 at the moment).
## Step 1: Prepare network configuration
## Step 1: Prepare network configuration
To start the network, you need a set of consensus nodes, the same number of
Alphabet nodes and any number of Storage nodes. While the number of Storage
nodes can be scaled almost infinitely, the number of consensus and Alphabet
To start a network, you need a set of consensus nodes, the same number of
Alphabet nodes and any number of Storage nodes. While the number of Storage
nodes can be scaled almost infinitely, the number of consensus and Alphabet
nodes can't be changed so easily right now. Consider this before going any further.
Note also that there is an upper limit on the number of alphabet nodes (currently 22).
It is easier to use`neofs-adm` with predefined configuration. First, create
network configuration file. In this example, there is going to be only one
It is easier to use`frostfs-adm` with a predefined configuration. First, create
a network configuration file. In this example, there is going to be only one
consensus / Alphabet node in the network.
```
$ neofs-adm config init --path foo.network.yml
$ frostfs-adm config init --path foo.network.yml
Initial config file saved to foo.network.yml
$ cat foo.network.yml
$ cat foo.network.yml
rpc-endpoint: https://neo.rpc.node:30333
alphabet-wallets: /home/user/deploy/alphabet-wallets
network:
@ -43,40 +44,40 @@ credentials:
az: hunter2
```
For private installation it is recommended to set all **fees** and **basic
income rate** to 0.
For private installation, it is recommended to set all **fees** and **basic
income rate** to 0.
As for **epoch duration**, consider consensus node block generation frequency.
With default 15 seconds per block, 240 blocks are going to be a 1-hour epoch.
As for **epoch duration**, consider consensus node block generation frequency.
With default 15 seconds per block, 240 blocks are going to be a 1-hour epoch.
For **max object size**, 67108864 (64 MiB) or 134217728 (128 MiB) should provide
For **max object size**, 67108864 (64 MiB) or 134217728 (128 MiB) should provide
good chunk distribution in most cases.
With this config, generate wallets (private keys) of consensus nodes. The same
wallets will be used for Alphabet nodes. Make sure, that dir for alphabet
wallets will be used for Alphabet nodes. Make sure, that dir for alphabet
wallets already exists.
```
$ neofs-adm -c foo.network.yml morph generate-alphabet --size 1
$ frostfs-adm -c foo.network.yml morph generate-alphabet --size 1
size: 1
alphabet-wallets: /home/user/deploy/alphabet-wallets
wallet[0]: hunter2
```
Do not lose wallet files and network config. Store it in encrypted backed up
Do not lose wallet files and network config. Store it in an encrypted backed up
storage.
## Step 2: Launch consensus nodes
Configure blockchain nodes with the generated wallets from the previous step.
Config examples can be found in
Config examples can be found in
[neo-go repository](https://github.com/nspcc-dev/neo-go/tree/master/config).
Gather public keys from **all** generated wallets. We are interested in first
Gather public keys from **all** generated wallets. We are interested in the first
`simple signature contract` public key.
```
$ neo-go wallet dump-keys -w alphabet-wallets/az.json
$ neo-go wallet dump-keys -w alphabet-wallets/az.json
NitdS4k4f1Hh5mbLJhAswBK3WC2gQgPN1o (simple signature contract):
02c1cc85f9c856dbe2d02017349bcb7b4e5defa78b8056a09b3240ba2a8c078869
@ -87,10 +88,10 @@ NiMKabp3ddi3xShmLAXhTfbnuWb4cSJT6E (1 out of 1 multisig contract):
02c1cc85f9c856dbe2d02017349bcb7b4e5defa78b8056a09b3240ba2a8c078869
```
Put the list of public keys into `ProtocolConfiguration.StandbyCommittee`
Put the list of public keys into `ProtocolConfiguration.StandbyCommittee`
section. Specify the wallet path and the password in `ApplicationConfiguration.P2PNotary`
and `ApplicationConfiguration.UnlockWallet` sections. If config includes
`ProtocolConfiguration.NativeActivations` section, then add notary
`ProtocolConfiguration.NativeActivations` section, add notary
contract `Notary: [0]`.
```yaml
@ -110,27 +111,27 @@ ApplicationConfiguration:
Password: "hunter2"
```
Then, launch consensus node. They should connect to each other and start
Then, launch consensus nodes. They should connect to each other and start
producing blocks in consensus. You might want to deploy additional RPC
nodes at this stage because Storage nodes should be connected to the chain too.
It is not recommended to use consensus node as an RPC node due to security policies
It is not recommended to use a consensus node as an RPC node due to security policies
and possible overload issues.
## Step 3: Initialize side chain
## Step 3: Initialize sidechain
Use archive with compiled NeoFS contracts to initialize side chain.
Use archive with compiled FrostFS contracts to initialize the sidechain.
```
$ tar -xzvf neofs-contract-v0.11.0.tar.gz
$ tar -xzvf frostfs-contract-v0.11.0.tar.gz
$ ./neofs-adm -c foo.network.yml morph init --contracts ./neofs-contract-v0.11.0
$ ./frostfs-adm -c foo.network.yml morph init --contracts ./frostfs-contract-v0.11.0
Stage 1: transfer GAS to alphabet nodes.
Waiting for transactions to persist...
Stage 2: set notary and alphabet nodes in designate contract.
Waiting for transactions to persist...
Stage 3: deploy NNS contract.
Waiting for transactions to persist...
Stage 4: deploy NeoFS contracts.
Stage 4: deploy FrostFS contracts.
Waiting for transactions to persist...
Stage 4.1: Transfer GAS to proxy contract.
Waiting for transactions to persist...
@ -140,21 +141,20 @@ Stage 6: transfer NEO to alphabet contracts.
Waiting for transactions to persist...
Stage 7: set addresses in NNS.
Waiting for transactions to persist...
NNS: Set alphabet0.neofs -> f692dfb4d43a15b464eb51a7041160fb29c44b6a
NNS: Set audit.neofs -> 7df847b993affb3852074345a7c2bd622171ee0d
NNS: Set balance.neofs -> 103519b3067a66307080a66570c0491ee8f68879
NNS: Set container.neofs -> cae60bdd689d185901e495352d0247752ce50846
NNS: Set neofsid.neofs -> c421fb60a3895865a8f24d197d6a80ef686041d2
NNS: Set netmap.neofs -> 894eb854632f50fb124412ce7951ebc00763525e
NNS: Set proxy.neofs -> ac6e6fe4b373d0ca0ca4969d1e58fa0988724e7d
NNS: Set reputation.neofs -> 6eda57c9d93d990573646762d1fea327ce41191f
NNS: Set alphabet0.frostfs -> f692dfb4d43a15b464eb51a7041160fb29c44b6a
NNS: Set audit.frostfs -> 7df847b993affb3852074345a7c2bd622171ee0d
NNS: Set balance.frostfs -> 103519b3067a66307080a66570c0491ee8f68879
NNS: Set container.frostfs -> cae60bdd689d185901e495352d0247752ce50846
NNS: Set frostfsid.frostfs -> c421fb60a3895865a8f24d197d6a80ef686041d2
NNS: Set netmap.frostfs -> 894eb854632f50fb124412ce7951ebc00763525e
NNS: Set proxy.frostfs -> ac6e6fe4b373d0ca0ca4969d1e58fa0988724e7d
Waiting for transactions to persist...
```
## Step 4: Launch Alphabet nodes
Configure Alphabet nodes with the wallets generated in step 1. For
`morph.validators` use list of public keys from
Configure Alphabet nodes with the wallets generated in step 1. For
`morph.validators` use a list of public keys from
`ProtocolConfiguration.StandbyCommittee`.
```yaml
@ -174,19 +174,19 @@ contracts:
## Step 4: Launch Storage node
Generate a new wallet for Storage node.
Generate a new wallet for a Storage node.
```
$ neofs-adm -c foo.network.yml morph generate-storage-wallet --storage-wallet ./sn01.json --initial-gas 10.0
New password >
$ frostfs-adm -c foo.network.yml morph generate-storage-wallet --storage-wallet ./sn01.json --initial-gas 10.0
New password >
Waiting for transactions to persist...
$ neo-go wallet dump-keys -w sn01.json
$ neo-go wallet dump-keys -w sn01.json
Ngr7p8Z9S22XDH6VkUG9oXobv8zZRAWwwv (simple signature contract):
0355eccb72cd46f09a3e5237eaa0f4949cceb5ecfa5a225bd3bb9fd021c4d75b85
```
Configure Storage node to use this wallet.
Configure the Storage node to use this wallet.
```
node:
@ -196,16 +196,16 @@ node:
password: "foobar"
```
Storage node will be included in the network map in next NeoFS epoch. To
The storage node will be included in the network map in the next FrostFS epoch. To
speed up this process, you can increment epoch counter immediately.
```
$ neofs-adm -c foo.network.yml morph force-new-epoch
$ frostfs-adm -c foo.network.yml morph force-new-epoch
Current epoch: 8, increase to 9.
Waiting for transactions to persist...
```
---
---
After that NeoFS Storage is ready to work. You can access it directly or
After that, FrostFS Storage is ready to work. You can access it directly or
with protocol gates.

View file

@ -0,0 +1,14 @@
package commonflags
const (
ConfigFlag = "config"
ConfigFlagShorthand = "c"
ConfigFlagUsage = "Config file"
ConfigDirFlag = "config-dir"
ConfigDirFlagUsage = "Config directory"
Verbose = "verbose"
VerboseShorthand = "v"
VerboseUsage = "Verbose output"
)

View file

@ -7,24 +7,25 @@ import (
"path/filepath"
"text/template"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neofs-node/pkg/innerring"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
type configTemplate struct {
Endpoint string
AlphabetDir string
MaxObjectSize int
EpochDuration int
BasicIncomeRate int
AuditFee int
CandidateFee int
ContainerFee int
ContainerAliasFee int
WithdrawFee int
Glagolitics []string
Endpoint string
AlphabetDir string
MaxObjectSize int
EpochDuration int
BasicIncomeRate int
AuditFee int
CandidateFee int
ContainerFee int
ContainerAliasFee int
WithdrawFee int
Glagolitics []string
HomomorphicHashDisabled bool
}
const configTxtTemplate = `rpc-endpoint: {{ .Endpoint}}
@ -33,19 +34,20 @@ network:
max_object_size: {{ .MaxObjectSize}}
epoch_duration: {{ .EpochDuration}}
basic_income_rate: {{ .BasicIncomeRate}}
homomorphic_hash_disabled: {{ .HomomorphicHashDisabled}}
fee:
audit: {{ .AuditFee}}
candidate: {{ .CandidateFee}}
container: {{ .ContainerFee}}
container_alias: {{ .ContainerAliasFee }}
withdraw: {{ .WithdrawFee}}
# if credentials section is omitted, then neofs-adm will require manual password input
# if credentials section is omitted, then frostfs-adm will require manual password input
credentials:
contract: password # wallet for contract group signature{{ range.Glagolitics}}
{{.}}: password{{end}}
`
func initConfig(cmd *cobra.Command, args []string) error {
func initConfig(cmd *cobra.Command, _ []string) error {
configPath, err := readConfigPathFromArgs(cmd)
if err != nil {
return nil
@ -97,7 +99,7 @@ func defaultConfigPath() (string, error) {
return "", fmt.Errorf("getting home dir path: %w", err)
}
return filepath.Join(home, ".neofs", "adm", "config.yml"), nil
return filepath.Join(home, ".frostfs", "adm", "config.yml"), nil
}
// generateConfigExample builds .yml representation of the config file. It is
@ -106,16 +108,17 @@ func defaultConfigPath() (string, error) {
// some comments as well.
func generateConfigExample(appDir string, credSize int) (string, error) {
tmpl := configTemplate{
Endpoint: "https://neo.rpc.node:30333",
MaxObjectSize: 67108864, // 64 MiB
EpochDuration: 240, // 1 hour with 15s per block
BasicIncomeRate: 1_0000_0000, // 0.0001 GAS per GiB (Fixed12)
AuditFee: 1_0000, // 0.00000001 GAS per audit (Fixed12)
CandidateFee: 100_0000_0000, // 100.0 GAS (Fixed8)
ContainerFee: 1000, // 0.000000001 * 7 GAS per container (Fixed12)
ContainerAliasFee: 500, // ContainerFee / 2
WithdrawFee: 1_0000_0000, // 1.0 GAS (Fixed8)
Glagolitics: make([]string, 0, credSize),
Endpoint: "https://neo.rpc.node:30333",
MaxObjectSize: 67108864, // 64 MiB
EpochDuration: 240, // 1 hour with 15s per block
BasicIncomeRate: 1_0000_0000, // 0.0001 GAS per GiB (Fixed12)
HomomorphicHashDisabled: false, // object homomorphic hash is enabled
AuditFee: 1_0000, // 0.00000001 GAS per audit (Fixed12)
CandidateFee: 100_0000_0000, // 100.0 GAS (Fixed8)
ContainerFee: 1000, // 0.000000001 * 7 GAS per container (Fixed12)
ContainerAliasFee: 500, // ContainerFee / 2
WithdrawFee: 1_0000_0000, // 1.0 GAS (Fixed8)
Glagolitics: make([]string, 0, credSize),
}
appDir, err := filepath.Abs(appDir)
@ -144,13 +147,20 @@ func generateConfigExample(appDir string, credSize int) (string, error) {
return buf.String(), nil
}
func AlphabetPassword(v *viper.Viper, index int) (string, error) {
letter := innerring.GlagoliticLetter(index)
key := "credentials." + letter.String()
func GetPassword(v *viper.Viper, name string) (string, error) {
key := "credentials." + name
if v.IsSet(key) {
return v.GetString(key), nil
}
prompt := "Password for " + letter.String() + " wallet > "
prompt := "Password for " + name + " wallet > "
return input.ReadPassword(prompt)
}
func GetStoragePassword(v *viper.Viper, name string) (string, error) {
key := "storage." + name
if name != "" && v.IsSet(key) {
return v.GetString(key), nil
}
return input.ReadPassword("New password > ")
}

View file

@ -5,7 +5,7 @@ import (
"path/filepath"
"testing"
"github.com/nspcc-dev/neofs-node/pkg/innerring"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
)
@ -13,7 +13,7 @@ import (
func TestGenerateConfigExample(t *testing.T) {
const (
n = 10
appDir = "/home/example/.neofs"
appDir = "/home/example/.frostfs"
)
configText, err := generateConfigExample(appDir, n)

View file

@ -0,0 +1,29 @@
package config
import (
"github.com/spf13/cobra"
)
const configPathFlag = "path"
var (
// RootCmd is a root command of config section.
RootCmd = &cobra.Command{
Use: "config",
Short: "Section for frostfs-adm config related commands",
}
initCmd = &cobra.Command{
Use: "init",
Short: "Initialize basic frostfs-adm configuration file",
Example: `frostfs-adm config init
frostfs-adm config init --path .config/frostfs-adm.yml`,
RunE: initConfig,
}
)
func init() {
RootCmd.AddCommand(initCmd)
initCmd.Flags().String(configPathFlag, "", "Path to config (default ~/.frostfs/adm/config.yml)")
}

View file

@ -0,0 +1,242 @@
package morph
import (
"crypto/elliptic"
"errors"
"fmt"
"math/big"
"git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/nspcc-dev/neo-go/pkg/core/native/noderoles"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/rolemgmt"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
type accBalancePair struct {
scriptHash util.Uint160
balance *big.Int
}
const (
dumpBalancesStorageFlag = "storage"
dumpBalancesAlphabetFlag = "alphabet"
dumpBalancesProxyFlag = "proxy"
dumpBalancesUseScriptHashFlag = "script-hash"
)
func dumpBalances(cmd *cobra.Command, _ []string) error {
var (
dumpStorage, _ = cmd.Flags().GetBool(dumpBalancesStorageFlag)
dumpAlphabet, _ = cmd.Flags().GetBool(dumpBalancesAlphabetFlag)
dumpProxy, _ = cmd.Flags().GetBool(dumpBalancesProxyFlag)
nnsCs *state.Contract
nmHash util.Uint160
)
c, err := getN3Client(viper.GetViper())
if err != nil {
return err
}
inv := invoker.New(c, nil)
if dumpStorage || dumpAlphabet || dumpProxy {
nnsCs, err = c.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err = nnsResolveHash(inv, nnsCs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
}
irList, err := fetchIRNodes(c, rolemgmt.Hash)
if err != nil {
return err
}
if err := fetchBalances(inv, gas.Hash, irList); err != nil {
return err
}
printBalances(cmd, "Inner ring nodes balances:", irList)
if dumpStorage {
if err := printStorageNodeBalances(cmd, inv, nmHash); err != nil {
return err
}
}
if dumpProxy {
if err := printProxyContractBalance(cmd, inv, nnsCs.Hash); err != nil {
return err
}
}
if dumpAlphabet {
if err := printAlphabetContractBalances(cmd, c, inv, len(irList), nnsCs.Hash); err != nil {
return err
}
}
return nil
}
func printStorageNodeBalances(cmd *cobra.Command, inv *invoker.Invoker, nmHash util.Uint160) error {
arr, err := unwrap.Array(inv.Call(nmHash, "netmap"))
if err != nil {
return errors.New("can't fetch the list of storage nodes")
}
snList := make([]accBalancePair, len(arr))
for i := range arr {
node, ok := arr[i].Value().([]stackitem.Item)
if !ok || len(node) == 0 {
return errors.New("can't parse the list of storage nodes")
}
bs, err := node[0].TryBytes()
if err != nil {
return errors.New("can't parse the list of storage nodes")
}
var ni netmap.NodeInfo
if err := ni.Unmarshal(bs); err != nil {
return fmt.Errorf("can't parse the list of storage nodes: %w", err)
}
pub, err := keys.NewPublicKeyFromBytes(ni.PublicKey(), elliptic.P256())
if err != nil {
return fmt.Errorf("can't parse storage node public key: %w", err)
}
snList[i].scriptHash = pub.GetScriptHash()
}
if err := fetchBalances(inv, gas.Hash, snList); err != nil {
return err
}
printBalances(cmd, "\nStorage node balances:", snList)
return nil
}
func printProxyContractBalance(cmd *cobra.Command, inv *invoker.Invoker, nnsHash util.Uint160) error {
h, err := nnsResolveHash(inv, nnsHash, proxyContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get hash of the proxy contract: %w", err)
}
proxyList := []accBalancePair{{scriptHash: h}}
if err := fetchBalances(inv, gas.Hash, proxyList); err != nil {
return err
}
printBalances(cmd, "\nProxy contract balance:", proxyList)
return nil
}
func printAlphabetContractBalances(cmd *cobra.Command, c Client, inv *invoker.Invoker, count int, nnsHash util.Uint160) error {
alphaList := make([]accBalancePair, count)
w := io.NewBufBinWriter()
for i := range alphaList {
emit.AppCall(w.BinWriter, nnsHash, "resolve", callflag.ReadOnly,
getAlphabetNNSDomain(i),
int64(nns.TXT))
}
if w.Err != nil {
panic(w.Err)
}
alphaRes, err := c.InvokeScript(w.Bytes(), nil)
if err != nil {
return fmt.Errorf("can't fetch info from NNS: %w", err)
}
for i := range alphaList {
h, err := parseNNSResolveResult(alphaRes.Stack[i])
if err != nil {
return fmt.Errorf("can't fetch the alphabet contract #%d hash: %w", i, err)
}
alphaList[i].scriptHash = h
}
if err := fetchBalances(inv, gas.Hash, alphaList); err != nil {
return err
}
printBalances(cmd, "\nAlphabet contracts balances:", alphaList)
return nil
}
func fetchIRNodes(c Client, desigHash util.Uint160) ([]accBalancePair, error) {
inv := invoker.New(c, nil)
height, err := c.GetBlockCount()
if err != nil {
return nil, fmt.Errorf("can't get block height: %w", err)
}
arr, err := getDesignatedByRole(inv, desigHash, noderoles.NeoFSAlphabet, height)
if err != nil {
return nil, errors.New("can't fetch list of IR nodes from the netmap contract")
}
irList := make([]accBalancePair, len(arr))
for i := range arr {
irList[i].scriptHash = arr[i].GetScriptHash()
}
return irList, nil
}
func printBalances(cmd *cobra.Command, prefix string, accounts []accBalancePair) {
useScriptHash, _ := cmd.Flags().GetBool(dumpBalancesUseScriptHashFlag)
cmd.Println(prefix)
for i := range accounts {
var addr string
if useScriptHash {
addr = accounts[i].scriptHash.StringLE()
} else {
addr = address.Uint160ToString(accounts[i].scriptHash)
}
cmd.Printf("%s: %s\n", addr, fixedn.ToString(accounts[i].balance, 8))
}
}
func fetchBalances(c *invoker.Invoker, gasHash util.Uint160, accounts []accBalancePair) error {
w := io.NewBufBinWriter()
for i := range accounts {
emit.AppCall(w.BinWriter, gasHash, "balanceOf", callflag.ReadStates, accounts[i].scriptHash)
}
if w.Err != nil {
panic(w.Err)
}
res, err := c.Run(w.Bytes())
if err != nil || res.State != vmstate.Halt.String() || len(res.Stack) != len(accounts) {
return errors.New("can't fetch account balances")
}
for i := range accounts {
bal, err := res.Stack[i].TryInteger()
if err != nil {
return fmt.Errorf("can't parse account balance: %w", err)
}
accounts[i].balance = bal
}
return nil
}

View file

@ -0,0 +1,166 @@
package morph
import (
"bytes"
"encoding/binary"
"encoding/hex"
"errors"
"fmt"
"strconv"
"strings"
"text/tabwriter"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const forceConfigSet = "force"
func dumpNetworkConfig(cmd *cobra.Command, _ []string) error {
c, err := getN3Client(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
inv := invoker.New(c, nil)
cs, err := c.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err := nnsResolveHash(inv, cs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
arr, err := unwrap.Array(inv.Call(nmHash, "listConfig"))
if err != nil {
return errors.New("can't fetch list of network config keys from the netmap contract")
}
buf := bytes.NewBuffer(nil)
tw := tabwriter.NewWriter(buf, 0, 2, 2, ' ', 0)
m, err := parseConfigFromNetmapContract(arr)
if err != nil {
return err
}
for k, v := range m {
switch k {
case netmap.AuditFeeConfig, netmap.BasicIncomeRateConfig,
netmap.ContainerFeeConfig, netmap.ContainerAliasFeeConfig,
netmap.EpochDurationConfig, netmap.IrCandidateFeeConfig,
netmap.MaxObjectSizeConfig, netmap.WithdrawFeeConfig:
nbuf := make([]byte, 8)
copy(nbuf[:], v)
n := binary.LittleEndian.Uint64(nbuf)
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%d (int)\n", k, n)))
case netmap.HomomorphicHashingDisabledKey, netmap.MaintenanceModeAllowedConfig:
if len(v) == 0 || len(v) > 1 {
return invalidConfigValueErr(k)
}
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%t (bool)\n", k, v[0] == 1)))
default:
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%s (hex)\n", k, hex.EncodeToString(v))))
}
}
_ = tw.Flush()
cmd.Print(buf.String())
return nil
}
func setConfigCmd(cmd *cobra.Command, args []string) error {
if len(args) == 0 {
return errors.New("empty config pairs")
}
wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return fmt.Errorf("can't initialize context: %w", err)
}
cs, err := wCtx.Client.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err := nnsResolveHash(wCtx.ReadOnlyInvoker, cs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
forceFlag, _ := cmd.Flags().GetBool(forceConfigSet)
bw := io.NewBufBinWriter()
for _, arg := range args {
k, v, err := parseConfigPair(arg, forceFlag)
if err != nil {
return err
}
// In NeoFS this is done via Notary contract. Here, however, we can form the
// transaction locally. The first `nil` argument is required only for notary
// disabled environment which is not supported by that command.
emit.AppCall(bw.BinWriter, nmHash, "setConfig", callflag.All, nil, k, v)
if bw.Err != nil {
return fmt.Errorf("can't form raw transaction: %w", bw.Err)
}
}
err = wCtx.sendConsensusTx(bw.Bytes())
if err != nil {
return err
}
return wCtx.awaitTx()
}
func parseConfigPair(kvStr string, force bool) (key string, val any, err error) {
k, v, found := strings.Cut(kvStr, "=")
if !found {
return "", nil, fmt.Errorf("invalid parameter format: must be 'key=val', got: %s", kvStr)
}
key = k
valRaw := v
switch key {
case netmap.AuditFeeConfig, netmap.BasicIncomeRateConfig,
netmap.ContainerFeeConfig, netmap.ContainerAliasFeeConfig,
netmap.EpochDurationConfig, netmap.IrCandidateFeeConfig,
netmap.MaxObjectSizeConfig, netmap.WithdrawFeeConfig:
val, err = strconv.ParseInt(valRaw, 10, 64)
if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as int: %w", key, valRaw, err)
}
case netmap.HomomorphicHashingDisabledKey, netmap.MaintenanceModeAllowedConfig:
val, err = strconv.ParseBool(valRaw)
if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as bool: %w", key, valRaw, err)
}
default:
if !force {
return "", nil, fmt.Errorf(
"'%s' key is not well-known, use '--%s' flag if want to set it anyway",
key, forceConfigSet)
}
val = valRaw
}
return
}
func invalidConfigValueErr(key string) error {
return fmt.Errorf("invalid %s config value from netmap contract", key)
}

View file

@ -0,0 +1,447 @@
package morph
import (
"encoding/json"
"errors"
"fmt"
"os"
"sort"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/nspcc-dev/neo-go/pkg/crypto/hash"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var errInvalidContainerResponse = errors.New("invalid response from container contract")
func getContainerContractHash(cmd *cobra.Command, inv *invoker.Invoker, c Client) (util.Uint160, error) {
s, err := cmd.Flags().GetString(containerContractFlag)
var ch util.Uint160
if err == nil {
ch, err = util.Uint160DecodeStringLE(s)
}
if err != nil {
nnsCs, err := c.GetContractStateByID(1)
if err != nil {
return util.Uint160{}, fmt.Errorf("can't get NNS contract state: %w", err)
}
ch, err = nnsResolveHash(inv, nnsCs.Hash, containerContract+".frostfs")
if err != nil {
return util.Uint160{}, err
}
}
return ch, nil
}
func iterateContainerList(inv *invoker.Invoker, ch util.Uint160, f func([]byte) error) error {
sid, r, err := unwrap.SessionIterator(inv.Call(ch, "containersOf", ""))
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
// Nothing bad, except live session on the server, do not report to the user.
defer func() { _ = inv.TerminateSession(sid) }()
items, err := inv.TraverseIterator(sid, &r, 0)
for err == nil && len(items) != 0 {
for j := range items {
b, err := items[j].TryBytes()
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if err := f(b); err != nil {
return err
}
}
items, err = inv.TraverseIterator(sid, &r, 0)
}
return err
}
func dumpContainers(cmd *cobra.Command, _ []string) error {
filename, err := cmd.Flags().GetString(containerDumpFlag)
if err != nil {
return fmt.Errorf("invalid filename: %w", err)
}
c, err := getN3Client(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
inv := invoker.New(c, nil)
ch, err := getContainerContractHash(cmd, inv, c)
if err != nil {
return fmt.Errorf("unable to get contaract hash: %w", err)
}
isOK, err := getCIDFilterFunc(cmd)
if err != nil {
return err
}
f, err := os.OpenFile(filename, os.O_RDWR|os.O_TRUNC|os.O_CREATE, 0o660)
if err != nil {
return err
}
defer f.Close()
_, err = f.Write([]byte{'['})
if err != nil {
return err
}
written := 0
enc := json.NewEncoder(f)
bw := io.NewBufBinWriter()
iterErr := iterateContainerList(inv, ch, func(id []byte) error {
if !isOK(id) {
return nil
}
cnt, err := dumpSingleContainer(bw, ch, inv, id)
if err != nil {
return err
}
// Writing directly to the file is ok, because json.Encoder does no internal buffering.
if written != 0 {
_, err = f.Write([]byte{','})
if err != nil {
return err
}
}
written++
return enc.Encode(cnt)
})
if iterErr != nil {
return iterErr
}
_, err = f.Write([]byte{']'})
return err
}
func dumpSingleContainer(bw *io.BufBinWriter, ch util.Uint160, inv *invoker.Invoker, id []byte) (*Container, error) {
bw.Reset()
emit.AppCall(bw.BinWriter, ch, "get", callflag.All, id)
emit.AppCall(bw.BinWriter, ch, "eACL", callflag.All, id)
res, err := inv.Run(bw.Bytes())
if err != nil {
return nil, fmt.Errorf("can't get container info: %w", err)
}
if len(res.Stack) != 2 {
return nil, fmt.Errorf("%w: expected 2 items on stack", errInvalidContainerResponse)
}
cnt := new(Container)
err = cnt.FromStackItem(res.Stack[0])
if err != nil {
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
ea := new(EACL)
err = ea.FromStackItem(res.Stack[1])
if err != nil {
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if len(ea.Value) != 0 {
cnt.EACL = ea
}
return cnt, nil
}
func listContainers(cmd *cobra.Command, _ []string) error {
c, err := getN3Client(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
inv := invoker.New(c, nil)
ch, err := getContainerContractHash(cmd, inv, c)
if err != nil {
return fmt.Errorf("unable to get contaract hash: %w", err)
}
return iterateContainerList(inv, ch, func(id []byte) error {
var idCnr cid.ID
err = idCnr.Decode(id)
if err != nil {
return fmt.Errorf("unable to decode container id: %w", err)
}
cmd.Println(idCnr)
return nil
})
}
func restoreContainers(cmd *cobra.Command, _ []string) error {
filename, err := cmd.Flags().GetString(containerDumpFlag)
if err != nil {
return fmt.Errorf("invalid filename: %w", err)
}
wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return err
}
defer wCtx.close()
containers, err := parseContainers(filename)
if err != nil {
return err
}
ch, err := fetchContainerContractHash(wCtx)
if err != nil {
return err
}
isOK, err := getCIDFilterFunc(cmd)
if err != nil {
return err
}
err = restoreOrPutContainers(containers, isOK, cmd, wCtx, ch)
if err != nil {
return err
}
return wCtx.awaitTx()
}
func restoreOrPutContainers(containers []Container, isOK func([]byte) bool, cmd *cobra.Command, wCtx *initializeContext, ch util.Uint160) error {
bw := io.NewBufBinWriter()
for _, cnt := range containers {
hv := hash.Sha256(cnt.Value)
if !isOK(hv[:]) {
continue
}
bw.Reset()
restored, err := isContainerRestored(cmd, wCtx, ch, bw, hv)
if err != nil {
return err
}
if restored {
continue
}
bw.Reset()
putContainer(bw, ch, cnt)
if bw.Err != nil {
panic(bw.Err)
}
if err := wCtx.sendConsensusTx(bw.Bytes()); err != nil {
return err
}
}
return nil
}
func putContainer(bw *io.BufBinWriter, ch util.Uint160, cnt Container) {
emit.AppCall(bw.BinWriter, ch, "put", callflag.All,
cnt.Value, cnt.Signature, cnt.PublicKey, cnt.Token)
if ea := cnt.EACL; ea != nil {
emit.AppCall(bw.BinWriter, ch, "setEACL", callflag.All,
ea.Value, ea.Signature, ea.PublicKey, ea.Token)
}
}
func isContainerRestored(cmd *cobra.Command, wCtx *initializeContext, containerHash util.Uint160, bw *io.BufBinWriter, hashValue util.Uint256) (bool, error) {
emit.AppCall(bw.BinWriter, containerHash, "get", callflag.All, hashValue.BytesBE())
res, err := wCtx.Client.InvokeScript(bw.Bytes(), nil)
if err != nil {
return false, fmt.Errorf("can't check if container is already restored: %w", err)
}
if len(res.Stack) == 0 {
return false, errors.New("empty stack")
}
old := new(Container)
if err := old.FromStackItem(res.Stack[0]); err != nil {
return false, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if len(old.Value) != 0 {
var id cid.ID
id.SetSHA256(hashValue)
cmd.Printf("Container %s is already deployed.\n", id)
return true, nil
}
return false, nil
}
func parseContainers(filename string) ([]Container, error) {
data, err := os.ReadFile(filename)
if err != nil {
return nil, fmt.Errorf("can't read dump file: %w", err)
}
var containers []Container
err = json.Unmarshal(data, &containers)
if err != nil {
return nil, fmt.Errorf("can't parse dump file: %w", err)
}
return containers, nil
}
func fetchContainerContractHash(wCtx *initializeContext) (util.Uint160, error) {
nnsCs, err := wCtx.Client.GetContractStateByID(1)
if err != nil {
return util.Uint160{}, fmt.Errorf("can't get NNS contract state: %w", err)
}
ch, err := nnsResolveHash(wCtx.ReadOnlyInvoker, nnsCs.Hash, containerContract+".frostfs")
if err != nil {
return util.Uint160{}, fmt.Errorf("can't fetch container contract hash: %w", err)
}
return ch, nil
}
// Container represents container struct in contract storage.
type Container struct {
Value []byte `json:"value"`
Signature []byte `json:"signature"`
PublicKey []byte `json:"public_key"`
Token []byte `json:"token"`
EACL *EACL `json:"eacl"`
}
// EACL represents extended ACL struct in contract storage.
type EACL struct {
Value []byte `json:"value"`
Signature []byte `json:"signature"`
PublicKey []byte `json:"public_key"`
Token []byte `json:"token"`
}
// ToStackItem implements stackitem.Convertible.
func (c *Container) ToStackItem() (stackitem.Item, error) {
return stackitem.NewStruct([]stackitem.Item{
stackitem.NewByteArray(c.Value),
stackitem.NewByteArray(c.Signature),
stackitem.NewByteArray(c.PublicKey),
stackitem.NewByteArray(c.Token),
}), nil
}
// FromStackItem implements stackitem.Convertible.
func (c *Container) FromStackItem(item stackitem.Item) error {
arr, ok := item.Value().([]stackitem.Item)
if !ok || len(arr) != 4 {
return errors.New("invalid stack item type")
}
value, err := arr[0].TryBytes()
if err != nil {
return errors.New("invalid container value")
}
sig, err := arr[1].TryBytes()
if err != nil {
return errors.New("invalid container signature")
}
pub, err := arr[2].TryBytes()
if err != nil {
return errors.New("invalid container public key")
}
tok, err := arr[3].TryBytes()
if err != nil {
return errors.New("invalid container token")
}
c.Value = value
c.Signature = sig
c.PublicKey = pub
c.Token = tok
return nil
}
// ToStackItem implements stackitem.Convertible.
func (c *EACL) ToStackItem() (stackitem.Item, error) {
return stackitem.NewStruct([]stackitem.Item{
stackitem.NewByteArray(c.Value),
stackitem.NewByteArray(c.Signature),
stackitem.NewByteArray(c.PublicKey),
stackitem.NewByteArray(c.Token),
}), nil
}
// FromStackItem implements stackitem.Convertible.
func (c *EACL) FromStackItem(item stackitem.Item) error {
arr, ok := item.Value().([]stackitem.Item)
if !ok || len(arr) != 4 {
return errors.New("invalid stack item type")
}
value, err := arr[0].TryBytes()
if err != nil {
return errors.New("invalid eACL value")
}
sig, err := arr[1].TryBytes()
if err != nil {
return errors.New("invalid eACL signature")
}
pub, err := arr[2].TryBytes()
if err != nil {
return errors.New("invalid eACL public key")
}
tok, err := arr[3].TryBytes()
if err != nil {
return errors.New("invalid eACL token")
}
c.Value = value
c.Signature = sig
c.PublicKey = pub
c.Token = tok
return nil
}
// getCIDFilterFunc returns filtering function for container IDs.
// Raw byte slices are used because it works with structures returned
// from contract.
func getCIDFilterFunc(cmd *cobra.Command) (func([]byte) bool, error) {
rawIDs, err := cmd.Flags().GetStringSlice(containerIDsFlag)
if err != nil {
return nil, err
}
if len(rawIDs) == 0 {
return func([]byte) bool { return true }, nil
}
for i := range rawIDs {
err := new(cid.ID).DecodeString(rawIDs[i])
if err != nil {
return nil, fmt.Errorf("can't parse CID %s: %w", rawIDs[i], err)
}
}
sort.Strings(rawIDs)
return func(rawID []byte) bool {
var v [32]byte
copy(v[:], rawID)
var id cid.ID
id.SetSHA256(v)
idStr := id.EncodeToString()
n := sort.Search(len(rawIDs), func(i int) bool { return rawIDs[i] >= idStr })
return n < len(rawIDs) && rawIDs[n] == idStr
}, nil
}

View file

@ -0,0 +1,232 @@
package morph
import (
"encoding/json"
"fmt"
"os"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
"github.com/nspcc-dev/neo-go/cli/cmdargs"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/management"
"github.com/nspcc-dev/neo-go/pkg/services/rpcsrv/params"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
contractPathFlag = "contract"
updateFlag = "update"
customZoneFlag = "domain"
)
var deployCmd = &cobra.Command{
Use: "deploy",
Short: "Deploy additional smart-contracts",
Long: `Deploy additional smart-contract which are not related to core.
All contracts are deployed by the committee, so access to the alphabet wallets is required.
Optionally, arguments can be provided to be passed to a contract's _deploy function.
The syntax is the same as for 'neo-go contract testinvokefunction' command.
Compiled contract file name must contain '_contract.nef' suffix.
Contract's manifest file name must be 'config.json'.
NNS name is taken by stripping '_contract.nef' from the NEF file (similar to frostfs contracts).`,
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: deployContractCmd,
}
func init() {
ff := deployCmd.Flags()
ff.String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
_ = deployCmd.MarkFlagFilename(alphabetWalletsFlag)
ff.StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
ff.String(contractPathFlag, "", "Path to the contract directory")
_ = deployCmd.MarkFlagFilename(contractPathFlag)
ff.Bool(updateFlag, false, "Update an existing contract")
ff.String(customZoneFlag, "frostfs", "Custom zone for NNS")
}
func deployContractCmd(cmd *cobra.Command, args []string) error {
v := viper.GetViper()
c, err := newInitializeContext(cmd, v)
if err != nil {
return fmt.Errorf("initialization error: %w", err)
}
defer c.close()
ctrPath, _ := cmd.Flags().GetString(contractPathFlag)
ctrName, err := probeContractName(ctrPath)
if err != nil {
return err
}
cs, err := readContract(ctrPath, ctrName)
if err != nil {
return err
}
nnsCs, err := c.Client.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't fetch NNS contract state: %w", err)
}
callHash := management.Hash
method := deployMethodName
zone, _ := cmd.Flags().GetString(customZoneFlag)
domain := ctrName + "." + zone
isUpdate, _ := cmd.Flags().GetBool(updateFlag)
if isUpdate {
cs.Hash, err = nnsResolveHash(c.ReadOnlyInvoker, nnsCs.Hash, domain)
if err != nil {
return fmt.Errorf("can't fetch contract hash from NNS: %w", err)
}
callHash = cs.Hash
method = updateMethodName
} else {
cs.Hash = state.CreateContractHash(
c.CommitteeAcc.Contract.ScriptHash(),
cs.NEF.Checksum,
cs.Manifest.Name)
}
writer := io.NewBufBinWriter()
if err := emitDeploymentArguments(writer.BinWriter, args); err != nil {
return err
}
emit.Bytes(writer.BinWriter, cs.RawManifest)
emit.Bytes(writer.BinWriter, cs.RawNEF)
emit.Int(writer.BinWriter, 3)
emit.Opcodes(writer.BinWriter, opcode.PACK)
emit.AppCallNoArgs(writer.BinWriter, callHash, method, callflag.All)
emit.Opcodes(writer.BinWriter, opcode.DROP) // contract state on stack
if !isUpdate {
err := registerNNS(nnsCs, c, zone, domain, cs, writer)
if err != nil {
return err
}
}
if writer.Err != nil {
panic(fmt.Errorf("BUG: can't create deployment script: %w", writer.Err))
}
if err := c.sendCommitteeTx(writer.Bytes(), false); err != nil {
return err
}
return c.awaitTx()
}
func registerNNS(nnsCs *state.Contract, c *initializeContext, zone string, domain string, cs *contractState, writer *io.BufBinWriter) error {
bw := io.NewBufBinWriter()
emit.Instruction(bw.BinWriter, opcode.INITSSLOT, []byte{1})
emit.AppCall(bw.BinWriter, nnsCs.Hash, "getPrice", callflag.All)
emit.Opcodes(bw.BinWriter, opcode.STSFLD0)
emit.AppCall(bw.BinWriter, nnsCs.Hash, "setPrice", callflag.All, 1)
start := bw.Len()
needRecord := false
ok, err := c.nnsRootRegistered(nnsCs.Hash, zone)
if err != nil {
return err
} else if !ok {
needRecord = true
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
zone, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
domain, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
} else {
s, ok, err := c.nnsRegisterDomainScript(nnsCs.Hash, cs.Hash, domain)
if err != nil {
return err
}
needRecord = !ok
if len(s) != 0 {
bw.WriteBytes(s)
}
}
if needRecord {
emit.AppCall(bw.BinWriter, nnsCs.Hash, "deleteRecords", callflag.All, domain, int64(nns.TXT))
emit.AppCall(bw.BinWriter, nnsCs.Hash, "addRecord", callflag.All,
domain, int64(nns.TXT), address.Uint160ToString(cs.Hash))
}
if bw.Err != nil {
panic(fmt.Errorf("BUG: can't create deployment script: %w", writer.Err))
} else if bw.Len() != start {
writer.WriteBytes(bw.Bytes())
emit.Opcodes(writer.BinWriter, opcode.LDSFLD0, opcode.PUSH1, opcode.PACK)
emit.AppCallNoArgs(writer.BinWriter, nnsCs.Hash, "setPrice", callflag.All)
if needRecord {
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
}
}
return nil
}
func emitDeploymentArguments(w *io.BinWriter, args []string) error {
_, ps, err := cmdargs.ParseParams(args, true)
if err != nil {
return err
}
if len(ps) == 0 {
emit.Opcodes(w, opcode.NEWARRAY0)
return nil
}
if len(ps) != 1 {
return fmt.Errorf("at most one argument is expected for deploy, got %d", len(ps))
}
// We could emit this directly, but round-trip through JSON is more robust.
// This a CLI, so optimizing the conversion is not worth the effort.
data, err := json.Marshal(ps)
if err != nil {
return err
}
var pp params.Params
if err := json.Unmarshal(data, &pp); err != nil {
return err
}
return params.ExpandArrayIntoScript(w, pp)
}
func probeContractName(ctrPath string) (string, error) {
ds, err := os.ReadDir(ctrPath)
if err != nil {
return "", fmt.Errorf("can't read directory: %w", err)
}
var ctrName string
for i := range ds {
if strings.HasSuffix(ds[i].Name(), "_contract.nef") {
ctrName = strings.TrimSuffix(ds[i].Name(), "_contract.nef")
break
}
}
if ctrName == "" {
return "", fmt.Errorf("can't find any NEF files in %s", ctrPath)
}
return ctrName, nil
}

View file

@ -0,0 +1,251 @@
package morph
import (
"bytes"
"errors"
"fmt"
"strings"
"text/tabwriter"
"git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
morphClient "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const lastGlagoliticLetter = 41
type contractDumpInfo struct {
hash util.Uint160
name string
version string
}
func dumpContractHashes(cmd *cobra.Command, _ []string) error {
c, err := getN3Client(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
cs, err := c.GetContractStateByID(1)
if err != nil {
return err
}
zone, _ := cmd.Flags().GetString(customZoneFlag)
if zone != "" {
return dumpCustomZoneHashes(cmd, cs.Hash, zone, c)
}
infos := []contractDumpInfo{{name: nnsContract, hash: cs.Hash}}
irSize := 0
for ; irSize < lastGlagoliticLetter; irSize++ {
ok, err := nnsIsAvailable(c, cs.Hash, getAlphabetNNSDomain(irSize))
if err != nil {
return err
} else if ok {
break
}
}
bw := io.NewBufBinWriter()
if irSize != 0 {
bw.Reset()
for i := 0; i < irSize; i++ {
emit.AppCall(bw.BinWriter, cs.Hash, "resolve", callflag.ReadOnly,
getAlphabetNNSDomain(i),
int64(nns.TXT))
}
alphaRes, err := c.InvokeScript(bw.Bytes(), nil)
if err != nil {
return fmt.Errorf("can't fetch info from NNS: %w", err)
}
for i := 0; i < irSize; i++ {
info := contractDumpInfo{name: fmt.Sprintf("alphabet %d", i)}
if h, err := parseNNSResolveResult(alphaRes.Stack[i]); err == nil {
info.hash = h
}
infos = append(infos, info)
}
}
for _, ctrName := range contractList {
bw.Reset()
emit.AppCall(bw.BinWriter, cs.Hash, "resolve", callflag.ReadOnly,
ctrName+".frostfs", int64(nns.TXT))
res, err := c.InvokeScript(bw.Bytes(), nil)
if err != nil {
return fmt.Errorf("can't fetch info from NNS: %w", err)
}
info := contractDumpInfo{name: ctrName}
if len(res.Stack) != 0 {
if h, err := parseNNSResolveResult(res.Stack[0]); err == nil {
info.hash = h
}
}
infos = append(infos, info)
}
fillContractVersion(cmd, c, infos)
printContractInfo(cmd, infos)
return nil
}
func dumpCustomZoneHashes(cmd *cobra.Command, nnsHash util.Uint160, zone string, c Client) error {
const nnsMaxTokens = 100
inv := invoker.New(c, nil)
if !strings.HasPrefix(zone, ".") {
zone = "." + zone
}
var infos []contractDumpInfo
processItem := func(item stackitem.Item) {
bs, err := item.TryBytes()
if err != nil {
cmd.PrintErrf("Invalid NNS record: %v\n", err)
return
}
if !bytes.HasSuffix(bs, []byte(zone)) || bytes.HasPrefix(bs, []byte(morphClient.NNSGroupKeyName)) {
// Related https://github.com/nspcc-dev/neofs-contract/issues/316.
return
}
h, err := nnsResolveHash(inv, nnsHash, string(bs))
if err != nil {
cmd.PrintErrf("Could not resolve name %s: %v\n", string(bs), err)
return
}
infos = append(infos, contractDumpInfo{
hash: h,
name: strings.TrimSuffix(string(bs), zone),
})
}
sessionID, iter, err := unwrap.SessionIterator(inv.Call(nnsHash, "tokens"))
if err != nil {
if errors.Is(err, unwrap.ErrNoSessionID) {
items, err := unwrap.Array(inv.CallAndExpandIterator(nnsHash, "tokens", nnsMaxTokens))
if err != nil {
return fmt.Errorf("can't get a list of NNS domains: %w", err)
}
if len(items) == nnsMaxTokens {
cmd.PrintErrln("Provided RPC endpoint doesn't support sessions, some hashes might be lost.")
}
for i := range items {
processItem(items[i])
}
} else {
return err
}
} else {
defer func() {
_ = inv.TerminateSession(sessionID)
}()
items, err := inv.TraverseIterator(sessionID, &iter, nnsMaxTokens)
for err == nil && len(items) != 0 {
for i := range items {
processItem(items[i])
}
items, err = inv.TraverseIterator(sessionID, &iter, nnsMaxTokens)
}
if err != nil {
return fmt.Errorf("error during NNS domains iteration: %w", err)
}
}
fillContractVersion(cmd, c, infos)
printContractInfo(cmd, infos)
return nil
}
func parseContractVersion(item stackitem.Item) string {
bi, err := item.TryInteger()
if err != nil || bi.Sign() == 0 || !bi.IsInt64() {
return "unknown"
}
v := bi.Int64()
major := v / 1_000_000
minor := (v % 1_000_000) / 1000
patch := v % 1_000
return fmt.Sprintf("v%d.%d.%d", major, minor, patch)
}
func printContractInfo(cmd *cobra.Command, infos []contractDumpInfo) {
if len(infos) == 0 {
return
}
buf := bytes.NewBuffer(nil)
tw := tabwriter.NewWriter(buf, 0, 2, 2, ' ', 0)
for _, info := range infos {
if info.version == "" {
info.version = "unknown"
}
_, _ = tw.Write([]byte(fmt.Sprintf("%s\t(%s):\t%s\n",
info.name, info.version, info.hash.StringLE())))
}
_ = tw.Flush()
cmd.Print(buf.String())
}
func fillContractVersion(cmd *cobra.Command, c Client, infos []contractDumpInfo) {
bw := io.NewBufBinWriter()
sub := io.NewBufBinWriter()
for i := range infos {
if infos[i].hash.Equals(util.Uint160{}) {
emit.Int(bw.BinWriter, 0)
} else {
sub.Reset()
emit.AppCall(sub.BinWriter, infos[i].hash, "version", callflag.NoneFlag)
if sub.Err != nil {
panic(fmt.Errorf("BUG: can't create version script: %w", bw.Err))
}
script := sub.Bytes()
emit.Instruction(bw.BinWriter, opcode.TRY, []byte{byte(3 + len(script) + 2), 0})
bw.BinWriter.WriteBytes(script)
emit.Instruction(bw.BinWriter, opcode.ENDTRY, []byte{2 + 1})
emit.Opcodes(bw.BinWriter, opcode.PUSH0)
}
}
emit.Opcodes(bw.BinWriter, opcode.NOP) // for the last ENDTRY target
if bw.Err != nil {
panic(fmt.Errorf("BUG: can't create version script: %w", bw.Err))
}
res, err := c.InvokeScript(bw.Bytes(), nil)
if err != nil {
cmd.Printf("Can't fetch version from NNS: %v\n", err)
return
}
if res.State == vmstate.Halt.String() {
for i := range res.Stack {
infos[i].version = parseContractVersion(res.Stack[i])
}
}
}

View file

@ -5,15 +5,15 @@ import (
"fmt"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
func forceNewEpochCmd(cmd *cobra.Command, args []string) error {
func forceNewEpochCmd(cmd *cobra.Command, _ []string) error {
wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return fmt.Errorf("can't to initialize context: %w", err)
@ -24,31 +24,34 @@ func forceNewEpochCmd(cmd *cobra.Command, args []string) error {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err := nnsResolveHash(wCtx.Client, cs.Hash, netmapContract+".neofs")
nmHash, err := nnsResolveHash(wCtx.ReadOnlyInvoker, cs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
res, err := wCtx.Client.InvokeFunction(nmHash, "epoch", []smartcontract.Parameter{}, nil)
if err != nil || res.State != vm.HaltState.String() || len(res.Stack) == 0 {
return errors.New("can't fetch current epoch from the netmap contract")
}
bi, err := res.Stack[0].TryInteger()
if err != nil {
return fmt.Errorf("can't parse current epoch: %w", err)
}
newEpoch := bi.Int64() + 1
cmd.Printf("Current epoch: %s, increase to %d.\n", bi, newEpoch)
// In NeoFS this is done via Notary contract. Here, however, we can form the
// transaction locally.
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, nmHash, "newEpoch", callflag.All, newEpoch)
if err := wCtx.sendCommitteeTx(bw.Bytes(), -1, true); err != nil {
if err := emitNewEpochCall(bw, wCtx, nmHash); err != nil {
return err
}
if err := wCtx.sendConsensusTx(bw.Bytes()); err != nil {
return err
}
return wCtx.awaitTx()
}
func emitNewEpochCall(bw *io.BufBinWriter, wCtx *initializeContext, nmHash util.Uint160) error {
curr, err := unwrap.Int64(wCtx.ReadOnlyInvoker.Call(nmHash, "epoch"))
if err != nil {
return errors.New("can't fetch current epoch from the netmap contract")
}
newEpoch := curr + 1
wCtx.Command.Printf("Current epoch: %d, increase to %d.\n", curr, newEpoch)
// In NeoFS this is done via Notary contract. Here, however, we can form the
// transaction locally.
emit.AppCall(bw.BinWriter, nmHash, "newEpoch", callflag.All, newEpoch)
return bw.Err
}

View file

@ -6,20 +6,22 @@ import (
"os"
"path/filepath"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/nspcc-dev/neofs-node/cmd/neofs-adm/internal/modules/config"
"github.com/nspcc-dev/neofs-node/pkg/innerring"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"golang.org/x/sync/errgroup"
)
const (
@ -28,7 +30,7 @@ const (
consensusAccountName = "consensus"
)
func generateAlphabetCreds(cmd *cobra.Command, args []string) error {
func generateAlphabetCreds(cmd *cobra.Command, _ []string) error {
// alphabet size is not part of the config
size, err := cmd.Flags().GetUint(alphabetSizeFlag)
if err != nil {
@ -37,18 +39,21 @@ func generateAlphabetCreds(cmd *cobra.Command, args []string) error {
if size == 0 {
return errors.New("size must be > 0")
}
if size > maxAlphabetNodes {
return ErrTooManyAlphabetNodes
}
v := viper.GetViper()
walletDir := config.ResolveHomePath(viper.GetString(alphabetWalletsFlag))
pwds, err := initializeWallets(walletDir, int(size))
pwds, err := initializeWallets(v, walletDir, int(size))
if err != nil {
return err
}
w, err := initializeContractWallet(walletDir)
_, err = initializeContractWallet(v, walletDir)
if err != nil {
return err
}
w.Close()
cmd.Println("size:", size)
cmd.Println("alphabet-wallets:", walletDir)
@ -59,13 +64,13 @@ func generateAlphabetCreds(cmd *cobra.Command, args []string) error {
return nil
}
func initializeWallets(walletDir string, size int) ([]string, error) {
func initializeWallets(v *viper.Viper, walletDir string, size int) ([]string, error) {
wallets := make([]*wallet.Wallet, size)
pubs := make(keys.PublicKeys, size)
passwords := make([]string, size)
for i := range wallets {
password, err := config.AlphabetPassword(viper.GetViper(), i)
password, err := config.GetPassword(v, innerring.GlagoliticLetter(i).String())
if err != nil {
return nil, fmt.Errorf("can't fetch password: %w", err)
}
@ -91,29 +96,32 @@ func initializeWallets(walletDir string, size int) ([]string, error) {
pubs[i] = w.Accounts[0].PrivateKey().PublicKey()
}
var errG errgroup.Group
// Create committee account with N/2+1 multi-signature.
majCount := smartcontract.GetMajorityHonestNodeCount(size)
for i, w := range wallets {
if err := addMultisigAccount(w, majCount, committeeAccountName, passwords[i], pubs); err != nil {
return nil, fmt.Errorf("can't create committee account: %w", err)
}
}
// Create consensus account with 2*N/3+1 multi-signature.
bftCount := smartcontract.GetDefaultHonestNodeCount(size)
for i, w := range wallets {
if err := addMultisigAccount(w, bftCount, consensusAccountName, passwords[i], pubs); err != nil {
return nil, fmt.Errorf("can't create consensus account: %w", err)
}
for i := range wallets {
i := i
ps := make(keys.PublicKeys, len(pubs))
copy(ps, pubs)
errG.Go(func() error {
if err := addMultisigAccount(wallets[i], majCount, committeeAccountName, passwords[i], ps); err != nil {
return fmt.Errorf("can't create committee account: %w", err)
}
if err := addMultisigAccount(wallets[i], bftCount, consensusAccountName, passwords[i], ps); err != nil {
return fmt.Errorf("can't create consentus account: %w", err)
}
if err := wallets[i].SavePretty(); err != nil {
return fmt.Errorf("can't save wallet: %w", err)
}
return nil
})
}
for _, w := range wallets {
if err := w.Save(); err != nil {
return nil, fmt.Errorf("can't save wallet: %w", err)
}
w.Close()
if err := errG.Wait(); err != nil {
return nil, err
}
return passwords, nil
}
@ -135,37 +143,55 @@ func generateStorageCreds(cmd *cobra.Command, _ []string) error {
return refillGas(cmd, storageGasConfigFlag, true)
}
func refillGas(cmd *cobra.Command, gasFlag string, createWallet bool) error {
func refillGas(cmd *cobra.Command, gasFlag string, createWallet bool) (err error) {
// storage wallet path is not part of the config
storageWalletPath, err := cmd.Flags().GetString(storageWalletFlag)
if err != nil {
return err
}
if storageWalletPath == "" {
return fmt.Errorf("missing wallet path (use '--%s <out.json>')", storageWalletFlag)
}
storageWalletPath, _ := cmd.Flags().GetString(storageWalletFlag)
// wallet address is not part of the config
walletAddress, _ := cmd.Flags().GetString(walletAddressFlag)
var w *wallet.Wallet
var gasReceiver util.Uint160
if createWallet {
w, err = wallet.NewWallet(storageWalletPath)
} else {
w, err = wallet.NewWalletFromFile(storageWalletPath)
}
if err != nil {
return fmt.Errorf("can't create wallet: %w", err)
}
if createWallet {
password, err := input.ReadPassword("New password > ")
if len(walletAddress) != 0 {
gasReceiver, err = address.StringToUint160(walletAddress)
if err != nil {
return fmt.Errorf("can't fetch password: %w", err)
return fmt.Errorf("invalid wallet address %s: %w", walletAddress, err)
}
} else {
if storageWalletPath == "" {
return fmt.Errorf("missing wallet path (use '--%s <out.json>')", storageWalletFlag)
}
if err := w.CreateAccount(singleAccountName, password); err != nil {
return fmt.Errorf("can't create account: %w", err)
var w *wallet.Wallet
if createWallet {
w, err = wallet.NewWallet(storageWalletPath)
} else {
w, err = wallet.NewWalletFromFile(storageWalletPath)
}
if err != nil {
return fmt.Errorf("can't create wallet: %w", err)
}
if createWallet {
var password string
label, _ := cmd.Flags().GetString(storageWalletLabelFlag)
password, err := config.GetStoragePassword(viper.GetViper(), label)
if err != nil {
return fmt.Errorf("can't fetch password: %w", err)
}
if label == "" {
label = singleAccountName
}
if err := w.CreateAccount(label, password); err != nil {
return fmt.Errorf("can't create account: %w", err)
}
}
gasReceiver = w.Accounts[0].Contract.ScriptHash()
}
gasStr := viper.GetString(gasFlag)
@ -180,17 +206,15 @@ func refillGas(cmd *cobra.Command, gasFlag string, createWallet bool) error {
return err
}
gasHash := wCtx.nativeHash(nativenames.Gas)
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, gasHash, "transfer", callflag.All,
wCtx.CommitteeAcc.Contract.ScriptHash(), w.Accounts[0].Contract.ScriptHash(), int64(gasAmount), nil)
emit.AppCall(bw.BinWriter, gas.Hash, "transfer", callflag.All,
wCtx.CommitteeAcc.Contract.ScriptHash(), gasReceiver, int64(gasAmount), nil)
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if bw.Err != nil {
return fmt.Errorf("BUG: invalid transfer arguments: %w", bw.Err)
}
if err := wCtx.sendCommitteeTx(bw.Bytes(), -1, false); err != nil {
if err := wCtx.sendCommitteeTx(bw.Bytes(), false); err != nil {
return err
}

View file

@ -7,21 +7,25 @@ import (
"os"
"path/filepath"
"strconv"
"sync"
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/nspcc-dev/neofs-node/pkg/innerring"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
"golang.org/x/term"
)
const testContractPassword = "grouppass"
func TestGenerateAlphabet(t *testing.T) {
const size = 4
walletDir := newTempDir(t)
walletDir := t.TempDir()
buf := setupTestTerminal(t)
cmd := generateAlphabetCmd
@ -60,40 +64,46 @@ func TestGenerateAlphabet(t *testing.T) {
buf.Reset()
v.Set(alphabetWalletsFlag, walletDir)
require.NoError(t, cmd.Flags().Set(alphabetSizeFlag, strconv.FormatUint(size, 10)))
require.NoError(t, generateAlphabetCmd.Flags().Set(alphabetSizeFlag, strconv.FormatUint(size, 10)))
for i := uint64(0); i < size; i++ {
buf.WriteString(strconv.FormatUint(i, 10) + "\r")
}
const groupPassword = "grouppass"
buf.WriteString(groupPassword + "\r")
require.NoError(t, generateAlphabetCreds(cmd, nil))
buf.WriteString(testContractPassword + "\r")
require.NoError(t, generateAlphabetCreds(generateAlphabetCmd, nil))
var wg sync.WaitGroup
for i := uint64(0); i < size; i++ {
p := filepath.Join(walletDir, innerring.GlagoliticLetter(i).String()+".json")
w, err := wallet.NewWalletFromFile(p)
require.NoError(t, err, "wallet doesn't exist")
require.Equal(t, 3, len(w.Accounts), "not all accounts were created")
for _, a := range w.Accounts {
err := a.Decrypt(strconv.FormatUint(i, 10), keys.NEP2ScryptParams())
require.NoError(t, err, "can't decrypt account")
switch a.Label {
case consensusAccountName:
require.Equal(t, size/2+1, len(a.Contract.Parameters))
case committeeAccountName:
require.Equal(t, size*2/3+1, len(a.Contract.Parameters))
default:
require.Equal(t, singleAccountName, a.Label)
i := i
go func() {
defer wg.Done()
p := filepath.Join(walletDir, innerring.GlagoliticLetter(i).String()+".json")
w, err := wallet.NewWalletFromFile(p)
require.NoError(t, err, "wallet doesn't exist")
require.Equal(t, 3, len(w.Accounts), "not all accounts were created")
for _, a := range w.Accounts {
err := a.Decrypt(strconv.FormatUint(i, 10), keys.NEP2ScryptParams())
require.NoError(t, err, "can't decrypt account")
switch a.Label {
case consensusAccountName:
require.Equal(t, smartcontract.GetDefaultHonestNodeCount(size), len(a.Contract.Parameters))
case committeeAccountName:
require.Equal(t, smartcontract.GetMajorityHonestNodeCount(size), len(a.Contract.Parameters))
default:
require.Equal(t, singleAccountName, a.Label)
}
}
}
}()
}
wg.Wait()
t.Run("check contract group wallet", func(t *testing.T) {
p := filepath.Join(walletDir, contractWalletFilename)
w, err := wallet.NewWalletFromFile(p)
require.NoError(t, err, "contract wallet doesn't exist")
require.Equal(t, 1, len(w.Accounts), "contract wallet must have 1 accout")
require.NoError(t, w.Accounts[0].Decrypt(groupPassword, keys.NEP2ScryptParams()))
require.NoError(t, w.Accounts[0].Decrypt(testContractPassword, keys.NEP2ScryptParams()))
})
}
@ -108,12 +118,3 @@ func setupTestTerminal(t *testing.T) *bytes.Buffer {
return in
}
func newTempDir(t *testing.T) string {
dir := filepath.Join(os.TempDir(), "neofs-adm.test."+strconv.FormatUint(rand.Uint64(), 10))
require.NoError(t, os.Mkdir(dir, os.ModePerm))
t.Cleanup(func() {
require.NoError(t, os.RemoveAll(dir))
})
return dir
}

View file

@ -4,10 +4,9 @@ import (
"encoding/json"
"fmt"
"os"
"path"
"path/filepath"
"github.com/nspcc-dev/neo-go/cli/input"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/manifest"
"github.com/nspcc-dev/neo-go/pkg/util"
@ -18,16 +17,16 @@ import (
const (
contractWalletFilename = "contract.json"
contractWalletPasswordKey = "credentials.contract"
contractWalletPasswordKey = "contract"
)
func initializeContractWallet(walletDir string) (*wallet.Wallet, error) {
password, err := getPassword(contractWalletPasswordKey, "Password for contract wallet > ")
func initializeContractWallet(v *viper.Viper, walletDir string) (*wallet.Wallet, error) {
password, err := config.GetPassword(v, contractWalletPasswordKey)
if err != nil {
return nil, err
}
w, err := wallet.NewWallet(path.Join(walletDir, contractWalletFilename))
w, err := wallet.NewWallet(filepath.Join(walletDir, contractWalletFilename))
if err != nil {
return nil, err
}
@ -43,30 +42,26 @@ func initializeContractWallet(walletDir string) (*wallet.Wallet, error) {
}
w.AddAccount(acc)
if err := w.Save(); err != nil {
if err := w.SavePretty(); err != nil {
return nil, err
}
return w, nil
}
func openContractWallet(cmd *cobra.Command, walletDir string) (*wallet.Wallet, error) {
p := path.Join(walletDir, contractWalletFilename)
func openContractWallet(v *viper.Viper, cmd *cobra.Command, walletDir string) (*wallet.Wallet, error) {
p := filepath.Join(walletDir, contractWalletFilename)
w, err := wallet.NewWalletFromFile(p)
if err != nil {
if !os.IsNotExist(err) {
return nil, fmt.Errorf("can't open wallet: %w", err)
}
cmd.Printf("Contract group wallet is missing, initialize at %s\n",
filepath.Join(walletDir, contractWalletFilename))
w, err = initializeContractWallet(walletDir)
if err != nil {
return nil, err
}
cmd.Printf("Contract group wallet is missing, initialize at %s\n", p)
return initializeContractWallet(v, walletDir)
}
password, err := getPassword(contractWalletPasswordKey, "Password for contract wallet > ")
password, err := config.GetPassword(v, contractWalletPasswordKey)
if err != nil {
return nil, err
}
@ -80,15 +75,6 @@ func openContractWallet(cmd *cobra.Command, walletDir string) (*wallet.Wallet, e
return w, nil
}
func getPassword(key string, promps string) (string, error) {
if viper.IsSet(key) {
return viper.GetString(key), nil
}
prompt := "Password for contract wallet > "
return input.ReadPassword(prompt)
}
func (c *initializeContext) addManifestGroup(h util.Uint160, cs *contractState) error {
priv := c.ContractWallet.Accounts[0].PrivateKey()
pub := priv.PublicKey()

View file

@ -0,0 +1,514 @@
package morph
import (
"errors"
"fmt"
"os"
"path/filepath"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
morphClient "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/trigger"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
// maxAlphabetNodes is the maximum number of candidates allowed, which is currently limited by the size
// of the invocation script.
// See: https://github.com/nspcc-dev/neo-go/blob/740488f7f35e367eaa99a71c0a609c315fe2b0fc/pkg/core/transaction/witness.go#L10
maxAlphabetNodes = 22
)
type cache struct {
nnsCs *state.Contract
groupKey *keys.PublicKey
}
type initializeContext struct {
clientContext
cache
// CommitteeAcc is used for retrieving the committee address and the verification script.
CommitteeAcc *wallet.Account
// ConsensusAcc is used for retrieving the committee address and the verification script.
ConsensusAcc *wallet.Account
Wallets []*wallet.Wallet
// ContractWallet is a wallet for providing the contract group signature.
ContractWallet *wallet.Wallet
// Accounts contains simple signature accounts in the same order as in Wallets.
Accounts []*wallet.Account
Contracts map[string]*contractState
Command *cobra.Command
ContractPath string
}
var ErrTooManyAlphabetNodes = fmt.Errorf("too many alphabet nodes (maximum allowed is %d)", maxAlphabetNodes)
func initializeSideChainCmd(cmd *cobra.Command, _ []string) error {
initCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return fmt.Errorf("initialization error: %w", err)
}
defer initCtx.close()
// 1. Transfer funds to committee accounts.
cmd.Println("Stage 1: transfer GAS to alphabet nodes.")
if err := initCtx.transferFunds(); err != nil {
return err
}
cmd.Println("Stage 2: set notary and alphabet nodes in designate contract.")
if err := initCtx.setNotaryAndAlphabetNodes(); err != nil {
return err
}
// 3. Deploy NNS contract.
cmd.Println("Stage 3: deploy NNS contract.")
if err := initCtx.deployNNS(deployMethodName); err != nil {
return err
}
// 4. Deploy NeoFS contracts.
cmd.Println("Stage 4: deploy NeoFS contracts.")
if err := initCtx.deployContracts(); err != nil {
return err
}
cmd.Println("Stage 4.1: Transfer GAS to proxy contract.")
if err := initCtx.transferGASToProxy(); err != nil {
return err
}
cmd.Println("Stage 5: register candidates.")
if err := initCtx.registerCandidates(); err != nil {
return err
}
cmd.Println("Stage 6: transfer NEO to alphabet contracts.")
if err := initCtx.transferNEOToAlphabetContracts(); err != nil {
return err
}
cmd.Println("Stage 7: set addresses in NNS.")
return initCtx.setNNS()
}
func (c *initializeContext) close() {
if local, ok := c.Client.(*localClient); ok {
err := local.dump()
if err != nil {
c.Command.PrintErrf("Can't write dump: %v\n", err)
os.Exit(1)
}
}
}
func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContext, error) {
walletDir := config.ResolveHomePath(viper.GetString(alphabetWalletsFlag))
wallets, err := openAlphabetWallets(v, walletDir)
if err != nil {
return nil, err
}
if len(wallets) > maxAlphabetNodes {
return nil, ErrTooManyAlphabetNodes
}
needContracts := cmd.Name() == "update-contracts" || cmd.Name() == "init"
var w *wallet.Wallet
w, err = getWallet(cmd, v, needContracts, walletDir)
if err != nil {
return nil, err
}
c, err := createClient(cmd, v, wallets)
if err != nil {
return nil, err
}
committeeAcc, err := getWalletAccount(wallets[0], committeeAccountName)
if err != nil {
return nil, fmt.Errorf("can't find committee account: %w", err)
}
consensusAcc, err := getWalletAccount(wallets[0], consensusAccountName)
if err != nil {
return nil, fmt.Errorf("can't find consensus account: %w", err)
}
if err := validateInit(cmd); err != nil {
return nil, err
}
ctrPath, err := getContractsPath(cmd, needContracts)
if err != nil {
return nil, err
}
if err := checkNotaryEnabled(c); err != nil {
return nil, err
}
accounts, err := createWalletAccounts(wallets)
if err != nil {
return nil, err
}
cliCtx, err := defaultClientContext(c, committeeAcc)
if err != nil {
return nil, fmt.Errorf("client context: %w", err)
}
initCtx := &initializeContext{
clientContext: *cliCtx,
ConsensusAcc: consensusAcc,
CommitteeAcc: committeeAcc,
ContractWallet: w,
Wallets: wallets,
Accounts: accounts,
Command: cmd,
Contracts: make(map[string]*contractState),
ContractPath: ctrPath,
}
if needContracts {
err := initCtx.readContracts(fullContractList)
if err != nil {
return nil, err
}
}
return initCtx, nil
}
func validateInit(cmd *cobra.Command) error {
if cmd.Name() != "init" {
return nil
}
if viper.GetInt64(epochDurationInitFlag) <= 0 {
return fmt.Errorf("epoch duration must be positive")
}
if viper.GetInt64(maxObjectSizeInitFlag) <= 0 {
return fmt.Errorf("max object size must be positive")
}
return nil
}
func createClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet) (Client, error) {
var c Client
var err error
if v.GetString(localDumpFlag) != "" {
if v.GetString(endpointFlag) != "" {
return nil, fmt.Errorf("`%s` and `%s` flags are mutually exclusive", endpointFlag, localDumpFlag)
}
c, err = newLocalClient(cmd, v, wallets)
} else {
c, err = getN3Client(v)
}
if err != nil {
return nil, fmt.Errorf("can't create N3 client: %w", err)
}
return c, nil
}
func getWallet(cmd *cobra.Command, v *viper.Viper, needContracts bool, walletDir string) (*wallet.Wallet, error) {
if !needContracts {
return nil, nil
}
return openContractWallet(v, cmd, walletDir)
}
func getContractsPath(cmd *cobra.Command, needContracts bool) (string, error) {
if !needContracts {
return "", nil
}
ctrPath, err := cmd.Flags().GetString(contractsInitFlag)
if err != nil {
return "", fmt.Errorf("invalid contracts path: %w", err)
}
return ctrPath, nil
}
func createWalletAccounts(wallets []*wallet.Wallet) ([]*wallet.Account, error) {
accounts := make([]*wallet.Account, len(wallets))
for i, w := range wallets {
acc, err := getWalletAccount(w, singleAccountName)
if err != nil {
return nil, fmt.Errorf("wallet %s is invalid (no single account): %w", w.Path(), err)
}
accounts[i] = acc
}
return accounts, nil
}
func openAlphabetWallets(v *viper.Viper, walletDir string) ([]*wallet.Wallet, error) {
walletFiles, err := os.ReadDir(walletDir)
if err != nil {
return nil, fmt.Errorf("can't read alphabet wallets dir: %w", err)
}
var size int
loop:
for i := 0; i < len(walletFiles); i++ {
name := innerring.GlagoliticLetter(i).String() + ".json"
for j := range walletFiles {
if walletFiles[j].Name() == name {
size++
continue loop
}
}
break
}
if size == 0 {
return nil, errors.New("alphabet wallets dir is empty (run `generate-alphabet` command first)")
}
wallets := make([]*wallet.Wallet, size)
for i := 0; i < size; i++ {
letter := innerring.GlagoliticLetter(i).String()
p := filepath.Join(walletDir, letter+".json")
w, err := wallet.NewWalletFromFile(p)
if err != nil {
return nil, fmt.Errorf("can't open wallet: %w", err)
}
password, err := config.GetPassword(v, letter)
if err != nil {
return nil, fmt.Errorf("can't fetch password: %w", err)
}
for i := range w.Accounts {
if err := w.Accounts[i].Decrypt(password, keys.NEP2ScryptParams()); err != nil {
return nil, fmt.Errorf("can't unlock wallet: %w", err)
}
}
wallets[i] = w
}
return wallets, nil
}
func (c *initializeContext) awaitTx() error {
return c.clientContext.awaitTx(c.Command)
}
func (c *initializeContext) nnsContractState() (*state.Contract, error) {
if c.nnsCs != nil {
return c.nnsCs, nil
}
cs, err := c.Client.GetContractStateByID(1)
if err != nil {
return nil, err
}
c.nnsCs = cs
return cs, nil
}
func (c *initializeContext) getSigner(tryGroup bool, acc *wallet.Account) transaction.Signer {
if tryGroup && c.groupKey != nil {
return transaction.Signer{
Account: acc.Contract.ScriptHash(),
Scopes: transaction.CustomGroups,
AllowedGroups: keys.PublicKeys{c.groupKey},
}
}
signer := transaction.Signer{
Account: acc.Contract.ScriptHash(),
Scopes: transaction.Global, // Scope is important, as we have nested call to container contract.
}
if !tryGroup {
return signer
}
nnsCs, err := c.nnsContractState()
if err != nil {
return signer
}
groupKey, err := nnsResolveKey(c.ReadOnlyInvoker, nnsCs.Hash, morphClient.NNSGroupKeyName)
if err == nil {
c.groupKey = groupKey
signer.Scopes = transaction.CustomGroups
signer.AllowedGroups = keys.PublicKeys{groupKey}
}
return signer
}
func (c *clientContext) awaitTx(cmd *cobra.Command) error {
if len(c.SentTxs) == 0 {
return nil
}
if local, ok := c.Client.(*localClient); ok {
if err := local.putTransactions(); err != nil {
return fmt.Errorf("can't persist transactions: %w", err)
}
}
err := awaitTx(cmd, c.Client, c.SentTxs)
c.SentTxs = c.SentTxs[:0]
return err
}
func awaitTx(cmd *cobra.Command, c Client, txs []hashVUBPair) error {
cmd.Println("Waiting for transactions to persist...")
const pollInterval = time.Second
tick := time.NewTicker(pollInterval)
defer tick.Stop()
at := trigger.Application
var retErr error
currBlock, err := c.GetBlockCount()
if err != nil {
return fmt.Errorf("can't fetch current block height: %w", err)
}
loop:
for i := range txs {
res, err := c.GetApplicationLog(txs[i].hash, &at)
if err == nil {
if retErr == nil && len(res.Executions) > 0 && res.Executions[0].VMState != vmstate.Halt {
retErr = fmt.Errorf("tx %d persisted in %s state: %s",
i, res.Executions[0].VMState, res.Executions[0].FaultException)
}
continue loop
}
if txs[i].vub < currBlock {
return fmt.Errorf("tx was not persisted: vub=%d, height=%d", txs[i].vub, currBlock)
}
for range tick.C {
// We must fetch current height before application log, to avoid race condition.
currBlock, err = c.GetBlockCount()
if err != nil {
return fmt.Errorf("can't fetch current block height: %w", err)
}
res, err := c.GetApplicationLog(txs[i].hash, &at)
if err == nil {
if retErr == nil && len(res.Executions) > 0 && res.Executions[0].VMState != vmstate.Halt {
retErr = fmt.Errorf("tx %d persisted in %s state: %s",
i, res.Executions[0].VMState, res.Executions[0].FaultException)
}
continue loop
}
if txs[i].vub < currBlock {
return fmt.Errorf("tx was not persisted: vub=%d, height=%d", txs[i].vub, currBlock)
}
}
}
return retErr
}
// sendCommitteeTx creates transaction from script, signs it by committee nodes and sends it to RPC.
// If tryGroup is false, global scope is used for the signer (useful when
// working with native contracts).
func (c *initializeContext) sendCommitteeTx(script []byte, tryGroup bool) error {
return c.sendMultiTx(script, tryGroup, false)
}
// sendConsensusTx creates transaction from script, signs it by alphabet nodes and sends it to RPC.
// Not that because this is used only after the contracts were initialized and deployed,
// we always try to have a group scope.
func (c *initializeContext) sendConsensusTx(script []byte) error {
return c.sendMultiTx(script, true, true)
}
func (c *initializeContext) sendMultiTx(script []byte, tryGroup bool, withConsensus bool) error {
var act *actor.Actor
var err error
withConsensus = withConsensus && !c.ConsensusAcc.Contract.ScriptHash().Equals(c.CommitteeAcc.ScriptHash())
if tryGroup {
// Even for consensus signatures we need the committee to pay.
signers := make([]actor.SignerAccount, 1, 2)
signers[0] = actor.SignerAccount{
Signer: c.getSigner(tryGroup, c.CommitteeAcc),
Account: c.CommitteeAcc,
}
if withConsensus {
signers = append(signers, actor.SignerAccount{
Signer: c.getSigner(tryGroup, c.ConsensusAcc),
Account: c.ConsensusAcc,
})
}
act, err = actor.New(c.Client, signers)
} else {
if withConsensus {
panic("BUG: should never happen")
}
act, err = c.CommitteeAct, nil
}
if err != nil {
return fmt.Errorf("could not create actor: %w", err)
}
tx, err := act.MakeUnsignedRun(script, []transaction.Attribute{{Type: transaction.HighPriority}})
if err != nil {
return fmt.Errorf("could not perform test invocation: %w", err)
}
if err := c.multiSign(tx, committeeAccountName); err != nil {
return err
}
if withConsensus {
if err := c.multiSign(tx, consensusAccountName); err != nil {
return err
}
}
return c.sendTx(tx, c.Command, false)
}
func getWalletAccount(w *wallet.Wallet, typ string) (*wallet.Account, error) {
for i := range w.Accounts {
if w.Accounts[i].Label == typ {
return w.Accounts[i], nil
}
}
return nil, fmt.Errorf("account for '%s' not found", typ)
}
func checkNotaryEnabled(c Client) error {
ns, err := c.GetNativeContracts()
if err != nil {
return fmt.Errorf("can't get native contract hashes: %w", err)
}
notaryEnabled := false
nativeHashes := make(map[string]util.Uint160, len(ns))
for i := range ns {
if ns[i].Manifest.Name == nativenames.Notary {
notaryEnabled = len(ns[i].UpdateHistory) > 0
}
nativeHashes[ns[i].Manifest.Name] = ns[i].Hash
}
if !notaryEnabled {
return errors.New("notary contract must be enabled")
}
return nil
}

View file

@ -0,0 +1,623 @@
package morph
import (
"archive/tar"
"compress/gzip"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-contract/common"
"git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
morphClient "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
io2 "github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/management"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/manifest"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/nef"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
)
const (
nnsContract = "nns"
frostfsContract = "frostfs" // not deployed in side-chain.
processingContract = "processing" // not deployed in side-chain.
alphabetContract = "alphabet"
auditContract = "audit"
balanceContract = "balance"
containerContract = "container"
frostfsIDContract = "frostfsid"
netmapContract = "netmap"
proxyContract = "proxy"
)
var (
contractList = []string{
auditContract,
balanceContract,
containerContract,
frostfsIDContract,
netmapContract,
proxyContract,
}
fullContractList = append([]string{
frostfsContract,
processingContract,
nnsContract,
alphabetContract,
}, contractList...)
netmapConfigKeys = []string{
netmap.EpochDurationConfig,
netmap.MaxObjectSizeConfig,
netmap.AuditFeeConfig,
netmap.ContainerFeeConfig,
netmap.ContainerAliasFeeConfig,
netmap.BasicIncomeRateConfig,
netmap.IrCandidateFeeConfig,
netmap.WithdrawFeeConfig,
netmap.HomomorphicHashingDisabledKey,
netmap.MaintenanceModeAllowedConfig,
}
)
type contractState struct {
NEF *nef.File
RawNEF []byte
Manifest *manifest.Manifest
RawManifest []byte
Hash util.Uint160
}
const (
updateMethodName = "update"
deployMethodName = "deploy"
)
func (c *initializeContext) deployNNS(method string) error {
cs := c.getContract(nnsContract)
h := cs.Hash
nnsCs, err := c.nnsContractState()
if err == nil {
if nnsCs.NEF.Checksum == cs.NEF.Checksum {
if method == deployMethodName {
c.Command.Println("NNS contract is already deployed.")
} else {
c.Command.Println("NNS contract is already updated.")
}
return nil
}
h = nnsCs.Hash
}
err = c.addManifestGroup(h, cs)
if err != nil {
return fmt.Errorf("can't sign manifest group: %v", err)
}
params := getContractDeployParameters(cs, nil)
signer := transaction.Signer{
Account: c.CommitteeAcc.Contract.ScriptHash(),
Scopes: transaction.CalledByEntry,
}
invokeHash := management.Hash
if method == updateMethodName {
invokeHash = nnsCs.Hash
}
res, err := invokeFunction(c.Client, invokeHash, method, params, []transaction.Signer{signer})
if err != nil {
return fmt.Errorf("can't deploy NNS contract: %w", err)
}
if res.State != vmstate.Halt.String() {
return fmt.Errorf("can't deploy NNS contract: %s", res.FaultException)
}
tx, err := c.Client.CreateTxFromScript(res.Script, c.CommitteeAcc, res.GasConsumed, 0, []rpcclient.SignerAccount{{
Signer: signer,
Account: c.CommitteeAcc,
}})
if err != nil {
return fmt.Errorf("failed to create deploy tx for %s: %w", nnsContract, err)
}
if err := c.multiSignAndSend(tx, committeeAccountName); err != nil {
return fmt.Errorf("can't send deploy transaction: %w", err)
}
return c.awaitTx()
}
func (c *initializeContext) updateContracts() error {
alphaCs := c.getContract(alphabetContract)
nnsCs, err := c.nnsContractState()
if err != nil {
return err
}
nnsHash := nnsCs.Hash
w := io2.NewBufBinWriter()
// Update script size for a single-node committee is close to the maximum allowed size of 65535.
// Because of this we want to reuse alphabet contract NEF and manifest for different updates.
// The generated script is as following.
// 1. Initialize static slot for alphabet NEF.
// 2. Store NEF into the static slot.
// 3. Push parameters for each alphabet contract on stack.
// 4. Add contract group to the manifest.
// 5. For each alphabet contract, invoke `update` using parameters on stack and
// NEF from step 2 and manifest from step 4.
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
emit.Bytes(w.BinWriter, alphaCs.RawNEF)
emit.Opcodes(w.BinWriter, opcode.STSFLD0)
keysParam, err := c.deployAlphabetAccounts(nnsHash, w, alphaCs)
if err != nil {
return err
}
w.Reset()
if err = c.deployOrUpdateContracts(w, nnsHash, keysParam); err != nil {
return err
}
groupKey := c.ContractWallet.Accounts[0].PrivateKey().PublicKey()
_, _, err = c.emitUpdateNNSGroupScript(w, nnsHash, groupKey)
if err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", morphClient.NNSGroupKeyName, hex.EncodeToString(groupKey.Bytes()))
emit.Opcodes(w.BinWriter, opcode.LDSFLD0)
emit.Int(w.BinWriter, 1)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsHash, "setPrice", callflag.All)
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
return err
}
return c.awaitTx()
}
func (c *initializeContext) deployOrUpdateContracts(w *io2.BufBinWriter, nnsHash util.Uint160, keysParam []any) error {
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
emit.AppCall(w.BinWriter, nnsHash, "getPrice", callflag.All)
emit.Opcodes(w.BinWriter, opcode.STSFLD0)
emit.AppCall(w.BinWriter, nnsHash, "setPrice", callflag.All, 1)
for _, ctrName := range contractList {
cs := c.getContract(ctrName)
method := updateMethodName
ctrHash, err := nnsResolveHash(c.ReadOnlyInvoker, nnsHash, ctrName+".frostfs")
if err != nil {
if errors.Is(err, errMissingNNSRecord) {
// if contract not found we deploy it instead of update
method = deployMethodName
} else {
return fmt.Errorf("can't resolve hash for contract update: %w", err)
}
}
err = c.addManifestGroup(ctrHash, cs)
if err != nil {
return fmt.Errorf("can't sign manifest group: %v", err)
}
invokeHash := management.Hash
if method == updateMethodName {
invokeHash = ctrHash
}
params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam, updateMethodName))
res, err := c.CommitteeAct.MakeCall(invokeHash, method, params...)
if err != nil {
if method != updateMethodName || !strings.Contains(err.Error(), common.ErrAlreadyUpdated) {
return fmt.Errorf("deploy contract: %w", err)
}
c.Command.Printf("%s contract is already updated.\n", ctrName)
continue
}
w.WriteBytes(res.Script)
if method == deployMethodName {
// same actions are done in initializeContext.setNNS, can be unified
domain := ctrName + ".frostfs"
script, ok, err := c.nnsRegisterDomainScript(nnsHash, cs.Hash, domain)
if err != nil {
return err
}
if !ok {
w.WriteBytes(script)
emit.AppCall(w.BinWriter, nnsHash, "deleteRecords", callflag.All, domain, int64(nns.TXT))
emit.AppCall(w.BinWriter, nnsHash, "addRecord", callflag.All,
domain, int64(nns.TXT), cs.Hash.StringLE())
emit.AppCall(w.BinWriter, nnsHash, "addRecord", callflag.All,
domain, int64(nns.TXT), address.Uint160ToString(cs.Hash))
}
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
}
}
return nil
}
func (c *initializeContext) deployAlphabetAccounts(nnsHash util.Uint160, w *io2.BufBinWriter, alphaCs *contractState) ([]any, error) {
var keysParam []any
baseGroups := alphaCs.Manifest.Groups
// alphabet contracts should be deployed by individual nodes to get different hashes.
for i, acc := range c.Accounts {
ctrHash, err := nnsResolveHash(c.ReadOnlyInvoker, nnsHash, getAlphabetNNSDomain(i))
if err != nil {
return nil, fmt.Errorf("can't resolve hash for contract update: %w", err)
}
keysParam = append(keysParam, acc.PrivateKey().PublicKey().Bytes())
params := c.getAlphabetDeployItems(i, len(c.Wallets))
emit.Array(w.BinWriter, params...)
alphaCs.Manifest.Groups = baseGroups
err = c.addManifestGroup(ctrHash, alphaCs)
if err != nil {
return nil, fmt.Errorf("can't sign manifest group: %v", err)
}
emit.Bytes(w.BinWriter, alphaCs.RawManifest)
emit.Opcodes(w.BinWriter, opcode.LDSFLD0)
emit.Int(w.BinWriter, 3)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, ctrHash, updateMethodName, callflag.All)
}
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
if !strings.Contains(err.Error(), common.ErrAlreadyUpdated) {
return nil, err
}
c.Command.Println("Alphabet contracts are already updated.")
}
return keysParam, nil
}
func (c *initializeContext) deployContracts() error {
alphaCs := c.getContract(alphabetContract)
var keysParam []any
baseGroups := alphaCs.Manifest.Groups
// alphabet contracts should be deployed by individual nodes to get different hashes.
for i, acc := range c.Accounts {
ctrHash := state.CreateContractHash(acc.Contract.ScriptHash(), alphaCs.NEF.Checksum, alphaCs.Manifest.Name)
if c.isUpdated(ctrHash, alphaCs) {
c.Command.Printf("Alphabet contract #%d is already deployed.\n", i)
continue
}
alphaCs.Manifest.Groups = baseGroups
err := c.addManifestGroup(ctrHash, alphaCs)
if err != nil {
return fmt.Errorf("can't sign manifest group: %v", err)
}
keysParam = append(keysParam, acc.PrivateKey().PublicKey().Bytes())
params := getContractDeployParameters(alphaCs, c.getAlphabetDeployItems(i, len(c.Wallets)))
act, err := actor.NewSimple(c.Client, acc)
if err != nil {
return fmt.Errorf("could not create actor: %w", err)
}
txHash, vub, err := act.SendCall(management.Hash, deployMethodName, params...)
if err != nil {
return fmt.Errorf("can't deploy alphabet #%d contract: %w", i, err)
}
c.SentTxs = append(c.SentTxs, hashVUBPair{hash: txHash, vub: vub})
}
for _, ctrName := range contractList {
cs := c.getContract(ctrName)
ctrHash := cs.Hash
if c.isUpdated(ctrHash, cs) {
c.Command.Printf("%s contract is already deployed.\n", ctrName)
continue
}
err := c.addManifestGroup(ctrHash, cs)
if err != nil {
return fmt.Errorf("can't sign manifest group: %v", err)
}
params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam, deployMethodName))
res, err := c.CommitteeAct.MakeCall(management.Hash, deployMethodName, params...)
if err != nil {
return fmt.Errorf("can't deploy %s contract: %w", ctrName, err)
}
if err := c.sendCommitteeTx(res.Script, false); err != nil {
return err
}
}
return c.awaitTx()
}
func (c *initializeContext) isUpdated(ctrHash util.Uint160, cs *contractState) bool {
realCs, err := c.Client.GetContractStateByHash(ctrHash)
return err == nil && realCs.NEF.Checksum == cs.NEF.Checksum
}
func (c *initializeContext) getContract(ctrName string) *contractState {
return c.Contracts[ctrName]
}
func (c *initializeContext) readContracts(names []string) error {
var (
fi os.FileInfo
err error
)
if c.ContractPath != "" {
fi, err = os.Stat(c.ContractPath)
if err != nil {
return fmt.Errorf("invalid contracts path: %w", err)
}
}
if c.ContractPath != "" && fi.IsDir() {
for _, ctrName := range names {
cs, err := readContract(filepath.Join(c.ContractPath, ctrName), ctrName)
if err != nil {
return err
}
c.Contracts[ctrName] = cs
}
} else {
var r io.ReadCloser
if c.ContractPath == "" {
return errors.New("contracts flag is missing")
}
r, err = os.Open(c.ContractPath)
if err != nil {
return fmt.Errorf("can't open contracts archive: %w", err)
}
defer r.Close()
m, err := readContractsFromArchive(r, names)
if err != nil {
return err
}
for _, name := range names {
if err := m[name].parse(); err != nil {
return err
}
c.Contracts[name] = m[name]
}
}
for _, ctrName := range names {
if ctrName != alphabetContract {
cs := c.Contracts[ctrName]
cs.Hash = state.CreateContractHash(c.CommitteeAcc.Contract.ScriptHash(),
cs.NEF.Checksum, cs.Manifest.Name)
}
}
return nil
}
func readContract(ctrPath, ctrName string) (*contractState, error) {
rawNef, err := os.ReadFile(filepath.Join(ctrPath, ctrName+"_contract.nef"))
if err != nil {
return nil, fmt.Errorf("can't read NEF file for %s contract: %w", ctrName, err)
}
rawManif, err := os.ReadFile(filepath.Join(ctrPath, "config.json"))
if err != nil {
return nil, fmt.Errorf("can't read manifest file for %s contract: %w", ctrName, err)
}
cs := &contractState{
RawNEF: rawNef,
RawManifest: rawManif,
}
return cs, cs.parse()
}
func (cs *contractState) parse() error {
nf, err := nef.FileFromBytes(cs.RawNEF)
if err != nil {
return fmt.Errorf("can't parse NEF file: %w", err)
}
m := new(manifest.Manifest)
if err := json.Unmarshal(cs.RawManifest, m); err != nil {
return fmt.Errorf("can't parse manifest file: %w", err)
}
cs.NEF = &nf
cs.Manifest = m
return nil
}
func readContractsFromArchive(file io.Reader, names []string) (map[string]*contractState, error) {
m := make(map[string]*contractState, len(names))
for i := range names {
m[names[i]] = new(contractState)
}
gr, err := gzip.NewReader(file)
if err != nil {
return nil, fmt.Errorf("contracts file must be tar.gz archive: %w", err)
}
r := tar.NewReader(gr)
for h, err := r.Next(); ; h, err = r.Next() {
if err != nil {
break
}
dir, _ := filepath.Split(h.Name)
ctrName := filepath.Base(dir)
cs, ok := m[ctrName]
if !ok {
continue
}
switch {
case strings.HasSuffix(h.Name, filepath.Join(ctrName, ctrName+"_contract.nef")):
cs.RawNEF, err = io.ReadAll(r)
if err != nil {
return nil, fmt.Errorf("can't read NEF file for %s contract: %w", ctrName, err)
}
case strings.HasSuffix(h.Name, "config.json"):
cs.RawManifest, err = io.ReadAll(r)
if err != nil {
return nil, fmt.Errorf("can't read manifest file for %s contract: %w", ctrName, err)
}
}
m[ctrName] = cs
}
for ctrName, cs := range m {
if cs.RawNEF == nil {
return nil, fmt.Errorf("NEF for %s contract wasn't found", ctrName)
}
if cs.RawManifest == nil {
return nil, fmt.Errorf("manifest for %s contract wasn't found", ctrName)
}
}
return m, nil
}
func getContractDeployParameters(cs *contractState, deployData []any) []any {
return []any{cs.RawNEF, cs.RawManifest, deployData}
}
func (c *initializeContext) getContractDeployData(ctrName string, keysParam []any, method string) []any {
items := make([]any, 1, 6)
items[0] = false // notaryDisabled is false
switch ctrName {
case frostfsContract:
items = append(items,
c.Contracts[processingContract].Hash,
keysParam,
smartcontract.Parameter{})
case processingContract:
items = append(items, c.Contracts[frostfsContract].Hash)
return items[1:] // no notary info
case auditContract:
items = append(items, c.Contracts[netmapContract].Hash)
case balanceContract:
items = append(items,
c.Contracts[netmapContract].Hash,
c.Contracts[containerContract].Hash)
case containerContract:
// In case if NNS is updated multiple times, we can't calculate
// it's actual hash based on local data, thus query chain.
nnsCs, err := c.Client.GetContractStateByID(1)
if err != nil {
panic("NNS is not yet deployed")
}
items = append(items,
c.Contracts[netmapContract].Hash,
c.Contracts[balanceContract].Hash,
c.Contracts[frostfsIDContract].Hash,
nnsCs.Hash,
"container")
case frostfsIDContract:
items = append(items,
c.Contracts[netmapContract].Hash,
c.Contracts[containerContract].Hash)
case netmapContract:
md := getDefaultNetmapContractConfigMap()
if method == updateMethodName {
arr, err := c.getNetConfigFromNetmapContract()
if err != nil {
panic(err)
}
m, err := parseConfigFromNetmapContract(arr)
if err != nil {
panic(err)
}
for k, v := range m {
for _, key := range netmapConfigKeys {
if k == key {
md[k] = v
break
}
}
}
}
var configParam []any
for k, v := range md {
configParam = append(configParam, k, v)
}
items = append(items,
c.Contracts[balanceContract].Hash,
c.Contracts[containerContract].Hash,
keysParam,
configParam)
case proxyContract:
items = nil
default:
panic(fmt.Sprintf("invalid contract name: %s", ctrName))
}
return items
}
func (c *initializeContext) getNetConfigFromNetmapContract() ([]stackitem.Item, error) {
cs, err := c.Client.GetContractStateByID(1)
if err != nil {
return nil, fmt.Errorf("NNS is not yet deployed: %w", err)
}
nmHash, err := nnsResolveHash(c.ReadOnlyInvoker, cs.Hash, netmapContract+".frostfs")
if err != nil {
return nil, fmt.Errorf("can't get netmap contract hash: %w", err)
}
arr, err := unwrap.Array(c.ReadOnlyInvoker.Call(nmHash, "listConfig"))
if err != nil {
return nil, fmt.Errorf("can't fetch list of network config keys from the netmap contract")
}
return arr, err
}
func (c *initializeContext) getAlphabetDeployItems(i, n int) []any {
items := make([]any, 6)
items[0] = false
items[1] = c.Contracts[netmapContract].Hash
items[2] = c.Contracts[proxyContract].Hash
items[3] = innerring.GlagoliticLetter(i).String()
items[4] = int64(i)
items[5] = int64(n)
return items
}

View file

@ -0,0 +1,301 @@
package morph
import (
"encoding/hex"
"errors"
"fmt"
"strconv"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
morphClient "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
nnsClient "github.com/nspcc-dev/neo-go/pkg/rpcclient/nns"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
)
const defaultExpirationTime = 10 * 365 * 24 * time.Hour / time.Second
func (c *initializeContext) setNNS() error {
nnsCs, err := c.Client.GetContractStateByID(1)
if err != nil {
return err
}
ok, err := c.nnsRootRegistered(nnsCs.Hash, "frostfs")
if err != nil {
return err
} else if !ok {
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
"frostfs", c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if err := c.sendCommitteeTx(bw.Bytes(), true); err != nil {
return fmt.Errorf("can't add domain root to NNS: %w", err)
}
if err := c.awaitTx(); err != nil {
return err
}
}
alphaCs := c.getContract(alphabetContract)
for i, acc := range c.Accounts {
alphaCs.Hash = state.CreateContractHash(acc.Contract.ScriptHash(), alphaCs.NEF.Checksum, alphaCs.Manifest.Name)
domain := getAlphabetNNSDomain(i)
if err := c.nnsRegisterDomain(nnsCs.Hash, alphaCs.Hash, domain); err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", domain, alphaCs.Hash.StringLE())
}
for _, ctrName := range contractList {
cs := c.getContract(ctrName)
domain := ctrName + ".frostfs"
if err := c.nnsRegisterDomain(nnsCs.Hash, cs.Hash, domain); err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
}
groupKey := c.ContractWallet.Accounts[0].PrivateKey().PublicKey()
err = c.updateNNSGroup(nnsCs.Hash, groupKey)
if err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", morphClient.NNSGroupKeyName, hex.EncodeToString(groupKey.Bytes()))
return c.awaitTx()
}
func (c *initializeContext) updateNNSGroup(nnsHash util.Uint160, pub *keys.PublicKey) error {
bw := io.NewBufBinWriter()
keyAlreadyAdded, domainRegCodeEmitted, err := c.emitUpdateNNSGroupScript(bw, nnsHash, pub)
if keyAlreadyAdded || err != nil {
return err
}
script := bw.Bytes()
if domainRegCodeEmitted {
w := io.NewBufBinWriter()
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
wrapRegisterScriptWithPrice(w, nnsHash, script)
script = w.Bytes()
}
return c.sendCommitteeTx(script, true)
}
// emitUpdateNNSGroupScript emits script for updating group key stored in NNS.
// First return value is true iff the key is already there and nothing should be done.
// Second return value is true iff a domain registration code was emitted.
func (c *initializeContext) emitUpdateNNSGroupScript(bw *io.BufBinWriter, nnsHash util.Uint160, pub *keys.PublicKey) (bool, bool, error) {
isAvail, err := nnsIsAvailable(c.Client, nnsHash, morphClient.NNSGroupKeyName)
if err != nil {
return false, false, err
}
if !isAvail {
currentPub, err := nnsResolveKey(c.ReadOnlyInvoker, nnsHash, morphClient.NNSGroupKeyName)
if err != nil {
return false, false, err
}
if pub.Equal(currentPub) {
return true, false, nil
}
}
if isAvail {
emit.AppCall(bw.BinWriter, nnsHash, "register", callflag.All,
morphClient.NNSGroupKeyName, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
}
emit.AppCall(bw.BinWriter, nnsHash, "deleteRecords", callflag.All, "group.frostfs", int64(nns.TXT))
emit.AppCall(bw.BinWriter, nnsHash, "addRecord", callflag.All,
"group.frostfs", int64(nns.TXT), hex.EncodeToString(pub.Bytes()))
return false, isAvail, nil
}
func getAlphabetNNSDomain(i int) string {
return alphabetContract + strconv.FormatUint(uint64(i), 10) + ".frostfs"
}
// wrapRegisterScriptWithPrice wraps a given script with `getPrice`/`setPrice` calls for NNS.
// It is intended to be used for a single transaction, and not as a part of other scripts.
// It is assumed that script already contains static slot initialization code, the first one
// (with index 0) is used to store the price.
func wrapRegisterScriptWithPrice(w *io.BufBinWriter, nnsHash util.Uint160, s []byte) {
if len(s) == 0 {
return
}
emit.AppCall(w.BinWriter, nnsHash, "getPrice", callflag.All)
emit.Opcodes(w.BinWriter, opcode.STSFLD0)
emit.AppCall(w.BinWriter, nnsHash, "setPrice", callflag.All, 1)
w.WriteBytes(s)
emit.Opcodes(w.BinWriter, opcode.LDSFLD0, opcode.PUSH1, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsHash, "setPrice", callflag.All)
if w.Err != nil {
panic(fmt.Errorf("BUG: can't wrap register script: %w", w.Err))
}
}
func (c *initializeContext) nnsRegisterDomainScript(nnsHash, expectedHash util.Uint160, domain string) ([]byte, bool, error) {
ok, err := nnsIsAvailable(c.Client, nnsHash, domain)
if err != nil {
return nil, false, err
}
if ok {
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, nnsHash, "register", callflag.All,
domain, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if bw.Err != nil {
panic(bw.Err)
}
return bw.Bytes(), false, nil
}
s, err := nnsResolveHash(c.ReadOnlyInvoker, nnsHash, domain)
if err != nil {
return nil, false, err
}
return nil, s == expectedHash, nil
}
func (c *initializeContext) nnsRegisterDomain(nnsHash, expectedHash util.Uint160, domain string) error {
script, ok, err := c.nnsRegisterDomainScript(nnsHash, expectedHash, domain)
if ok || err != nil {
return err
}
w := io.NewBufBinWriter()
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
wrapRegisterScriptWithPrice(w, nnsHash, script)
emit.AppCall(w.BinWriter, nnsHash, "deleteRecords", callflag.All, domain, int64(nns.TXT))
emit.AppCall(w.BinWriter, nnsHash, "addRecord", callflag.All,
domain, int64(nns.TXT), expectedHash.StringLE())
emit.AppCall(w.BinWriter, nnsHash, "addRecord", callflag.All,
domain, int64(nns.TXT), address.Uint160ToString(expectedHash))
return c.sendCommitteeTx(w.Bytes(), true)
}
func (c *initializeContext) nnsRootRegistered(nnsHash util.Uint160, zone string) (bool, error) {
res, err := c.CommitteeAct.Call(nnsHash, "isAvailable", "name."+zone)
if err != nil {
return false, err
}
return res.State == vmstate.Halt.String(), nil
}
var errMissingNNSRecord = errors.New("missing NNS record")
// Returns errMissingNNSRecord if invocation fault exception contains "token not found".
func nnsResolveHash(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (util.Uint160, error) {
item, err := nnsResolve(inv, nnsHash, domain)
if err != nil {
return util.Uint160{}, err
}
return parseNNSResolveResult(item)
}
func nnsResolve(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (stackitem.Item, error) {
return unwrap.Item(inv.Call(nnsHash, "resolve", domain, int64(nns.TXT)))
}
func nnsResolveKey(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (*keys.PublicKey, error) {
res, err := nnsResolve(inv, nnsHash, domain)
if err != nil {
return nil, err
}
if _, ok := res.Value().(stackitem.Null); ok {
return nil, errors.New("NNS record is missing")
}
arr, ok := res.Value().([]stackitem.Item)
if !ok {
return nil, errors.New("API of the NNS contract method `resolve` has changed")
}
for i := range arr {
var bs []byte
bs, err = arr[i].TryBytes()
if err != nil {
continue
}
return keys.NewPublicKeyFromString(string(bs))
}
return nil, errors.New("no valid keys are found")
}
// parseNNSResolveResult parses the result of resolving NNS record.
// It works with multiple formats (corresponding to multiple NNS versions).
// If array of hashes is provided, it returns only the first one.
func parseNNSResolveResult(res stackitem.Item) (util.Uint160, error) {
arr, ok := res.Value().([]stackitem.Item)
if !ok {
arr = []stackitem.Item{res}
}
if _, ok := res.Value().(stackitem.Null); ok || len(arr) == 0 {
return util.Uint160{}, errors.New("NNS record is missing")
}
for i := range arr {
bs, err := arr[i].TryBytes()
if err != nil {
continue
}
// We support several formats for hash encoding, this logic should be maintained in sync
// with nnsResolve from pkg/morph/client/nns.go
h, err := util.Uint160DecodeStringLE(string(bs))
if err == nil {
return h, nil
}
h, err = address.StringToUint160(string(bs))
if err == nil {
return h, nil
}
}
return util.Uint160{}, errors.New("no valid hashes are found")
}
func nnsIsAvailable(c Client, nnsHash util.Uint160, name string) (bool, error) {
switch c.(type) {
case *rpcclient.Client:
inv := invoker.New(c, nil)
reader := nnsClient.NewReader(inv, nnsHash)
return reader.IsAvailable(name)
default:
b, err := unwrap.Bool(invokeFunction(c, nnsHash, "isAvailable", []any{name}, nil))
if err != nil {
return false, fmt.Errorf("`isAvailable`: invalid response: %w", err)
}
return b, nil
}
}

View file

@ -0,0 +1,164 @@
package morph
import (
"errors"
"fmt"
"github.com/nspcc-dev/neo-go/pkg/core/native"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/neo"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
)
// initialAlphabetNEOAmount represents the total amount of GAS distributed between alphabet nodes.
const (
initialAlphabetNEOAmount = native.NEOTotalSupply
registerBatchSize = transaction.MaxAttributes - 1
)
func (c *initializeContext) registerCandidateRange(start, end int) error {
regPrice, err := c.getCandidateRegisterPrice()
if err != nil {
return fmt.Errorf("can't fetch registration price: %w", err)
}
w := io.NewBufBinWriter()
emit.AppCall(w.BinWriter, neo.Hash, "setRegisterPrice", callflag.States, 1)
for _, acc := range c.Accounts[start:end] {
emit.AppCall(w.BinWriter, neo.Hash, "registerCandidate", callflag.States, acc.PrivateKey().PublicKey().Bytes())
emit.Opcodes(w.BinWriter, opcode.ASSERT)
}
emit.AppCall(w.BinWriter, neo.Hash, "setRegisterPrice", callflag.States, regPrice)
if w.Err != nil {
panic(fmt.Sprintf("BUG: %v", w.Err))
}
signers := []rpcclient.SignerAccount{{
Signer: c.getSigner(false, c.CommitteeAcc),
Account: c.CommitteeAcc,
}}
for _, acc := range c.Accounts[start:end] {
signers = append(signers, rpcclient.SignerAccount{
Signer: transaction.Signer{
Account: acc.Contract.ScriptHash(),
Scopes: transaction.CustomContracts,
AllowedContracts: []util.Uint160{neo.Hash},
},
Account: acc,
})
}
tx, err := c.Client.CreateTxFromScript(w.Bytes(), c.CommitteeAcc, -1, 0, signers)
if err != nil {
return fmt.Errorf("can't create tx: %w", err)
}
if err := c.multiSign(tx, committeeAccountName); err != nil {
return fmt.Errorf("can't sign a transaction: %w", err)
}
network := c.CommitteeAct.GetNetwork()
for _, acc := range c.Accounts[start:end] {
if err := acc.SignTx(network, tx); err != nil {
return fmt.Errorf("can't sign a transaction: %w", err)
}
}
return c.sendTx(tx, c.Command, true)
}
func (c *initializeContext) registerCandidates() error {
cc, err := unwrap.Array(c.ReadOnlyInvoker.Call(neo.Hash, "getCandidates"))
if err != nil {
return fmt.Errorf("`getCandidates`: %w", err)
}
need := len(c.Accounts)
have := len(cc)
if need == have {
c.Command.Println("Candidates are already registered.")
return nil
}
// Register candidates in batches in order to overcome the signers amount limit.
// See: https://github.com/nspcc-dev/neo-go/blob/master/pkg/core/transaction/transaction.go#L27
for i := 0; i < need; i += registerBatchSize {
start, end := i, i+registerBatchSize
if end > need {
end = need
}
// This check is sound because transactions are accepted/rejected atomically.
if have >= end {
continue
}
if err := c.registerCandidateRange(start, end); err != nil {
return fmt.Errorf("registering candidates %d..%d: %q", start, end-1, err)
}
}
return nil
}
func (c *initializeContext) transferNEOToAlphabetContracts() error {
neoHash := neo.Hash
ok, err := c.transferNEOFinished(neoHash)
if ok || err != nil {
return err
}
cs := c.getContract(alphabetContract)
amount := initialAlphabetNEOAmount / len(c.Wallets)
bw := io.NewBufBinWriter()
for _, acc := range c.Accounts {
h := state.CreateContractHash(acc.Contract.ScriptHash(), cs.NEF.Checksum, cs.Manifest.Name)
emit.AppCall(bw.BinWriter, neoHash, "transfer", callflag.All,
c.CommitteeAcc.Contract.ScriptHash(), h, int64(amount), nil)
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
}
if err := c.sendCommitteeTx(bw.Bytes(), false); err != nil {
return err
}
return c.awaitTx()
}
func (c *initializeContext) transferNEOFinished(neoHash util.Uint160) (bool, error) {
bal, err := c.Client.NEP17BalanceOf(neoHash, c.CommitteeAcc.Contract.ScriptHash())
return bal < native.NEOTotalSupply, err
}
var errGetPriceInvalid = errors.New("`getRegisterPrice`: invalid response")
func (c *initializeContext) getCandidateRegisterPrice() (int64, error) {
switch c.Client.(type) {
case *rpcclient.Client:
inv := invoker.New(c.Client, nil)
reader := neo.NewReader(inv)
return reader.GetRegisterPrice()
default:
neoHash := neo.Hash
res, err := invokeFunction(c.Client, neoHash, "getRegisterPrice", nil, nil)
if err != nil {
return 0, err
}
if len(res.Stack) == 0 {
return 0, errGetPriceInvalid
}
bi, err := res.Stack[0].TryInteger()
if err != nil || !bi.IsInt64() {
return 0, errGetPriceInvalid
}
return bi.Int64(), nil
}
}

View file

@ -1,9 +1,9 @@
package morph
import (
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/core/native/noderoles"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/rolemgmt"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
)
@ -16,20 +16,18 @@ func (c *initializeContext) setNotaryAndAlphabetNodes() error {
return err
}
designateHash := c.nativeHash(nativenames.Designation)
var pubs []interface{}
var pubs []any
for _, acc := range c.Accounts {
pubs = append(pubs, acc.PrivateKey().PublicKey().Bytes())
}
w := io.NewBufBinWriter()
emit.AppCall(w.BinWriter, designateHash, "designateAsRole",
emit.AppCall(w.BinWriter, rolemgmt.Hash, "designateAsRole",
callflag.States|callflag.AllowNotify, int64(noderoles.P2PNotary), pubs)
emit.AppCall(w.BinWriter, designateHash, "designateAsRole",
emit.AppCall(w.BinWriter, rolemgmt.Hash, "designateAsRole",
callflag.States|callflag.AllowNotify, int64(noderoles.NeoFSAlphabet), pubs)
if err := c.sendCommitteeTx(w.Bytes(), -1, false); err != nil {
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
return err
}
@ -42,6 +40,6 @@ func (c *initializeContext) setRolesFinished() (bool, error) {
return false, err
}
pubs, err := c.Client.GetDesignatedByRole(noderoles.NeoFSAlphabet, height)
pubs, err := getDesignatedByRole(c.ReadOnlyInvoker, rolemgmt.Hash, noderoles.NeoFSAlphabet, height)
return len(pubs) == len(c.Wallets), err
}

View file

@ -0,0 +1,141 @@
package morph
import (
"encoding/hex"
"fmt"
"os"
"path/filepath"
"strconv"
"testing"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/pkg/config"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/vm"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
)
const (
contractsPath = "../../../../../../frostfs-contract/frostfs-contract-v0.16.0.tar.gz"
protoFileName = "proto.yml"
)
func TestInitialize(t *testing.T) {
// This test needs frostfs-contract tarball, so it is skipped by default.
// It is here for performing local testing after the changes.
t.Skip()
t.Run("1 nodes", func(t *testing.T) {
testInitialize(t, 1)
})
t.Run("4 nodes", func(t *testing.T) {
testInitialize(t, 4)
})
t.Run("7 nodes", func(t *testing.T) {
testInitialize(t, 7)
})
t.Run("16 nodes", func(t *testing.T) {
testInitialize(t, 16)
})
t.Run("max nodes", func(t *testing.T) {
testInitialize(t, maxAlphabetNodes)
})
t.Run("too many nodes", func(t *testing.T) {
require.ErrorIs(t, generateTestData(t, t.TempDir(), maxAlphabetNodes+1), ErrTooManyAlphabetNodes)
})
}
func testInitialize(t *testing.T, committeeSize int) {
testdataDir := t.TempDir()
v := viper.GetViper()
require.NoError(t, generateTestData(t, testdataDir, committeeSize))
v.Set(protoConfigPath, filepath.Join(testdataDir, protoFileName))
// Set to the path or remove the next statement to download from the network.
require.NoError(t, initCmd.Flags().Set(contractsInitFlag, contractsPath))
v.Set(localDumpFlag, filepath.Join(testdataDir, "out"))
v.Set(alphabetWalletsFlag, testdataDir)
v.Set(epochDurationInitFlag, 1)
v.Set(maxObjectSizeInitFlag, 1024)
setTestCredentials(v, committeeSize)
require.NoError(t, initializeSideChainCmd(initCmd, nil))
t.Run("force-new-epoch", func(t *testing.T) {
require.NoError(t, forceNewEpochCmd(forceNewEpoch, nil))
})
t.Run("set-config", func(t *testing.T) {
require.NoError(t, setConfigCmd(setConfig, []string{"MaintenanceModeAllowed=true"}))
})
t.Run("set-policy", func(t *testing.T) {
require.NoError(t, setPolicyCmd(setPolicy, []string{"ExecFeeFactor=1"}))
})
t.Run("remove-node", func(t *testing.T) {
pk, err := keys.NewPrivateKey()
require.NoError(t, err)
pub := hex.EncodeToString(pk.PublicKey().Bytes())
require.NoError(t, removeNodesCmd(removeNodes, []string{pub}))
})
}
func generateTestData(t *testing.T, dir string, size int) error {
v := viper.GetViper()
v.Set(alphabetWalletsFlag, dir)
sizeStr := strconv.FormatUint(uint64(size), 10)
if err := generateAlphabetCmd.Flags().Set(alphabetSizeFlag, sizeStr); err != nil {
return err
}
setTestCredentials(v, size)
if err := generateAlphabetCreds(generateAlphabetCmd, nil); err != nil {
return err
}
var pubs []string
for i := 0; i < size; i++ {
p := filepath.Join(dir, innerring.GlagoliticLetter(i).String()+".json")
w, err := wallet.NewWalletFromFile(p)
if err != nil {
return fmt.Errorf("wallet doesn't exist: %w", err)
}
for _, acc := range w.Accounts {
if acc.Label == singleAccountName {
pub, ok := vm.ParseSignatureContract(acc.Contract.Script)
if !ok {
return fmt.Errorf("could not parse signature script for %s", acc.Address)
}
pubs = append(pubs, hex.EncodeToString(pub))
continue
}
}
}
cfg := config.Config{}
cfg.ProtocolConfiguration.Magic = 12345
cfg.ProtocolConfiguration.ValidatorsCount = uint32(size)
cfg.ProtocolConfiguration.TimePerBlock = time.Second
cfg.ProtocolConfiguration.StandbyCommittee = pubs // sorted by glagolic letters
cfg.ProtocolConfiguration.P2PSigExtensions = true
cfg.ProtocolConfiguration.VerifyTransactions = true
data, err := yaml.Marshal(cfg)
if err != nil {
return err
}
protoPath := filepath.Join(dir, protoFileName)
return os.WriteFile(protoPath, data, os.ModePerm)
}
func setTestCredentials(v *viper.Viper, size int) {
for i := 0; i < size; i++ {
v.Set("credentials."+innerring.GlagoliticLetter(i).String(), strconv.FormatUint(uint64(i), 10))
}
v.Set("credentials.contract", testContractPassword)
}

View file

@ -4,17 +4,23 @@ import (
"fmt"
"github.com/nspcc-dev/neo-go/pkg/core/native"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/rpc/client"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/neo"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
scContext "github.com/nspcc-dev/neo-go/pkg/smartcontract/context"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/wallet"
)
const (
gasInitialTotalSupply = 30000000 * native.GASFactor
// initialAlphabetGASAmount represents amount of GAS given to each alphabet node.
// initialAlphabetGASAmount represents the amount of GAS given to each alphabet node.
initialAlphabetGASAmount = 10_000 * native.GASFactor
// initialProxyGASAmount represents amount of GAS given to proxy contract.
// initialProxyGASAmount represents the amount of GAS given to a proxy contract.
initialProxyGASAmount = 50_000 * native.GASFactor
)
@ -27,15 +33,12 @@ func (c *initializeContext) transferFunds() error {
return err
}
gasHash := c.nativeHash(nativenames.Gas)
neoHash := c.nativeHash(nativenames.Neo)
var transfers []client.TransferTarget
var transfers []rpcclient.TransferTarget
for _, acc := range c.Accounts {
to := acc.Contract.ScriptHash()
transfers = append(transfers,
client.TransferTarget{
Token: gasHash,
rpcclient.TransferTarget{
Token: gas.Hash,
Address: to,
Amount: initialAlphabetGASAmount,
},
@ -44,19 +47,19 @@ func (c *initializeContext) transferFunds() error {
// It is convenient to have all funds at the committee account.
transfers = append(transfers,
client.TransferTarget{
Token: gasHash,
rpcclient.TransferTarget{
Token: gas.Hash,
Address: c.CommitteeAcc.Contract.ScriptHash(),
Amount: (gasInitialTotalSupply - initialAlphabetGASAmount*int64(len(c.Wallets))) / 2,
},
client.TransferTarget{
Token: neoHash,
rpcclient.TransferTarget{
Token: neo.Hash,
Address: c.CommitteeAcc.Contract.ScriptHash(),
Amount: native.NEOTotalSupply,
},
)
tx, err := c.Client.CreateNEP17MultiTransferTx(c.ConsensusAcc, 0, transfers, []client.SignerAccount{{
tx, err := createNEP17MultiTransferTx(c.Client, c.ConsensusAcc, 0, transfers, []rpcclient.SignerAccount{{
Signer: transaction.Signer{
Account: c.ConsensusAcc.Contract.ScriptHash(),
Scopes: transaction.CalledByEntry,
@ -75,10 +78,9 @@ func (c *initializeContext) transferFunds() error {
}
func (c *initializeContext) transferFundsFinished() (bool, error) {
gasHash := c.nativeHash(nativenames.Gas)
acc := c.Accounts[0]
res, err := c.Client.NEP17BalanceOf(gasHash, acc.Contract.ScriptHash())
res, err := c.Client.NEP17BalanceOf(gas.Hash, acc.Contract.ScriptHash())
return res > initialAlphabetGASAmount/2, err
}
@ -120,25 +122,40 @@ func (c *initializeContext) multiSign(tx *transaction.Transaction, accType strin
}
}
w, err := pc.GetWitness(tx.Signers[0].Account)
w, err := pc.GetWitness(h)
if err != nil {
return fmt.Errorf("incomplete signature: %w", err)
}
tx.Scripts = append(tx.Scripts, *w)
return nil
for i := range tx.Signers {
if tx.Signers[i].Account == h {
if i < len(tx.Scripts) {
tx.Scripts[i] = *w
} else if i == len(tx.Scripts) {
tx.Scripts = append(tx.Scripts, *w)
} else {
panic("BUG: invalid signing order")
}
return nil
}
}
return fmt.Errorf("%s account was not found among transaction signers", accType)
}
func (c *initializeContext) transferGASToProxy() error {
gasHash := c.nativeHash(nativenames.Gas)
proxyCs := c.getContract(proxyContract)
bal, err := c.Client.NEP17BalanceOf(gasHash, proxyCs.Hash)
bal, err := c.Client.NEP17BalanceOf(gas.Hash, proxyCs.Hash)
if err != nil || bal > 0 {
return err
}
tx, err := c.Client.CreateNEP17TransferTx(c.CommitteeAcc, proxyCs.Hash, gasHash, initialProxyGASAmount, 0, nil, nil)
tx, err := createNEP17MultiTransferTx(c.Client, c.CommitteeAcc, 0, []rpcclient.TransferTarget{{
Token: gas.Hash,
Address: proxyCs.Hash,
Amount: initialProxyGASAmount,
}}, nil)
if err != nil {
return err
}
@ -149,3 +166,25 @@ func (c *initializeContext) transferGASToProxy() error {
return c.awaitTx()
}
func createNEP17MultiTransferTx(c Client, acc *wallet.Account, netFee int64,
recipients []rpcclient.TransferTarget, cosigners []rpcclient.SignerAccount) (*transaction.Transaction, error) {
from := acc.Contract.ScriptHash()
w := io.NewBufBinWriter()
for i := range recipients {
emit.AppCall(w.BinWriter, recipients[i].Token, "transfer", callflag.All,
from, recipients[i].Address, recipients[i].Amount, recipients[i].Data)
emit.Opcodes(w.BinWriter, opcode.ASSERT)
}
if w.Err != nil {
return nil, fmt.Errorf("failed to create transfer script: %w", w.Err)
}
return c.CreateTxFromScript(w.Bytes(), acc, -1, netFee, append([]rpcclient.SignerAccount{{
Signer: transaction.Signer{
Account: from,
Scopes: transaction.CalledByEntry,
},
Account: acc,
}}, cosigners...))
}

View file

@ -0,0 +1,505 @@
package morph
import (
"crypto/elliptic"
"errors"
"fmt"
"os"
"sort"
"time"
"github.com/google/uuid"
"github.com/nspcc-dev/neo-go/pkg/config"
"github.com/nspcc-dev/neo-go/pkg/config/netmode"
"github.com/nspcc-dev/neo-go/pkg/core"
"github.com/nspcc-dev/neo-go/pkg/core/block"
"github.com/nspcc-dev/neo-go/pkg/core/chaindump"
"github.com/nspcc-dev/neo-go/pkg/core/fee"
"github.com/nspcc-dev/neo-go/pkg/core/native/noderoles"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/storage"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/hash"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/neorpc/result"
"github.com/nspcc-dev/neo-go/pkg/network/payload"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/trigger"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"go.uber.org/zap"
)
type localClient struct {
bc *core.Blockchain
transactions []*transaction.Transaction
dumpPath string
accounts []*wallet.Account
maxGasInvoke int64
}
func newLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet) (*localClient, error) {
cfg, err := config.LoadFile(v.GetString(protoConfigPath))
if err != nil {
return nil, err
}
bc, err := core.NewBlockchain(storage.NewMemoryStore(), cfg.Blockchain(), zap.NewNop())
if err != nil {
return nil, err
}
m := smartcontract.GetDefaultHonestNodeCount(int(cfg.ProtocolConfiguration.ValidatorsCount))
accounts := make([]*wallet.Account, len(wallets))
for i := range accounts {
accounts[i], err = getWalletAccount(wallets[i], consensusAccountName)
if err != nil {
return nil, err
}
}
indexMap := make(map[string]int)
for i, pub := range cfg.ProtocolConfiguration.StandbyCommittee {
indexMap[pub] = i
}
sort.Slice(accounts, func(i, j int) bool {
pi := accounts[i].PrivateKey().PublicKey().Bytes()
pj := accounts[j].PrivateKey().PublicKey().Bytes()
return indexMap[string(pi)] < indexMap[string(pj)]
})
sort.Slice(accounts[:cfg.ProtocolConfiguration.ValidatorsCount], func(i, j int) bool {
return accounts[i].PublicKey().Cmp(accounts[j].PublicKey()) == -1
})
go bc.Run()
dumpPath := v.GetString(localDumpFlag)
if cmd.Name() != "init" {
f, err := os.OpenFile(dumpPath, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("can't open local dump: %w", err)
}
defer f.Close()
r := io.NewBinReaderFromIO(f)
var skip uint32
if bc.BlockHeight() != 0 {
skip = bc.BlockHeight() + 1
}
count := r.ReadU32LE() - skip
if err := chaindump.Restore(bc, r, skip, count, nil); err != nil {
return nil, fmt.Errorf("can't restore local dump: %w", err)
}
}
return &localClient{
bc: bc,
dumpPath: dumpPath,
accounts: accounts[:m],
maxGasInvoke: 15_0000_0000,
}, nil
}
func (l *localClient) GetBlockCount() (uint32, error) {
return l.bc.BlockHeight(), nil
}
func (l *localClient) GetContractStateByID(id int32) (*state.Contract, error) {
h, err := l.bc.GetContractScriptHash(id)
if err != nil {
return nil, err
}
return l.GetContractStateByHash(h)
}
func (l *localClient) GetContractStateByHash(h util.Uint160) (*state.Contract, error) {
if cs := l.bc.GetContractState(h); cs != nil {
return cs, nil
}
return nil, storage.ErrKeyNotFound
}
func (l *localClient) GetNativeContracts() ([]state.NativeContract, error) {
return l.bc.GetNatives(), nil
}
func (l *localClient) GetNetwork() (netmode.Magic, error) {
return l.bc.GetConfig().Magic, nil
}
func (l *localClient) GetApplicationLog(h util.Uint256, t *trigger.Type) (*result.ApplicationLog, error) {
aer, err := l.bc.GetAppExecResults(h, *t)
if err != nil {
return nil, err
}
a := result.NewApplicationLog(h, aer, *t)
return &a, nil
}
func (l *localClient) CreateTxFromScript(script []byte, acc *wallet.Account, sysFee int64, netFee int64, cosigners []rpcclient.SignerAccount) (*transaction.Transaction, error) {
signers, accounts, err := getSigners(acc, cosigners)
if err != nil {
return nil, fmt.Errorf("failed to construct tx signers: %w", err)
}
if sysFee < 0 {
res, err := l.InvokeScript(script, signers)
if err != nil {
return nil, fmt.Errorf("can't add system fee to transaction: %w", err)
}
if res.State != "HALT" {
return nil, fmt.Errorf("can't add system fee to transaction: bad vm state: %s due to an error: %s", res.State, res.FaultException)
}
sysFee = res.GasConsumed
}
tx := transaction.New(script, sysFee)
tx.Signers = signers
tx.ValidUntilBlock = l.bc.BlockHeight() + 2
err = l.AddNetworkFee(tx, netFee, accounts...)
if err != nil {
return nil, fmt.Errorf("failed to add network fee: %w", err)
}
return tx, nil
}
func (l *localClient) GetCommittee() (keys.PublicKeys, error) {
// not used by `morph init` command
panic("unexpected call")
}
// InvokeFunction is implemented via `InvokeScript`.
func (l *localClient) InvokeFunction(h util.Uint160, method string, sPrm []smartcontract.Parameter, ss []transaction.Signer) (*result.Invoke, error) {
var err error
pp := make([]any, len(sPrm))
for i, p := range sPrm {
pp[i], err = smartcontract.ExpandParameterToEmitable(p)
if err != nil {
return nil, fmt.Errorf("incorrect parameter type %s: %w", p.Type, err)
}
}
return invokeFunction(l, h, method, pp, ss)
}
func (l *localClient) CalculateNotaryFee(_ uint8) (int64, error) {
// not used by `morph init` command
panic("unexpected call")
}
func (l *localClient) SignAndPushP2PNotaryRequest(_ *transaction.Transaction, _ []byte, _ int64, _ int64, _ uint32, _ *wallet.Account) (*payload.P2PNotaryRequest, error) {
// not used by `morph init` command
panic("unexpected call")
}
func (l *localClient) SignAndPushInvocationTx(_ []byte, _ *wallet.Account, _ int64, _ fixedn.Fixed8, _ []rpcclient.SignerAccount) (util.Uint256, error) {
// not used by `morph init` command
panic("unexpected call")
}
func (l *localClient) TerminateSession(_ uuid.UUID) (bool, error) {
// not used by `morph init` command
panic("unexpected call")
}
func (l *localClient) TraverseIterator(_, _ uuid.UUID, _ int) ([]stackitem.Item, error) {
// not used by `morph init` command
panic("unexpected call")
}
// GetVersion return default version.
func (l *localClient) GetVersion() (*result.Version, error) {
c := l.bc.GetConfig()
return &result.Version{
Protocol: result.Protocol{
AddressVersion: address.NEO3Prefix,
Network: c.Magic,
MillisecondsPerBlock: int(c.TimePerBlock / time.Millisecond),
MaxTraceableBlocks: c.MaxTraceableBlocks,
MaxValidUntilBlockIncrement: c.MaxValidUntilBlockIncrement,
MaxTransactionsPerBlock: c.MaxTransactionsPerBlock,
MemoryPoolMaxTransactions: c.MemPoolSize,
ValidatorsCount: byte(c.ValidatorsCount),
InitialGasDistribution: c.InitialGASSupply,
CommitteeHistory: c.CommitteeHistory,
P2PSigExtensions: c.P2PSigExtensions,
StateRootInHeader: c.StateRootInHeader,
ValidatorsHistory: c.ValidatorsHistory,
},
}, nil
}
func (l *localClient) InvokeContractVerify(util.Uint160, []smartcontract.Parameter, []transaction.Signer, ...transaction.Witness) (*result.Invoke, error) {
// not used by `morph init` command
panic("unexpected call")
}
// CalculateNetworkFee calculates network fee for the given transaction.
// Copied from neo-go with minor corrections (no need to support non-notary mode):
// https://github.com/nspcc-dev/neo-go/blob/v0.99.2/pkg/services/rpcsrv/server.go#L744
func (l *localClient) CalculateNetworkFee(tx *transaction.Transaction) (int64, error) {
hashablePart, err := tx.EncodeHashableFields()
if err != nil {
return 0, fmt.Errorf("failed to compute tx size: %w", err)
}
size := len(hashablePart) + io.GetVarSize(len(tx.Signers))
ef := l.bc.GetBaseExecFee()
var netFee int64
for i, signer := range tx.Signers {
var verificationScript []byte
for _, w := range tx.Scripts {
if w.VerificationScript != nil && hash.Hash160(w.VerificationScript).Equals(signer.Account) {
verificationScript = w.VerificationScript
break
}
}
if verificationScript == nil {
gasConsumed, err := l.bc.VerifyWitness(signer.Account, tx, &tx.Scripts[i], l.maxGasInvoke)
if err != nil {
return 0, fmt.Errorf("invalid signature: %w", err)
}
netFee += gasConsumed
size += io.GetVarSize([]byte{}) + io.GetVarSize(tx.Scripts[i].InvocationScript)
continue
}
fee, sizeDelta := fee.Calculate(ef, verificationScript)
netFee += fee
size += sizeDelta
}
fee := l.bc.FeePerByte()
netFee += int64(size) * fee
return netFee, nil
}
// AddNetworkFee adds network fee for each witness script and optional extra
// network fee to transaction. `accs` is an array signer's accounts.
// Copied from neo-go with minor corrections (no need to support contract signers):
// https://github.com/nspcc-dev/neo-go/blob/6ff11baa1b9e4c71ef0d1de43b92a8c541ca732c/pkg/rpc/client/rpc.go#L960
func (l *localClient) AddNetworkFee(tx *transaction.Transaction, extraFee int64, accs ...*wallet.Account) error {
if len(tx.Signers) != len(accs) {
return errors.New("number of signers must match number of scripts")
}
size := io.GetVarSize(tx)
ef := l.bc.GetBaseExecFee()
for i := range tx.Signers {
netFee, sizeDelta := fee.Calculate(ef, accs[i].Contract.Script)
tx.NetworkFee += netFee
size += sizeDelta
}
tx.NetworkFee += int64(size)*l.bc.FeePerByte() + extraFee
return nil
}
// getSigners returns an array of transaction signers and corresponding accounts from
// given sender and cosigners. If cosigners list already contains sender, the sender
// will be placed at the start of the list.
// Copied from neo-go with minor corrections:
// https://github.com/nspcc-dev/neo-go/blob/6ff11baa1b9e4c71ef0d1de43b92a8c541ca732c/pkg/rpc/client/rpc.go#L735
func getSigners(sender *wallet.Account, cosigners []rpcclient.SignerAccount) ([]transaction.Signer, []*wallet.Account, error) {
var (
signers []transaction.Signer
accounts []*wallet.Account
)
from := sender.Contract.ScriptHash()
s := transaction.Signer{
Account: from,
Scopes: transaction.None,
}
for _, c := range cosigners {
if c.Signer.Account == from {
s = c.Signer
continue
}
signers = append(signers, c.Signer)
accounts = append(accounts, c.Account)
}
signers = append([]transaction.Signer{s}, signers...)
accounts = append([]*wallet.Account{sender}, accounts...)
return signers, accounts, nil
}
func (l *localClient) NEP17BalanceOf(h util.Uint160, acc util.Uint160) (int64, error) {
res, err := invokeFunction(l, h, "balanceOf", []any{acc}, nil)
if err != nil {
return 0, err
}
if res.State != vmstate.Halt.String() || len(res.Stack) == 0 {
return 0, fmt.Errorf("`balance`: invalid response (empty: %t): %s",
len(res.Stack) == 0, res.FaultException)
}
bi, err := res.Stack[0].TryInteger()
if err != nil || !bi.IsInt64() {
return 0, fmt.Errorf("`balance`: invalid response")
}
return bi.Int64(), nil
}
func (l *localClient) InvokeScript(script []byte, signers []transaction.Signer) (*result.Invoke, error) {
lastBlock, err := l.bc.GetBlock(l.bc.CurrentBlockHash())
if err != nil {
return nil, err
}
tx := transaction.New(script, 0)
tx.Signers = signers
tx.ValidUntilBlock = l.bc.BlockHeight() + 2
ic, err := l.bc.GetTestVM(trigger.Application, tx, &block.Block{
Header: block.Header{
Index: lastBlock.Index + 1,
Timestamp: lastBlock.Timestamp + 1,
},
})
if err != nil {
return nil, fmt.Errorf("get test VM: %w", err)
}
ic.VM.GasLimit = 100_0000_0000
ic.VM.LoadScriptWithFlags(script, callflag.All)
var errStr string
if err := ic.VM.Run(); err != nil {
errStr = err.Error()
}
return &result.Invoke{
State: ic.VM.State().String(),
GasConsumed: ic.VM.GasConsumed(),
Script: script,
Stack: ic.VM.Estack().ToArray(),
FaultException: errStr,
}, nil
}
func (l *localClient) SendRawTransaction(tx *transaction.Transaction) (util.Uint256, error) {
// We need to test that transaction was formed correctly to catch as many errors as we can.
bs := tx.Bytes()
_, err := transaction.NewTransactionFromBytes(bs)
if err != nil {
return tx.Hash(), fmt.Errorf("invalid transaction: %w", err)
}
l.transactions = append(l.transactions, tx)
return tx.Hash(), nil
}
func (l *localClient) putTransactions() error {
// 1. Prepare new block.
lastBlock, err := l.bc.GetBlock(l.bc.CurrentBlockHash())
if err != nil {
panic(err)
}
defer func() { l.transactions = l.transactions[:0] }()
b := &block.Block{
Header: block.Header{
NextConsensus: l.accounts[0].Contract.ScriptHash(),
Script: transaction.Witness{
VerificationScript: l.accounts[0].Contract.Script,
},
Timestamp: lastBlock.Timestamp + 1,
},
Transactions: l.transactions,
}
if l.bc.GetConfig().StateRootInHeader {
b.StateRootEnabled = true
b.PrevStateRoot = l.bc.GetStateModule().CurrentLocalStateRoot()
}
b.PrevHash = lastBlock.Hash()
b.Index = lastBlock.Index + 1
b.RebuildMerkleRoot()
// 2. Sign prepared block.
var invocationScript []byte
magic := l.bc.GetConfig().Magic
for _, acc := range l.accounts {
sign := acc.PrivateKey().SignHashable(uint32(magic), b)
invocationScript = append(invocationScript, byte(opcode.PUSHDATA1), 64)
invocationScript = append(invocationScript, sign...)
}
b.Script.InvocationScript = invocationScript
// 3. Persist block.
return l.bc.AddBlock(b)
}
func invokeFunction(c Client, h util.Uint160, method string, parameters []any, signers []transaction.Signer) (*result.Invoke, error) {
w := io.NewBufBinWriter()
emit.Array(w.BinWriter, parameters...)
emit.AppCallNoArgs(w.BinWriter, h, method, callflag.All)
if w.Err != nil {
panic(fmt.Sprintf("BUG: invalid parameters for '%s': %v", method, w.Err))
}
return c.InvokeScript(w.Bytes(), signers)
}
var errGetDesignatedByRoleResponse = errors.New("`getDesignatedByRole`: invalid response")
func getDesignatedByRole(inv *invoker.Invoker, h util.Uint160, role noderoles.Role, u uint32) (keys.PublicKeys, error) {
arr, err := unwrap.Array(inv.Call(h, "getDesignatedByRole", int64(role), int64(u)))
if err != nil {
return nil, errGetDesignatedByRoleResponse
}
pubs := make(keys.PublicKeys, len(arr))
for i := range arr {
bs, err := arr[i].TryBytes()
if err != nil {
return nil, errGetDesignatedByRoleResponse
}
pubs[i], err = keys.NewPublicKeyFromBytes(bs, elliptic.P256())
if err != nil {
return nil, errGetDesignatedByRoleResponse
}
}
return pubs, nil
}
func (l *localClient) dump() (err error) {
defer l.bc.Close()
f, err := os.Create(l.dumpPath)
if err != nil {
return err
}
defer func() {
closeErr := f.Close()
if err == nil && closeErr != nil {
err = closeErr
}
}()
w := io.NewBinWriterFromIO(f)
w.WriteU32LE(l.bc.BlockHeight() + 1)
err = chaindump.Dump(l.bc, w, 0, l.bc.BlockHeight()+1)
return
}

View file

@ -0,0 +1,122 @@
package morph
import (
"context"
"errors"
"fmt"
"time"
"github.com/nspcc-dev/neo-go/pkg/config/netmode"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/neorpc/result"
"github.com/nspcc-dev/neo-go/pkg/network/payload"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/trigger"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// Client represents N3 client interface capable of test-invoking scripts
// and sending signed transactions to chain.
type Client interface {
invoker.RPCInvoke
GetBlockCount() (uint32, error)
GetContractStateByID(int32) (*state.Contract, error)
GetContractStateByHash(util.Uint160) (*state.Contract, error)
GetNativeContracts() ([]state.NativeContract, error)
GetNetwork() (netmode.Magic, error)
GetApplicationLog(util.Uint256, *trigger.Type) (*result.ApplicationLog, error)
GetVersion() (*result.Version, error)
CreateTxFromScript([]byte, *wallet.Account, int64, int64, []rpcclient.SignerAccount) (*transaction.Transaction, error)
NEP17BalanceOf(util.Uint160, util.Uint160) (int64, error)
SendRawTransaction(*transaction.Transaction) (util.Uint256, error)
GetCommittee() (keys.PublicKeys, error)
CalculateNotaryFee(uint8) (int64, error)
CalculateNetworkFee(tx *transaction.Transaction) (int64, error)
AddNetworkFee(*transaction.Transaction, int64, ...*wallet.Account) error
SignAndPushInvocationTx([]byte, *wallet.Account, int64, fixedn.Fixed8, []rpcclient.SignerAccount) (util.Uint256, error)
SignAndPushP2PNotaryRequest(*transaction.Transaction, []byte, int64, int64, uint32, *wallet.Account) (*payload.P2PNotaryRequest, error)
}
type hashVUBPair struct {
hash util.Uint256
vub uint32
}
type clientContext struct {
Client Client // a raw neo-go client OR a local chain implementation
CommitteeAct *actor.Actor // committee actor with the Global witness scope
ReadOnlyInvoker *invoker.Invoker // R/O contract invoker, does not contain any signer
SentTxs []hashVUBPair
}
func getN3Client(v *viper.Viper) (Client, error) {
// number of opened connections
// by neo-go client per one host
const (
maxConnsPerHost = 10
requestTimeout = time.Second * 10
)
ctx := context.Background()
endpoint := v.GetString(endpointFlag)
if endpoint == "" {
return nil, errors.New("missing endpoint")
}
c, err := rpcclient.New(ctx, endpoint, rpcclient.Options{
MaxConnsPerHost: maxConnsPerHost,
RequestTimeout: requestTimeout,
})
if err != nil {
return nil, err
}
if err := c.Init(); err != nil {
return nil, err
}
return c, nil
}
func defaultClientContext(c Client, committeeAcc *wallet.Account) (*clientContext, error) {
commAct, err := actor.New(c, []actor.SignerAccount{{
Signer: transaction.Signer{
Account: committeeAcc.Contract.ScriptHash(),
Scopes: transaction.Global,
},
Account: committeeAcc,
}})
if err != nil {
return nil, err
}
return &clientContext{
Client: c,
CommitteeAct: commAct,
ReadOnlyInvoker: invoker.New(c, nil),
}, nil
}
func (c *clientContext) sendTx(tx *transaction.Transaction, cmd *cobra.Command, await bool) error {
h, err := c.Client.SendRawTransaction(tx)
if err != nil {
return err
}
if h != tx.Hash() {
return fmt.Errorf("sent and actual tx hashes mismatch:\n\tsent: %v\n\tactual: %v", tx.Hash().StringLE(), h.StringLE())
}
c.SentTxs = append(c.SentTxs, hashVUBPair{hash: h, vub: tx.ValidUntilBlock})
if await {
return c.awaitTx(cmd)
}
return nil
}

View file

@ -0,0 +1,29 @@
package morph
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
func listNetmapCandidatesNodes(cmd *cobra.Command, _ []string) {
c, err := getN3Client(viper.GetViper())
commonCmd.ExitOnErr(cmd, "can't create N3 client: %w", err)
inv := invoker.New(c, nil)
cs, err := c.GetContractStateByID(1)
commonCmd.ExitOnErr(cmd, "can't get NNS contract info: %w", err)
nmHash, err := nnsResolveHash(inv, cs.Hash, netmapContract+".frostfs")
commonCmd.ExitOnErr(cmd, "can't get netmap contract hash: %w", err)
res, err := inv.Call(nmHash, "netmapCandidates")
commonCmd.ExitOnErr(cmd, "can't fetch list of network config keys from the netmap contract", err)
nm, err := netmap.DecodeNetMap(res.Stack)
commonCmd.ExitOnErr(cmd, "unable to decode netmap: %w", err)
commonCmd.PrettyPrintNetMap(cmd, *nm, !viper.GetBool(commonflags.Verbose))
}

View file

@ -0,0 +1,46 @@
package morph
import (
"errors"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/spf13/viper"
)
func getDefaultNetmapContractConfigMap() map[string]any {
m := make(map[string]any)
m[netmap.EpochDurationConfig] = viper.GetInt64(epochDurationInitFlag)
m[netmap.MaxObjectSizeConfig] = viper.GetInt64(maxObjectSizeInitFlag)
m[netmap.AuditFeeConfig] = viper.GetInt64(auditFeeInitFlag)
m[netmap.ContainerFeeConfig] = viper.GetInt64(containerFeeInitFlag)
m[netmap.ContainerAliasFeeConfig] = viper.GetInt64(containerAliasFeeInitFlag)
m[netmap.BasicIncomeRateConfig] = viper.GetInt64(incomeRateInitFlag)
m[netmap.IrCandidateFeeConfig] = viper.GetInt64(candidateFeeInitFlag)
m[netmap.WithdrawFeeConfig] = viper.GetInt64(withdrawFeeInitFlag)
m[netmap.HomomorphicHashingDisabledKey] = viper.GetBool(homomorphicHashDisabledInitFlag)
m[netmap.MaintenanceModeAllowedConfig] = viper.GetBool(maintenanceModeAllowedInitFlag)
return m
}
func parseConfigFromNetmapContract(arr []stackitem.Item) (map[string][]byte, error) {
m := make(map[string][]byte, len(arr))
for _, param := range arr {
tuple, ok := param.Value().([]stackitem.Item)
if !ok || len(tuple) != 2 {
return nil, errors.New("invalid ListConfig response from netmap contract")
}
k, err := tuple[0].TryBytes()
if err != nil {
return nil, errors.New("invalid config key from netmap contract")
}
v, err := tuple[1].TryBytes()
if err != nil {
return nil, invalidConfigValueErr(string(k))
}
m[string(k)] = v
}
return m, nil
}

View file

@ -2,18 +2,19 @@ package morph
import (
"fmt"
"math/big"
"strconv"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpc/client"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/nep17"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/notary"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
@ -24,16 +25,9 @@ import (
const defaultNotaryDepositLifetime = 5760
func depositNotary(cmd *cobra.Command, _ []string) error {
p, err := cmd.Flags().GetString(storageWalletFlag)
w, err := openWallet(cmd)
if err != nil {
return err
} else if p == "" {
return fmt.Errorf("missing wallet path (use '--%s <out.json>')", storageWalletFlag)
}
w, err := wallet.NewWalletFromFile(p)
if err != nil {
return fmt.Errorf("can't open wallet: %v", err)
}
accHash := w.GetChangeAddress()
@ -81,23 +75,17 @@ func depositNotary(cmd *cobra.Command, _ []string) error {
}
}
return transferGas(cmd, acc, accHash, gasAmount, till)
}
func transferGas(cmd *cobra.Command, acc *wallet.Account, accHash util.Uint160, gasAmount fixedn.Fixed8, till int64) error {
c, err := getN3Client(viper.GetViper())
if err != nil {
return err
}
nhs, err := getNativeHashes(c)
if err != nil {
return fmt.Errorf("can't get native contract hashes: %w", err)
}
gasHash, ok := nhs[nativenames.Gas]
if !ok {
return fmt.Errorf("can't retrieve %s contract hash", nativenames.Gas)
}
notaryHash, ok := nhs[nativenames.Notary]
if !ok {
return fmt.Errorf("can't retrieve %s contract hash", nativenames.Notary)
if err := checkNotaryEnabled(c); err != nil {
return err
}
height, err := c.GetBlockCount()
@ -105,15 +93,7 @@ func depositNotary(cmd *cobra.Command, _ []string) error {
return fmt.Errorf("can't get current height: %v", err)
}
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, gasHash, "transfer", callflag.All,
accHash, notaryHash.BytesBE(), int64(gasAmount), []interface{}{nil, int64(height) + till})
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if bw.Err != nil {
return fmt.Errorf("BUG: invalid transfer arguments: %w", bw.Err)
}
tx, err := c.CreateTxFromScript(bw.Bytes(), acc, -1, 0, []client.SignerAccount{{
act, err := actor.New(c, []actor.SignerAccount{{
Signer: transaction.Signer{
Account: acc.Contract.ScriptHash(),
Scopes: transaction.Global,
@ -121,21 +101,35 @@ func depositNotary(cmd *cobra.Command, _ []string) error {
Account: acc,
}})
if err != nil {
return fmt.Errorf("can't create tx: %w", err)
return fmt.Errorf("could not create actor: %w", err)
}
mn, err := c.GetNetwork()
gasActor := nep17.New(act, gas.Hash)
txHash, vub, err := gasActor.Transfer(
accHash,
notary.Hash,
big.NewInt(int64(gasAmount)),
[]any{nil, int64(height) + till},
)
if err != nil {
// error appears only if client
// has not been initialized
panic(err)
return fmt.Errorf("could not send tx: %w", err)
}
err = acc.SignTx(mn, tx)
if err != nil {
return fmt.Errorf("can't sign tx: %w", err)
}
cc := defaultClientContext(c)
return cc.sendTx(tx, cmd, true)
return awaitTx(cmd, c, []hashVUBPair{{hash: txHash, vub: vub}})
}
func openWallet(cmd *cobra.Command) (*wallet.Wallet, error) {
p, err := cmd.Flags().GetString(storageWalletFlag)
if err != nil {
return nil, err
} else if p == "" {
return nil, fmt.Errorf("missing wallet path (use '--%s <out.json>')", storageWalletFlag)
}
w, err := wallet.NewWalletFromFile(p)
if err != nil {
return nil, fmt.Errorf("can't open wallet: %v", err)
}
return w, nil
}

View file

@ -5,8 +5,8 @@ import (
"strconv"
"strings"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/policy"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/spf13/cobra"
@ -25,33 +25,28 @@ func setPolicyCmd(cmd *cobra.Command, args []string) error {
return fmt.Errorf("can't to initialize context: %w", err)
}
policyHash, err := wCtx.Client.GetNativeContractHash(nativenames.Policy)
if err != nil {
return fmt.Errorf("can't get policy contract hash: %w", err)
}
bw := io.NewBufBinWriter()
for i := range args {
kv := strings.SplitN(args[i], "=", 2)
if len(kv) != 2 {
k, v, found := strings.Cut(args[i], "=")
if !found {
return fmt.Errorf("invalid parameter format, must be Parameter=Value")
}
switch kv[0] {
switch k {
case execFeeParam, storagePriceParam, setFeeParam:
default:
return fmt.Errorf("parameter must be one of %s, %s and %s", execFeeParam, storagePriceParam, setFeeParam)
}
value, err := strconv.ParseUint(kv[1], 10, 32)
value, err := strconv.ParseUint(v, 10, 32)
if err != nil {
return fmt.Errorf("can't parse parameter value '%s': %w", args[1], err)
}
emit.AppCall(bw.BinWriter, policyHash, "set"+kv[0], callflag.All, int64(value))
emit.AppCall(bw.BinWriter, policy.Hash, "set"+k, callflag.All, int64(value))
}
if err := wCtx.sendCommitteeTx(bw.Bytes(), -1, false); err != nil {
if err := wCtx.sendCommitteeTx(bw.Bytes(), false); err != nil {
return err
}

View file

@ -0,0 +1,61 @@
package morph
import (
"errors"
"fmt"
netmapcontract "git.frostfs.info/TrueCloudLab/frostfs-contract/netmap"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
func removeNodesCmd(cmd *cobra.Command, args []string) error {
if len(args) == 0 {
return errors.New("at least one node key must be provided")
}
nodeKeys := make(keys.PublicKeys, len(args))
for i := range args {
var err error
nodeKeys[i], err = keys.NewPublicKeyFromString(args[i])
if err != nil {
return fmt.Errorf("can't parse node public key: %w", err)
}
}
wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return fmt.Errorf("can't initialize context: %w", err)
}
defer wCtx.close()
cs, err := wCtx.Client.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err)
}
nmHash, err := nnsResolveHash(wCtx.ReadOnlyInvoker, cs.Hash, netmapContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get netmap contract hash: %w", err)
}
bw := io.NewBufBinWriter()
for i := range nodeKeys {
emit.AppCall(bw.BinWriter, nmHash, "updateStateIR", callflag.All,
int64(netmapcontract.NodeStateOffline), nodeKeys[i].Bytes())
}
if err := emitNewEpochCall(bw, wCtx, nmHash); err != nil {
return err
}
if err := wCtx.sendConsensusTx(bw.Bytes()); err != nil {
return err
}
return wCtx.awaitTx()
}

View file

@ -0,0 +1,394 @@
package morph
import (
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
alphabetWalletsFlag = "alphabet-wallets"
alphabetSizeFlag = "size"
endpointFlag = "rpc-endpoint"
storageWalletFlag = "storage-wallet"
storageWalletLabelFlag = "label"
storageGasCLIFlag = "initial-gas"
storageGasConfigFlag = "storage.initial_gas"
contractsInitFlag = "contracts"
maxObjectSizeInitFlag = "network.max_object_size"
maxObjectSizeCLIFlag = "max-object-size"
epochDurationInitFlag = "network.epoch_duration"
epochDurationCLIFlag = "epoch-duration"
incomeRateInitFlag = "network.basic_income_rate"
incomeRateCLIFlag = "basic-income-rate"
auditFeeInitFlag = "network.fee.audit"
auditFeeCLIFlag = "audit-fee"
containerFeeInitFlag = "network.fee.container"
containerAliasFeeInitFlag = "network.fee.container_alias"
containerFeeCLIFlag = "container-fee"
containerAliasFeeCLIFlag = "container-alias-fee"
candidateFeeInitFlag = "network.fee.candidate"
candidateFeeCLIFlag = "candidate-fee"
homomorphicHashDisabledInitFlag = "network.homomorphic_hash_disabled"
maintenanceModeAllowedInitFlag = "network.maintenance_mode_allowed"
homomorphicHashDisabledCLIFlag = "homomorphic-disabled"
withdrawFeeInitFlag = "network.fee.withdraw"
withdrawFeeCLIFlag = "withdraw-fee"
containerDumpFlag = "dump"
containerContractFlag = "container-contract"
containerIDsFlag = "cid"
refillGasAmountFlag = "gas"
walletAccountFlag = "account"
notaryDepositTillFlag = "till"
localDumpFlag = "local-dump"
protoConfigPath = "protocol"
walletAddressFlag = "wallet-address"
)
var (
// RootCmd is a root command of config section.
RootCmd = &cobra.Command{
Use: "morph",
Short: "Section for morph network configuration commands",
}
generateAlphabetCmd = &cobra.Command{
Use: "generate-alphabet",
Short: "Generate alphabet wallets for consensus nodes of the morph network",
PreRun: func(cmd *cobra.Command, _ []string) {
// PreRun fixes https://github.com/spf13/viper/issues/233
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
},
RunE: generateAlphabetCreds,
}
initCmd = &cobra.Command{
Use: "init",
Short: "Initialize side chain network with smart-contracts and network settings",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
_ = viper.BindPFlag(epochDurationInitFlag, cmd.Flags().Lookup(epochDurationCLIFlag))
_ = viper.BindPFlag(maxObjectSizeInitFlag, cmd.Flags().Lookup(maxObjectSizeCLIFlag))
_ = viper.BindPFlag(incomeRateInitFlag, cmd.Flags().Lookup(incomeRateCLIFlag))
_ = viper.BindPFlag(homomorphicHashDisabledInitFlag, cmd.Flags().Lookup(homomorphicHashDisabledCLIFlag))
_ = viper.BindPFlag(auditFeeInitFlag, cmd.Flags().Lookup(auditFeeCLIFlag))
_ = viper.BindPFlag(candidateFeeInitFlag, cmd.Flags().Lookup(candidateFeeCLIFlag))
_ = viper.BindPFlag(containerFeeInitFlag, cmd.Flags().Lookup(containerFeeCLIFlag))
_ = viper.BindPFlag(containerAliasFeeInitFlag, cmd.Flags().Lookup(containerAliasFeeCLIFlag))
_ = viper.BindPFlag(withdrawFeeInitFlag, cmd.Flags().Lookup(withdrawFeeCLIFlag))
_ = viper.BindPFlag(protoConfigPath, cmd.Flags().Lookup(protoConfigPath))
_ = viper.BindPFlag(localDumpFlag, cmd.Flags().Lookup(localDumpFlag))
},
RunE: initializeSideChainCmd,
}
generateStorageCmd = &cobra.Command{
Use: "generate-storage-wallet",
Short: "Generate storage node wallet for the morph network",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
_ = viper.BindPFlag(storageGasConfigFlag, cmd.Flags().Lookup(storageGasCLIFlag))
},
RunE: generateStorageCreds,
}
refillGasCmd = &cobra.Command{
Use: "refill-gas",
Short: "Refill GAS of storage node's wallet in the morph network",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
_ = viper.BindPFlag(refillGasAmountFlag, cmd.Flags().Lookup(refillGasAmountFlag))
},
RunE: func(cmd *cobra.Command, args []string) error {
return refillGas(cmd, refillGasAmountFlag, false)
},
}
forceNewEpoch = &cobra.Command{
Use: "force-new-epoch",
Short: "Create new FrostFS epoch event in the side chain",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: forceNewEpochCmd,
}
removeNodes = &cobra.Command{
Use: "remove-nodes key1 [key2 [...]]",
Short: "Remove storage nodes from the netmap",
Long: `Move nodes to the Offline state in the candidates list and tick an epoch to update the netmap`,
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: removeNodesCmd,
}
setConfig = &cobra.Command{
Use: "set-config key1=val1 [key2=val2 ...]",
DisableFlagsInUseLine: true,
Short: "Add/update global config value in the FrostFS network",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
Args: cobra.MinimumNArgs(1),
RunE: setConfigCmd,
}
setPolicy = &cobra.Command{
Use: "set-policy [ExecFeeFactor=<n1>] [StoragePrice=<n2>] [FeePerByte=<n3>]",
DisableFlagsInUseLine: true,
Short: "Set global policy values",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: setPolicyCmd,
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
return []string{"ExecFeeFactor=", "StoragePrice=", "FeePerByte="}, cobra.ShellCompDirectiveNoSpace
},
}
dumpContractHashesCmd = &cobra.Command{
Use: "dump-hashes",
Short: "Dump deployed contract hashes",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: dumpContractHashes,
}
dumpNetworkConfigCmd = &cobra.Command{
Use: "dump-config",
Short: "Dump FrostFS network config",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: dumpNetworkConfig,
}
dumpBalancesCmd = &cobra.Command{
Use: "dump-balances",
Short: "Dump GAS balances",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: dumpBalances,
}
updateContractsCmd = &cobra.Command{
Use: "update-contracts",
Short: "Update FrostFS contracts",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: updateContracts,
}
dumpContainersCmd = &cobra.Command{
Use: "dump-containers",
Short: "Dump FrostFS containers to file",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: dumpContainers,
}
restoreContainersCmd = &cobra.Command{
Use: "restore-containers",
Short: "Restore FrostFS containers from file",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: restoreContainers,
}
listContainersCmd = &cobra.Command{
Use: "list-containers",
Short: "List FrostFS containers",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: listContainers,
}
depositNotaryCmd = &cobra.Command{
Use: "deposit-notary",
Short: "Deposit GAS for notary service",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
},
RunE: depositNotary,
}
netmapCandidatesCmd = &cobra.Command{
Use: "netmap-candidates",
Short: "List netmap candidates nodes",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
_ = viper.BindPFlag(alphabetWalletsFlag, cmd.Flags().Lookup(alphabetWalletsFlag))
},
Run: listNetmapCandidatesNodes,
}
)
func init() {
initGenerateAlphabetCmd()
initInitCmd()
initDeployCmd()
initGenerateStorageCmd()
initForceNewEpochCmd()
initRemoveNodesCmd()
initSetPolicyCmd()
initDumpContractHashesCmd()
initDumpNetworkConfigCmd()
initSetConfigCmd()
initDumpBalancesCmd()
initUpdateContractsCmd()
initDumpContainersCmd()
initRestoreContainersCmd()
initListContainersCmd()
initRefillGasCmd()
initDepositoryNotaryCmd()
initNetmapCandidatesCmd()
}
func initNetmapCandidatesCmd() {
RootCmd.AddCommand(netmapCandidatesCmd)
netmapCandidatesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initDepositoryNotaryCmd() {
RootCmd.AddCommand(depositNotaryCmd)
depositNotaryCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
depositNotaryCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
depositNotaryCmd.Flags().String(walletAccountFlag, "", "Wallet account address")
depositNotaryCmd.Flags().String(refillGasAmountFlag, "", "Amount of GAS to deposit")
depositNotaryCmd.Flags().String(notaryDepositTillFlag, "", "Notary deposit duration in blocks")
}
func initRefillGasCmd() {
RootCmd.AddCommand(refillGasCmd)
refillGasCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
refillGasCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
refillGasCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
refillGasCmd.Flags().String(walletAddressFlag, "", "Address of wallet")
refillGasCmd.Flags().String(refillGasAmountFlag, "", "Additional amount of GAS to transfer")
refillGasCmd.MarkFlagsMutuallyExclusive(walletAddressFlag, storageWalletFlag)
}
func initListContainersCmd() {
RootCmd.AddCommand(listContainersCmd)
listContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
listContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
}
func initRestoreContainersCmd() {
RootCmd.AddCommand(restoreContainersCmd)
restoreContainersCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
restoreContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
restoreContainersCmd.Flags().String(containerDumpFlag, "", "File to restore containers from")
restoreContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to restore")
}
func initDumpContainersCmd() {
RootCmd.AddCommand(dumpContainersCmd)
dumpContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContainersCmd.Flags().String(containerDumpFlag, "", "File where to save dumped containers")
dumpContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
dumpContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to dump")
}
func initUpdateContractsCmd() {
RootCmd.AddCommand(updateContractsCmd)
updateContractsCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
updateContractsCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
updateContractsCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts")
_ = updateContractsCmd.MarkFlagRequired(contractsInitFlag)
}
func initDumpBalancesCmd() {
RootCmd.AddCommand(dumpBalancesCmd)
dumpBalancesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpBalancesCmd.Flags().BoolP(dumpBalancesStorageFlag, "s", false, "Dump balances of storage nodes from the current netmap")
dumpBalancesCmd.Flags().BoolP(dumpBalancesAlphabetFlag, "a", false, "Dump balances of alphabet contracts")
dumpBalancesCmd.Flags().BoolP(dumpBalancesProxyFlag, "p", false, "Dump balances of the proxy contract")
dumpBalancesCmd.Flags().Bool(dumpBalancesUseScriptHashFlag, false, "Use script-hash format for addresses")
}
func initSetConfigCmd() {
RootCmd.AddCommand(setConfig)
setConfig.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setConfig.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
setConfig.Flags().Bool(forceConfigSet, false, "Force setting not well-known configuration key")
}
func initDumpNetworkConfigCmd() {
RootCmd.AddCommand(dumpNetworkConfigCmd)
dumpNetworkConfigCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initDumpContractHashesCmd() {
RootCmd.AddCommand(dumpContractHashesCmd)
dumpContractHashesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContractHashesCmd.Flags().String(customZoneFlag, "", "Custom zone to search.")
}
func initSetPolicyCmd() {
RootCmd.AddCommand(setPolicy)
setPolicy.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setPolicy.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initRemoveNodesCmd() {
RootCmd.AddCommand(removeNodes)
removeNodes.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
removeNodes.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initForceNewEpochCmd() {
RootCmd.AddCommand(forceNewEpoch)
forceNewEpoch.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
forceNewEpoch.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initGenerateStorageCmd() {
RootCmd.AddCommand(generateStorageCmd)
generateStorageCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
generateStorageCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
generateStorageCmd.Flags().String(storageWalletFlag, "", "Path to new storage node wallet")
generateStorageCmd.Flags().String(storageGasCLIFlag, "", "Initial amount of GAS to transfer")
generateStorageCmd.Flags().StringP(storageWalletLabelFlag, "l", "", "Wallet label")
}
func initInitCmd() {
RootCmd.AddCommand(initCmd)
initCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
initCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
initCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts")
_ = initCmd.MarkFlagRequired(contractsInitFlag)
initCmd.Flags().Uint(epochDurationCLIFlag, 240, "Amount of side chain blocks in one FrostFS epoch")
initCmd.Flags().Uint(maxObjectSizeCLIFlag, 67108864, "Max single object size in bytes")
initCmd.Flags().Bool(homomorphicHashDisabledCLIFlag, false, "Disable object homomorphic hashing")
// Defaults are taken from neo-preodolenie.
initCmd.Flags().Uint64(containerFeeCLIFlag, 1000, "Container registration fee")
initCmd.Flags().Uint64(containerAliasFeeCLIFlag, 500, "Container alias fee")
initCmd.Flags().String(protoConfigPath, "", "Path to the consensus node configuration")
initCmd.Flags().String(localDumpFlag, "", "Path to the blocks dump file")
}
func initGenerateAlphabetCmd() {
RootCmd.AddCommand(generateAlphabetCmd)
generateAlphabetCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
generateAlphabetCmd.Flags().Uint(alphabetSizeFlag, 7, "Amount of alphabet wallets to generate")
}
func initDeployCmd() {
RootCmd.AddCommand(deployCmd)
}

View file

@ -0,0 +1,85 @@
package modules
import (
"os"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/storagecfg"
"git.frostfs.info/TrueCloudLab/frostfs-node/misc"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/autocomplete"
utilConfig "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/gendoc"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var (
rootCmd = &cobra.Command{
Use: "frostfs-adm",
Short: "FrostFS Administrative Tool",
Long: `FrostFS Administrative Tool provides functions to setup and
manage FrostFS network deployment.`,
RunE: entryPoint,
SilenceUsage: true,
}
)
func init() {
cobra.OnInitialize(func() { initConfig(rootCmd) })
// we need to init viper config to bind viper and cobra configurations for
// rpc endpoint, alphabet wallet dir, key credentials, etc.
// use stdout as default output for cmd.Print()
rootCmd.SetOut(os.Stdout)
rootCmd.PersistentFlags().StringP(commonflags.ConfigFlag, commonflags.ConfigFlagShorthand, "", commonflags.ConfigFlagUsage)
rootCmd.PersistentFlags().String(commonflags.ConfigDirFlag, "", commonflags.ConfigDirFlagUsage)
rootCmd.PersistentFlags().BoolP(commonflags.Verbose, commonflags.VerboseShorthand, false, commonflags.VerboseUsage)
_ = viper.BindPFlag(commonflags.Verbose, rootCmd.PersistentFlags().Lookup(commonflags.Verbose))
rootCmd.Flags().Bool("version", false, "Application version")
rootCmd.AddCommand(config.RootCmd)
rootCmd.AddCommand(morph.RootCmd)
rootCmd.AddCommand(storagecfg.RootCmd)
rootCmd.AddCommand(autocomplete.Command("frostfs-adm"))
rootCmd.AddCommand(gendoc.Command(rootCmd))
}
func Execute() error {
return rootCmd.Execute()
}
func entryPoint(cmd *cobra.Command, _ []string) error {
printVersion, _ := cmd.Flags().GetBool("version")
if printVersion {
cmd.Print(misc.BuildInfo("FrostFS Adm"))
return nil
}
return cmd.Usage()
}
func initConfig(cmd *cobra.Command) {
configFile, err := cmd.Flags().GetString(commonflags.ConfigFlag)
if err != nil {
return
}
if configFile != "" {
viper.SetConfigType("yml")
viper.SetConfigFile(configFile)
_ = viper.ReadInConfig() // if config file is set but unavailable, ignore it
}
configDir, err := cmd.Flags().GetString(commonflags.ConfigDirFlag)
if err != nil {
return
}
if configDir != "" {
_ = utilConfig.ReadConfigDir(viper.GetViper(), configDir) // if config files cannot be read, ignore it
}
}

View file

@ -12,9 +12,6 @@ node:
- {{ .AnnouncedAddress }}
attribute_0: UN-LOCODE:{{ .Attribute.Locode }}
relay: {{ .Relay }} # start Storage node in relay mode without bootstrapping into the Network map
subnet:
exit_zero: false # toggle entrance to zero subnet (overrides corresponding attribute and occurrence in entries)
entries: [] # list of IDs of subnets to enter in a text format of NeoFS API protocol (overrides corresponding attributes)
grpc:
num: 1 # total number of listener endpoints
@ -37,39 +34,37 @@ control:
morph:
dial_timeout: 20s # timeout for side chain NEO RPC client connection
disable_cache: false # use TTL cache for side chain GET operations
cache_ttl: 15s # use TTL cache for side chain GET operations
rpc_endpoint: # side chain N3 RPC endpoints
{{- range .MorphRPC }}
- wss://{{.}}/ws{{end}}
- address: wss://{{.}}/ws{{end}}
{{if not .Relay }}
storage:
shard_pool_size: 15 # size of per-shard worker pools used for PUT operations
shard_num: 1 # total number of shards
default: # section with the default shard parameters
metabase:
perm: 0644 # permissions for metabase files(directories: +x for current user and group)
blobstor:
perm: 0644 # permissions for blobstor files(directories: +x for current user and group)
depth: 2 # max depth of object tree storage in FS
small_object_size: 102400 # 100KiB, size threshold for "small" objects which are stored in key-value DB, not in FS, bytes
compress: true # turn on/off Zstandard compression (level 3) of stored objects
compression_exclude_content_types:
- audio/*
- video/*
blobovnicza:
size: 1073741824 # approximate size limit of single blobovnicza instance, total size will be: size*width^(depth+1), bytes
depth: 1 # max depth of object tree storage in key-value DB
width: 4 # max width of object tree storage in key-value DB
opened_cache_capacity: 50 # maximum number of opened database files
gc:
remover_batch_size: 200 # number of objects to be removed by the garbage collector
remover_sleep_interval: 5m # frequency of the garbage collector invocation
shard:
default: # section with the default shard parameters
metabase:
perm: 0644 # permissions for metabase files(directories: +x for current user and group)
blobstor:
perm: 0644 # permissions for blobstor files(directories: +x for current user and group)
depth: 2 # max depth of object tree storage in FS
small_object_size: 102400 # 100KiB, size threshold for "small" objects which are stored in key-value DB, not in FS, bytes
compress: true # turn on/off Zstandard compression (level 3) of stored objects
compression_exclude_content_types:
- audio/*
- video/*
blobovnicza:
size: 1073741824 # approximate size limit of single blobovnicza instance, total size will be: size*width^(depth+1), bytes
depth: 1 # max depth of object tree storage in key-value DB
width: 4 # max width of object tree storage in key-value DB
opened_cache_capacity: 50 # maximum number of opened database files
gc:
remover_batch_size: 200 # number of objects to be removed by the garbage collector
remover_sleep_interval: 5m # frequency of the garbage collector invocation
0:
mode: "read-write" # mode of the shard, must be one of the: "read-write" (default), "read-only"

View file

@ -6,9 +6,9 @@ import (
"encoding/hex"
"errors"
"fmt"
"math/big"
"math/rand"
"net"
"net/url"
"os"
"path/filepath"
"strconv"
@ -16,19 +16,21 @@ import (
"text/template"
"time"
netutil "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network"
"github.com/chzyer/readline"
"github.com/nspcc-dev/neo-go/cli/flags"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/rpc/client"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/nep17"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/trigger"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
)
@ -38,22 +40,22 @@ const (
)
const (
defaultControlEndpoint = "127.0.0.1:8090"
defaultDataEndpoint = "127.0.0.1"
defaultControlEndpoint = "localhost:8090"
defaultDataEndpoint = "localhost"
)
// RootCmd is a root command of config section.
var RootCmd = &cobra.Command{
Use: "storage-config [-w wallet] [-a acccount] [<path-to-config>]",
Short: "Section for storage node configuration commands.",
Short: "Section for storage node configuration commands",
Run: storageConfig,
}
func init() {
fs := RootCmd.Flags()
fs.StringP(walletFlag, "w", "", "path to wallet")
fs.StringP(accountFlag, "a", "", "wallet account")
fs.StringP(walletFlag, "w", "", "Path to wallet")
fs.StringP(accountFlag, "a", "", "Wallet account")
}
type config struct {
@ -78,17 +80,9 @@ type config struct {
}
func storageConfig(cmd *cobra.Command, args []string) {
var outPath string
if len(args) != 0 {
outPath = args[0]
} else {
outPath = getPath("File to write config at [./config.yml]: ")
if outPath == "" {
outPath = "./config.yml"
}
}
outPath := getOutputPath(args)
historyPath := filepath.Join(os.TempDir(), "neofs-adm.history")
historyPath := filepath.Join(os.TempDir(), "frostfs-adm.history")
readline.SetHistoryPath(historyPath)
var c config
@ -101,14 +95,7 @@ func storageConfig(cmd *cobra.Command, args []string) {
w, err := wallet.NewWalletFromFile(c.Wallet.Path)
fatalOnErr(err)
c.Wallet.Account, _ = cmd.Flags().GetString(accountFlag)
if c.Wallet.Account == "" {
addr := address.Uint160ToString(w.GetChangeAddress())
c.Wallet.Account = getWalletAccount(w, fmt.Sprintf("Wallet account [%s]: ", addr))
if c.Wallet.Account == "" {
c.Wallet.Account = addr
}
}
fillWalletAccount(cmd, &c, w)
accH, err := flags.ParseAddress(c.Wallet.Account)
fatalOnErr(err)
@ -126,59 +113,18 @@ func storageConfig(cmd *cobra.Command, args []string) {
c.AuthorizedKeys = append(c.AuthorizedKeys, hex.EncodeToString(acc.PrivateKey().PublicKey().Bytes()))
var network string
for {
network = getString("Choose network [mainnet]/testnet: ")
switch network {
case "":
network = "mainnet"
case "testnet", "mainnet":
default:
cmd.Println(`Network must be either "mainnet" or "testnet"`)
continue
}
break
}
network := readNetwork(cmd)
c.MorphRPC = n3config[network].MorphRPC
depositGas(cmd, acc, network)
c.Attribute.Locode = getString("UN-LOCODE attribute in [XX YYY] format: ")
var addr, port string
for {
c.AnnouncedAddress = getString("Publicly announced address: ")
addr, port, err = net.SplitHostPort(c.AnnouncedAddress)
if err != nil {
cmd.Println("Address must have form A.B.C.D:PORT")
continue
}
ip, err := net.ResolveIPAddr("ip", addr)
if err != nil {
cmd.Printf("Can't resolve IP address %s: %v\n", addr, err)
continue
}
if !ip.IP.IsGlobalUnicast() {
cmd.Println("IP must be global unicast.")
continue
}
cmd.Printf("Resolved IP address: %s\n", ip.String())
_, err = strconv.ParseUint(port, 10, 16)
if err != nil {
cmd.Println("Port must be an integer.")
continue
}
break
}
defaultAddr := net.JoinHostPort(defaultDataEndpoint, port)
c.Endpoint = getString(fmt.Sprintf("Listening address [%s]: ", defaultAddr))
endpoint := getDefaultEndpoint(cmd, &c)
c.Endpoint = getString(fmt.Sprintf("Listening address [%s]: ", endpoint))
if c.Endpoint == "" {
c.Endpoint = defaultAddr
c.Endpoint = endpoint
}
c.ControlEndpoint = getString(fmt.Sprintf("Listening address (control endpoint) [%s]: ", defaultControlEndpoint))
@ -201,7 +147,85 @@ func storageConfig(cmd *cobra.Command, args []string) {
out := applyTemplate(c)
fatalOnErr(os.WriteFile(outPath, out, 0644))
cmd.Println("Node is ready for work! Run `neofs-node -config " + outPath + "`")
cmd.Println("Node is ready for work! Run `frostfs-node -config " + outPath + "`")
}
func getDefaultEndpoint(cmd *cobra.Command, c *config) string {
var addr, port string
for {
c.AnnouncedAddress = getString("Publicly announced address: ")
validator := netutil.Address{}
err := validator.FromString(c.AnnouncedAddress)
if err != nil {
cmd.Println("Incorrect address format. See https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/network/address.go for details.")
continue
}
uriAddr, err := url.Parse(validator.URIAddr())
if err != nil {
panic(fmt.Errorf("unexpected error: %w", err))
}
addr = uriAddr.Hostname()
port = uriAddr.Port()
ip, err := net.ResolveIPAddr("ip", addr)
if err != nil {
cmd.Printf("Can't resolve IP address %s: %v\n", addr, err)
continue
}
if !ip.IP.IsGlobalUnicast() {
cmd.Println("IP must be global unicast.")
continue
}
cmd.Printf("Resolved IP address: %s\n", ip.String())
_, err = strconv.ParseUint(port, 10, 16)
if err != nil {
cmd.Println("Port must be an integer.")
continue
}
break
}
return net.JoinHostPort(defaultDataEndpoint, port)
}
func fillWalletAccount(cmd *cobra.Command, c *config, w *wallet.Wallet) {
c.Wallet.Account, _ = cmd.Flags().GetString(accountFlag)
if c.Wallet.Account == "" {
addr := address.Uint160ToString(w.GetChangeAddress())
c.Wallet.Account = getWalletAccount(w, fmt.Sprintf("Wallet account [%s]: ", addr))
if c.Wallet.Account == "" {
c.Wallet.Account = addr
}
}
}
func readNetwork(cmd *cobra.Command) string {
var network string
for {
network = getString("Choose network [mainnet]/testnet: ")
switch network {
case "":
network = "mainnet"
case "testnet", "mainnet":
default:
cmd.Println(`Network must be either "mainnet" or "testnet"`)
continue
}
break
}
return network
}
func getOutputPath(args []string) string {
if len(args) != 0 {
return args[0]
}
outPath := getPath("File to write config at [./config.yml]: ")
if outPath == "" {
outPath = "./config.yml"
}
return outPath
}
func getWalletAccount(w *wallet.Wallet, prompt string) string {
@ -313,23 +337,17 @@ func depositGas(cmd *cobra.Command, acc *wallet.Account, network string) {
sideClient := initClient(n3config[network].MorphRPC)
balanceHash, _ := util.Uint160DecodeStringLE(n3config[network].BalanceContract)
res, err := sideClient.InvokeFunction(balanceHash, "balanceOf", []smartcontract.Parameter{{
Type: smartcontract.Hash160Type,
Value: acc.Contract.ScriptHash(),
}}, nil)
fatalOnErr(err)
if res.State != vm.HaltState.String() {
fatalOnErr(fmt.Errorf("invalid response from balance contract: %s", res.FaultException))
sideActor, err := actor.NewSimple(sideClient, acc)
if err != nil {
fatalOnErr(fmt.Errorf("creating actor over side chain client: %w", err))
}
var balance *big.Int
if len(res.Stack) != 0 {
balance, _ = res.Stack[0].TryInteger()
}
sideGas := nep17.NewReader(sideActor, balanceHash)
accSH := acc.Contract.ScriptHash()
if balance == nil {
fatalOnErr(errors.New("invalid response from balance contract"))
balance, err := sideGas.BalanceOf(accSH)
if err != nil {
fatalOnErr(fmt.Errorf("side chain balance: %w", err))
}
ok := getConfirmation(false, fmt.Sprintf("Current NeoFS balance is %s, make a deposit? y/[n]: ",
@ -347,14 +365,17 @@ func depositGas(cmd *cobra.Command, acc *wallet.Account, network string) {
mainClient := initClient(n3config[network].RPC)
neofsHash, _ := util.Uint160DecodeStringLE(n3config[network].NeoFSContract)
gasHash, err := mainClient.GetNativeContractHash(nativenames.Gas)
fatalOnErr(err)
mainActor, err := actor.NewSimple(mainClient, acc)
if err != nil {
fatalOnErr(fmt.Errorf("creating actor over main chain client: %w", err))
}
tx, err := mainClient.CreateNEP17TransferTx(acc, neofsHash, gasHash, amount.Int64(), 0, nil, nil)
fatalOnErr(err)
mainGas := nep17.New(mainActor, gas.Hash)
txHash, err := mainClient.SignAndPushTx(tx, acc, nil)
fatalOnErr(err)
txHash, _, err := mainGas.Transfer(accSH, neofsHash, amount, nil)
if err != nil {
fatalOnErr(fmt.Errorf("sending TX to the NeoFS contract: %w", err))
}
cmd.Print("Waiting for transactions to persist.")
tick := time.NewTicker(time.Second / 2)
@ -385,8 +406,8 @@ loop:
}
}
func initClient(rpc []string) *client.Client {
var c *client.Client
func initClient(rpc []string) *rpcclient.Client {
var c *rpcclient.Client
var err error
shuffled := make([]string, len(rpc))
@ -394,7 +415,7 @@ func initClient(rpc []string) *client.Client {
rand.Shuffle(len(shuffled), func(i, j int) { shuffled[i], shuffled[j] = shuffled[j], shuffled[i] })
for _, endpoint := range shuffled {
c, err = client.New(context.Background(), "https://"+endpoint, client.Options{
c, err = rpcclient.New(context.Background(), "https://"+endpoint, rpcclient.Options{
DialTimeout: time.Second * 2,
RequestTimeout: time.Second * 5,
})

View file

@ -3,7 +3,7 @@ package main
import (
"os"
"github.com/nspcc-dev/neofs-node/cmd/neofs-adm/internal/modules"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules"
)
func main() {

View file

@ -0,0 +1,75 @@
# How FrostFS CLI uses session mechanism of the FrostFS
## Overview
FrostFS sessions implement a mechanism for issuing a power of attorney by one
party to another. A trusted party can provide a so-called session token as
proof of the right to act on behalf of another member of the network. The
client of operations carried out with such a token will be the user who opened
the session. The token contains information which limits power of attorney like
action context or lifetime.
The client confirms trust in a third party by signing its public (session) key
with his private key. Any operation signed using private session key with
attached session token is treated as performed by the original client.
## Types
FrostFS CLI supports two ways to execute operation within a session depending on
whether the user of the command application is an original user (1) or a trusted
one (2).
### Dynamic
For case (1) CLI user can only open dynamic sessions. Protocol call
`SessionService.Create` is used for this purpose. As a result of the call, a
private session key will be generated on the server, thus making the remote
server trusted. This type of session is useful when the client needs to
transfer part of the responsibility for the formation of strict system elements
to the trusted server. At the moment, the approach is applicable only to
creating objects.
```shell
$ frostfs-cli session create --rpc-endpoint <server_ip> --out ./blank_token
```
After this example command remote node holds session private key while its
public part is written into the session token encoded into the output file.
Later this token can be attached to the operations which support dynamic
sessions. Then the token will be finally formed and signed by CLI itself.
### Static
For case (2) CLI user can act on behalf of the person who issued the session
token to him. Unlike (1) the token must be fully prepared on the side of the
original client, and the CLI uses it only for reading. Ready token MUST have:
- correct context (object, container, etc.)
- valid lifetime
- public session key corresponding to the CLI key
- valid client signature
To sign the session token, exec:
```shell
$ frostfs-cli --wallet <client_wallet> util sign session-token --from ./blank_token --to ./token
```
Once the token is signed, it MUST NOT be modified.
## Commands
### Object
Here are sub-commands of `object` command which support only dynamic sessions (1):
- `put`
- `delete`
- `lock`
These commands accept blank token of the dynamically opened session or open
session internally if it has not been opened yet.
All other `object` sub-commands support only static sessions (2).
### Container
List of commands supporting sessions (static only):
- `create`
- `delete`
- `set-eacl`

View file

@ -0,0 +1,33 @@
# Extended headers
## Overview
Extended headers are used for request/response. They may contain any
user-defined headers to be interpreted on application level. Key name must be a
unique valid UTF-8 string. Value can't be empty. Requests or Responses with
duplicated header names or headers with empty values are considered invalid.
## Existing headers
There are some "well-known" headers starting with `__SYSTEM__` prefix that
affect system behaviour. For backward compatibility, the same set of
"well-known" headers may also use `__NEOFS__` prefix:
* `__SYSTEM__NETMAP_EPOCH` - netmap epoch to use for object placement calculation. The `value` is string
encoded `uint64` in decimal presentation. If set to '0' or omitted, the
current epoch only will be used.
* `__SYSTEM__NETMAP_LOOKUP_DEPTH` - if object can't be found using current epoch's netmap, this header limits
how many past epochs the node can look up through. Depth is applied to a current epoch or the value
of `__SYSTEM__NETMAP_EPOCH` attribute. The `value` is string encoded `uint64` in decimal presentation.
If set to '0' or not set, only the current epoch is used.
## `frostfs-cli` commands with `--xhdr`
List of commands with support of extended headers:
* `container list-objects`
* `object delete/get/hash/head/lock/put/range/search`
Example:
```shell
$ frostfs-cli object put -r s01.frostfs.devenv:8080 -w wallet.json --cid CID --file FILE --xhdr "__SYSTEM__NETMAP_EPOCH=777"
```

View file

@ -0,0 +1,930 @@
package internal
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/accounting"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
containerSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/version"
)
// BalanceOfPrm groups parameters of BalanceOf operation.
type BalanceOfPrm struct {
commonPrm
client.PrmBalanceGet
}
// BalanceOfRes groups the resulting values of BalanceOf operation.
type BalanceOfRes struct {
cliRes *client.ResBalanceGet
}
// Balance returns the current balance.
func (x BalanceOfRes) Balance() accounting.Decimal {
return x.cliRes.Amount()
}
// BalanceOf requests the current balance of a FrostFS user.
//
// Returns any error which prevented the operation from completing correctly in error return.
func BalanceOf(ctx context.Context, prm BalanceOfPrm) (res BalanceOfRes, err error) {
res.cliRes, err = prm.cli.BalanceGet(ctx, prm.PrmBalanceGet)
return
}
// ListContainersPrm groups parameters of ListContainers operation.
type ListContainersPrm struct {
commonPrm
client.PrmContainerList
}
// ListContainersRes groups the resulting values of ListContainers operation.
type ListContainersRes struct {
cliRes *client.ResContainerList
}
// IDList returns list of identifiers of user's containers.
func (x ListContainersRes) IDList() []cid.ID {
return x.cliRes.Containers()
}
// ListContainers requests a list of FrostFS user's containers.
//
// Returns any error which prevented the operation from completing correctly in error return.
func ListContainers(ctx context.Context, prm ListContainersPrm) (res ListContainersRes, err error) {
res.cliRes, err = prm.cli.ContainerList(ctx, prm.PrmContainerList)
return
}
// PutContainerPrm groups parameters of PutContainer operation.
type PutContainerPrm struct {
commonPrm
client.PrmContainerPut
}
// PutContainerRes groups the resulting values of PutContainer operation.
type PutContainerRes struct {
cnr cid.ID
}
// ID returns identifier of the created container.
func (x PutContainerRes) ID() cid.ID {
return x.cnr
}
// PutContainer sends a request to save the container in FrostFS.
//
// Operation is asynchronous and not guaranteed even in the absence of errors.
// The required time is also not predictable.
//
// Success can be verified by reading by identifier.
//
// Returns any error which prevented the operation from completing correctly in error return.
func PutContainer(ctx context.Context, prm PutContainerPrm) (res PutContainerRes, err error) {
cliRes, err := prm.cli.ContainerPut(ctx, prm.PrmContainerPut)
if err == nil {
res.cnr = cliRes.ID()
}
return
}
// GetContainerPrm groups parameters of GetContainer operation.
type GetContainerPrm struct {
commonPrm
cliPrm client.PrmContainerGet
}
// SetContainer sets identifier of the container to be read.
func (x *GetContainerPrm) SetContainer(id cid.ID) {
x.cliPrm.SetContainer(id)
}
// GetContainerRes groups the resulting values of GetContainer operation.
type GetContainerRes struct {
cliRes *client.ResContainerGet
}
// Container returns structured of the requested container.
func (x GetContainerRes) Container() containerSDK.Container {
return x.cliRes.Container()
}
// GetContainer reads a container from FrostFS by ID.
//
// Returns any error which prevented the operation from completing correctly in error return.
func GetContainer(ctx context.Context, prm GetContainerPrm) (res GetContainerRes, err error) {
res.cliRes, err = prm.cli.ContainerGet(ctx, prm.cliPrm)
return
}
// IsACLExtendable checks if ACL of the container referenced by the given identifier
// can be extended. Client connection MUST BE correctly established in advance.
func IsACLExtendable(ctx context.Context, c *client.Client, cnr cid.ID) (bool, error) {
var prm GetContainerPrm
prm.SetClient(c)
prm.SetContainer(cnr)
res, err := GetContainer(ctx, prm)
if err != nil {
return false, fmt.Errorf("get container from the FrostFS: %w", err)
}
return res.Container().BasicACL().Extendable(), nil
}
// DeleteContainerPrm groups parameters of DeleteContainerPrm operation.
type DeleteContainerPrm struct {
commonPrm
client.PrmContainerDelete
}
// DeleteContainerRes groups the resulting values of DeleteContainer operation.
type DeleteContainerRes struct{}
// DeleteContainer sends a request to remove a container from FrostFS by ID.
//
// Operation is asynchronous and not guaranteed even in the absence of errors.
// The required time is also not predictable.
//
// Success can be verified by reading by identifier.
//
// Returns any error which prevented the operation from completing correctly in error return.
func DeleteContainer(ctx context.Context, prm DeleteContainerPrm) (res DeleteContainerRes, err error) {
_, err = prm.cli.ContainerDelete(ctx, prm.PrmContainerDelete)
return
}
// EACLPrm groups parameters of EACL operation.
type EACLPrm struct {
commonPrm
client.PrmContainerEACL
}
// EACLRes groups the resulting values of EACL operation.
type EACLRes struct {
cliRes *client.ResContainerEACL
}
// EACL returns requested eACL table.
func (x EACLRes) EACL() eacl.Table {
return x.cliRes.Table()
}
// EACL reads eACL table from FrostFS by container ID.
//
// Returns any error which prevented the operation from completing correctly in error return.
func EACL(ctx context.Context, prm EACLPrm) (res EACLRes, err error) {
res.cliRes, err = prm.cli.ContainerEACL(ctx, prm.PrmContainerEACL)
return
}
// SetEACLPrm groups parameters of SetEACL operation.
type SetEACLPrm struct {
commonPrm
client.PrmContainerSetEACL
}
// SetEACLRes groups the resulting values of SetEACL operation.
type SetEACLRes struct{}
// SetEACL requests to save an eACL table in FrostFS.
//
// Operation is asynchronous and no guaranteed even in the absence of errors.
// The required time is also not predictable.
//
// Success can be verified by reading by container identifier.
//
// Returns any error which prevented the operation from completing correctly in error return.
func SetEACL(ctx context.Context, prm SetEACLPrm) (res SetEACLRes, err error) {
_, err = prm.cli.ContainerSetEACL(ctx, prm.PrmContainerSetEACL)
return
}
// NetworkInfoPrm groups parameters of NetworkInfo operation.
type NetworkInfoPrm struct {
commonPrm
client.PrmNetworkInfo
}
// NetworkInfoRes groups the resulting values of NetworkInfo operation.
type NetworkInfoRes struct {
cliRes *client.ResNetworkInfo
}
// NetworkInfo returns structured information about the FrostFS network.
func (x NetworkInfoRes) NetworkInfo() netmap.NetworkInfo {
return x.cliRes.Info()
}
// NetworkInfo reads information about the FrostFS network.
//
// Returns any error which prevented the operation from completing correctly in error return.
func NetworkInfo(ctx context.Context, prm NetworkInfoPrm) (res NetworkInfoRes, err error) {
res.cliRes, err = prm.cli.NetworkInfo(ctx, prm.PrmNetworkInfo)
return
}
// NodeInfoPrm groups parameters of NodeInfo operation.
type NodeInfoPrm struct {
commonPrm
client.PrmEndpointInfo
}
// NodeInfoRes groups the resulting values of NodeInfo operation.
type NodeInfoRes struct {
cliRes *client.ResEndpointInfo
}
// NodeInfo returns information about the node from netmap.
func (x NodeInfoRes) NodeInfo() netmap.NodeInfo {
return x.cliRes.NodeInfo()
}
// LatestVersion returns the latest FrostFS API version in use.
func (x NodeInfoRes) LatestVersion() version.Version {
return x.cliRes.LatestVersion()
}
// NodeInfo requests information about the remote server from FrostFS netmap.
//
// Returns any error which prevented the operation from completing correctly in error return.
func NodeInfo(ctx context.Context, prm NodeInfoPrm) (res NodeInfoRes, err error) {
res.cliRes, err = prm.cli.EndpointInfo(ctx, prm.PrmEndpointInfo)
return
}
// NetMapSnapshotPrm groups parameters of NetMapSnapshot operation.
type NetMapSnapshotPrm struct {
commonPrm
}
// NetMapSnapshotRes groups the resulting values of NetMapSnapshot operation.
type NetMapSnapshotRes struct {
cliRes *client.ResNetMapSnapshot
}
// NetMap returns current local snapshot of the FrostFS network map.
func (x NetMapSnapshotRes) NetMap() netmap.NetMap {
return x.cliRes.NetMap()
}
// NetMapSnapshot requests current network view of the remote server.
//
// Returns any error which prevented the operation from completing correctly in error return.
func NetMapSnapshot(ctx context.Context, prm NetMapSnapshotPrm) (res NetMapSnapshotRes, err error) {
res.cliRes, err = prm.cli.NetMapSnapshot(ctx, client.PrmNetMapSnapshot{})
return
}
// CreateSessionPrm groups parameters of CreateSession operation.
type CreateSessionPrm struct {
commonPrm
client.PrmSessionCreate
}
// CreateSessionRes groups the resulting values of CreateSession operation.
type CreateSessionRes struct {
cliRes *client.ResSessionCreate
}
// ID returns session identifier.
func (x CreateSessionRes) ID() []byte {
return x.cliRes.ID()
}
// SessionKey returns public session key in a binary format.
func (x CreateSessionRes) SessionKey() []byte {
return x.cliRes.PublicKey()
}
// CreateSession opens a new unlimited session with the remote node.
//
// Returns any error which prevented the operation from completing correctly in error return.
func CreateSession(ctx context.Context, prm CreateSessionPrm) (res CreateSessionRes, err error) {
res.cliRes, err = prm.cli.SessionCreate(ctx, prm.PrmSessionCreate)
return
}
// PutObjectPrm groups parameters of PutObject operation.
type PutObjectPrm struct {
commonObjectPrm
copyNum []uint32
hdr *objectSDK.Object
rdr io.Reader
headerCallback func(*objectSDK.Object)
prepareLocally bool
}
// SetHeader sets object header.
func (x *PutObjectPrm) SetHeader(hdr *objectSDK.Object) {
x.hdr = hdr
}
// SetPayloadReader sets reader of the object payload.
func (x *PutObjectPrm) SetPayloadReader(rdr io.Reader) {
x.rdr = rdr
}
// SetHeaderCallback sets callback which is called on the object after the header is received
// but before the payload is written.
func (x *PutObjectPrm) SetHeaderCallback(f func(*objectSDK.Object)) {
x.headerCallback = f
}
// SetCopiesNumberByVectors sets ordered list of minimal required object copies numbers
// per placement vector.
func (x *PutObjectPrm) SetCopiesNumberByVectors(copiesNumbers []uint32) {
x.copyNum = copiesNumbers
}
// PrepareLocally generate object header on the client side.
// For big object - split locally too.
func (x *PutObjectPrm) PrepareLocally() {
x.prepareLocally = true
}
func (x *PutObjectPrm) convertToSDKPrm(ctx context.Context) (client.PrmObjectPutInit, error) {
var putPrm client.PrmObjectPutInit
if !x.prepareLocally && x.sessionToken != nil {
putPrm.WithinSession(*x.sessionToken)
}
if x.bearerToken != nil {
putPrm.WithBearerToken(*x.bearerToken)
}
if x.local {
putPrm.MarkLocal()
}
putPrm.WithXHeaders(x.xHeaders...)
putPrm.SetCopiesNumberByVectors(x.copyNum)
if x.prepareLocally {
res, err := x.cli.NetworkInfo(ctx, client.PrmNetworkInfo{})
if err != nil {
return client.PrmObjectPutInit{}, err
}
putPrm.WithObjectMaxSize(res.Info().MaxObjectSize())
putPrm.WithEpochSource(epochSource(res.Info().CurrentEpoch()))
putPrm.WithoutHomomorphicHash(res.Info().HomomorphicHashingDisabled())
}
return putPrm, nil
}
// PutObjectRes groups the resulting values of PutObject operation.
type PutObjectRes struct {
id oid.ID
}
// ID returns identifier of the created object.
func (x PutObjectRes) ID() oid.ID {
return x.id
}
type epochSource uint64
func (s epochSource) CurrentEpoch() uint64 {
return uint64(s)
}
// PutObject saves the object in FrostFS network.
//
// Returns any error which prevented the operation from completing correctly in error return.
func PutObject(ctx context.Context, prm PutObjectPrm) (*PutObjectRes, error) {
sdkPrm, err := prm.convertToSDKPrm(ctx)
if err != nil {
return nil, fmt.Errorf("unable to create parameters of object put operation: %w", err)
}
wrt, err := prm.cli.ObjectPutInit(ctx, sdkPrm)
if err != nil {
return nil, fmt.Errorf("init object writing: %w", err)
}
if wrt.WriteHeader(ctx, *prm.hdr) {
if prm.headerCallback != nil {
prm.headerCallback(prm.hdr)
}
sz := prm.hdr.PayloadSize()
if data := prm.hdr.Payload(); len(data) > 0 {
if prm.rdr != nil {
prm.rdr = io.MultiReader(bytes.NewReader(data), prm.rdr)
} else {
prm.rdr = bytes.NewReader(data)
sz = uint64(len(data))
}
}
if prm.rdr != nil {
const defaultBufferSizePut = 3 << 20 // Maximum chunk size is 3 MiB in the SDK.
if sz == 0 || sz > defaultBufferSizePut {
sz = defaultBufferSizePut
}
buf := make([]byte, sz)
var n int
for {
n, err = prm.rdr.Read(buf)
if n > 0 {
if !wrt.WritePayloadChunk(ctx, buf[:n]) {
break
}
continue
}
if errors.Is(err, io.EOF) {
break
}
return nil, fmt.Errorf("read payload: %w", err)
}
}
}
cliRes, err := wrt.Close(ctx)
if err != nil { // here err already carries both status and client errors
return nil, fmt.Errorf("client failure: %w", err)
}
return &PutObjectRes{
id: cliRes.StoredObjectID(),
}, nil
}
// DeleteObjectPrm groups parameters of DeleteObject operation.
type DeleteObjectPrm struct {
commonObjectPrm
objectAddressPrm
}
// DeleteObjectRes groups the resulting values of DeleteObject operation.
type DeleteObjectRes struct {
tomb oid.ID
}
// Tombstone returns the ID of the created object with tombstone.
func (x DeleteObjectRes) Tombstone() oid.ID {
return x.tomb
}
// DeleteObject marks an object to be removed from FrostFS through tombstone placement.
//
// Returns any error which prevented the operation from completing correctly in error return.
func DeleteObject(ctx context.Context, prm DeleteObjectPrm) (*DeleteObjectRes, error) {
var delPrm client.PrmObjectDelete
delPrm.FromContainer(prm.objAddr.Container())
delPrm.ByID(prm.objAddr.Object())
if prm.sessionToken != nil {
delPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
delPrm.WithBearerToken(*prm.bearerToken)
}
delPrm.WithXHeaders(prm.xHeaders...)
cliRes, err := prm.cli.ObjectDelete(ctx, delPrm)
if err != nil {
return nil, fmt.Errorf("remove object via client: %w", err)
}
return &DeleteObjectRes{
tomb: cliRes.Tombstone(),
}, nil
}
// GetObjectPrm groups parameters of GetObject operation.
type GetObjectPrm struct {
commonObjectPrm
objectAddressPrm
rawPrm
payloadWriterPrm
headerCallback func(*objectSDK.Object)
}
// SetHeaderCallback sets callback which is called on the object after the header is received
// but before the payload is written.
func (p *GetObjectPrm) SetHeaderCallback(f func(*objectSDK.Object)) {
p.headerCallback = f
}
// GetObjectRes groups the resulting values of GetObject operation.
type GetObjectRes struct {
hdr *objectSDK.Object
}
// Header returns the header of the request object.
func (x GetObjectRes) Header() *objectSDK.Object {
return x.hdr
}
// GetObject reads an object by address.
//
// Interrupts on any writer error. If successful, payload is written to the writer.
//
// Returns any error which prevented the operation from completing correctly in error return.
// For raw reading, returns *object.SplitInfoError error if object is virtual.
func GetObject(ctx context.Context, prm GetObjectPrm) (*GetObjectRes, error) {
var getPrm client.PrmObjectGet
getPrm.FromContainer(prm.objAddr.Container())
getPrm.ByID(prm.objAddr.Object())
if prm.sessionToken != nil {
getPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
getPrm.WithBearerToken(*prm.bearerToken)
}
if prm.raw {
getPrm.MarkRaw()
}
if prm.local {
getPrm.MarkLocal()
}
getPrm.WithXHeaders(prm.xHeaders...)
rdr, err := prm.cli.ObjectGetInit(ctx, getPrm)
if err != nil {
return nil, fmt.Errorf("init object reading on client: %w", err)
}
var hdr objectSDK.Object
if !rdr.ReadHeader(&hdr) {
_, err = rdr.Close()
return nil, fmt.Errorf("read object header: %w", err)
}
if prm.headerCallback != nil {
prm.headerCallback(&hdr)
}
_, err = io.Copy(prm.wrt, rdr)
if err != nil {
return nil, fmt.Errorf("copy payload: %w", err)
}
return &GetObjectRes{
hdr: &hdr,
}, nil
}
// HeadObjectPrm groups parameters of HeadObject operation.
type HeadObjectPrm struct {
commonObjectPrm
objectAddressPrm
rawPrm
mainOnly bool
}
// SetMainOnlyFlag sets flag to get only main fields of an object header in terms of FrostFS API.
func (x *HeadObjectPrm) SetMainOnlyFlag(v bool) {
x.mainOnly = v
}
// HeadObjectRes groups the resulting values of HeadObject operation.
type HeadObjectRes struct {
hdr *objectSDK.Object
}
// Header returns the requested object header.
func (x HeadObjectRes) Header() *objectSDK.Object {
return x.hdr
}
// HeadObject reads an object header by address.
//
// Returns any error which prevented the operation from completing correctly in error return.
// For raw reading, returns *object.SplitInfoError error if object is virtual.
func HeadObject(ctx context.Context, prm HeadObjectPrm) (*HeadObjectRes, error) {
var cliPrm client.PrmObjectHead
cliPrm.FromContainer(prm.objAddr.Container())
cliPrm.ByID(prm.objAddr.Object())
if prm.sessionToken != nil {
cliPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
cliPrm.WithBearerToken(*prm.bearerToken)
}
if prm.raw {
cliPrm.MarkRaw()
}
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.WithXHeaders(prm.xHeaders...)
res, err := prm.cli.ObjectHead(ctx, cliPrm)
if err != nil {
return nil, fmt.Errorf("read object header via client: %w", err)
}
var hdr objectSDK.Object
if !res.ReadHeader(&hdr) {
return nil, fmt.Errorf("missing header in response")
}
return &HeadObjectRes{
hdr: &hdr,
}, nil
}
// SearchObjectsPrm groups parameters of SearchObjects operation.
type SearchObjectsPrm struct {
commonObjectPrm
containerIDPrm
filters objectSDK.SearchFilters
}
// SetFilters sets search filters.
func (x *SearchObjectsPrm) SetFilters(filters objectSDK.SearchFilters) {
x.filters = filters
}
// SearchObjectsRes groups the resulting values of SearchObjects operation.
type SearchObjectsRes struct {
ids []oid.ID
}
// IDList returns identifiers of the matched objects.
func (x SearchObjectsRes) IDList() []oid.ID {
return x.ids
}
// SearchObjects selects objects from the container which match the filters.
//
// Returns any error which prevented the operation from completing correctly in error return.
func SearchObjects(ctx context.Context, prm SearchObjectsPrm) (*SearchObjectsRes, error) {
var cliPrm client.PrmObjectSearch
cliPrm.InContainer(prm.cnrID)
cliPrm.SetFilters(prm.filters)
if prm.sessionToken != nil {
cliPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
cliPrm.WithBearerToken(*prm.bearerToken)
}
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.WithXHeaders(prm.xHeaders...)
rdr, err := prm.cli.ObjectSearchInit(ctx, cliPrm)
if err != nil {
return nil, fmt.Errorf("init object search: %w", err)
}
buf := make([]oid.ID, 10)
var list []oid.ID
var n int
var ok bool
for {
n, ok = rdr.Read(buf)
for i := 0; i < n; i++ {
list = append(list, buf[i])
}
if !ok {
break
}
}
_, err = rdr.Close()
if err != nil {
return nil, fmt.Errorf("read object list: %w", err)
}
return &SearchObjectsRes{
ids: list,
}, nil
}
// HashPayloadRangesPrm groups parameters of HashPayloadRanges operation.
type HashPayloadRangesPrm struct {
commonObjectPrm
objectAddressPrm
tz bool
rngs []*objectSDK.Range
salt []byte
}
// TZ sets flag to request Tillich-Zemor hashes.
func (x *HashPayloadRangesPrm) TZ() {
x.tz = true
}
// SetRanges sets a list of payload ranges to hash.
func (x *HashPayloadRangesPrm) SetRanges(rngs []*objectSDK.Range) {
x.rngs = rngs
}
// SetSalt sets data for each range to be XOR'ed with.
func (x *HashPayloadRangesPrm) SetSalt(salt []byte) {
x.salt = salt
}
// HashPayloadRangesRes groups the resulting values of HashPayloadRanges operation.
type HashPayloadRangesRes struct {
cliRes *client.ResObjectHash
}
// HashList returns a list of hashes of the payload ranges keeping order.
func (x HashPayloadRangesRes) HashList() [][]byte {
return x.cliRes.Checksums()
}
// HashPayloadRanges requests hashes (by default SHA256) of the object payload ranges.
//
// Returns any error which prevented the operation from completing correctly in error return.
// Returns an error if number of received hashes differs with the number of requested ranges.
func HashPayloadRanges(ctx context.Context, prm HashPayloadRangesPrm) (*HashPayloadRangesRes, error) {
var cliPrm client.PrmObjectHash
cliPrm.FromContainer(prm.objAddr.Container())
cliPrm.ByID(prm.objAddr.Object())
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.UseSalt(prm.salt)
rngs := make([]uint64, 2*len(prm.rngs))
for i := range prm.rngs {
rngs[2*i] = prm.rngs[i].GetOffset()
rngs[2*i+1] = prm.rngs[i].GetLength()
}
cliPrm.SetRangeList(rngs...)
if prm.tz {
cliPrm.TillichZemorAlgo()
}
if prm.sessionToken != nil {
cliPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
cliPrm.WithBearerToken(*prm.bearerToken)
}
cliPrm.WithXHeaders(prm.xHeaders...)
res, err := prm.cli.ObjectHash(ctx, cliPrm)
if err != nil {
return nil, fmt.Errorf("read payload hashes via client: %w", err)
}
return &HashPayloadRangesRes{
cliRes: res,
}, nil
}
// PayloadRangePrm groups parameters of PayloadRange operation.
type PayloadRangePrm struct {
commonObjectPrm
objectAddressPrm
rawPrm
payloadWriterPrm
rng *objectSDK.Range
}
// SetRange sets payload range to read.
func (x *PayloadRangePrm) SetRange(rng *objectSDK.Range) {
x.rng = rng
}
// PayloadRangeRes groups the resulting values of PayloadRange operation.
type PayloadRangeRes struct{}
// PayloadRange reads object payload range from FrostFS and writes it to the specified writer.
//
// Interrupts on any writer error.
//
// Returns any error which prevented the operation from completing correctly in error return.
// For raw reading, returns *object.SplitInfoError error if object is virtual.
func PayloadRange(ctx context.Context, prm PayloadRangePrm) (*PayloadRangeRes, error) {
var cliPrm client.PrmObjectRange
cliPrm.FromContainer(prm.objAddr.Container())
cliPrm.ByID(prm.objAddr.Object())
if prm.sessionToken != nil {
cliPrm.WithinSession(*prm.sessionToken)
}
if prm.bearerToken != nil {
cliPrm.WithBearerToken(*prm.bearerToken)
}
if prm.raw {
cliPrm.MarkRaw()
}
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.SetOffset(prm.rng.GetOffset())
cliPrm.SetLength(prm.rng.GetLength())
cliPrm.WithXHeaders(prm.xHeaders...)
rdr, err := prm.cli.ObjectRangeInit(ctx, cliPrm)
if err != nil {
return nil, fmt.Errorf("init payload reading: %w", err)
}
_, err = io.Copy(prm.wrt, rdr)
if err != nil {
return nil, fmt.Errorf("copy payload: %w", err)
}
return new(PayloadRangeRes), nil
}
// SyncContainerPrm groups parameters of SyncContainerSettings operation.
type SyncContainerPrm struct {
commonPrm
c *containerSDK.Container
}
// SetContainer sets a container that is required to be synced.
func (s *SyncContainerPrm) SetContainer(c *containerSDK.Container) {
s.c = c
}
// SyncContainerRes groups resulting values of SyncContainerSettings
// operation.
type SyncContainerRes struct{}
// SyncContainerSettings reads global network config from FrostFS and
// syncs container settings with it.
//
// Interrupts on any writer error.
//
// Panics if a container passed as a parameter is nil.
func SyncContainerSettings(ctx context.Context, prm SyncContainerPrm) (*SyncContainerRes, error) {
if prm.c == nil {
panic("sync container settings with the network: nil container")
}
err := client.SyncContainerWithNetwork(ctx, prm.c, prm.cli)
if err != nil {
return nil, err
}
return new(SyncContainerRes), nil
}

View file

@ -0,0 +1,15 @@
// Package internal provides functionality for FrostFS CLI application
// communication with FrostFS network.
//
// The base client for accessing remote nodes via FrostFS API is a FrostFS SDK
// Go API client. However, although it encapsulates a useful piece of business
// logic (e.g. the signature mechanism), the FrostFS CLI application does not
// fully use the client's flexible interface.
//
// In this regard, this package provides functions over base API client
// necessary for the application. This allows you to concentrate the entire
// spectrum of the client's use in one place (this will be convenient both when
// updating the base client and for evaluating the UX of SDK library). So it is
// expected that all application packages will be limited to this package for
// the development of functionality requiring FrostFS API communication.
package internal

View file

@ -0,0 +1,92 @@
package internal
import (
"io"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
)
// here are small structures with public setters to share between parameter structures
type commonPrm struct {
cli *client.Client
}
// SetClient sets the base client for FrostFS API communication.
func (x *commonPrm) SetClient(cli *client.Client) {
x.cli = cli
}
type containerIDPrm struct {
cnrID cid.ID
}
// SetContainerID sets the container identifier.
func (x *containerIDPrm) SetContainerID(id cid.ID) {
x.cnrID = id
}
type bearerTokenPrm struct {
bearerToken *bearer.Token
}
// SetBearerToken sets the bearer token to be attached to the request.
func (x *bearerTokenPrm) SetBearerToken(tok *bearer.Token) {
x.bearerToken = tok
}
type objectAddressPrm struct {
objAddr oid.Address
}
func (x *objectAddressPrm) SetAddress(addr oid.Address) {
x.objAddr = addr
}
type rawPrm struct {
raw bool
}
// SetRawFlag sets flag of raw request.
func (x *rawPrm) SetRawFlag(raw bool) {
x.raw = raw
}
type payloadWriterPrm struct {
wrt io.Writer
}
// SetPayloadWriter sets the writer of the object payload.
func (x *payloadWriterPrm) SetPayloadWriter(wrt io.Writer) {
x.wrt = wrt
}
type commonObjectPrm struct {
commonPrm
bearerTokenPrm
sessionToken *session.Object
local bool
xHeaders []string
}
// SetTTL sets request TTL value.
func (x *commonObjectPrm) SetTTL(ttl uint32) {
x.local = ttl < 2
}
// SetXHeaders sets request X-Headers.
func (x *commonObjectPrm) SetXHeaders(hs []string) {
x.xHeaders = hs
}
// SetSessionToken sets the token of the session within which the request should be sent.
func (x *commonObjectPrm) SetSessionToken(tok *session.Object) {
x.sessionToken = tok
}

View file

@ -0,0 +1,101 @@
package internal
import (
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"errors"
"fmt"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network"
tracing "git.frostfs.info/TrueCloudLab/frostfs-observability/tracing/grpc"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"google.golang.org/grpc"
)
var errInvalidEndpoint = errors.New("provided RPC endpoint is incorrect")
// GetSDKClientByFlag returns default frostfs-sdk-go client using the specified flag for the address.
// On error, outputs to stderr of cmd and exits with non-zero code.
func GetSDKClientByFlag(cmd *cobra.Command, key *ecdsa.PrivateKey, endpointFlag string) *client.Client {
cli, err := getSDKClientByFlag(cmd, key, endpointFlag)
if err != nil {
commonCmd.ExitOnErr(cmd, "can't create API client: %w", err)
}
return cli
}
func getSDKClientByFlag(cmd *cobra.Command, key *ecdsa.PrivateKey, endpointFlag string) (*client.Client, error) {
var addr network.Address
err := addr.FromString(viper.GetString(endpointFlag))
if err != nil {
return nil, fmt.Errorf("%v: %w", errInvalidEndpoint, err)
}
return GetSDKClient(cmd.Context(), cmd, key, addr)
}
// GetSDKClient returns default frostfs-sdk-go client.
func GetSDKClient(ctx context.Context, cmd *cobra.Command, key *ecdsa.PrivateKey, addr network.Address) (*client.Client, error) {
var (
c client.Client
prmInit client.PrmInit
prmDial client.PrmDial
)
prmInit.SetDefaultPrivateKey(*key)
prmInit.ResolveFrostFSFailures()
prmDial.SetServerURI(addr.URIAddr())
if timeout := viper.GetDuration(commonflags.Timeout); timeout > 0 {
// In CLI we can only set a timeout for the whole operation.
// By also setting stream timeout we ensure that no operation hands
// for too long.
prmDial.SetTimeout(timeout)
prmDial.SetStreamTimeout(timeout)
common.PrintVerbose(cmd, "Set request timeout to %s.", timeout)
}
prmDial.SetGRPCDialOptions(
grpc.WithChainUnaryInterceptor(tracing.NewUnaryClientInteceptor()),
grpc.WithChainStreamInterceptor(tracing.NewStreamClientInterceptor()))
c.Init(prmInit)
if err := c.Dial(ctx, prmDial); err != nil {
return nil, fmt.Errorf("can't init SDK client: %w", err)
}
return &c, nil
}
// GetCurrentEpoch returns current epoch.
func GetCurrentEpoch(ctx context.Context, cmd *cobra.Command, endpoint string) (uint64, error) {
var addr network.Address
if err := addr.FromString(endpoint); err != nil {
return 0, fmt.Errorf("can't parse RPC endpoint: %w", err)
}
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
return 0, fmt.Errorf("can't generate key to sign query: %w", err)
}
c, err := GetSDKClient(ctx, cmd, key, addr)
if err != nil {
return 0, err
}
ni, err := c.NetworkInfo(ctx, client.PrmNetworkInfo{})
if err != nil {
return 0, err
}
return ni.Info().CurrentEpoch(), nil
}

View file

@ -0,0 +1,50 @@
package common
import (
"errors"
"os"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/core/version"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
versionSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/version"
"github.com/spf13/cobra"
)
var errUnsupportedEACLFormat = errors.New("unsupported eACL format")
// ReadEACL reads extended ACL table from eaclPath.
func ReadEACL(cmd *cobra.Command, eaclPath string) *eacl.Table {
_, err := os.Stat(eaclPath) // check if `eaclPath` is an existing file
if err != nil {
commonCmd.ExitOnErr(cmd, "", errors.New("incorrect path to file with EACL"))
}
PrintVerbose(cmd, "Reading EACL from file: %s", eaclPath)
data, err := os.ReadFile(eaclPath)
commonCmd.ExitOnErr(cmd, "can't read file with EACL: %w", err)
table := eacl.NewTable()
if err = table.UnmarshalJSON(data); err == nil {
validateAndFixEACLVersion(table)
PrintVerbose(cmd, "Parsed JSON encoded EACL table")
return table
}
if err = table.Unmarshal(data); err == nil {
validateAndFixEACLVersion(table)
PrintVerbose(cmd, "Parsed binary encoded EACL table")
return table
}
commonCmd.ExitOnErr(cmd, "", errUnsupportedEACLFormat)
return nil
}
func validateAndFixEACLVersion(table *eacl.Table) {
if !version.IsValid(table.Version()) {
table.SetVersion(versionSDK.Current())
}
}

View file

@ -0,0 +1,28 @@
package common
import (
"fmt"
"strconv"
"github.com/spf13/cobra"
)
// ParseEpoch parses epoch argument. Second return value is true if
// the specified epoch is relative, and false otherwise.
func ParseEpoch(cmd *cobra.Command, flag string) (uint64, bool, error) {
s, _ := cmd.Flags().GetString(flag)
if len(s) == 0 {
return 0, false, nil
}
relative := s[0] == '+'
if relative {
s = s[1:]
}
epoch, err := strconv.ParseUint(s, 10, 64)
if err != nil {
return 0, relative, fmt.Errorf("can't parse epoch for %s argument: %w", flag, err)
}
return epoch, relative, nil
}

View file

@ -0,0 +1,23 @@
package common
import (
"bytes"
"encoding/json"
"github.com/spf13/cobra"
)
// PrettyPrintJSON prints m as an indented JSON to the cmd output.
func PrettyPrintJSON(cmd *cobra.Command, m json.Marshaler, entity string) {
data, err := m.MarshalJSON()
if err != nil {
PrintVerbose(cmd, "Can't convert %s to json: %w", entity, err)
return
}
buf := new(bytes.Buffer)
if err := json.Indent(buf, data, "", " "); err != nil {
PrintVerbose(cmd, "Can't pretty print json: %w", err)
return
}
cmd.Println(buf)
}

View file

@ -0,0 +1,67 @@
package common
import (
"encoding/json"
"errors"
"fmt"
"os"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
"github.com/spf13/cobra"
)
// ReadBearerToken reads bearer token from the path provided in a specified flag.
func ReadBearerToken(cmd *cobra.Command, flagname string) *bearer.Token {
path, err := cmd.Flags().GetString(flagname)
commonCmd.ExitOnErr(cmd, "", err)
if len(path) == 0 {
return nil
}
PrintVerbose(cmd, "Reading bearer token from file [%s]...", path)
var tok bearer.Token
err = ReadBinaryOrJSON(cmd, &tok, path)
commonCmd.ExitOnErr(cmd, "invalid bearer token: %v", err)
return &tok
}
// BinaryOrJSON is an interface of entities which provide json.Unmarshaler
// and FrostFS binary decoder.
type BinaryOrJSON interface {
Unmarshal([]byte) error
json.Unmarshaler
}
// ReadBinaryOrJSON reads file data using provided path and decodes
// BinaryOrJSON from the data.
func ReadBinaryOrJSON(cmd *cobra.Command, dst BinaryOrJSON, fPath string) error {
PrintVerbose(cmd, "Reading file [%s]...", fPath)
// try to read session token from file
data, err := os.ReadFile(fPath)
if err != nil {
return fmt.Errorf("read file <%s>: %w", fPath, err)
}
PrintVerbose(cmd, "Trying to decode binary...")
err = dst.Unmarshal(data)
if err != nil {
PrintVerbose(cmd, "Failed to decode binary: %v", err)
PrintVerbose(cmd, "Trying to decode JSON...")
err = dst.UnmarshalJSON(data)
if err != nil {
PrintVerbose(cmd, "Failed to decode JSON: %v", err)
return errors.New("invalid format")
}
}
return nil
}

View file

@ -0,0 +1,62 @@
package common
import (
"context"
"sort"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/misc"
"git.frostfs.info/TrueCloudLab/frostfs-observability/tracing"
"github.com/spf13/cobra"
"go.opentelemetry.io/otel/trace"
)
type spanKey struct{}
// StopClientCommandSpan stops tracing span for the command and prints trace ID on the standard output.
func StopClientCommandSpan(cmd *cobra.Command, _ []string) {
span, ok := cmd.Context().Value(spanKey{}).(trace.Span)
if !ok {
return
}
span.End()
// Noop provider cannot fail on flush.
_ = tracing.Shutdown(cmd.Context())
cmd.PrintErrf("Trace ID: %s\n", span.SpanContext().TraceID())
}
// StartClientCommandSpan starts tracing span for the command.
func StartClientCommandSpan(cmd *cobra.Command) {
enableTracing, err := cmd.Flags().GetBool(commonflags.TracingFlag)
if err != nil || !enableTracing {
return
}
_, err = tracing.Setup(cmd.Context(), tracing.Config{
Enabled: true,
Exporter: tracing.NoOpExporter,
Service: "frostfs-cli",
Version: misc.Version,
})
commonCmd.ExitOnErr(cmd, "init tracing: %w", err)
var components sort.StringSlice
for c := cmd; c != nil; c = c.Parent() {
components = append(components, c.Name())
}
for i, j := 0, len(components)-1; i < j; {
components.Swap(i, j)
i++
j--
}
operation := strings.Join(components, ".")
ctx, span := tracing.StartSpanFromContext(cmd.Context(), operation)
ctx = context.WithValue(ctx, spanKey{}, span)
cmd.SetContext(ctx)
}

View file

@ -0,0 +1,46 @@
package common
import (
"encoding/hex"
"strconv"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/checksum"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// PrintVerbose prints to the stdout if the commonflags.Verbose flag is on.
func PrintVerbose(cmd *cobra.Command, format string, a ...any) {
if viper.GetBool(commonflags.Verbose) {
cmd.Printf(format+"\n", a...)
}
}
// PrettyPrintUnixTime interprets s as unix timestamp and prints it as
// a date. Is s is invalid, "malformed" is returned.
func PrettyPrintUnixTime(s string) string {
unixTime, err := strconv.ParseInt(s, 10, 64)
if err != nil {
return "malformed"
}
timestamp := time.Unix(unixTime, 0)
return timestamp.String()
}
// PrintChecksum prints checksum.
func PrintChecksum(cmd *cobra.Command, name string, recv func() (checksum.Checksum, bool)) {
var strVal string
cs, csSet := recv()
if csSet {
strVal = hex.EncodeToString(cs.Value())
} else {
strVal = "<empty>"
}
cmd.Printf("%s: %s\n", name, strVal)
}

View file

@ -0,0 +1,33 @@
package commonflags
import (
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
TTL = "ttl"
TTLShorthand = ""
TTLDefault = 2
TTLUsage = "TTL value in request meta header"
XHeadersKey = "xhdr"
XHeadersShorthand = "x"
XHeadersUsage = "Request X-Headers in form of Key=Value"
)
// InitAPI inits common flags for storage node services.
func InitAPI(cmd *cobra.Command) {
ff := cmd.Flags()
ff.StringSliceP(XHeadersKey, XHeadersShorthand, []string{}, XHeadersUsage)
ff.Uint32P(TTL, TTLShorthand, TTLDefault, TTLUsage)
}
// BindAPI binds API flags of storage node services to the viper.
func BindAPI(cmd *cobra.Command) {
ff := cmd.Flags()
_ = viper.BindPFlag(TTL, ff.Lookup(TTL))
_ = viper.BindPFlag(XHeadersKey, ff.Lookup(XHeadersKey))
}

View file

@ -0,0 +1,9 @@
package commonflags
const (
// ExpireAt is a flag for setting last epoch of an object or a token.
ExpireAt = "expire-at"
// Lifetime is a flag for setting the lifetime of an object or a token,
// starting from the current epoch.
Lifetime = "lifetime"
)

View file

@ -0,0 +1,89 @@
package commonflags
import (
"time"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// Common CLI flag keys, shorthands, default
// values and their usage descriptions.
const (
GenerateKey = "generate-key"
generateKeyShorthand = "g"
generateKeyDefault = false
generateKeyUsage = "Generate new private key"
WalletPath = "wallet"
WalletPathShorthand = "w"
WalletPathDefault = ""
WalletPathUsage = "Path to the wallet or binary key"
Account = "address"
AccountShorthand = ""
AccountDefault = ""
AccountUsage = "Address of wallet account"
RPC = "rpc-endpoint"
RPCShorthand = "r"
RPCDefault = ""
RPCUsage = "Remote node address (as 'multiaddr' or '<host>:<port>')"
Timeout = "timeout"
TimeoutShorthand = "t"
TimeoutDefault = 15 * time.Second
TimeoutUsage = "Timeout for an operation"
Verbose = "verbose"
VerboseShorthand = "v"
VerboseUsage = "Verbose output"
ForceFlag = "force"
ForceFlagShorthand = "f"
CIDFlag = "cid"
CIDFlagUsage = "Container ID."
OIDFlag = "oid"
OIDFlagUsage = "Object ID."
TracingFlag = "trace"
TracingFlagUsage = "Generate trace ID and print it."
)
// Init adds common flags to the command:
// - GenerateKey,
// - WalletPath,
// - Account,
// - RPC,
// - Tracing,
// - Timeout.
func Init(cmd *cobra.Command) {
InitWithoutRPC(cmd)
ff := cmd.Flags()
ff.StringP(RPC, RPCShorthand, RPCDefault, RPCUsage)
ff.Bool(TracingFlag, false, TracingFlagUsage)
ff.DurationP(Timeout, TimeoutShorthand, TimeoutDefault, TimeoutUsage)
}
// InitWithoutRPC is similar to Init but doesn't create the RPC flag.
func InitWithoutRPC(cmd *cobra.Command) {
ff := cmd.Flags()
ff.BoolP(GenerateKey, generateKeyShorthand, generateKeyDefault, generateKeyUsage)
ff.StringP(WalletPath, WalletPathShorthand, WalletPathDefault, WalletPathUsage)
ff.StringP(Account, AccountShorthand, AccountDefault, AccountUsage)
}
// Bind binds common command flags to the viper.
func Bind(cmd *cobra.Command) {
ff := cmd.Flags()
_ = viper.BindPFlag(GenerateKey, ff.Lookup(GenerateKey))
_ = viper.BindPFlag(WalletPath, ff.Lookup(WalletPath))
_ = viper.BindPFlag(Account, ff.Lookup(Account))
_ = viper.BindPFlag(RPC, ff.Lookup(RPC))
_ = viper.BindPFlag(Timeout, ff.Lookup(Timeout))
}

View file

@ -0,0 +1,3 @@
package commonflags
const JSON = "json"

View file

@ -0,0 +1,19 @@
package commonflags
import (
"fmt"
"github.com/spf13/cobra"
)
const SessionToken = "session"
// InitSession registers SessionToken flag representing file path to the token of
// the session with the given name. Supports FrostFS-binary and JSON files.
func InitSession(cmd *cobra.Command, name string) {
cmd.Flags().String(
SessionToken,
"",
fmt.Sprintf("Filepath to a JSON- or binary-encoded token of the %s session", name),
)
}

View file

@ -1,28 +1,38 @@
package cmd
package key
import (
"bytes"
"errors"
"io"
"os"
"path/filepath"
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
"golang.org/x/term"
)
func Test_getKey(t *testing.T) {
var testCmd = &cobra.Command{
Use: "test",
Short: "test",
Run: func(cmd *cobra.Command, args []string) {},
}
func Test_getOrGenerate(t *testing.T) {
dir := t.TempDir()
wallPath := filepath.Join(dir, "wallet.json")
w, err := wallet.NewWallet(wallPath)
require.NoError(t, err)
badWallPath := filepath.Join(dir, "bad_wallet.json")
require.NoError(t, os.WriteFile(badWallPath, []byte("bad content"), os.ModePerm))
acc1, err := wallet.NewAccount()
require.NoError(t, err)
require.NoError(t, acc1.Encrypt("pass", keys.NEP2ScryptParams()))
@ -34,7 +44,6 @@ func Test_getKey(t *testing.T) {
acc2.Default = true
w.AddAccount(acc2)
require.NoError(t, w.Save())
w.Close()
keyPath := filepath.Join(dir, "binary.key")
rawKey, err := keys.NewPrivateKey()
@ -55,52 +64,37 @@ func Test_getKey(t *testing.T) {
Writer: io.Discard,
}, "")
checkKeyError(t, filepath.Join(dir, "badfile"), errInvalidKey)
checkKeyError(t, filepath.Join(dir, "badfile"), ErrFs)
checkKeyError(t, badWallPath, ErrInvalidKey)
t.Run("wallet", func(t *testing.T) {
checkKeyError(t, wallPath, errInvalidPassword)
checkKeyError(t, wallPath, ErrInvalidPassword)
in.WriteString("invalid\r")
checkKeyError(t, wallPath, errInvalidPassword)
checkKeyError(t, wallPath, ErrInvalidPassword)
in.WriteString("pass\r")
checkKey(t, wallPath, acc2.PrivateKey()) // default account
viper.Set(address, acc1.Address)
viper.Set(commonflags.Account, acc1.Address)
in.WriteString("pass\r")
checkKey(t, wallPath, acc1.PrivateKey())
viper.Set(address, "not an address")
checkKeyError(t, wallPath, errInvalidAddress)
viper.Set(commonflags.Account, "not an address")
checkKeyError(t, wallPath, ErrInvalidAddress)
acc, err := wallet.NewAccount()
require.NoError(t, err)
viper.Set(address, acc.Address)
checkKeyError(t, wallPath, errInvalidAddress)
viper.Set(commonflags.Account, acc.Address)
checkKeyError(t, wallPath, ErrInvalidAddress)
})
t.Run("WIF", func(t *testing.T) {
checkKey(t, wifKey.WIF(), wifKey)
checkKeyError(t, wifKey.WIF(), ErrFs)
})
t.Run("NEP-2", func(t *testing.T) {
checkKeyError(t, nep2, errInvalidPassword)
in.WriteString("invalid\r")
checkKeyError(t, nep2, errInvalidPassword)
in.WriteString("pass\r")
checkKey(t, nep2, nep2Key)
t.Run("password from config", func(t *testing.T) {
viper.Set(password, "invalid")
in.WriteString("pass\r")
checkKeyError(t, nep2, errInvalidPassword)
viper.Set(password, "pass")
in.WriteString("invalid\r")
checkKey(t, nep2, nep2Key)
})
checkKeyError(t, nep2, ErrFs)
})
t.Run("raw key", func(t *testing.T) {
@ -108,8 +102,8 @@ func Test_getKey(t *testing.T) {
})
t.Run("generate", func(t *testing.T) {
viper.Set(generateKey, true)
actual, err := getKey()
viper.Set(commonflags.GenerateKey, true)
actual, err := getOrGenerate(testCmd)
require.NoError(t, err)
require.NotNil(t, actual)
for _, p := range []*keys.PrivateKey{nep2Key, rawKey, wifKey, acc1.PrivateKey(), acc2.PrivateKey()} {
@ -119,14 +113,14 @@ func Test_getKey(t *testing.T) {
}
func checkKeyError(t *testing.T, desc string, err error) {
viper.Set(walletPath, desc)
_, actualErr := getKey()
require.True(t, errors.Is(actualErr, err), "got: %v", actualErr)
viper.Set(commonflags.WalletPath, desc)
_, actualErr := getOrGenerate(testCmd)
require.ErrorIs(t, actualErr, err)
}
func checkKey(t *testing.T, desc string, expected *keys.PrivateKey) {
viper.Set(walletPath, desc)
actual, err := getKey()
viper.Set(commonflags.WalletPath, desc)
actual, err := getOrGenerate(testCmd)
require.NoError(t, err)
require.Equal(t, &expected.PrivateKey, actual)
}

View file

@ -0,0 +1,62 @@
package key
import (
"crypto/ecdsa"
"errors"
"fmt"
"os"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var errCantGenerateKey = errors.New("can't generate new private key")
// Get returns private key from wallet or binary file.
// Ideally we want to touch file-system on the last step.
// This function assumes that all flags were bind to viper in a `PersistentPreRun`.
func Get(cmd *cobra.Command) *ecdsa.PrivateKey {
pk, err := get(cmd)
commonCmd.ExitOnErr(cmd, "can't fetch private key: %w", err)
return pk
}
func get(cmd *cobra.Command) (*ecdsa.PrivateKey, error) {
keyDesc := viper.GetString(commonflags.WalletPath)
data, err := os.ReadFile(keyDesc)
if err != nil {
return nil, fmt.Errorf("%w: %v", ErrFs, err)
}
priv, err := keys.NewPrivateKeyFromBytes(data)
if err != nil {
w, err := wallet.NewWalletFromFile(keyDesc)
if err == nil {
return FromWallet(cmd, w, viper.GetString(commonflags.Account))
}
return nil, fmt.Errorf("%w: %v", ErrInvalidKey, err)
}
return &priv.PrivateKey, nil
}
// GetOrGenerate is similar to get but generates a new key if commonflags.GenerateKey is set.
func GetOrGenerate(cmd *cobra.Command) *ecdsa.PrivateKey {
pk, err := getOrGenerate(cmd)
commonCmd.ExitOnErr(cmd, "can't fetch private key: %w", err)
return pk
}
func getOrGenerate(cmd *cobra.Command) (*ecdsa.PrivateKey, error) {
if viper.GetBool(commonflags.GenerateKey) {
priv, err := keys.NewPrivateKey()
if err != nil {
return nil, fmt.Errorf("%w: %v", errCantGenerateKey, err)
}
return &priv.PrivateKey, nil
}
return get(cmd)
}

View file

@ -0,0 +1,70 @@
package key
import (
"crypto/ecdsa"
"errors"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/nspcc-dev/neo-go/cli/flags"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// Key-related errors.
var (
ErrFs = errors.New("unable to read file from given path")
ErrInvalidKey = errors.New("provided key is incorrect, only wallet or binary key supported")
ErrInvalidAddress = errors.New("--address option must be specified and valid")
ErrInvalidPassword = errors.New("invalid password for the encrypted key")
)
// FromWallet returns private key of the wallet account.
func FromWallet(cmd *cobra.Command, w *wallet.Wallet, addrStr string) (*ecdsa.PrivateKey, error) {
var (
addr util.Uint160
err error
)
if addrStr == "" {
common.PrintVerbose(cmd, "Using default wallet address")
addr = w.GetChangeAddress()
} else {
addr, err = flags.ParseAddress(addrStr)
if err != nil {
common.PrintVerbose(cmd, "Can't parse address: %s", addrStr)
return nil, ErrInvalidAddress
}
}
acc := w.GetAccount(addr)
if acc == nil {
common.PrintVerbose(cmd, "Can't find wallet account for %s", addrStr)
return nil, ErrInvalidAddress
}
pass, err := getPassword()
if err != nil {
common.PrintVerbose(cmd, "Can't read password: %v", err)
return nil, ErrInvalidPassword
}
if err := acc.Decrypt(pass, keys.NEP2ScryptParams()); err != nil {
common.PrintVerbose(cmd, "Can't decrypt account: %v", err)
return nil, ErrInvalidPassword
}
return &acc.PrivateKey().PrivateKey, nil
}
func getPassword() (string, error) {
// this check allows empty passwords
if viper.IsSet("password") {
return viper.GetString("password"), nil
}
return input.ReadPassword("Enter password > ")
}

7
cmd/frostfs-cli/main.go Normal file
View file

@ -0,0 +1,7 @@
package main
import cmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules"
func main() {
cmd.Execute()
}

View file

@ -0,0 +1,70 @@
package accounting
import (
"math/big"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/precision"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/accounting"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
ownerFlag = "owner"
)
var accountingBalanceCmd = &cobra.Command{
Use: "balance",
Short: "Get internal balance of FrostFS account",
Long: `Get internal balance of FrostFS account`,
Run: func(cmd *cobra.Command, args []string) {
var idUser user.ID
pk := key.GetOrGenerate(cmd)
balanceOwner, _ := cmd.Flags().GetString(ownerFlag)
if balanceOwner == "" {
user.IDFromKey(&idUser, pk.PublicKey)
} else {
commonCmd.ExitOnErr(cmd, "can't decode owner ID wallet address: %w", idUser.DecodeString(balanceOwner))
}
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
var prm internalclient.BalanceOfPrm
prm.SetClient(cli)
prm.SetAccount(idUser)
res, err := internalclient.BalanceOf(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
// print to stdout
prettyPrintDecimal(cmd, res.Balance())
},
}
func initAccountingBalanceCmd() {
ff := accountingBalanceCmd.Flags()
ff.StringP(commonflags.WalletPath, commonflags.WalletPathShorthand, commonflags.WalletPathDefault, commonflags.WalletPathUsage)
ff.StringP(commonflags.Account, commonflags.AccountShorthand, commonflags.AccountDefault, commonflags.AccountUsage)
ff.StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage)
ff.String(ownerFlag, "", "owner of balance account (omit to use owner from private key)")
}
func prettyPrintDecimal(cmd *cobra.Command, decimal accounting.Decimal) {
if viper.GetBool(commonflags.Verbose) {
cmd.Println("value:", decimal.Value())
cmd.Println("precision:", decimal.Precision())
} else {
amountF8 := precision.Convert(decimal.Precision(), 8, big.NewInt(decimal.Value()))
cmd.Println(fixedn.ToString(amountF8, 8))
}
}

View file

@ -0,0 +1,30 @@
package accounting
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// Cmd represents the accounting command.
var Cmd = &cobra.Command{
Use: "accounting",
Short: "Operations with accounts and balances",
Long: `Operations with accounts and balances`,
PersistentPreRun: func(cmd *cobra.Command, args []string) {
flags := cmd.Flags()
_ = viper.BindPFlag(commonflags.WalletPath, flags.Lookup(commonflags.WalletPath))
_ = viper.BindPFlag(commonflags.Account, flags.Lookup(commonflags.Account))
_ = viper.BindPFlag(commonflags.RPC, flags.Lookup(commonflags.RPC))
common.StartClientCommandSpan(cmd)
},
PersistentPostRun: common.StopClientCommandSpan,
}
func init() {
Cmd.AddCommand(accountingBalanceCmd)
initAccountingBalanceCmd()
}

View file

@ -0,0 +1,28 @@
package basic
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/acl"
"github.com/spf13/cobra"
)
var printACLCmd = &cobra.Command{
Use: "print",
Short: "Pretty print basic ACL from the HEX representation",
Example: `frostfs-cli acl basic print 0x1C8C8CCC`,
Long: `Pretty print basic ACL from the HEX representation.
Few roles have exclusive default access to set of operation, even if particular bit deny it.
Container have access to the operations of the data replication mechanism:
Get, Head, Put, Search, Hash.
InnerRing members are allowed to data audit ops only:
Get, Head, Hash, Search.`,
Run: printACL,
Args: cobra.ExactArgs(1),
}
func printACL(cmd *cobra.Command, args []string) {
var bacl acl.Basic
commonCmd.ExitOnErr(cmd, "unable to parse basic acl: %w", bacl.DecodeString(args[0]))
util.PrettyPrintTableBACL(cmd, &bacl)
}

View file

@ -0,0 +1,14 @@
package basic
import (
"github.com/spf13/cobra"
)
var Cmd = &cobra.Command{
Use: "basic",
Short: "Operations with Basic Access Control Lists",
}
func init() {
Cmd.AddCommand(printACLCmd)
}

View file

@ -0,0 +1,127 @@
package extended
import (
"bytes"
"encoding/json"
"os"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/spf13/cobra"
)
var createCmd = &cobra.Command{
Use: "create",
Short: "Create extended ACL from the text representation",
Long: `Create extended ACL from the text representation.
Rule consist of these blocks: <action> <operation> [<filter1> ...] [<target1> ...]
Action is 'allow' or 'deny'.
Operation is an object service verb: 'get', 'head', 'put', 'search', 'delete', 'getrange', or 'getrangehash'.
Filter consists of <typ>:<key><match><value>
Typ is 'obj' for object applied filter or 'req' for request applied filter.
Key is a valid unicode string corresponding to object or request header key.
Well-known system object headers start with '$Object:' prefix.
User defined headers start without prefix.
Read more about filter keys at git.frostfs.info.com/TrueCloudLab/frostfs-api/src/branch/master/proto-docs/acl.md#message-eaclrecordfilter
Match is '=' for matching and '!=' for non-matching filter.
Value is a valid unicode string corresponding to object or request header value.
Target is
'user' for container owner,
'system' for Storage nodes in container and Inner Ring nodes,
'others' for all other request senders,
'pubkey:<key1>,<key2>,...' for exact request sender, where <key> is a hex-encoded 33-byte public key.
When both '--rule' and '--file' arguments are used, '--rule' records will be placed higher in resulting extended ACL table.
`,
Example: `frostfs-cli acl extended create --cid EutHBsdT1YCzHxjCfQHnLPL1vFrkSyLSio4vkphfnEk -f rules.txt --out table.json
frostfs-cli acl extended create --cid EutHBsdT1YCzHxjCfQHnLPL1vFrkSyLSio4vkphfnEk -r 'allow get obj:Key=Value others' -r 'deny put others'`,
Run: createEACL,
}
func init() {
createCmd.Flags().StringArrayP("rule", "r", nil, "Extended ACL table record to apply")
createCmd.Flags().StringP("file", "f", "", "Read list of extended ACL table records from text file")
createCmd.Flags().StringP("out", "o", "", "Save JSON formatted extended ACL table in file")
createCmd.Flags().StringP(commonflags.CIDFlag, "", "", commonflags.CIDFlagUsage)
_ = cobra.MarkFlagFilename(createCmd.Flags(), "file")
_ = cobra.MarkFlagFilename(createCmd.Flags(), "out")
}
func createEACL(cmd *cobra.Command, _ []string) {
rules, _ := cmd.Flags().GetStringArray("rule")
fileArg, _ := cmd.Flags().GetString("file")
outArg, _ := cmd.Flags().GetString("out")
cidArg, _ := cmd.Flags().GetString(commonflags.CIDFlag)
var containerID cid.ID
if cidArg != "" {
if err := containerID.DecodeString(cidArg); err != nil {
cmd.PrintErrf("invalid container ID: %v\n", err)
os.Exit(1)
}
}
rulesFile, err := getRulesFromFile(fileArg)
if err != nil {
cmd.PrintErrf("can't read rules from file: %v\n", err)
os.Exit(1)
}
rules = append(rules, rulesFile...)
if len(rules) == 0 {
cmd.PrintErrln("no extended ACL rules has been provided")
os.Exit(1)
}
tb := eacl.NewTable()
commonCmd.ExitOnErr(cmd, "unable to parse provided rules: %w", util.ParseEACLRules(tb, rules))
tb.SetCID(containerID)
data, err := tb.MarshalJSON()
if err != nil {
cmd.PrintErrln(err)
os.Exit(1)
}
buf := new(bytes.Buffer)
err = json.Indent(buf, data, "", " ")
if err != nil {
cmd.PrintErrln(err)
os.Exit(1)
}
if len(outArg) == 0 {
cmd.Println(buf)
return
}
err = os.WriteFile(outArg, buf.Bytes(), 0644)
if err != nil {
cmd.PrintErrln(err)
os.Exit(1)
}
}
func getRulesFromFile(filename string) ([]string, error) {
if len(filename) == 0 {
return nil, nil
}
data, err := os.ReadFile(filename)
if err != nil {
return nil, err
}
return strings.Split(strings.TrimSpace(string(data)), "\n"), nil
}

View file

@ -1,10 +1,10 @@
package extended
import (
"strings"
"testing"
"github.com/nspcc-dev/neofs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/stretchr/testify/require"
)
@ -63,8 +63,7 @@ func TestParseTable(t *testing.T) {
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
ss := strings.Split(test.rule, " ")
err := parseTable(eaclTable, ss)
err := util.ParseEACLRule(eaclTable, test.rule)
ok := len(test.jsonRecord) > 0
require.Equal(t, ok, err == nil, err)
if ok {

View file

@ -0,0 +1,38 @@
package extended
import (
"os"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/spf13/cobra"
)
var printEACLCmd = &cobra.Command{
Use: "print",
Short: "Pretty print extended ACL from the file(in text or json format) or for given container.",
Run: printEACL,
}
func init() {
flags := printEACLCmd.Flags()
flags.StringP("file", "f", "",
"Read list of extended ACL table records from text or json file")
_ = printEACLCmd.MarkFlagRequired("file")
}
func printEACL(cmd *cobra.Command, _ []string) {
file, _ := cmd.Flags().GetString("file")
eaclTable := new(eacl.Table)
data, err := os.ReadFile(file)
commonCmd.ExitOnErr(cmd, "can't read file with EACL: %w", err)
if strings.HasSuffix(file, ".json") {
commonCmd.ExitOnErr(cmd, "unable to parse json: %w", eaclTable.UnmarshalJSON(data))
} else {
rules := strings.Split(strings.TrimSpace(string(data)), "\n")
commonCmd.ExitOnErr(cmd, "can't parse file with EACL: %w", util.ParseEACLRules(eaclTable, rules))
}
util.PrettyPrintTableEACL(cmd, eaclTable)
}

View file

@ -1,6 +1,8 @@
package extended
import "github.com/spf13/cobra"
import (
"github.com/spf13/cobra"
)
var Cmd = &cobra.Command{
Use: "extended",
@ -9,4 +11,5 @@ var Cmd = &cobra.Command{
func init() {
Cmd.AddCommand(createCmd)
Cmd.AddCommand(printEACLCmd)
}

View file

@ -0,0 +1,17 @@
package acl
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/acl/basic"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/acl/extended"
"github.com/spf13/cobra"
)
var Cmd = &cobra.Command{
Use: "acl",
Short: "Operations with Access Control Lists",
}
func init() {
Cmd.AddCommand(extended.Cmd)
Cmd.AddCommand(basic.Cmd)
}

View file

@ -0,0 +1,126 @@
package bearer
import (
"context"
"encoding/json"
"fmt"
"os"
"time"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
eaclSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra"
)
const (
eaclFlag = "eacl"
issuedAtFlag = "issued-at"
notValidBeforeFlag = "not-valid-before"
ownerFlag = "owner"
outFlag = "out"
jsonFlag = commonflags.JSON
)
var createCmd = &cobra.Command{
Use: "create",
Short: "Create bearer token",
Long: `Create bearer token.
All epoch flags can be specified relative to the current epoch with the +n syntax.
In this case --` + commonflags.RPC + ` flag should be specified and the epoch in bearer token
is set to current epoch + n.
`,
Run: createToken,
}
func init() {
createCmd.Flags().StringP(eaclFlag, "e", "", "Path to the extended ACL table")
createCmd.Flags().StringP(issuedAtFlag, "i", "", "Epoch to issue token at")
createCmd.Flags().StringP(notValidBeforeFlag, "n", "", "Not valid before epoch")
createCmd.Flags().StringP(commonflags.ExpireAt, "x", "", "The last active epoch for the token")
createCmd.Flags().StringP(ownerFlag, "o", "", "Token owner")
createCmd.Flags().String(outFlag, "", "File to write token to")
createCmd.Flags().Bool(jsonFlag, false, "Output token in JSON")
createCmd.Flags().StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage)
_ = cobra.MarkFlagFilename(createCmd.Flags(), eaclFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), issuedAtFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), notValidBeforeFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), commonflags.ExpireAt)
_ = cobra.MarkFlagRequired(createCmd.Flags(), ownerFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), outFlag)
}
func createToken(cmd *cobra.Command, _ []string) {
iat, iatRelative, err := common.ParseEpoch(cmd, issuedAtFlag)
commonCmd.ExitOnErr(cmd, "can't parse --"+issuedAtFlag+" flag: %w", err)
exp, expRelative, err := common.ParseEpoch(cmd, commonflags.ExpireAt)
commonCmd.ExitOnErr(cmd, "can't parse --"+commonflags.ExpireAt+" flag: %w", err)
nvb, nvbRelative, err := common.ParseEpoch(cmd, notValidBeforeFlag)
commonCmd.ExitOnErr(cmd, "can't parse --"+notValidBeforeFlag+" flag: %w", err)
if iatRelative || expRelative || nvbRelative {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*30)
defer cancel()
endpoint, _ := cmd.Flags().GetString(commonflags.RPC)
currEpoch, err := internalclient.GetCurrentEpoch(ctx, cmd, endpoint)
commonCmd.ExitOnErr(cmd, "can't fetch current epoch: %w", err)
if iatRelative {
iat += currEpoch
}
if expRelative {
exp += currEpoch
}
if nvbRelative {
nvb += currEpoch
}
}
if exp < nvb {
commonCmd.ExitOnErr(cmd, "",
fmt.Errorf("expiration epoch is less than not-valid-before epoch: %d < %d", exp, nvb))
}
ownerStr, _ := cmd.Flags().GetString(ownerFlag)
var ownerID user.ID
commonCmd.ExitOnErr(cmd, "can't parse recipient: %w", ownerID.DecodeString(ownerStr))
var b bearer.Token
b.SetExp(exp)
b.SetNbf(nvb)
b.SetIat(iat)
b.ForUser(ownerID)
eaclPath, _ := cmd.Flags().GetString(eaclFlag)
if eaclPath != "" {
table := eaclSDK.NewTable()
raw, err := os.ReadFile(eaclPath)
commonCmd.ExitOnErr(cmd, "can't read extended ACL file: %w", err)
commonCmd.ExitOnErr(cmd, "can't parse extended ACL: %w", json.Unmarshal(raw, table))
b.SetEACLTable(*table)
}
var data []byte
toJSON, _ := cmd.Flags().GetBool(jsonFlag)
if toJSON {
data, err = json.Marshal(b)
commonCmd.ExitOnErr(cmd, "can't mashal token to JSON: %w", err)
} else {
data = b.Marshal()
}
out, _ := cmd.Flags().GetString(outFlag)
err = os.WriteFile(out, data, 0644)
commonCmd.ExitOnErr(cmd, "can't write token to file: %w", err)
}

View file

@ -1,11 +1,11 @@
package token
package bearer
import (
"github.com/spf13/cobra"
)
var Cmd = &cobra.Command{
Use: "token",
Use: "bearer",
Short: "Operations with bearer token",
}

View file

@ -0,0 +1,9 @@
package cmd
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/autocomplete"
)
func init() {
rootCmd.AddCommand(autocomplete.Command("frostfs-cli"))
}

View file

@ -0,0 +1,218 @@
package container
import (
"errors"
"fmt"
"os"
"strings"
"time"
containerApi "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/acl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra"
)
var (
containerACL string
containerPolicy string
containerAttributes []string
containerAwait bool
containerName string
containerNnsName string
containerNnsZone string
containerNoTimestamp bool
force bool
)
var createContainerCmd = &cobra.Command{
Use: "create",
Short: "Create new container",
Long: `Create new container and register it in the FrostFS.
It will be stored in sidechain when inner ring will accepts it.`,
Run: func(cmd *cobra.Command, args []string) {
placementPolicy, err := parseContainerPolicy(cmd, containerPolicy)
commonCmd.ExitOnErr(cmd, "", err)
key := key.Get(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, key, commonflags.RPC)
if !force {
var prm internalclient.NetMapSnapshotPrm
prm.SetClient(cli)
resmap, err := internalclient.NetMapSnapshot(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "unable to get netmap snapshot to validate container placement, "+
"use --force option to skip this check: %w", err)
nodesByRep, err := resmap.NetMap().ContainerNodes(*placementPolicy, nil)
commonCmd.ExitOnErr(cmd, "could not build container nodes based on given placement policy, "+
"use --force option to skip this check: %w", err)
for i, nodes := range nodesByRep {
if placementPolicy.ReplicaNumberByIndex(i) > uint32(len(nodes)) {
commonCmd.ExitOnErr(cmd, "", fmt.Errorf(
"the number of nodes '%d' in selector is not enough for the number of replicas '%d', "+
"use --force option to skip this check",
len(nodes),
placementPolicy.ReplicaNumberByIndex(i),
))
}
}
}
var cnr container.Container
cnr.Init()
err = parseAttributes(&cnr, containerAttributes)
commonCmd.ExitOnErr(cmd, "", err)
var basicACL acl.Basic
commonCmd.ExitOnErr(cmd, "decode basic ACL string: %w", basicACL.DecodeString(containerACL))
tok := getSession(cmd)
if tok != nil {
issuer := tok.Issuer()
cnr.SetOwner(issuer)
} else {
var idOwner user.ID
user.IDFromKey(&idOwner, key.PublicKey)
cnr.SetOwner(idOwner)
}
cnr.SetPlacementPolicy(*placementPolicy)
cnr.SetBasicACL(basicACL)
var syncContainerPrm internalclient.SyncContainerPrm
syncContainerPrm.SetClient(cli)
syncContainerPrm.SetContainer(&cnr)
_, err = internalclient.SyncContainerSettings(cmd.Context(), syncContainerPrm)
commonCmd.ExitOnErr(cmd, "syncing container's settings rpc error: %w", err)
var putPrm internalclient.PutContainerPrm
putPrm.SetClient(cli)
putPrm.SetContainer(cnr)
if tok != nil {
putPrm.WithinSession(*tok)
}
res, err := internalclient.PutContainer(cmd.Context(), putPrm)
commonCmd.ExitOnErr(cmd, "put container rpc error: %w", err)
id := res.ID()
cmd.Println("container ID:", id)
if containerAwait {
cmd.Println("awaiting...")
var getPrm internalclient.GetContainerPrm
getPrm.SetClient(cli)
getPrm.SetContainer(id)
for i := 0; i < awaitTimeout; i++ {
time.Sleep(1 * time.Second)
_, err := internalclient.GetContainer(cmd.Context(), getPrm)
if err == nil {
cmd.Println("container has been persisted on sidechain")
return
}
}
commonCmd.ExitOnErr(cmd, "", errCreateTimeout)
}
},
}
func initContainerCreateCmd() {
flags := createContainerCmd.Flags()
// Init common flags
flags.StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage)
flags.Bool(commonflags.TracingFlag, false, commonflags.TracingFlagUsage)
flags.DurationP(commonflags.Timeout, commonflags.TimeoutShorthand, commonflags.TimeoutDefault, commonflags.TimeoutUsage)
flags.StringP(commonflags.WalletPath, commonflags.WalletPathShorthand, commonflags.WalletPathDefault, commonflags.WalletPathUsage)
flags.StringP(commonflags.Account, commonflags.AccountShorthand, commonflags.AccountDefault, commonflags.AccountUsage)
flags.StringVar(&containerACL, "basic-acl", acl.NamePrivate, fmt.Sprintf("HEX encoded basic ACL value or keywords like '%s', '%s', '%s'",
acl.NamePublicRW, acl.NamePrivate, acl.NamePublicROExtended,
))
flags.StringVarP(&containerPolicy, "policy", "p", "", "QL-encoded or JSON-encoded placement policy or path to file with it")
flags.StringSliceVarP(&containerAttributes, "attributes", "a", nil, "Comma separated pairs of container attributes in form of Key1=Value1,Key2=Value2")
flags.BoolVar(&containerAwait, "await", false, "Block execution until container is persisted")
flags.StringVar(&containerName, "name", "", "Container name attribute")
flags.StringVar(&containerNnsName, "nns-name", "", "Container nns name attribute")
flags.StringVar(&containerNnsZone, "nns-zone", "", "Container nns zone attribute")
flags.BoolVar(&containerNoTimestamp, "disable-timestamp", false, "Disable timestamp container attribute")
flags.BoolVarP(&force, commonflags.ForceFlag, commonflags.ForceFlagShorthand, false,
"Skip placement validity check")
}
func parseContainerPolicy(cmd *cobra.Command, policyString string) (*netmap.PlacementPolicy, error) {
_, err := os.Stat(policyString) // check if `policyString` is a path to file with placement policy
if err == nil {
common.PrintVerbose(cmd, "Reading placement policy from file: %s", policyString)
data, err := os.ReadFile(policyString)
if err != nil {
return nil, fmt.Errorf("can't read file with placement policy: %w", err)
}
policyString = string(data)
}
var result netmap.PlacementPolicy
err = result.DecodeString(policyString)
if err == nil {
common.PrintVerbose(cmd, "Parsed QL encoded policy")
return &result, nil
}
if err = result.UnmarshalJSON([]byte(policyString)); err == nil {
common.PrintVerbose(cmd, "Parsed JSON encoded policy")
return &result, nil
}
return nil, errors.New("can't parse placement policy")
}
func parseAttributes(dst *container.Container, attributes []string) error {
for i := range attributes {
k, v, found := strings.Cut(attributes[i], attributeDelimiter)
if !found {
return errors.New("invalid container attribute")
}
dst.SetAttribute(k, v)
}
if !containerNoTimestamp {
container.SetCreationTime(dst, time.Now())
}
if containerName != "" {
container.SetName(dst, containerName)
}
if containerNnsName != "" {
dst.SetAttribute(containerApi.SysAttributeName, containerNnsName)
}
if containerNnsZone != "" {
dst.SetAttribute(containerApi.SysAttributeZone, containerNnsZone)
}
return nil
}

Some files were not shown because too many files have changed in this diff Show more