Compare commits

...

723 commits

Author SHA1 Message Date
f2437f7ae9 [#734] shard: Fix Delete method
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
f26233b47a [#734] metabase: Include UpdateStorageID in metrics and traces
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
7e0c5a55de [#734] writecache: Fix flush
Now UpdateStorageID doesn't return error in case of logical error.
If object is in graveyard or GC market, it is still required to
update storage ID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:17 +03:00
d5c10612f4 [#735] policer: Register metrics
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-12 09:31:36 +03:00
3a997d1207 [#680] metrics: Initialize log metrics together with services
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
bf082348d4 [#680] metrics: Add step export-metrics in Makefile
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
994f48f8bb [#680] metrics: Export log and morph with script
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
aca11d7474 [#735] policer: Allow to provide metrics from the outside
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-11 15:14:13 +03:00
5e229dc248 [#701] metrics: add metric to evaluate policer performance
Add processed objects counter in policerMetrics,
add policer field to NodeMetrics

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-10-09 19:02:08 +00:00
4caa934eea [#729] containersvc: Remove load announcement
IR code was removed in 8879c6ea.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-09 19:01:13 +00:00
d07afd803c [#726] writecache: Fix small object flush for Badger
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-06 11:32:50 +03:00
997ac7cd8d [#726] writecache: Fix small object flush for BBolt
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-06 11:32:44 +03:00
bd5bf8b1a9 [#721] netmap: Drop already bootstraped check
Because of this check, under certain conditions,
the node could be removed from the network map,
although the node was functioning normally.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-05 11:47:06 +03:00
f3278d76a9 [#721] netmap: Send bootstrap at each epoch tick
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-05 11:46:56 +03:00
627b302745 [#709] node: Put in log info about listening endpoints
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-02 13:31:10 +00:00
a0a35ffbec [#702] node: Update SDK version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-29 18:41:48 +03:00
c1e4130020 [#146] node: Add trace_id to logs
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-09-27 11:05:27 +03:00
b8c3c2486d [#333] Sort containers by ID
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-25 11:30:36 +03:00
c14c9a023c [#333] Sort objects by ID in SearchObjects
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-25 11:30:36 +03:00
d9b93b12c1 [#333] Sort shards by shard_ID in cli output
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-25 10:04:29 +03:00
3889e829e6 [#667] writecache: Add logs for report error func in tests
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-09-14 17:02:54 +00:00
10570fc035 [#690] go.mod: Update contract and api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-14 14:50:30 +03:00
c6af4a3ec8 [#679] engine: Do not increase error counter on meta mismatch
It was introduced in 69e1e6ca to help node determine faulty shards.
However, the situation is possible in a real-life scenario:
1. Object O is evacuated from shard A to B.
2. Shard A is unmounted because of lower-level errors.
3. We now have object in meta on A and in blobstor on B. Technically we
   have it in meta on shard B too, but we still got the error if B goes
   to a degraded mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-14 10:39:18 +03:00
58239d1b2c [#683] cli: Add context to policy parsing errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-12 09:47:21 +00:00
3c76884182 [#682] cli: Unify array of ranges type
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-12 10:42:34 +03:00
f435ab1b26 [#682] go.mod: Update sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-12 10:42:34 +03:00
aa9f8dce3d [#677] client: Refactor PrmAnnounceSpace/EndpointInfo/NetworkInfo usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-09-08 09:42:28 +03:00
8a81af5a3b [#653] Add context parameter to Open functions
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-07 18:03:29 +03:00
a716db99db [#668] shard/test: Do not alter rootPath option
Supposedly, this was added to allow creating 2 different shards without
subtest. Now we use t.TempDir() everywhere, so this should not be a
problem.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
36759f8434 [#668] shard/test: Properly check event processing
See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
39879fa868 [#668] shard/test: Add dontRelease options
Most of the time we would like to close shard with minor exceptions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
c661ba1312 [#668] shard/test: Use sane defaults in the test constructor
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
268adb79cb [#668] shard/test: Simplify shard construction
newCustomShard() has many parameters but only the first is obligatory.
`enableWriteCache` is left as-is, because it directly affects the
functionality.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
429f941cda [#668] shard/test: Release shard in t.Cleanup()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
382eb8a485 [#668] shard/test: Disable GC where it is not needed
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
42696016de [#668] shard: Close stopChannel in GC
It is done once, but now we could read it from multiple places.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
bdecfbc1be [#668] shard/test: Move tests to the main package
Semantic patch (also, duplicate definitions are removed):
```
@@
var e identifier
@@
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"

-shard.e
+e
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
aa23c6a83a [#668] shard/test: Remove subtest from TestCounters
Otherwise, individual tests cannot be run.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
da8f384324 [#668] shard/test: Fix typo in existence
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
aeeb8193d2 [#676] node: Fix header source creation when checking eacl
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-09-06 17:06:54 +03:00
d9de9e2bbb [#675] client: Refactor PrmObjectDelete usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-09-06 08:05:47 +00:00
88d50e4c77 [#656] policer: Add "bad" testcase
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-09-06 08:04:59 +00:00
054e3ef3d3 [#674] pre-commit: Update shellcheck-py
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-04 15:34:24 +03:00
a54b4472de [#674] network: Close connections on address updates
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-04 15:34:24 +03:00
7456c8556a [#536] blobovnicza: Add blobovniczatree DB cache
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:53:11 +03:00
c672f59ab8 [#536] blobovnicza: Drop cache
Each blobovnicza instance is opened
while is in use.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:51:26 +03:00
b9b86d2ec8 [#666] shard/test: Fix data race in metrics tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-31 08:39:42 +00:00
4dff9555f1 [#568] writecache: Improve flushing scheme for badger
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-30 17:22:28 +00:00
806cc13d9f [#658] client: Refactor PrmObjectGet/Head/Range usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-30 17:13:23 +00:00
1daef2ceeb [#660] writecache: Fix remaining addr2key uses
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-30 17:12:33 +00:00
fe5aa06a75 [#665] node: Bind length of copies number to number of replicas
Allow to use one digit in copies number array for backward compatibility.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-30 17:11:55 +00:00
91f3745b58 [#659] debian: Remove nspcc email
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-30 08:29:26 +00:00
7654847f79 [#659] adm: Remove nspcc.ru from the default email
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-30 08:29:26 +00:00
a724debb19 [#632] .forgejo: Print --version
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-29 12:41:45 +03:00
55b82e744b [#529] objectcore: Use common sender classifier
Use common sender classifier for ACL service and format validator.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:33:06 +03:00
ae81d6660a [#529] objectcore: Fix object content validation
There are old objects where the owner of the object
may not match the one who issued the token.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:33:06 +03:00
ab2614ec2d [#528] objectcore: Validate token issuer
Add token issuer against object owner validation.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:10:10 +03:00
4ea0df77d0 [#574] policer: Check if the container was really removed
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
554ff2c06b [#574] core: Extend Source interface with DeletionInfo method
* Introduce common method EverExisted
* Define DeletionInfo for struct that must implement Source
* Refactor tree srv

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
9072772a09 [#649] shard/test: Increase GC remover interval
This was set in #348 to speed up tests.
It seems 100ms doesn't increase overall test time,
but it reduces the amount of logs by 100x factor.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
c4db8e7690 [#637] shard/test: Fix data race
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
f8ba60aa0c [#648] objsvc/delete: Handle errors in Go style
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 09:45:35 +00:00
d2084ece41 [#648] objsvc/delete: Remove redundant logs
We never propagate delete requests to the container node, because
tombstone broadcast is done via PUT. No need to pollute logs.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 09:45:35 +00:00
40b556fc19 [#647] objsvc/search: Improve testing coverage
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
4db2cbc927 [#647] objsvc/search: Wrap in uniqueIDWriter during parameter setting
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
966ad22abf [#647] objsvc/search: Simplify error handling
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
56f841b022 [#647] objsvc/search: Remove TraverserGenerator wrapper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
ba58144de1 [#647] objsvc/search: Remove netmap.Source wrapper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
c9e3c9956e [#643] objsvc/put: Unify extraBroadcastEnabled usage
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
facd3b2c4b [#643] objsvc/put: Unify placement iterators
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
3fcf56f2fb [#643] objsvc/put: Copy config to distributedTarget
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
96e690883f [#638] Unify test loggers
In some places we have debug=false, in others debug=true.
Let's be consistent.

Semantic patch:
```
@@
@@
-test.NewLogger(..., false)
+test.NewLogger(..., true)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
322c1dc273 [#638] Use test.NewLogger() in tests
Semantic patch (restricted to **/*_test.go):
```
@@
@@
+import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger/test"
-import "go.uber.org/zap"
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger"

-&logger.Logger{Logger: zap.L()}
+test.NewLogger(t, false)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
02b03d9c4f [#638] logger: Remove sampling from test loggers
Losing logs is always a bad idea, especially when we debug tests.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
82cc453be9 [#xx] shard: Fix data race in metrics tests
Protect test metric store fields with a mutex. Probably, not every field
should be protected, but better safe than sorry.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 10:26:12 +00:00
238b8f10a0 [#630] cli: Fix SDK SetEACLPrm usage for PrmContainerSetEACL
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-22 14:25:39 +00:00
345a1a69a2 [#635] Use internal key type when deleting from badger wc
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-22 10:53:19 +03:00
dc3bc08c07 [#631] lens: Fix db type flag name
Typo from 1a0cb0f34a.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-21 17:18:05 +00:00
23be3eb627 [#574] tree: Check if container is really removed
* Use DeletionInfo method from morph client to check if
  the container has been really removed from neo-go

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-21 12:50:20 +03:00
42fb6fb372 [#574] morph: Add DeletionInfo method for morph client
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-21 12:49:06 +03:00
62c2ad4b22 [#626] logs: Remove autogenerated comments
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-21 11:15:06 +03:00
84ea075587 [#625] cli: Fix SDK EACLPrm usage for PrmContainerEACL
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-21 10:36:47 +03:00
354a92ea2c [#602] blobovnicza: Add leaf width implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-21 10:27:32 +03:00
d3904ec599 [#602] config: Add blobovnicza leaf width parameter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-21 10:27:32 +03:00
4d9a6c07fb [#618] core: Replace fmt.Sprintf with strconv.FormatUint
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-21 07:14:50 +00:00
a1f1d233cc [#618] linters: bump truecloudlab-linters to 0.0.2
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-21 07:14:50 +00:00
f2811f8585 [#602] metrics: Add blobovnicza items counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-18 13:01:27 +03:00
c4e1d8eb07 [#602] node: Fix blobovnicza typos
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-18 11:14:10 +03:00
10e63537b2 [#602] metrics: Rename blobovnicza size metric
`Size` is not size, but open db size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-17 19:17:35 +00:00
809e97626b [#602] blobovnicza: Fix size counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-17 19:17:35 +00:00
2e49d7ea7e [#602] blobovnicza: Init before using
Fix blobovnicza size: after restart size metric resets.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-17 19:17:35 +00:00
f7042c5a6f [#609] Replace zaptest.NewLogger() with zap.L()
Semantic patch:
```
@@
@@
-import "go.uber.org/zap/zaptest"
+import "go.uber.org/zap"

-zaptest.NewLogger(t)
+zap.L()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:51:10 +00:00
127c676786 [#607] *: Use keys.PublicKeys.Copy() where possible
Semantic patch:
```
@@
var dst identifier
var src identifier
var keys identifier
@@
 import keys "github.com/nspcc-dev/neo-go/pkg/crypto/keys"

-dst := make(keys.PublicKeys, len(src))
-copy(dst, src)
+dst := src.Copy()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
e604a3d749 [#607] *: Use zap.Stringer() where possible
Semantic patch:
```
@@
var f expression
var t expression
var a expression
@@
 f(
    ...,
-    zap.String(t, a.String()),
+    zap.Stringer(t, a),
    ...,
)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
a8de37c8a2 [#607] *: Remove redundant if on error returns
Semantic patch:
```
@@
@@
-if err != nil { return err }
-return nil
+return err
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
5a51b78946 [#620] object: Send status response for server-side streams
Previously status responses were wrapped in the gRPC error and thus
couldn't be correctly handled on client.

Introduced in c2617baf63, thanks @ale64bit for having found.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 12:52:38 +00:00
6407bb5bd1 [#619] node: Fix object put when copies numbers contains only zeros
In this case object should placement according to replicas.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-17 11:31:34 +03:00
5335e7089e [#615] pilorama: Speedup TestForest_ApplyRandom()
Some of our pilorama tests fail on CI.
The reasons are not obvious, but one possible improvement
is using `WithNoSync` option for these. It should have much effect,
because we are writing on the tmpfs, but doesn't hurt anyway.

If I replace `t.TempDir()` with a local directory, test execution time
goes down from 5s (sync) to 0.4s (nosync), which is the same time as
with `t.TempDir()`. Maybe we have some strange CI configuration.

```
panic: test timed out after 10m0s
running tests:
	TestForest_ApplyRandom (8m22s)
	TestForest_ApplyRandom/bbolt (8m21s)
...
goroutine 170 [syscall]:
syscall.Syscall(0xc000100000?, 0xc00047b758?, 0x6aff9a?, 0xc00041c1b0?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/syscall_linux.go:69 +0x27
syscall.Fdatasync(0x9e35c0?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/zsyscall_linux_amd64.go:418 +0x2a
go.etcd.io/bbolt.fdatasync(0xc000189000?)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-16 19:09:08 +00:00
2efe9cc1be [#585] writecache: Fix DB counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
58c8722c81 [#585] fstree: Add optional file counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
baad49990c [#585] fstree: Return logical error if object deleted
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
0c52186572 [#585] fstree: Remove unused method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
eec97d177e [#585] writecache: Count items periodically
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
d15199c5d8 [#596] engine: Consider context errors as logical
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-16 10:39:41 +03:00
bc425b5bad [#608] pre-commit: Fix linter
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-15 07:42:46 +00:00
88b6755c5e [#598] Fix use-after-close bug in badger writecache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-15 07:04:06 +00:00
ae8be495c8 [#xx] Avoid manual management of files in tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-14 14:01:39 +03:00
376f03a445 [#598] Hold mode mutex when setting mode
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-14 07:12:27 +00:00
ad87493c41 [#8] Bump required go version to go1.20
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-11 15:51:06 +03:00
21800e9fcc [#162] core: Move literals to constants
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-11 15:48:42 +03:00
a5f51add25 [#162] ci: Add noliteral linter
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-11 15:48:42 +03:00
abdb0910cc [#600] adm/tests: Add missing WaitGroup.Add()
```
panic: sync: negative WaitGroup counter
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-11 09:50:07 +00:00
4d2af137e9 [#500] .forgejo: Add DCO action
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-11 09:47:41 +00:00
b44a8dd46c [#597] *: Fix linter warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-11 09:55:51 +03:00
20af34ecdb [#591] cli: fix SDK PrmContainerDelete usage for DeleteContainerPrm
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-10 16:40:35 +00:00
dd988a5912 [#592] adm: Allow to tick epoch concurrently
Epoch can be ticked by IR in the background. Because the only usecase
for this is to apply some changes, the right behaviour is ignoring an
error, not retrying.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:40:00 +00:00
32a9f51586 [#562] logs: Remove superfluous comment
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
e084c47bd6 [#248] innerring: Remove audit from tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
779da6ec35 [#248] morph: Remove audit contract name
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
2685b1d548 [#248] adm: Do not deploy audit contract
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
55ce4dc075 [#248] morph: Remove obsolete network parameters
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
c54fcb297d [#248] cli,node: Remove obsolete network parameters
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
26a78aa366 [#248] adm: Remove obsolete network parameters
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
4ad0ebb32f [#565] Add metrics for current GRPC endpoint status
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-10 16:37:49 +00:00
c3e23a1448 [#578] gendoc: Allow to override flags
The command is used in multiple places across the whole FrostFS
ecosystem. While we want to have uniform interfaces everywhere,
sometimes we can't: already defined global flags can be harder to change
because of our obligations to the users. Cobra framework doesn't allow
conflicting flags (we can have global ones), so allow to override them.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
6bcba27757 [#578] gendoc: Remove flag shorthands
Make it harder to encounter conflicts in already existing commands.
Because the command is executed once, I don't think the usability is
worse.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
dca31e8888 [#578] gendoc: Add date to man pages
```
HISTORY
       8-Aug-2023 Auto generated by spf13/cobra
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
b02a1a34c1 [#578] gendoc: Allow to customize man pages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
082370583f [#578] gendoc: Replace NSPCC with FrostFS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
34b5d90441 [#590] cli: fix SDK PrmContainerPut usage for PutContainerPrm
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com
2023-08-10 11:09:31 +00:00
6186329aec [#590] cli: fix SDK PrmContainerGet usage for GetContainerPrm
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-10 11:09:31 +00:00
c3c0574e3c [#589] cli: Add control shards evacuate command
It was accidentally removed in f7c0b50d70.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 09:06:06 +00:00
8f994163ee [#586] Fix writecache benchmarks and refactor hacky NeedsCompression
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-10 08:05:18 +00:00
023b90342c [#584] Disable compression in badger writecache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-10 10:14:06 +03:00
d641cba2fc [#587] Do not use math/rand.Read
Fix staticcheck warnings after go1.20 update.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-09 16:02:44 +03:00
33c11be0cf [#587] .golangci.yml: Fix pre-commit warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-09 16:02:44 +03:00
5b7e4a51b7 [#481] Update frostfs-sdk-go and error pointer receivers
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-09 10:26:53 +00:00
de3d1eb99c [#581] Bump required go version to go1.20
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-09 10:16:48 +00:00
ae322e9f73 [#576] Set SyncWrites for badger writecache by default
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-09 11:28:26 +03:00
8d589314b5 [#560] node: Fix Put in multi REP with intersecting sets of nodes
Once the node was processed it skipped, at the step of forming
result in case when all nodes skipped, because processed for
previous REP, service mark the whole request as incomplete.

Example of policies which are unblocked:
- REP 1 REP 1 CBF 1
- REP 4 IN X REP 4 IN Y
  CBF 4
  SELECT 2 FROM FX AS X SELECT 2 FROM FY AS Y
  FILTER Country EQ Russia OR Country EQ Sweden OR Country EQ Finland AS FY
  FILTER Price GE 0 AS FX

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-08 10:22:53 +00:00
7da4306e38 [#575] writecache: Fix log level for badger writecache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-08 08:16:36 +00:00
d3a52ec73a [#516] node: Send bootstrap request if attributes were updated
Consider following situation:
1. Epoch tick.
2. Update node attributes in the config.
3. Restart node.

Because we already sent bootstrap query after (1) and we are still
online, new bootstrap query won't be sent. This leads to the node
attributes being updated after another epoch.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-08 07:40:26 +00:00
0e697266c3 [#563] writecache: Fix metrics and bolt delete
Estimate cache size after delete objects to update metric.
Update counters on small object deletion.
Do not count bbolt DB file as FSTree object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-07 12:53:28 +00:00
5bbfebba2d [#570] Fix writecache type constant copy-pasta bug
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-07 12:06:44 +03:00
1a0cb0f34a [#421] Try using badger for the write-cache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-07 08:16:57 +00:00
65c72f3e0b [#559] Remove manual path handling in fstree tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-03 10:03:41 +03:00
1e8b4b8a17 [#557] services: Regenerate stable marshalers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-02 13:37:40 +00:00
435a581b5e [#557] go.mod: Update sdk-go and api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-02 13:37:40 +00:00
93c46cfdf0 [#550] cli: make get-op-log meta pretty formatted
Close #550

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-02 10:27:15 +00:00
1b7b54ba89 [#540] cli: refactor client parameters usage
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-02 09:44:51 +00:00
b3695411d9 [#553] eacl: Fix bug with casting to ObjectAccessDenied error
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-02 07:22:48 +00:00
ec8a631d31 [#542] Update test for SizeInBytesSafe function
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-01 13:38:00 +00:00
9ca63ac8c3 [#542] Fix bug in SizeInBytesSafe function
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-01 13:38:00 +00:00
b8052c794e [#547] go.mod: Update api-go, sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
35dc64bd7b [#547] metabase: Fix datarace in tests
Quite an old one bf9e938a3b.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
05ac9e3637 [#547] objectsvc: Work with traversal struct from a single thread
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
7b0fdf0202 [#533] services: Assume API supports status codes
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
ec8b4fdc48 [#541] writecache/test: Close writecache on exit
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-29 10:57:25 +00:00
ad5f527bd3 [#541] writecache/test: Remove initWC()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-29 10:57:25 +00:00
ea32913430 [#543] putsvc: Fix PutSingle implementation
Add Lock and Delete handlers to local PutSingle.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-28 12:09:41 +00:00
99bb488ebd [#539] getsvc: Write payload direct to out stream
To reduce memory allocations.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-27 17:02:08 +03:00
286242cad0 [#539] getsvc: Use buffer to assemble object
To reduce memory consumption.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-27 17:02:08 +03:00
32c77f3a23 [#537] node: Add runtime.memory_limit config parameter
This parameter allows to set soft memory limit for Go GC.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-25 17:27:09 +03:00
5ff82ff04f [#6] services/object: Simplify local/remote targets
We do not use the return result from Close() and we always execute both
methods in succession. It makes sense to unite them.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:39:12 +03:00
448b48287c [#6] services/util: Do not panic in sign function
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:39:12 +03:00
c2617baf63 [#6] services/util: Remove remaining stream wrappers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:39:09 +03:00
372160d048 [#6] services/util: Remove SignService.HandleUnaryRequest
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:36:08 +03:00
fef172c5b0 [#6] services/util: Simplify response.Service
It has only 1 parameter, which is obligatory.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:29:57 +03:00
5a4054eeb6 [#535] .forgejo: Add vulncheck workflow
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-20 09:52:03 +00:00
eed594431f [#335] treesvc: Add GetSubTree ordering unit test
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
af82c2865e [#335] treesvc: Fix inmemory unit tests and nil meta items
Bolt forest saves empty slice of items. Now inmemory forest
does it the same way.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
b4e72a2dfd [#335] treesvc: Sort nodes by Filename in GetSubTree
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
94df541426 [#530] go.mod: Update frostfs-sdk-go and frostfs-api-go versions
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-07-19 16:15:21 +03:00
57e7fb5ccf [#521] cli: Add common aliases to policy playground commands
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-19 11:34:08 +00:00
24dffdac6f [#521] cli: Add netmap load command to policy playground
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-19 11:34:08 +00:00
a9d04ba86f [#244] Remove --local-dump from frostfs-adm config
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-18 13:51:25 +00:00
6429975584 [#511] docs: Remove GitHub mentions from CONTRIBUTING.md
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 13:24:02 +00:00
4680087711 [#527] Add support for select-filter expressions in policy playground
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-18 13:15:58 +00:00
f0355a453e [#463] policer: Remove capacity rebalance logic
Current implementation has some quirks. For example,
using only half of object.put.pool_size_remote threads
tells replicator that is node is 50% loaded,
but in reality we could be putting lot's of big objects.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
8b78db74bc [#463] replicator: Add tracing span
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
8966dd8e35 [#463] putsvc: Use PutSingle RPC for remote target
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
b2487e8cc5 [#516] node: Do not bootstrap if node is online candidate
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:51:02 +00:00
3b66f98f27 [#519] Use Address.Equals in policer tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-17 10:45:07 +00:00
3e8de14e7d [#382] evacuate: Fix unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-17 10:24:05 +00:00
a0c7045f29 [#512] cli: Refactor UX for bearer create command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-17 11:45:27 +03:00
d8e37a827f [#497] config: Add examples and unit tests
Add examples and unit tests for tree.authorized_keys section.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-17 08:23:28 +00:00
486287c2f7 [#524] cli: Add impersonate flag for bearer token creation
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-17 08:20:50 +00:00
397131b0ea [#520] objectcore: Refactor format validator
Remove redundant FIXME.
Move error to consts.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-17 07:07:21 +00:00
11027945d8 [#479] writecache: Fix writecache fstree flush premature ctx cancel
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-14 10:25:52 +03:00
8a9fc2c372 [#510] treesvc: Rename tableFromBearer to useBearer
With impersonation, the old name is no longer descriptive.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-12 10:23:21 +00:00
b8bcfac531 [#510] treesvc: Fix panic in bearer token processing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-12 10:23:21 +00:00
24eb988897 [#294] deletesvc: Drop cast
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
c83e7c875f [#294] searchsvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
e8091101c7 [#294] searchsvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
ec9b738465 [#294] putsvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
800a685e84 [#294] putsvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
1420b8b9ea [#294] deletesvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
a476d8285a [#294] deletesvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
70a1081988 [#294] aclsvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
18d8898b00 [#294] aclsvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
61541eaec2 [#294] aclsvc: Refactor checker constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
7da284f3e8 [#509] go.mod: Update sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-11 17:38:00 +03:00
80481c015c [#509] putsvc: Omit AccessIdentifiers from iteratePlacement()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-11 17:38:00 +03:00
0754e6e654 [#390] cli: Fix bearer token reading for tree subcommands
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-07-11 14:11:30 +00:00
676a3efa79 [#390] cli: Make tree commands errors and messages more verbose
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-07-11 14:11:30 +00:00
c42db4e761 [#390] cli: Support more tree service API calls in CLI
* Support healthcheck API call
* Support move API call
* Support remove API call
* Support getSubtree API call
* Support getOpLog API call

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-07-11 14:11:30 +00:00
a65e26878b [#486] put service: Fix error typo
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
fcbf90d31b [#486] node: Add PutSingle implemetation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
7b76527759 [#486] node: Add PutSingle wrappers
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
9be5d44a46 [#486] node: Update api-go and sdk-go versions
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
040a623d39 [#476] cli: Fix object nodes command
Do not fail if client creation failed.
Use external addresses to create the client too.
Use public key as node ID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 09:58:31 +03:00
140d970a95 [#498] policer: Fix objectSDK import
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-07 18:52:40 +03:00
2310a5c7ba [#498] policer: Allow to set sleep duration between iterations
Speed up tests on CI.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-07 14:52:33 +00:00
e858479a74 [#498] policer: Explicitly Rewind() iterator after finish
Previously, we can continue to return `EndOfListing` infinitely.
Reflect iterator reuse via Rewind() method.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-07 14:52:33 +00:00
a0d51090a4 [#482] Enable staticcheck in foregjo actions
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-07 10:12:49 +03:00
033eaf77e1 [#496] node: Fix linter importas
Standardize the alias of the
import frostfs-sdk-go/object as objectSDK.

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-07-06 15:36:41 +03:00
4bbe9cc936 [#496] .golangci.yml: Add importas linter
Add importas linter

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-07-06 15:35:23 +03:00
6eefe9747e treesvc: Do not provide credentials unless TLS is used
4f413fe86e was perfect, except it did the opposite of what we needed.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-06 08:04:28 +00:00
f354b8a270 [#473] Add tests for rem values of traverser
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-07-05 20:13:23 +03:00
c6df6c84ae [#473] Do not modify traverse plan when [0] copies number vector is provided
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-07-05 19:54:43 +03:00
b520a3049e [#483] cli: Fix object put cmd with flag --prepare-locally
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-04 18:58:26 +03:00
4c248d573e [#483] cli: Allow to split object on the client side
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-03 18:31:46 +03:00
90e9a85acc [#483] Update dependencies
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-03 15:44:59 +03:00
8d16d95376 [#484] morph: Add expired tx logging
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-03 08:03:25 +00:00
26acf5689e [#92] Ensure policer objects cannot be worked on concurrently
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
3223402c90 [#92] Embed policer's objectsInWork mutex
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
f9730f090d [#92] Refactor policer and add some unit tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
0c5b025788 [#470] grpc: Increase message limits
For send message limit set to 2GiB, but there are custom
GET/GET RANGE limiters.
For receive message limit set to 256 MiB, but actual chunk size
will be managed by client.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-30 16:32:15 +03:00
f437ab8f15 [#197] object: Make Delete method return correct status
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-06-30 12:58:56 +00:00
d01c064674 [#455] Fix non-informative message when parse node attributes
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-28 15:02:29 +00:00
d0ab552a90 [#478] *: Fix funlen linter warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 15:01:49 +00:00
33d9ebbe7f [#478] .golangci.yml: Increase timeout
Make it enough for CI.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 15:01:49 +00:00
f91cfb36e6 [#478] .forgejo: Add build/test workflows
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 15:01:49 +00:00
8a4e250dae [#468] *: replace outdated TODO crypto-related links
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
4f413fe86e [#1] treesvc: Properly check for secure transport
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
cab51c8cbe [#1] metabase: Rename blindlyProcess()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
73a71a71b0 [#1] node: Use a proper validation of a substorage type
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
f4c71cea65 [#1] *: Replace outdated FIXME/TODO links
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
64e1383df6 [#1] Makefile: Remove image-storage-testnet target
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
13b53258b4 [#1] go.mod: Remove retract version
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
0c866f62c5 [#473] placement: Fix backwards compatibility for copies_number
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-27 11:13:16 +03:00
43d263c3d5 [#428] linter: Fix unkeyed assignment
Thanks to gopls.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 13:29:39 +00:00
cac4ed93d6 [#428] engine: Add low_mem config parameter
Concurrent initialization in case of the metabase resync leads to
high memory consumption and potential OOM.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 13:29:39 +00:00
71a63b8e9c [#439] node: Update SDK-Go version
This fixed version: null header for complex object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 07:56:32 +00:00
4bf345225c [#447] pilorama: Use named constant for the key size
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
b4ce0b0412 [#447] pilorama: Do not undo log for create ops
```
goos: linux
goarch: amd64
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
         │     old      │                 new                  │
         │    sec/op    │    sec/op     vs base                │
Create-8   36.48µ ± 11%   30.34µ ± 14%  -16.84% (p=0.000 n=10)

         │     old      │                 new                  │
         │     B/op     │     B/op      vs base                │
Create-8   43.01Ki ± 4%   37.78Ki ± 5%  -12.15% (p=0.000 n=10)

         │    old     │                new                 │
         │ allocs/op  │ allocs/op   vs base                │
Create-8   166.0 ± 3%   146.0 ± 3%  -12.05% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
dd3874eff1 [#447] pilorama: Add benchmark for create ops
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
72fedff7ad [#426] cli: Add object nodes command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 07:41:34 +00:00
ab489265b3 [#449] cli: Allow to exec tree get-by-path without bearer token
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-23 14:44:19 +03:00
71889234b7 [#449] tree: Allow reading requests signed by keys from allow list
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-23 14:44:19 +03:00
40eae22109 [#460] services/util: Remove CreateRequestStreamer
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:08:04 +03:00
a64dc9ad70 [#460] services/util: Remove HandleServerStreamRequest
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:08:01 +03:00
167a67f0b8 [#460] services/util: Remove HandleUnaryRequest
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:07:56 +03:00
785d81a68a [#460] services/object: Reduce distibutedTarget memory footprint
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:07:50 +03:00
4d48377cec [#459] blobovniczatree: Fix get error
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 16:44:53 +03:00
03aa210145 [#373] metrics: Move labels to consts
To unify label naming all lable keys and other consts are moved to
one file.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
b5d9f4a285 [#373] metrics: Add pilorama metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
e89fa110c7 [#373] metrics: Add metabase metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
af608da952 [#373] metrics: Add blobovnizca metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
56f320dd85 [#373] metrics: Add blobstor metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
16a142cd0c [#373] metrics: Add FSTree metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
d8ecc69d00 [#373] local storage: Pass parent ID
This is required to add shard ID as metric label.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
d5aaec1107 [#373] pilorama: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
059e9e88a2 [#373] metabase: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
f54cc0b607 [#373] blobstor: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
8318d90ad0 [#373] blobovniczatree: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
3ae3c8dfdb [#373] fstree: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
a8526d45e9 [#373] blobovnizca: Add missed/fix tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
028d4a8058 [#373] blobovnicza: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
01a0c97760 [#453] engine: Set Disabled mode to deleted shard
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-20 12:04:07 +03:00
50caa388b0 [#303] ir: Use pub key when validate container deletion
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-20 08:02:48 +00:00
69df0d21c2 [#446] engine: Move to read-only on blobstor errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:53:32 +03:00
fe01781811 [#446] los: Wrap SSD errors in a separate type
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:53:31 +03:00
20b84f183a [#446] engine: Simplify logs for shard mode change
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:51:29 +03:00
69b788a90b [#424] metrics: Rename mode to mode_info
To follow best practice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 18:39:49 +03:00
26b305f82b [#424] morph: Fix cache metrics
Use separate morph cache metrics for node and IR

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 18:26:19 +03:00
4449006862 [#424] metrics: Use mode value as metric value for shard
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 18:26:19 +03:00
847732605c [#424] metrics: Rename counter to total
Suffix total should be used for a unit-less accumulating count.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
c348ae35b0 [#424] metrics: Drop embedded metrics
It was not obvious where metrics are used.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
1b364d8cf4 [#424] metrics: Refactor engine metrics
Use histogram vector to measure request duration.
Fix naming like in Prometheus best practice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
c8023a9c8d [#424] morphcache: Use labels for method duration
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
85deb12f4d [#424] writecache: Drop metrics when close
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:52:40 +03:00
07f155ac77 [#424] metrics: Use labels for writecache methods and operations
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:52:40 +03:00
71bbeddb64 [#424] metrics: Drop unused arg
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:51:40 +03:00
fb8fee0c8e [#442] Use strconv.FormatBool instead of untyped conversion
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-14 11:08:36 +00:00
344d6b2ae1 [#xx] Fix invalid log metric namespace identifier
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-13 15:36:34 +03:00
4887f489a1 [#17] Add morph client metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-13 14:06:04 +03:00
90e9247b69 [#371] ir: Add morph cache metrics
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-06-13 10:10:13 +00:00
4f83ab0fb4 [#409] node: Do not sent initial bootstrap under maintenance
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:02:36 +00:00
e68384d4e3 [#409] node: Fetch last bootstrap info on startup
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:02:36 +00:00
898f0686b1 [#409] node: Log maintenance state on startup
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:02:36 +00:00
957a43a124 [#266] services/tree: Add sync check
Do not accept requests until initial sync is finished.
`Apply` is deliberately left out -- we don't want to miss anything new.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
e69a1e8482 [#266] services/tree: Return operation log up to some height
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
2541d319de [#266] pilorama: Allow to get current tree height
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
0c40d98f7a [#375] Add log metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-13 09:52:45 +00:00
83d600ed77 [#424] node: Update api-go version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 07:19:55 +00:00
0400153b7d [#424] object: Do not store large slices in pool
Dynamically growing an unbounded buffers can cause a large
amount of memory to be pinned and never be freed.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 07:19:55 +00:00
41ab4d070e [#423] *: Use hrw.StringHash() where possible
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-13 07:18:25 +00:00
cd92d8a9e7 [#423] go.mod: Update sdk-go and hrw
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-13 07:18:25 +00:00
9bc1a25c07 [#434] metrics: Fix writecache duration
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 07:18:01 +00:00
508e2064eb [#438] cli: Drop tracing debug print
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-09 14:12:25 +03:00
5b75432ca2 [#431] metrics: Add missed label
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-07 13:51:36 +00:00
263c6fdc50 [#372] node: Add metrics for the error counter in the engine
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-07 13:04:47 +00:00
189a367ef2 [#390] frostfs-cli: Pass bearer token to Tree srv
* Add --bearer flag for "tree" subcommand

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-06-07 11:08:34 +03:00
a770b89fd8 [#422] adm: Fix ValidatorCount problems after neo-go update
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 14:48:51 +03:00
f8c1e0639d [#422] cli: Provide context to NetmapSnapshot()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 14:48:47 +03:00
f1f56ef871 [#406] cli: Add --trace flag
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 11:44:04 +00:00
96c9843591 [#406] cmd/common: Execute PersistentPostRun() on errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 11:44:04 +00:00
9562123c49 [#406] cli: Pass context to internal client
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 11:44:04 +00:00
63473d0806 [#417] go.mod: Update neo-go
Update to master, because after the API update notary signer is being
incorrectly formed.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 11:43:48 +00:00
2dd3fc8b7e [#xx] cli: Add policy-playground command
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-02 09:23:40 +00:00
55c28fd5f4 [#412] cache: Pass DialOptions to create new conn
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
dcdfb6ed41 [#412] node: Use observability interceptors
Use metrics and tracing interceptors.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
c09144ecf1 [#412] node: Replace metrics package
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
74578052f9 [#412] node: Replace tracing package
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
7e9a1f394a [#412] node: Update deps
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
f7c4c07453 [#xx] Create innerring metrics before metrics usage
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-01 10:07:48 +03:00
4476a1dbaf [#413] morph/client: Fix govet warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-31 15:38:17 +03:00
dbf41391b5 [#401] engine: Extend evacuation logs
Add operation-tag to logger.
Log evacuation results.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 13:24:30 +03:00
3220c4df9f [#376] metrics: Add GC metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 10:22:12 +00:00
faca861451 [#411] Remove unnecessary pointers for sync objects
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-31 10:19:14 +00:00
f934abed8f [#402] doc: Update evacuation docs
Add estimated time left output to examples.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 10:18:44 +00:00
f64322576a [#402] cli: Add estimated evacuation time left
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 10:18:44 +00:00
ebcc8afbee [#374] Add inner-ring event metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-31 10:11:48 +00:00
8dcd06c587 [#394] node: Use Context in Blobovniczas.Iterate()
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-31 10:09:18 +00:00
a3e30062df [#325] node: Introduce unsafe_disable param to disable policer
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-05-31 10:09:05 +00:00
365a7ca0f4 [#366] node: Stop GC once termination signal received
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-29 09:35:08 +03:00
802168c0c6 [#364] node: Stop flushing big object when termination signal received
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-26 16:46:58 +03:00
9c54a24101 [#364] Fix trailing whitespace
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-26 16:46:58 +03:00
271a56c2ab [#395] metrics: Drop redundant metrics
HistogramVec already has labeled counter.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 13:39:37 +00:00
bc34fee6a7 [#370] Add tree service metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-26 13:39:12 +00:00
f2e5dead7e [#398] pilorama: Disallow applying same operations
1. In redo() we save the old state.
2. If we do redo() for the same operation twice, the old state will be
   overritten with the new one.
3. This in turn affects undo() and subsequent isAncestor() check.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-26 12:14:29 +00:00
5983617069 [#393] shard: Create tombstone source when reload
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 12:14:02 +00:00
20a489bdb5 [#393] gc: Use defer to mark handler done
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 12:14:02 +00:00
9119199f6e [#397] cli: Fix evacuation method names
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 11:28:30 +03:00
2613351008 [#387] gc: Cancel GC is change mode requested
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-25 09:38:16 +03:00
a1823423cf [#312] node: Add WC metrics info to CHANGELOG
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
2ce43935f9 [#312] metrics: Add writecache metrcis
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
d212d908b5 [#312] wc: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
4503a61997 [#312] wc: Delete unused Iterate method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
53a1b15693 [#338] ir: Drop notaryless code from governance
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
0ab589dd52 [#338] ir: Always enable notary on sidechain
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
81718afb39 [#338] ir: Drop named named put fee
Named put fee value used only when notary disabled.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
656fd7f376 [#338] ir: Drop notaryless code from netmap
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
fb708b3a2d [#338] ir: Drop container notaryless code
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
03ab0ca30f [#338] adm: Drop notaryless code
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
cf8531ccd7 [#381] go.mod: Update api-go and sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-23 08:18:01 +03:00
4b768fd115 [#381] *: Move to sync/atomic
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-23 08:18:01 +03:00
ff570847a4 [#381] go.mod: Update bbolt
Adopt new `ForEachBucket` function where possible.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-22 11:49:14 +03:00
7eb8fa6350 [#381] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-22 11:49:14 +03:00
731bf5d0ee [#380] node: go version up
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-22 08:48:14 +00:00
e3ad3c2965 [#283] cli: Move control healthcheck command under control ir
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-22 08:13:52 +00:00
35c9b6b26d [#314] writecache: remove objects right after they are flushed
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-19 09:37:05 +00:00
Pavel Karpy
bf79d06f03 [#314] writecache: Do not lose small objects on disk errors
Do return error if an object could not been stored on WC's disk.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-19 09:37:05 +00:00
Pavel Karpy
9e56592be3 [#314] writecache: Simplify background workers naming
Also, drop not used arg.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-19 09:37:05 +00:00
483fac03d6 [#329] docs: Add shard evacuation description
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
f7c0b50d70 [#329] cli: Add async evacuate commands
Add start, stop evacuate and evacuate status commands.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
e4889e06ba [#329] node: Make evacuate async
Now it's possible to run evacuate shard in async.
Also only one evacuate process can be in progress.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
100b1b5128 [#329] node: Add async evacuate proto methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
0fc494637f [#367] node: Disable unit tests cache
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-18 15:14:55 +00:00
aabcd8d3e4 [#362] node: Cancel ctx in the same routine as for signal watcher
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-18 09:28:10 +03:00
df9e099fa7 [#352] Add explicit max number of alphabet nodes
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-17 15:01:26 +00:00
13a7a90101 [#355] Increase tree svc client cache size to test hypotheses
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-17 14:11:46 +03:00
6f47c75e43 [#125] ir: Set extra wallets on SIGHUP
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-16 12:47:43 +00:00
0624820909 [#336] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-16 12:45:13 +00:00
869fcbf591 [#332] gc: Fix expired complex object deletion
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-16 12:44:57 +00:00
ab07bad33d [#332] gc: Add complex object unit test
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-16 12:44:57 +00:00
8da6530f41 [#351] cli: Support copies number parameter in object put
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-16 08:22:56 +00:00
079b28fa0f [#341] Register candidates in separate transactions
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-15 12:46:12 +00:00
d4d921dcaf [#346] adm: fix race in wallet generation
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-15 11:50:20 +03:00
429a87e83b [#39] ir: Use defer for wg.Done()
Not important, but `exitOnErr` can alter control flow, let's be
explicit.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 17:49:47 +03:00
Pavel Karpy
f604d6bbdc [#39] ir: Add optional profilers
Includes `block` and `mutex` profiles configuration.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 17:49:47 +03:00
f989bc52be [#39] ir: Do not store config keys in httpComponent
Pprof will have specific options, it seems wrong to have them in a
generic struct.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 17:49:46 +03:00
61776033c2 [#39] ir: Do not reload services if they are disabled
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 15:49:04 +03:00
Pavel Karpy
14c35d776e [#39] node: Add optional profilers
Include settings for block and mutex profilers.
They are disabled by default, as in Go runtime itself.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 15:49:04 +03:00
a6ee7a3087 [#324] Add replicator metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-12 11:00:17 +00:00
6055b18362 [#342] morph: Remove unused toStackParameter()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 13:24:49 +03:00
02c02974b3 [#309] locode: Parallelize DB generation
For v0.4.0 release:
Before:
```
Executed in  571.64 secs    fish           external
   usr time  283.07 secs  744.00 micros  283.07 secs
   sys time    8.41 secs  179.00 micros    8.41 secs
```

After:
```
Executed in   54.23 secs    fish           external
   usr time  418.65 secs    1.01 millis  418.65 secs
   sys time    0.61 secs    0.25 millis    0.60 secs
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:59:13 +00:00
147ae8728a [#309] go.mod: Update paulmach/orb to v0.9.1
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:59:13 +00:00
c62025c836 [#321] metabase/test: execute tests in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
c8c5f14e2e [#321] adm/test: Check wallet correctness in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
fe4082799a [#321] adm: Create multisig accounts in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
945454f60c [#321] engine/test: Execute tests in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
4578d00619 [#321] shard/test: Execute tests in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
d35e4c389f [#321] shard/test: Parallelize TestWriteCacheObjectLoss
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
969bfb603f [#321] shard/test: Parallelize TestShard_List
```
go test -count=1 -run TestShard_List -race .
Before: 2.492s
After:  0.109s
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
47b0ec33c3 [#337] netmap: Remove unused events
Done in TrueCloudLab/frostfs-contract#16.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
b480df4985 [#337] container: Remove unused events
Done in TrueCloudLab/frostfs-contract#16.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
Anna Shaleva
bcdb0f330d [#337] morph: Completely remove fallbackTime from client cfg
It's unused and not needed, default fallback lifetime is set by Notary
actor.

Signed-off-by: Anna Shaleva <anna@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
Anna Shaleva
ddcc156ecc [#337] morph: Use Notary Actor for notary requests
Signed-off-by: Anna Shaleva <anna@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
35fdf6f315 [#337] morph: Move subscription logic to subscriber
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
Roman Khimov
a5f118a987 [#337] subscriber: Drop unused UnsubscribeForNotification
It's not really needed, closing the connection works fine when exiting and
normally the app doesn't need to unsubscribe at all.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
800eb5e983 [#125] ir: Reconfigure pprof and metrics on SIGHUP
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-11 12:38:34 +03:00
a181c9e434 [#332] gc: Add additional logging
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 17:34:40 +03:00
90799497d3 [#114] Add remove-node IR control command
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-10 14:31:44 +00:00
ea10abb42a [#296] morph: Add parser unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 09:40:26 +03:00
1309622b20 [#296] morph: Add listener unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 09:40:26 +03:00
b2ffd7df53 [#291] object: Use PayloadSizeLimiter from SDK
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-05 19:07:06 +03:00
35ea207df6 [#291] object: Split validating target in two
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-05 19:07:06 +03:00
Pavel Karpy
ee58b390bb [#221] node: Allow using vector copies_number
Also, take into account that value in general (it was not used before at
all).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-05 16:07:13 +00:00
d02950ad63 [#330] morph: Fix linter issue
Revive became smarter after go 1.20.4 upgrade.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-05 18:02:38 +03:00
973af12854 [#327] tests: replace os.MkdirTemp with t.TempDir
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-05 11:24:51 +00:00
cedd07bbc8 [#304] morph: Iterate endpoints when create ws client in constructor
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-05 10:07:30 +03:00
Pavel Karpy
479c5a65e1 [#322] node: Fix tree svc panic
If a connection has not been established earlier, it stores `nil` in LRU
cache. Cache eviction tries to close every connection (even a `nil` one) and
panics but not crash the app because we are using pools.
That ugly bug also leads to a deadlock where `Unlock` is not called via
`defer` func (and that is the way I found it).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-04 19:48:51 +03:00
2f6757c828 [#311] ir: Fix data race in unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-04 16:59:15 +03:00
872fe90c40 [#317] pre-commit: Add go-mod-tidy hook
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-04 13:58:19 +00:00
d1661ae7dc [#308] Remove downloadContractsFromGithub from the frostfs-adm
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-04 14:08:42 +03:00
5f1af84587 [#297] go.mod: Update compress to v1.16.5
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 16:28:29 +03:00
a1b4ba9980 [#209] compression: Do not store uncompressible data
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 16:28:29 +03:00
529d0bc710 [#302] tree: Drop unused ctx
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 16:54:06 +03:00
eca5c210dd [#299] evacuate: Add context cancel checks
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 15:55:30 +03:00
b939e4e5c5 [#299] ir: Drop unused structs
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 14:05:35 +03:00
235fe84ea3 [#298] innerring: Fix broken tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-02 11:52:29 +03:00
d00b1c0d29 [#280] ir: Add netmap processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
fb5dcc15d2 [#280] ir: Add governance processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
31b4da225a [#280] ir: Add frostfs processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
5010b35466 [#280] ir: Add container processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
686f01bce5 [#280] ir: Add balance processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
e89fa7f69f [#280] ir: Add alphabet processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
53693071de [#280] ir: Add block timer unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
e70f808dc3 [#280] ir: Add parser unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
2dbe382b5f [#280] ir: Add state unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
500611f3c8 [#280] ir: Add indexer tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
3b64dffda2 [#280] ir: Add fee config tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
22d47376a6 [#277] getsvc: Refactor errors
Move errors to separate files.
Use zap.Error for error logging.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
45438e7b06 [#277] getsvc: Rename and reorder code
Rename execCtx to request.
Move code to appropriate files.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
1440450606 [#277] getsvc: Drop cyclic struct dependency
Drop cyclic dependency between execCtx and Service.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
591c4e7d50 [#277] getsvc: Move headOnly to request params
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
265d2326a0 [#277] getsvc: Extract remote storage
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
30e1b62b67 [#277] getsvc: Fix service deps
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
8fc082b688 [#277] getsvc: Do not return status error
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
3bac5a485d [#248] config: Remove audit-related parameters
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
f368ccbdf0 [#248] morph: Remove audit client
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
b2bc1fccbc [#248] logs: Remove unused messages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
a9c4ba62c3 [#248] metabase: Remove storage group bucket
Backwards compatible change, so no version increase.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
586f5986bc [#248] core: Remove storage group
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
f1ea8fec93 [#248] object_manager: Remove storage group
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
7f49f07255 [#248] services: Remove audit service
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
8879c6ea4a [#248] innerring: Remove audit and settlement code
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
8b2aae73c6 [#248] cli: Remove storagegroup commands
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
f73ac6e02d [#290] control: Use generics for response wrappers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 10:57:24 +03:00
ff25521204 [#270] Add IR epoch tick control call
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-04-28 07:57:00 +00:00
58f1ba4b51 [#288] pilorama: Add missing operation in log
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
daa26f6e9b [#288] pilorama/test: Check operation order for TreeGetByPath()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
291f9e809a [#288] pilorama: Remove getMeta() wrapper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
0045f1bcd4 [#288] pilorama: Use more descriptive names for memory tree
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
f856ad7480 [#288] pilorama: Remove childMap from memory forest
Memory forest is here to check the correctness of boltdb optimized
implementation. Let's keep it simple.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
ada081dfd5 [#19] node: Make policier read shards concurrently
* Introduce ListWithMultiCursor that simultaneously reads objects
  from different shards

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-27 11:11:47 +03:00
1f4061c0e2 [#285] blobonicza: Optimize upperPowerOfTwo()
The real reason is this:
```
pkg/local_object_storage/blobovnicza/sizes.go:36:69
    revive empty-block: this block is empty, you can remove it
```

Didn't want to make this function longer or to add `nolint`, thus this
change. To justify:
```
UpperBound/size=1-8          0.4924n ± 1%   0.2472n ± 2%  -49.80% (p=0.000 n=10)
UpperBound/size=1023-8       0.4936n ± 3%   0.2442n ± 1%  -50.52% (p=0.000 n=10)
UpperBound/size=66560-8      0.8201n ± 2%   0.2436n ± 1%  -70.29% (p=0.000 n=10)
UpperBound/size=41943040-8   6.6900n ± 5%   0.2432n ± 0%  -96.36% (p=0.000 n=10)
geomean                       1.075n        0.2446n       -77.24%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:11 +03:00
dfe4ada838 [#285] lint: Resolve revive/if-return
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:08 +03:00
f07e2d4812 [#285] lint: Fix revive/unused-parameter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:04 +03:00
57718bd6b4 [#285] *: Update golangci-lint to v1.52.2
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:04 +03:00
14f83b8aa9 [#125] ir: Change log level on SIGHUP
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:54:19 +03:00
563780057d [#125] ir: Use internal/common/config for reading config
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
ef222e2487 [#125] cmd: Refactor internal/common/viper
Add `opts.WithViper`, set opts struct as private.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
e61aec4a7d [#125] node: Remove unused config.Prm
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
eb7be82e87 [#125] node: Avoid panic when reading config
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
d390f093e0 [#125] node: Move viper creation to internal/common/config
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
b2123bfd1a [#125] node: Add new option WithEnvPrefix
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
ee01275d25 [#125] node: Remove redundant env from config/internal
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
9d01029733 [#166] node: Parallelize background tree service sync by batching
* Merge operations

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-26 10:17:56 +00:00
299b24b974 [#166] node: Parallelize background tree service sync by batching
* Concurrently dispatch TreeApply operations for batching in forest

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-26 10:17:56 +00:00
700f39c3f8 [#229] Update CHANGELOG.md
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-26 10:23:33 +03:00
Denis Kirillov
dce5924a89 [#229] services/tree: Use bearer owner as signer
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-26 10:23:33 +03:00
89530534a1 [#229] service/tree: Disable container owner check in tree service
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-04-26 10:23:33 +03:00
c04f6c5e59 [#229] acl: Allow Impersonate
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-04-26 10:23:33 +03:00
04be9415d9 [#231] node: Fix race condition in TTL cache
Use key locker to lock by key.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-25 09:54:12 +03:00
ddbc9e255f [#231] node: Invalidate container cache on PutSuccess event
For example: frostfs-cli creates container and makes polling
GetContainer requests. These requests go through container cache,
so not found error stores in container cache.
So container cache can contain not found error when PutSuccess event received.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-25 09:42:02 +03:00
c70306b324 [#164] scripts: Add metrics description exporter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00
6fef2726b8 [#164] metrics: Allow to export metrics description
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00
4ade5339da [#164] metrics: Rename metrics.go -> node.go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00
015d62425b [#164] metrics: Fill local registry explicitly
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00
Pavel Karpy
59822f7fb4 [#1248] node: Do not update cache twice
Do not request morph values on morph cache misses concurrently.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-21 09:45:53 +00:00
dc2e9d70c7 [#271] go.mod: Update prometheus to v1.15.0
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-20 19:30:26 +03:00
Pavel Karpy
09938a9841 Revert "[#262] meta: Do not return old expired objects"
This reverts commit 3d23b087

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 19:22:55 +03:00
Pavel Karpy
e9461686b8 [#274] wc: Resolve possible deadlock
If operation with WC are _fast enough_ (e.g. `Init` failed and `Close` is
called immediately) there is a race and a deadlock that do not allow finish
(and start, in fact) an initialization routine because of taken `modeMtx`
and also do not allow finish `Close` call because of awaiting initialization
finish. So do stop initialization _before_ any mutex is taken.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 19:22:54 +03:00
Pavel Karpy
6b6f33ed71 [#274] wc: Make wait groups work more explicit
Do not run routine that calls `wg.Done()` inside, it is hard to read.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 19:22:54 +03:00
Pavel Karpy
4f5f832137 [#268] notary_preparator: Actualize notary requests parsing
After 75d7891ca1
`neo-go` does claim that an empty invocation script is the only way to
fill missing signature for unsigned notary requests. The new notary actor
does it that way and, therefore, breaks notary request parsing by the
Alphabet because of skipping any request that is not filled with a dummy (64
zeros) invocation script. Support both way. The "Dummy" approach will be
dropped later.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 10:29:34 +03:00
6c90bb87f1 [#118] node: add ctx for unit tests for blobstor
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-19 18:09:33 +03:00
Pavel Karpy
3d23b08773 [#262] meta: Do not return old expired objects
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-19 13:37:38 +00:00
13c8afcb02 [#118] node: add unit concurrent tests for blobstor
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-19 10:22:50 +00:00
Pavel Karpy
20cd080323 [#255] write-cache: Fix init race condition
Do not use WC's internals in the initialization routines without mode
protection. WC should be able to change its mode even if the initialization
is not finished yet.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-19 09:29:07 +00:00
3d43b0f7f9 [#265] node: Fix after SDK & API-Go version up
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-18 12:09:19 +03:00
a358255c1b [#265] node: Drop unused
Resolve unused linter.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-18 12:09:16 +03:00
b447ff99aa [#265] node: Up SDK and API-Go versions
Fix tracing panic.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-18 12:09:12 +03:00
7b981bfe97 [#249] logs: Drop unused consts
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
f07d4158f5 [#249] node: Drop subnet from IR and morph
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
d757d881d0 [#249] node: Drop subnet from config
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
05c870f39a [#249] cli: Drop subnet support
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
160147b05d [#249] adm: Drop subnet support
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
Pavel Karpy
262c9c2b93 [#256] blobovniczaTree: Make Exists test stable
Corrupt and request _the same_ file.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 16:46:24 +03:00
0b42a00a60 [#254] innerring: Remove unused TimersHandlers() method from processors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 15:34:15 +03:00
8466894fdf [#250] control: remove DumpShard and RestoreShard RPC
We have `Evacuate` with a cleaner interface.
Also, remove them from CLI and engine.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 12:28:49 +00:00
Pavel Karpy
070154d506 [#247] client: Drop reputation related RPCs
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:54:28 +03:00
Pavel Karpy
beabed788c [#247] network_config: Drop reputation
Drop the code that was expected to work with global reputation network
parameters.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:53:22 +03:00
Pavel Karpy
b453bb754c [#247] logs: Drop reputation log messages
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:44:14 +03:00
Pavel Karpy
8799138fcb [#247] morph: Drop reputation contract
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:44:14 +03:00
Pavel Karpy
960e3c219e [#247] config, doc: Drop reputation references
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:44:14 +03:00
Pavel Karpy
560f73ab7e [#247] node, ir: Drop reputation related code
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:44:14 +03:00
6121b541b5 [#242] treesvc: Add tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
d62c6e4ce6 [#242] node: Add tracing spans
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
200fc8b882 [#242] put: Pass context to relay function
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
995db117d0 [#238] node: Read cfg from dir even if cfg file not set
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 10:19:10 +00:00
8d2f443868 [#238] Fix linter error
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 10:19:10 +00:00
41eb3129ae [#139] Refactor blobovnicza exist test to not use chmod
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-04-14 12:16:14 +03:00
adcfce39cf [#246] .gitattributes: Do not show diff for go.sum
When we update dependencies it can be rather big. However it is
generated automatically with `go mod tidy`, no need to review.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 06:02:04 +00:00
299b6a6938 [#100] adm: Use netmap constants from pkg
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 05:12:50 +00:00
0c6aeaaf18 [#100] adm: Take net settings into account during netmap contract update
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 05:12:50 +00:00
4496999e52 [#100] Fix CHANGELOG
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 05:12:50 +00:00
cffcc7745e [#240] logs: Factor out common service log messages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 05:06:09 +00:00
0e31c12e63 [#240] logs: Move log messages to constants
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 05:06:09 +00:00
Roman Khimov
d29b13454f [#239] morph/client: Simplify code interacting with magic numbers
It can't be uint64 in fact, but this error is buried deeply in the NetworkInfo
API structure, so we're not touching MagicNumber() for now.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
d686ab49e8 [#202] adm: Remove deprecated RPC client methods
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
Roman Khimov
f41ad9d419 [#239] morph/client: Recreate actor/wrappers in SetGroupSignerScope
That's the reason #2230 and #2263 were not detected earlier, we actually had
Global scope being used before reconnection to RPC node.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
Roman Khimov
be4df989e5 [#239] morph/client: Deduplicate signers in Client a bit
One signer in the cfg is enough.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
Roman Khimov
96b38f7e86 [#239] morph/client: Add CalledByEntry into the "grouped" scope
Fixes #2230, fixes #2263. CustomGroups are nice while we're only calling NeoFS
contracts, but it doesn't work at all for standard ones like GAS or Notary.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
2f1beddfd3 [#202] adm: Remove deprecated warnings in tests
`VerifyBlocks` is now `SkipBlockVerification` and is false by default.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
01c0c90a86 [#113] cli: add "name" option for "get container" command
* Make get container command filter out the container by attribute name

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-12 18:25:37 +00:00
7d39fecc6a Release v0.36.0
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-12 16:57:02 +03:00
04727ce1d6 Revert "[#135] signature: Add tracing"
This reverts commit 5778980252.
2023-04-12 16:57:02 +03:00
08769f413f Revert "[#135] acl: Add tracing spans"
This reverts commit b2ca730547.
2023-04-12 16:54:13 +03:00
5d2affa5cd testutil: Fix linter warning
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-12 11:00:02 +03:00
5778980252 [#135] signature: Add tracing
Add tracing to verify request and sign response.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
b2ca730547 [#135] acl: Add tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
0920d848d0 [#135] get-object: Add tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
5af9f58469 [#135] tracing: Add tracing to node gRPC endpoints
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
72565a91ef [#135] node: Update api-go version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
c4865783fc [#236] blobstor/test: Prefill storage in parallel in read benchmark
`blobovniczatree` takes a really long time to prefill, because each
batch takes at least 10ms, so for 10k iterations we have at least 100s of
prefill.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:11 +03:00
6ad5c38225 [#236] testutil: Use random object id in RandObjGenerator
Before this commit it was like this:
```
BenchmarkSubstorageWritePerf/memstore-rand10-8            227425              4859 ns/op
BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8     --- FAIL: BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
BenchmarkSubstorageWritePerf/fstree-rand10-8            --- FAIL: BenchmarkSubstorageWritePerf/fstree-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
c85a0bc866 [#236] blobstor/test: Reduce test descriptions
I tried to add 4 more tests and suddenly, it became harder to navigate in
code. Move directory creation in a common function.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
6bf11f7cca [#230] CHANGELOG.md: Remove older entries
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 10:07:27 +00:00
2c07f831c7 [#223] node: Refactor cache usage
Drop excess type args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
93eba19a8e [#223] objectsvc: Refactor split-tree traverse
Resolve funlen & gocognit linters for traverseSplitChain method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
2ed9fd3f94 [#223] objectsvc: Refactor request parameters
Resolve containedctx linter for commonPrm.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
ccf8463e69 [#223] controlsvc: Drop unnecessary nolint
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
ae86cda58c [#223] sg: Refactor storage group parameters
Resolve containedctx linter for SearchSGPrm and GetSGPrm  structs.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
6016d78a45 [#223] core: Refactor object format validator
Resolve funlen linter for FormatValidator.ValidateContent method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 13:56:09 +03:00
02831d427b [#213] metrics: Refactor object metrics
Resolve funlen linter for newObjectServiceMetrics function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 09:43:24 +03:00
38ae71cc7d [#213] metrics: Refactor engine metrics
Resolve funlen linter for newEngineMetrics function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 09:43:24 +03:00
b689027d57 [#191] cli: Add control shards doctor command
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
0b9622c418 [#191] control: Add Doctor RPC
Doctor RPC performs complex operations on the storage engine.
Currently only duplicate removal is supported.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
dbc3811ff4 [#191] engine: Allow to remove redundant object copies
RemoveDuplicates() removes all duplicate object copies stored on
multiple shards. All shards are processed and the command tries to leave
a copy on the best shard according to HRW.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
cb172e73a6 [#228] node: Use uber atomic package instead standard
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 15:37:27 +00:00
c236b54a65 [#212] reputationsvc: Resolve funlen linter
Resolve funlen linter for Calculator.iterateDaughter method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
7ebbfa3358 [#212] reputationsvc: Resolve linters and rename
Resolved containedctx linters.
Renamed context structs and interfaces to more understandble names.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
469e8a6e59 [#212] reputationsvc: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
e2f13d03d7 [#222] auditsvc: Refactor PoR audit
Resolve funlen linter for Context.checkStorageGroupPoR method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
e8d340287f [#222] auditsvc: Refactor audit task
Resolve containedctx linter. Cancel task by listen cancel.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
3dbff0a478 [#222] auditsvc: Resolve containedctx linter
Resolve containedctx linter for commonCommunicatorPrm type.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
fe87735073 [#219] morph: Refactor notary preparator
Resolve funlen linter for Preparator.Prepare method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
d07e40d6fe [#219] morph: Refactor moprh event listener
Resolve funlen and gocognit linters for listener.listenLoop method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
775179f823 [#219] morph: Refactor notary invoke
Resolve funlen linter for Client.notaryInvoke method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
e815b19101 [#219] morph: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
56282edf02 [#166] node: Parallelize background tree service sync
* Run sync task for nodes in parallel within errgroup worker pool

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 04:43:32 +00:00
9027695371 [#203] node: Add staticcheck target and pre-commit
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
4ec69cbbf8 [#203] control: Fix test
Fix shard compare loop.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
f32f61df87 [#203] pilorama: Refactor tests
Do not pass 0 as channel capacity.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
8908798f59 [#203] node: Resolve unused vars
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
bab11492ad [#203] node: Resolve never used errors
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
e21c5bea21 [#203] cli: Fix error message
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
68a2f36636 [#203] morph: Mark depracated methods
Skip staticcheck for depracated methods. Will be fixed soon.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
9e2df4b7c7 [#203] node: Fix double imports
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
ab891517de [#116] node: Fix bug with extra generated files in TestReload
* Create testNewEngine in engineWithShards without default opts

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-05 18:49:34 +03:00
c58ab0c369 [#193] getsvc: Reduce private key requests
Get private key only once for request forwaring.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
89924071cd [#193] getsvc: Edit request forwarder signature
Pass context to forwarder direct, without closure.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
6c7b708a98 [#193] getsvc: Refactor get range params creation
Resolve funlen linter for toRangePrm function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
b0786d2e5c [#193] getsvc: Refactor get params creation
Resolve funlen linter for toPrm function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
f889893216 [#193] getsvc: Refactor head param creation
Resolve funlen linter for toHeadPrm method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
91ead04fa4 [#193] getsvc: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
1bf21dbb47 [#193] getsvc: Resolve context linters
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
206458c841 [#217] containersvc: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:00 +00:00
279261ace3 [#217] containersvc: Refactor route passing
Resolve containedctx for routeCtx.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:00 +00:00
6f7b6a8813 [#116] node: Improve shard/engine construction in tests
* Introduce testEngineWrapper that can be constructed with different options

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-05 14:36:40 +00:00
d6486d172e [#210] policier: Refactor nodes processing
Resolve funlen linter for processNodes method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
080be5cfcd [#210] policier: Refactor object placement
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
23575e1ac0 [#210] policier: Resolve contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
9098d0eec0 [#211] engine: Unify shard mode checks for tree operations
All operations must ensure the shard is not in a degraded mode.
Write operations must also ensure the shard is not in a read-only mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-05 11:10:39 +00:00
760af6b912 [#211] fstree: Consider ENOSPC a logical error
We already do this for file writing, however directory creation can also
fail.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-05 11:10:39 +00:00
d85703a963 [#208] searchsvc: Refactor request forwarding
Resolve funlen & gocognit linters for toPrm method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 16:24:55 +03:00
0b38419fbf [#208] searchsvc: Resolve context linters
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 16:24:50 +03:00
5f2a1531fe [#208] deletesvc: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 16:24:40 +03:00
4941926c9d [#207] aclsvc: Drop outdated tag
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:14 +00:00
585415fa92 [#207] aclsvc: Refactor send checker
Resolve funlen linter for putStreamBasicChecker.Send method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:14 +00:00
9ef790f782 [#207] aclsvc: Refactor object headers read
Resolve funlen linter for readObjectHeaders method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:13 +00:00
cd33a57f44 [#207] aclsvc: Refactor EACL check
Resolve funlen linter for CheckEACL method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:13 +00:00
1f1aed87be [#188] metabase: Refactor object inhume
Resolve funlen linter for db.Inhume method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
5a66db80c5 [#188] engine: Refactor shard evacuation
Resolve funlen and gocognit linter for StorageEngine.Evacuate method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
456bc097f7 [#188] engine: Refactor get range from engine
Resolve funlen linter for StorageEngine.getRange method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
3010ca2649 [#188] engine: Refactor get object from engine
Resolve funlen linter for StorageEngine.get method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
0739c36a3b [#188] metabase: Refactor object put to metabase
Resolve funlen linter for db.put method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
8273a3dfb2 [#188] blobstor: Refactor blobstor test
Resolve funlen linter for TestIterate function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
594b5821ed [#188] blobstor: Refactor put data to blobovniczas
Resolve funlen linter for Blobovniczas.Put method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
ee7468daa7 [#205] innerring: Provide alphabetState param to epochTimer
Fix NPE, introduced in f09ee27a.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-04 10:47:31 +00:00
49cc23e03c [#175] adm: pipeline container iteration
Do not accumulate everything in memory.
Also, CLI should be responsive.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-04 07:26:23 +00:00
e85e5382e4 [#175] adm: list containers using containersOf
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-04 07:26:23 +00:00
8e5a0dcf27 [#204] gc: Fix GC handlers start
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 06:48:27 +00:00
0948a280fa [#195] morph: use blocking unlimited pool for notifications
With non-blocking pool restricted by 10 in capacity, the probability of
dropping events is unexpectedly big. Notifications are an essential part of the FrostFS,
we should not drop anything, especially new epochs.
```
Mar 31 07:07:03 vedi neofs-ir[19164]: 2023-03-31T07:07:03.901Z        debug        subscriber/subscriber.go:154        new notification event from sidechain        {"name": "NewEpoch"}
Mar 31 07:07:03 vedi neofs-ir[19164]: 2023-03-31T07:07:03.901Z        warn        event/listener.go:248        listener worker pool drained        {"chain": "morph", "capacity": 10}
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-03 15:58:16 +00:00
ece6c820e7 [#199] putsvc: Refactor streamer initialization
Resolve funlen linter for initTarget method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
27bdddc48f [#199] putsvc: Refactor put object
Resolve containedctx linter for streamer and remote target

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
cecea8053a [#199] putsvc: Refactor streamer pool
Resolve staticcheck linter for putBytesPool

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
14d894178e [#199] putsvc: Refactor placement iterator
Resolve funlen linter for iteratePlacement method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
Pavel Karpy
a69c6d1ec9 [#2272] morph: Do not subscribe to events without listening
It led to a neo-go dead-lock in the `subscriber` component. Subscribing to
notifications is the same RPC as any others, so it could also be blocked
forever if no async listening (reading the notification channel) routine
exists. If a number of subscriptions is big enough (or a caller is lucky
enough) subscribing loop might have not finished subscribing before the
first notification is received and then: subscribing RPC is blocked by
received notification (non)handling and listening notifications routine is
blocked by not finished subscription loop.
That commit starts listening notification channel _before_ any subscription
actions.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-03 08:42:41 +00:00
Pavel Karpy
2bdf7126b8 [#181] Update CHANGELOG
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:54:05 +03:00
Pavel Karpy
aa92f977ef [#181] ir: Do not pay for audit by non-alphabet nodes
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:53:37 +03:00
Pavel Karpy
f09ee27af9 [#181] ir: Do not process container estimations by non-alphabet nodes
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:53:09 +03:00
Pavel Karpy
db5321309d [#181] ir: Do not sync the Alphabet by non-alphabet nodes
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:52:16 +03:00
Pavel Karpy
44d5412e10 [#181] ir: Do not deposit notary GAS by non-alphabet nodes
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:51:12 +03:00
ed28ce24cd [#168] node: Refactor reputation service
Resolve funlen linter for initReputationService function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
1f929fdd57 [#168] node: Refactor netmap service init
Resolve funlen linter for initNetmapService function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
dd572825c7 [#168] node: Refactor object service init
Resolve funlen linter for initObjectService function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
56161d39b4 [#168] node: Refactor container services
Resolve containedctx for remoteLoadAnnounceWriter struct

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
28dc9e2190 [#168] node: Refactor container service init
Resolve funlen linter for initContainerService function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
dcd39f8fdd [#168] node: Refactor shard opts initialization
Resolve funlen linter for shardOpts method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
c94372e6f9 [#168] node: Refactor config initialization
Resolve funlen linter for initCfg function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
3bbb516528 [#168] node: Refactor node config read
Resolve funlen linter for readConfig method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
a7c79c773a [#168] node: Refactor node config
Resolve containedctx linter for cfg

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
8426d25f4b [#185] ir: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
c1cbe6ff2d [#185] ir: Refactor signature verification
Resolve funlen linter for verifySignature method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
aeb4bbc51e [#185] ir: Refactor alphabet update
Resolve funlen linter for processAlphabetSync method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
30c18d46cc [#185] ir: Refactor audit
Resolve funlen linter for processStartAudit method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
6be2688fb4 [#185] ir: Refactor gas emit
Resolve funlen linter for processEmit method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
4d160bd4ab [#185] ir: Refactor ir service creation
Resolve funlen linter for New function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
c8a6978563 [#185] ir: Refactor ir start
Resolve funlen linter for Server.Start method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
85fb9e77c4 [#185] ir: Resolve funlen linter
Resolve funlen linter for GlagoliticLetter.String method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
7be5a0fd79 [#189] node: Fix bug with sync/atomic package
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-03-30 14:50:43 +00:00
ab32067152 [#183] gc: Fix drop expired locked complex objects
Do not delete bucket keys during iteration

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
9f0bce5c15 [#183] gc: Fix drop expired locked simple objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
341fe1688f [#139] test: Add test storage implementation
This aims to reduce the usage of chmod hackery to induce or simulate
OS-related failures.

Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-29 14:28:49 +00:00
Pavel Karpy
e843e7f090 [#184] ir: Delete unused config reader param
Not used since notary environments do endless and non-configurable
deposits.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
Pavel Karpy
ba58a77f8c [#184] node, ir: Do not wait for already made notary deposits
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
Pavel Karpy
3646723ae3 [#184] morph: Make deposit logs INFO
We have already had and solved plenty of deposit issues and notary balance
is a really important thing. Deserves to be INFO even before the huge logs
severity refactor, happens on an app start only.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
Pavel Karpy
97e201993b [#184] *: Unify done contexts handling
If `ctx.Done()`, return `ctx.Err()` in every function that returns an error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
221203beeb [#180] node: Refactor panics in unit test
* Replace panics in unit tests by require.NoError and t.Fatalf

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-03-29 12:39:07 +03:00
91717d4b98 [#176] morph: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 18:49:55 +03:00
382ecae96a [#172] Use ContainersOf() for container list fetching
Previously we were limited by ~2048 containers because of
neo-go VM limits.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 14:32:55 +00:00
9e54646248 [#172] client/container: Support listing containers with containersOf
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 14:32:55 +00:00
bf7d80f44b [#172] morph/client: Support iterators via neo-go session API
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 14:32:55 +00:00
5bf1ec348f [#161] adm: Refactor storage-config command
Resolve funlen linter for storageConfig function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
93cb9e3a94 [#161] morph: Refactor invokeNotary method
Resolve funlen linter

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
512b72591a [#161] adm: Refactor subnet commands
Resolve funlen linter for init method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
e1b99dacad [#161] adm: Refactor netmap add/remove commands
Resolve funlen linter for manageSubnetAdmins function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
92f8810970 [#161] adm: Refactor commands initialization
Resolve funlen linter for init function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
b0fefcb21f [#161] adm: Refactor deposit-notary command
Resolve funlen linter for depositNotary function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
d62b11f5e7 [#161] adm: Refactor init context creation
Resolve funlen linter for newInitializeContext function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
ecbc211016 [#161] adm: Refactor update-contracts command
Resolve funlen linter for updateContracts method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
5f7d70c59c [#161] adm: Refactor deploy command
Resolve funlen linter for deployContractCmd function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
9534ade716 [#161] adm: Refactor restore-containers command
Resolve funlen linter for restoreContainers function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
f2e880465e [#165] ir: Refactor config default init
Resolve funlen linter for defaultConfiguration function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
04b3d9d068 [#165] cli: Drop redundant nolint comment
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
1d5f2dd681 [#165] cli: Refactor collectObjectRelatives
Resolve funlen linter

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
c2cf708e0e [#165] cli: Refactor put command
Resolve funlen linter for putObject method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
c78e9cc857 [#165] cli: Refactor get command
Resolve funlen linter for getObject method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
Pavel Karpy
3c7ed21f74 [#141] Add big object's parts to a lock object's body
That will prevent part/link object from being removed by both an external
`DELETE` call and the object expiration procedure.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-27 07:28:32 +00:00
a5ece7889d [#174] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 07:24:58 +00:00
Pavel Karpy
f3ff9fd251 [#73] morph: Rename vars that collide with package names
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-24 09:42:30 +00:00
Pavel Karpy
533e9f8b75 [#59] morph: Adopt updated neo-go client API for subs
It does not use deprecated methods anymore but also adds more code that
removes. Future refactor that will affect more components will optimize
usage of the updated API.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-24 09:42:30 +00:00
9ffa0d8fea [#171] Remove unit tests from pipeline
Unit tests are included as pre-commit hook. They are
triggered by the change of .go files.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-24 10:50:11 +03:00
d857ffeb2e [#171] Syncrhonize pre-commit settings across FrostFS repos
This change allows to use `[#xx]` placeholders for issue number.
This change adds go unit test run if .go files were changed.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-24 10:43:55 +03:00
aarifullin
34329d67ff [#86] node: Fix unit test and linter errors
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 12:42:58 +03:00
9808dec591 [#86] node: Move testing utils to one package
Move testing utils from tests in local_object_storage package to
unified testutil package

Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 08:19:15 +00:00
342e571d89 [#159] Add handle __SYSTEM__ sys attributes
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-22 17:35:20 +03:00
f111704ceb [#158] Fix gitlint run in pre-commit under CI
Wrong stage was used.
It still may run gitlint against only latest commit in PR.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-22 08:58:34 +00:00
Pavel Karpy
da8da1c63a [#98] fstree: Do not fail iteration over just removed files
A directory is read and files are saved to a local variable. The iteration
over such files may lead to a non-existing files reading due to a normal SN
operation cycle and, therefore, may lead to a returning the OS error to a
caller. Skip just removed (or lost) files as the golang std library does in
similar situations:
5f1a0320b9/src/os/dir_unix.go (L128-L133).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-22 07:26:27 +00:00
49234b915e [#155] search-service: Fix search with ST
Search should return only objects allowed in static session

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-22 07:21:09 +00:00
1637a3edce [#155] search-service: Add search with ST test
In case of session token (ST) with object IDs search should
return only objects allowed in static session

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-22 07:21:09 +00:00
cbc2efb1d6 [#156] object/get: Make toHeadPrm() pass gocognit
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
484ac502ca [#156] frostfs-adm: Make dumpBalances() pass linter checks
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
8014fdb21a [#156] metabase: Make freePotentialLocks() pass linter checks
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
fb13902db9 [#156] shard: Make refillMetabase() pass linter checks
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
3f6b962349 [#156] services/tree: Pass context to replicationWorker()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
5368c4207a [#156] services/tree: Split syncLoop() in functions
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
47e8c5bf23 [#156] pilorama: Remove CIDDescriptor from TreeApply()
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
   only used in controlled environments.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
ec2c5d45b4 Add to Possible Solutions
Signed-off-by: Liza <e.chichindaeva@yadro.com>
2023-03-21 15:41:20 +03:00
7eb9e88f8f Add bug label; Delte label addition
Signed-off-by: Liza <e.chichindaeva@yadro.com>
2023-03-21 15:09:22 +03:00
9aeea0b974 [#153] ci: Minor pipeline fixes
- We can skip full pre-commit run
- On a very slow agent golangci run may take up to 10 minutes

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-21 09:33:45 +00:00
Pavel Karpy
9a4f40626c [#128] IR: Do not try to emit GAS to nobody
Fix sending GAS to an empty extra wallets receivers list. Also, send GAS to
extra wallets even if netmap is empty.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-21 08:59:46 +00:00
7a31988a36 [#145] docs: Add expired object collector params
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
5059dcc19d [#145] shard-gc: Delete expired objects after locks
GC deletes expired locks and objects sequentially. Expired locks and
objects are now being deleted concurrently in batches. Added a config
parameter that controls the number of concurrent workers and batch size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
6c4a1699ef [#145] shard-gc: Expired locked unit test
Added unit test that verifies that GC deletes expired
locked objects in one epoch.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
Pavel Karpy
9cd8f7cea0 [#152] IR: Process empty basic incomes
If network is not configured for basic income earnings, do not distribute
GAS by the Alphabet nodes.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-21 06:59:33 +00:00
44b86bac5a [#148] linter: Add contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
481a1ca6f3 [#148] linter: Add gocognit linter
Code with high cognitive complexity is hard intuitively to understand

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
97c36ed3ec [#148] linter: Add funlen linter
Long functions are hard to understand and source of errors

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
cc8ff015b4 [#148] linter: Add containedctx linter
Context has to be passed as an argument: https://pkg.go.dev/context

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:52:39 +03:00
2dc86058c3 [#148] memstore: Drop space line
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:52:39 +03:00
573d920821 [#149] Use custom image and kludges for node
Until #139 is fixed, we can't use root inside Docker container running
CI, but Woodpecker CI can't run non-root containers until they fix
https://github.com/woodpecker-ci/woodpecker/issues/1077, hence we use
temporary kludges with custom image and manual permissions in pipelines.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-20 18:37:37 +03:00
d64fb887ff [#149] Reorder pre-commit hooks
Minor changes to see what fails first

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-20 18:37:34 +03:00
392be818e5 [#149] Fix gitlint regex to match our policy
In our policy we mark commits not having a PR/Issue yet with a `[#XX]`
reference to be replaced after PR creation.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-20 18:37:27 +03:00
db3ccd2762 [#128] innerring: Add GAS pouring mechanism for a configurable list of wallets
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-03-20 12:50:05 +00:00
Pavel Karpy
abd21f8099 [#136] Revert "[#2260] services/object: Do not assemble object with TTL=1"
This reverts commit 2567f8020e. It assumes
that assembling logic could break some failover scenarios if request
forwarding is done. However, it also breaks requesting big objects via a
non-container node with TTL=2. Failover has been rechecked without that
commit and no problems were found. Any (if found) other bugs related to
the forwarding and object assembling must be solved more carefully.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-20 10:52:18 +03:00
Pavel Karpy
10c419adf0 [#67] node: Fix infinite recursion in SE's wrapper
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-17 18:47:59 +03:00
b70caa216b [#144] Don't copy cache inside Docker environment
May make docker builds not so clean.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-17 10:36:50 +03:00
Pavel Karpy
64bde68fb9 [#67] node: Accept expired locked objects
Allow replication of any (expired too) locked object. Information about
object locking is considered to be presented on the _container nodes_.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-16 16:22:19 +03:00
Pavel Karpy
f006f3b342 [#67] node: Make engine's IsLocked public
It will allow reusing that method in expiration checks.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-16 16:20:45 +03:00
22be532cbd object/put: Persist session token till the end of a session
Previously a token could've expired in the middle of an object.PUT
stream, leading to upload being interrupted. This is bad, because user
doesn't always now what is the right values for the session token
lifetime. More than that, setting it to a very high value will
eventually blow up the session token database.

In this commit we read the session token once and reuse it for the whole
stream duration.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-16 06:45:50 +00:00
724debfdcd [#81] node: Add basic read/write benchmarks for substorages
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-15 16:37:04 +00:00
b1c165a93b [#83] Fix shellcheck pre-commit action
Original shellcheck action requires Docker to run and it's not always
available, especially inside Docker containers. Replacing it with python
wrapper to simplify usage with Docker-based CI systems like Drone and
WoodpeckerCI.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-15 12:24:41 +00:00
ac0a278a05 [#85] get-service: Drop unused assemble flag
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
b8e93d4c08 [#85] get-service: Use assembler to assemble LOB
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
07de839f18 [#85] get-service: Fix corrupted chain logic
Should return an error in case of a broken LOB reference chain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
2886b1581b [#85] get-service: Add unit tests
Add unit tests to cover all assemble statements

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
8b9e40a848 [#85] get-service: Add assembler
Extract assemble logic to assembler

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
b4582239bf [#130] adm: Fix adding of pub key for group.frostfs at init step
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-13 15:05:53 +03:00
4e244686cf [#83] Makefile fixes for pre-commit
Add make targets to simplify pre-commit setup for individual developers.

Signed-off-by: Stanislav Bogatyrev <realloc@realloc.spb.ru>
2023-03-13 11:39:18 +00:00
6cd806f998 [#82] services/tree: Save last synchronized height in a persistent storage
Remember the last synchronized height and use it after service restart.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 11:25:44 +00:00
3e6fd4c611 [#82] pilorama: Allow to store last sync height
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 11:25:44 +00:00
5ae4446280 [#50] ir: Add Health status
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-13 11:24:42 +00:00
5890cd4d7d [#50] ir: Fix config property name for prometheus
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-13 11:24:42 +00:00
365adb4ebd [#133] .github: Restore logo.svg
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 10:39:12 +03:00
bce5827f64 [#83] pre-commit: Add shellcheck hook
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:30 +00:00
05471d3827 [#83] util/autocomplete: Fix deprecated warning
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:30 +00:00
8226d49376 [#83] pre-commit: Add gitlint hook
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:30 +00:00
0893689c6a [#83] pre-commit: Add golangci-lint hook
Skip deprecated warning for now, adopting new neo-go API will be done in
another task.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:29 +00:00
a4931ea4c7 [#83] .github: Remove CODEOWNERS and actions
Issue templates are still supported by Gitea:
https://docs.gitea.io/en-us/issue-pull-request-templates/ .

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:29 +00:00
861e9ab59a [#83] pre-commit: Add initial configuration
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:29 +00:00
Leonard Lyubich
24a540caa8 [#132] cli/util: Fix basic ACL rendering
In previous implementation pretty-printer of basic ACL in NeoFS CLI had
mistakes:
 * F-bit was set to `Extendable()` property instead of its inversion
 * B-bits were set to `acl.RoleInnerRing` rights

Make `PrettyPrintTableBACL` to correctly render mentioned bits.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2023-03-10 14:55:45 +03:00
6226c3ba86 [#129] policer: Use safer defaults
If `processNodes` exits earlier for some reason, `needLocalCopy` could
be false.
See https://github.com/nspcc-dev/neofs-node/issues/2267

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-10 10:59:15 +00:00
f2250a316f [#129] tree: Do not remove tree if the netmap is empty
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-10 10:59:15 +00:00
9929dcf50b [#126] adm: Exclude group.frostfs key from output of the dump-hashes
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-09 18:29:36 +03:00
7486c02bbc [#88] adm: Fix method nnsResolveKey
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-09 15:41:28 +03:00
Pavel Karpy
f1f3c80dbf [#32] node: Init write-cache asynchronously
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-09 11:07:33 +00:00
Pavel Karpy
381e363a8b [#32] node: Always close general components after testing
It will prevent test fails with `-race` flag on components that have
background processes and make some actions on test framework.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-09 11:07:33 +00:00
20de74a505 Rename package name
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
1145 changed files with 38256 additions and 38842 deletions

View file

@ -1,4 +1,4 @@
FROM golang:1.18 as builder FROM golang:1.21 as builder
ARG BUILD=now ARG BUILD=now
ARG VERSION=dev ARG VERSION=dev
ARG REPO=repository ARG REPO=repository

25
.docker/Dockerfile.ci Normal file
View file

@ -0,0 +1,25 @@
FROM golang:1.21
WORKDIR /tmp
# Install apt packages
RUN apt-get update && apt-get install --no-install-recommends -y \
pip \
&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
&& rm -rf /var/lib/apt/lists/*
# Dash → Bash
RUN echo "dash dash/sh boolean false" | debconf-set-selections
RUN DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
RUN useradd -u 1234 -d /home/ci -m ci
USER ci
ENV PATH="$PATH:/home/ci/.local/bin"
COPY .pre-commit-config.yaml .
RUN pip install "pre-commit==3.1.1" \
&& git init . \
&& pre-commit install-hooks \
&& rm -rf /tmp/*

View file

@ -1,4 +1,4 @@
FROM golang:1.18 as builder FROM golang:1.21 as builder
ARG BUILD=now ARG BUILD=now
ARG VERSION=dev ARG VERSION=dev
ARG REPO=repository ARG REPO=repository

View file

@ -1,4 +1,4 @@
FROM golang:1.18 as builder FROM golang:1.21 as builder
ARG BUILD=now ARG BUILD=now
ARG VERSION=dev ARG VERSION=dev
ARG REPO=repository ARG REPO=repository

View file

@ -1,4 +1,4 @@
FROM golang:1.18 as builder FROM golang:1.21 as builder
ARG BUILD=now ARG BUILD=now
ARG VERSION=dev ARG VERSION=dev
ARG REPO=repository ARG REPO=repository

View file

@ -1,19 +0,0 @@
FROM golang:1.18 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/frostfs-node
# Executable image
FROM alpine AS frostfs-node
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/frostfs-node /bin/frostfs-node
COPY --from=builder /src/config/testnet/config.yml /config.yml
CMD ["frostfs-node", "--config", "/config.yml"]

View file

@ -5,4 +5,5 @@ docker-compose.yml
Dockerfile Dockerfile
temp temp
.dockerignore .dockerignore
docker docker
.cache

View file

@ -0,0 +1,41 @@
name: Build
on: [pull_request]
jobs:
build:
name: Build Components
runs-on: ubuntu-latest
strategy:
matrix:
go_versions: [ '1.20', '1.21' ]
steps:
- uses: actions/checkout@v3
with:
# Allows to fetch all history for all branches and tags.
# Need this for proper versioning.
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '${{ matrix.go_versions }}'
- name: Build CLI
run: make bin/frostfs-cli
- run: bin/frostfs-cli --version
- name: Build NODE
run: make bin/frostfs-node
- name: Build IR
run: make bin/frostfs-ir
- name: Build ADM
run: make bin/frostfs-adm
- run: bin/frostfs-adm --version
- name: Build LENS
run: make bin/frostfs-lens
- run: bin/frostfs-lens --version

View file

@ -0,0 +1,21 @@
name: DCO action
on: [pull_request]
jobs:
dco:
name: DCO
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
- name: Run commit format checker
uses: https://git.frostfs.info/TrueCloudLab/dco-go@v2
with:
from: 'origin/${{ github.event.pull_request.base.ref }}'

View file

@ -0,0 +1,73 @@
name: Tests and linters
on: [pull_request]
jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
cache: true
- name: Install linters
run: make lint-install
- name: Run linters
run: make lint
tests:
name: Tests
runs-on: ubuntu-latest
strategy:
matrix:
go_versions: [ '1.20', '1.21' ]
fail-fast: false
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '${{ matrix.go_versions }}'
cache: true
- name: Run tests
run: make test
tests-race:
name: Tests with -race
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
cache: true
- name: Run tests
run: go test ./... -count=1 -race
staticcheck:
name: Staticcheck
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
cache: true
- name: Install staticcheck
run: make staticcheck-install
- name: Run staticcheck
run: make staticcheck-run

View file

@ -0,0 +1,22 @@
name: Vulncheck
on: [pull_request]
jobs:
vulncheck:
name: Vulncheck
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
- name: Install govulncheck
run: go install golang.org/x/vuln/cmd/govulncheck@latest
- name: Run govulncheck
run: govulncheck ./...

1
.gitattributes vendored
View file

@ -1,2 +1,3 @@
/**/*.pb.go -diff -merge /**/*.pb.go -diff -merge
/**/*.pb.go linguist-generated=true /**/*.pb.go linguist-generated=true
/go.sum -diff

1
.github/CODEOWNERS vendored
View file

@ -1 +0,0 @@
* @TrueCloudLab/storage-core @TrueCloudLab/committers

View file

@ -2,7 +2,7 @@
name: Bug report name: Bug report
about: Create a report to help us improve about: Create a report to help us improve
title: '' title: ''
labels: community, triage labels: community, triage, bug
assignees: '' assignees: ''
--- ---
@ -18,8 +18,11 @@ assignees: ''
If suggesting a change/improvement, explain the difference from current behavior --> If suggesting a change/improvement, explain the difference from current behavior -->
## Possible Solution ## Possible Solution
<!-- Not obligatory, but suggest a fix/reason for the bug, <!-- Not obligatory
or ideas how to implement the addition or change --> If no reason/fix/additions for the bug can be suggested,
uncomment the following phrase:
No fix can be suggested by a QA engineer. Further solutions shall be up to developers. -->
## Steps to Reproduce (for bugs) ## Steps to Reproduce (for bugs)
<!-- Provide a link to a live example, or an unambiguous set of steps <!-- Provide a link to a live example, or an unambiguous set of steps
@ -41,10 +44,3 @@ assignees: ''
* Version used: * Version used:
* Server setup and configuration: * Server setup and configuration:
* Operating System and version (`uname -a`): * Operating System and version (`uname -a`):
## Don't forget to add labels!
- component label (`frostfs-adm`, `frostfs-storage`, ...)
- `goodfirstissue`, `helpwanted` if needed
- does this issue belong to an epic?
- priority (`P0`-`P4`) if already triaged
- quarter label (`202XQY`) if possible

View file

@ -1,29 +0,0 @@
name: CHANGELOG check
on:
pull_request:
branches:
- master
- support/**
jobs:
build:
runs-on: ubuntu-latest
name: Check for updates
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Get changed CHANGELOG
id: changelog-diff
uses: tj-actions/changed-files@v29
with:
files: CHANGELOG.md
- name: Fail if changelog not updated
if: steps.changelog-diff.outputs.any_changed == 'false'
uses: actions/github-script@v3
with:
script: |
core.setFailed('CHANGELOG.md has not been updated')

View file

@ -1,37 +0,0 @@
name: Configuration check
on:
pull_request:
branches:
- master
- support/**
jobs:
build:
runs-on: ubuntu-latest
name: config-check
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Get changed config-related files
id: config-diff
uses: tj-actions/changed-files@v29
with:
files: |
config/**
cmd/neofs-node/config/**
- name: Get changed doc files
id: docs-diff
uses: tj-actions/changed-files@v29
with:
files: docs/**
- name: Fail if config files are changed but the documentation is not updated
if: steps.config-diff.outputs.any_changed == 'true' && steps.docs-diff.outputs.any_changed == 'false'
uses: actions/github-script@v3
with:
script: |
core.setFailed('Documentation has not been updated')

View file

@ -1,22 +0,0 @@
name: DCO check
on:
pull_request:
branches:
- master
- support/**
jobs:
commits_check_job:
runs-on: ubuntu-latest
name: Commits Check
steps:
- name: Get PR Commits
id: 'get-pr-commits'
uses: tim-actions/get-pr-commits@master
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: DCO Check
uses: tim-actions/dco@master
with:
commits: ${{ steps.get-pr-commits.outputs.commits }}

View file

@ -1,60 +0,0 @@
name: frostfs-node tests
on:
push:
branches:
- master
- support/**
paths-ignore:
- '*.md'
pull_request:
branches:
- master
- support/**
paths-ignore:
- '*.md'
jobs:
test:
runs-on: ubuntu-20.04
strategy:
matrix:
go: [ '1.18.x', '1.19.x' ]
steps:
- name: Setup go
uses: actions/setup-go@v3
with:
go-version: ${{ matrix.go }}
- name: Check out code
uses: actions/checkout@v3
- name: Cache go mod
uses: actions/cache@v3
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-go-${{ matrix.go }}-
- name: Run go test
run: go test -coverprofile=coverage.txt -covermode=atomic ./...
- name: Codecov
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
run: bash <(curl -s https://codecov.io/bash)
lint:
runs-on: ubuntu-20.04
steps:
- uses: actions/setup-go@v3
with:
go-version: 1.19
- uses: actions/checkout@v3
- name: golangci-lint
uses: golangci/golangci-lint-action@v3
with:
version: v1.50.0
args: --timeout=5m
only-new-issues: true

11
.gitlint Normal file
View file

@ -0,0 +1,11 @@
[general]
fail-without-commits=True
regex-style-search=True
contrib=CC1
[title-match-regex]
regex=^\[\#[0-9Xx]+\]\s
[ignore-by-title]
regex=^Release(.*)
ignore=title-match-regex

View file

@ -4,7 +4,7 @@
# options for analysis running # options for analysis running
run: run:
# timeout for analysis, e.g. 30s, 5m, default is 1m # timeout for analysis, e.g. 30s, 5m, default is 1m
timeout: 5m timeout: 20m
# include test files or not, default is true # include test files or not, default is true
tests: false tests: false
@ -24,6 +24,28 @@ linters-settings:
govet: govet:
# report about shadowed variables # report about shadowed variables
check-shadowing: false check-shadowing: false
staticcheck:
checks: ["all", "-SA1019"] # TODO Enable SA1019 after deprecated warning are fixed.
funlen:
lines: 80 # default 60
statements: 60 # default 40
gocognit:
min-complexity: 40 # default 30
importas:
no-unaliased: true
no-extra-aliases: false
alias:
pkg: git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object
alias: objectSDK
custom:
truecloudlab-linters:
path: bin/external_linters.so
original-url: git.frostfs.info/TrueCloudLab/linters.git
settings:
noliteral:
target-methods : ["reportFlushError", "reportError"]
disable-packages: ["codes", "err", "res","exec"]
constants-package: "git.frostfs.info/TrueCloudLab/frostfs-node/internal/logs"
linters: linters:
enable: enable:
@ -51,6 +73,11 @@ linters:
- predeclared - predeclared
- reassign - reassign
- whitespace - whitespace
- containedctx
- funlen
- gocognit
- contextcheck
- importas
- truecloudlab-linters
disable-all: true disable-all: true
fast: false fast: false

54
.pre-commit-config.yaml Normal file
View file

@ -0,0 +1,54 @@
ci:
autofix_prs: false
repos:
- repo: https://github.com/jorisroovers/gitlint
rev: v0.19.1
hooks:
- id: gitlint
stages: [commit-msg]
- id: gitlint-ci
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: check-added-large-files
- id: check-case-conflict
- id: check-executables-have-shebangs
- id: check-shebang-scripts-are-executable
- id: check-merge-conflict
- id: check-json
- id: check-xml
- id: check-yaml
- id: trailing-whitespace
args: [--markdown-linebreak-ext=md]
- id: end-of-file-fixer
exclude: ".key$"
- repo: https://github.com/shellcheck-py/shellcheck-py
rev: v0.9.0.5
hooks:
- id: shellcheck
- repo: local
hooks:
- id: make-lint
name: Run Make Lint
entry: make lint
language: system
pass_filenames: false
- repo: local
hooks:
- id: go-unit-tests
name: go unit tests
entry: make test
pass_filenames: false
types: [go]
language: system
- repo: https://github.com/TekWizely/pre-commit-golang
rev: v1.0.0-rc.1
hooks:
- id: go-staticcheck-repo-mod
- id: go-mod-tidy

View file

@ -0,0 +1,11 @@
pipeline:
# Kludge for non-root containers under WoodPecker
fix-ownership:
image: alpine:latest
commands: chown -R 1234:1234 .
pre-commit:
image: git.frostfs.info/truecloudlab/frostfs-ci:v0.36
commands:
- export HOME="$(getent passwd $(id -u) | cut '-d:' -f6)"
- pre-commit run --hook-stage manual

File diff suppressed because it is too large Load diff

View file

@ -3,8 +3,8 @@
First, thank you for contributing! We love and encourage pull requests from First, thank you for contributing! We love and encourage pull requests from
everyone. Please follow the guidelines: everyone. Please follow the guidelines:
- Check the open [issues](https://github.com/TrueCloudLab/frostfs-node/issues) and - Check the open [issues](https://git.frostfs.info/TrueCloudLab/frostfs-node/issues) and
[pull requests](https://github.com/TrueCloudLab/frostfs-node/pulls) for existing [pull requests](https://git.frostfs.info/TrueCloudLab/frostfs-node/pulls) for existing
discussions. discussions.
- Open an issue first, to discuss a new feature or enhancement. - Open an issue first, to discuss a new feature or enhancement.
@ -27,19 +27,19 @@ Start by forking the `frostfs-node` repository, make changes in a branch and the
send a pull request. We encourage pull requests to discuss code changes. Here send a pull request. We encourage pull requests to discuss code changes. Here
are the steps in details: are the steps in details:
### Set up your GitHub Repository ### Set up your Forgejo repository
Fork [FrostFS node upstream](https://github.com/TrueCloudLab/frostfs-node/fork) source Fork [FrostFS node upstream](https://git.frostfs.info/TrueCloudLab/frostfs-node) source
repository to your own personal repository. Copy the URL of your fork (you will repository to your own personal repository. Copy the URL of your fork (you will
need it for the `git clone` command below). need it for the `git clone` command below).
```sh ```sh
$ git clone https://github.com/TrueCloudLab/frostfs-node $ git clone https://git.frostfs.info/TrueCloudLab/frostfs-node
``` ```
### Set up git remote as ``upstream`` ### Set up git remote as ``upstream``
```sh ```sh
$ cd frostfs-node $ cd frostfs-node
$ git remote add upstream https://github.com/TrueCloudLab/frostfs-node $ git remote add upstream https://git.frostfs.info/TrueCloudLab/frostfs-node
$ git fetch upstream $ git fetch upstream
$ git merge upstream/master $ git merge upstream/master
... ...
@ -58,7 +58,7 @@ $ git checkout -b feature/123-something_awesome
After your code changes, make sure After your code changes, make sure
- To add test cases for the new code. - To add test cases for the new code.
- To run `make lint` - To run `make lint` and `make staticcheck-run`
- To squash your commits into a single commit or a series of logically separated - To squash your commits into a single commit or a series of logically separated
commits run `git rebase -i`. It's okay to force update your pull request. commits run `git rebase -i`. It's okay to force update your pull request.
- To run `make test` and `make all` completes. - To run `make test` and `make all` completes.
@ -89,8 +89,8 @@ $ git push origin feature/123-something_awesome
``` ```
### Create a Pull Request ### Create a Pull Request
Pull requests can be created via GitHub. Refer to [this Pull requests can be created via Forgejo. Refer to [this
document](https://help.github.com/articles/creating-a-pull-request/) for document](https://docs.codeberg.org/collaborating/pull-requests-and-git-flow/) for
detailed steps on how to create a pull request. After a Pull Request gets peer detailed steps on how to create a pull request. After a Pull Request gets peer
reviewed and approved, it will be merged. reviewed and approved, it will be merged.

View file

@ -10,7 +10,7 @@ In alphabetical order:
- Alexey Vanin - Alexey Vanin
- Anastasia Prasolova - Anastasia Prasolova
- Anatoly Bogatyrev - Anatoly Bogatyrev
- Evgeny Kulikov - Evgeny Kulikov
- Evgeny Stratonikov - Evgeny Stratonikov
- Leonard Liubich - Leonard Liubich
- Sergei Liubich - Sergei Liubich

59
Makefile Normal file → Executable file
View file

@ -7,8 +7,9 @@ VERSION ?= $(shell git describe --tags --dirty --match "v*" --always --abbrev=8
HUB_IMAGE ?= truecloudlab/frostfs HUB_IMAGE ?= truecloudlab/frostfs
HUB_TAG ?= "$(shell echo ${VERSION} | sed 's/^v//')" HUB_TAG ?= "$(shell echo ${VERSION} | sed 's/^v//')"
GO_VERSION ?= 1.19 GO_VERSION ?= 1.21
LINT_VERSION ?= 1.50.0 LINT_VERSION ?= 1.54.0
TRUECLOUDLAB_LINT_VERSION ?= 0.0.2
ARCH = amd64 ARCH = amd64
BIN = bin BIN = bin
@ -25,8 +26,12 @@ PKG_VERSION ?= $(shell echo $(VERSION) | sed "s/^v//" | \
sed -E "s/(.*)-(g[a-fA-F0-9]{6,8})(.*)/\1\3~\2/" | \ sed -E "s/(.*)-(g[a-fA-F0-9]{6,8})(.*)/\1\3~\2/" | \
sed "s/-/~/")-${OS_RELEASE} sed "s/-/~/")-${OS_RELEASE}
OUTPUT_LINT_DIR ?= $(shell pwd)/bin
LINT_DIR = $(OUTPUT_LINT_DIR)/golangci-lint-$(LINT_VERSION)-v$(TRUECLOUDLAB_LINT_VERSION)
TMP_DIR := .cache
.PHONY: help all images dep clean fmts fmt imports test lint docker/lint .PHONY: help all images dep clean fmts fmt imports test lint docker/lint
prepare-release debpackage prepare-release debpackage pre-commit unpre-commit
# To build a specific binary, use it's name prefix with bin/ as a target # To build a specific binary, use it's name prefix with bin/ as a target
# For example `make bin/frostfs-node` will build only storage node binary # For example `make bin/frostfs-node` will build only storage node binary
@ -65,12 +70,18 @@ dep:
CGO_ENABLED=0 \ CGO_ENABLED=0 \
go mod tidy -v && echo OK go mod tidy -v && echo OK
# Build export-metrics
export-metrics: dep
@printf "⇒ Build export-metrics\n"
CGO_ENABLED=0 \
go build -v -trimpath -o bin/export-metrics ./scripts/export-metrics
# Regenerate proto files: # Regenerate proto files:
protoc: protoc:
@GOPRIVATE=github.com/TrueCloudLab go mod vendor @GOPRIVATE=github.com/TrueCloudLab go mod vendor
# Install specific version for protobuf lib # Install specific version for protobuf lib
@go list -f '{{.Path}}/...@{{.Version}}' -m github.com/golang/protobuf | xargs go install -v @go list -f '{{.Path}}/...@{{.Version}}' -m github.com/golang/protobuf | xargs go install -v
@GOBIN=$(abspath $(BIN)) go install -mod=mod -v github.com/TrueCloudLab/frostfs-api-go/v2/util/protogen @GOBIN=$(abspath $(BIN)) go install -mod=mod -v git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/util/protogen
# Protoc generate # Protoc generate
@for f in `find . -type f -name '*.proto' -not -path './vendor/*'`; do \ @for f in `find . -type f -name '*.proto' -not -path './vendor/*'`; do \
echo "⇒ Processing $$f "; \ echo "⇒ Processing $$f "; \
@ -95,7 +106,7 @@ image-%:
-t $(HUB_IMAGE)-$*:$(HUB_TAG) . -t $(HUB_IMAGE)-$*:$(HUB_TAG) .
# Build all Docker images # Build all Docker images
images: image-storage image-ir image-cli image-adm image-storage-testnet images: image-storage image-ir image-cli image-adm
# Build dirty local Docker images # Build dirty local Docker images
dirty-images: image-dirty-storage image-dirty-ir image-dirty-cli image-dirty-adm dirty-images: image-dirty-storage image-dirty-ir image-dirty-cli image-dirty-adm
@ -126,11 +137,36 @@ imports:
# Run Unit Test with go test # Run Unit Test with go test
test: test:
@echo "⇒ Running go test" @echo "⇒ Running go test"
@go test ./... @go test ./... -count=1
pre-commit-run:
@pre-commit run -a --hook-stage manual
# Install linters
lint-install:
@mkdir -p $(TMP_DIR)
@rm -rf $(TMP_DIR)/linters
@git -c advice.detachedHead=false clone --branch v$(TRUECLOUDLAB_LINT_VERSION) https://git.frostfs.info/TrueCloudLab/linters.git $(TMP_DIR)/linters
@@make -C $(TMP_DIR)/linters lib CGO_ENABLED=1 OUT_DIR=$(OUTPUT_LINT_DIR)
@rm -rf $(TMP_DIR)/linters
@rmdir $(TMP_DIR) 2>/dev/null || true
@CGO_ENABLED=1 GOBIN=$(LINT_DIR) go install github.com/golangci/golangci-lint/cmd/golangci-lint@v$(LINT_VERSION)
# Run linters # Run linters
lint: lint:
@golangci-lint --timeout=5m run @if [ ! -d "$(LINT_DIR)" ]; then \
echo "Run make lint-install"; \
exit 1; \
fi
$(LINT_DIR)/golangci-lint run
# Install staticcheck
staticcheck-install:
@go install honnef.co/go/tools/cmd/staticcheck@latest
# Run staticcheck
staticcheck-run:
@staticcheck ./...
# Run linters in Docker # Run linters in Docker
docker/lint: docker/lint:
@ -140,10 +176,19 @@ docker/lint:
--env HOME=/src \ --env HOME=/src \
golangci/golangci-lint:v$(LINT_VERSION) bash -c 'cd /src/ && make lint' golangci/golangci-lint:v$(LINT_VERSION) bash -c 'cd /src/ && make lint'
# Activate pre-commit hooks
pre-commit:
pre-commit install -t pre-commit -t commit-msg
# Deactivate pre-commit hooks
unpre-commit:
pre-commit uninstall -t pre-commit -t commit-msg
# Print version # Print version
version: version:
@echo $(VERSION) @echo $(VERSION)
# Delete built artifacts
clean: clean:
rm -rf vendor rm -rf vendor
rm -rf .cache rm -rf .cache

View file

@ -31,7 +31,7 @@ dApps directly from
code level. This way dApps are not limited to on-chain storage and can code level. This way dApps are not limited to on-chain storage and can
manipulate large amounts of data without paying a prohibitive price. manipulate large amounts of data without paying a prohibitive price.
FrostFS has a native [gRPC API](https://github.com/TrueCloudLab/frostfs-api) and has FrostFS has a native [gRPC API](https://git.frostfs.info/TrueCloudLab/frostfs-api) and has
protocol gateways for popular protocols such as [AWS protocol gateways for popular protocols such as [AWS
S3](https://github.com/TrueCloudLab/frostfs-s3-gw), S3](https://github.com/TrueCloudLab/frostfs-s3-gw),
[HTTP](https://github.com/TrueCloudLab/frostfs-http-gw), [HTTP](https://github.com/TrueCloudLab/frostfs-http-gw),
@ -49,7 +49,7 @@ The latest version of frostfs-node works with frostfs-contract
# Building # Building
To make all binaries you need Go 1.18+ and `make`: To make all binaries you need Go 1.20+ and `make`:
``` ```
make all make all
``` ```

View file

@ -1 +1 @@
v0.35.0 v0.36.0

View file

@ -3,23 +3,22 @@
## Overview ## Overview
Admin tool provides an easier way to deploy and maintain private installation Admin tool provides an easier way to deploy and maintain private installation
of FrostFS. Private installation provides a set of N3 consensus nodes, FrostFS of FrostFS. Private installation provides a set of N3 consensus nodes, FrostFS
Alphabet, and Storage nodes. Admin tool generates consensus keys, initializes Alphabet, and Storage nodes. Admin tool generates consensus keys, initializes
the sidechain, and provides functions to update the network and register new the sidechain, and provides functions to update the network and register new
Storage nodes. Storage nodes.
## Build ## Build
To build binary locally, use `make bin/frostfs-adm` command. To build binary locally, use `make bin/frostfs-adm` command.
For clean build inside a docker container, use `make docker/bin/frostfs-adm`. For clean build inside a docker container, use `make docker/bin/frostfs-adm`.
Build docker image with `make image-adm`. Build docker image with `make image-adm`.
At FrostFS private install deployment, frostfs-adm requires compiled FrostFS At FrostFS private install deployment, frostfs-adm requires compiled FrostFS
contracts. Find them in the latest release of contracts. Find them in the latest release of
[frostfs-contract repository](https://github.com/TrueCloudLab/frostfs-contract/releases). [frostfs-contract repository](https://git.frostfs.info/TrueCloudLab/frostfs-contract/releases).
## Commands ## Commands
@ -27,7 +26,7 @@ contracts. Find them in the latest release of
Config section provides `init` command that creates a configuration file for Config section provides `init` command that creates a configuration file for
private installation deployment and updates. Config file is optional, all private installation deployment and updates. Config file is optional, all
parameters can be passed by arguments or read from standard input (wallet parameters can be passed by arguments or read from standard input (wallet
passwords). passwords).
Config example: Config example:
@ -37,9 +36,7 @@ alphabet-wallets: /path # path to consensus node / alphabet wallets s
network: network:
max_object_size: 67108864 # max size of a single FrostFS object, bytes max_object_size: 67108864 # max size of a single FrostFS object, bytes
epoch_duration: 240 # duration of a FrostFS epoch in blocks, consider block generation frequency in the sidechain epoch_duration: 240 # duration of a FrostFS epoch in blocks, consider block generation frequency in the sidechain
basic_income_rate: 0 # basic income rate, for private consider 0
fee: fee:
audit: 0 # network audit fee, for private installation consider 0
candidate: 0 # inner ring candidate registration fee, for private installation consider 0 candidate: 0 # inner ring candidate registration fee, for private installation consider 0
container: 0 # container creation fee, for private installation consider 0 container: 0 # container creation fee, for private installation consider 0
container_alias: 0 # container nice-name registration fee, for private installation consider 0 container_alias: 0 # container nice-name registration fee, for private installation consider 0
@ -58,14 +55,14 @@ credentials: # passwords for consensus node / alphabet wallets
#### Network deployment #### Network deployment
- `generate-alphabet` generates a set of wallets for consensus and - `generate-alphabet` generates a set of wallets for consensus and
Alphabet nodes. Alphabet nodes.
- `init` initializes the sidechain by deploying smart contracts and - `init` initializes the sidechain by deploying smart contracts and
setting provided FrostFS network configuration. setting provided FrostFS network configuration.
- `generate-storage-wallet` generates a wallet for the Storage node that - `generate-storage-wallet` generates a wallet for the Storage node that
is ready for deployment. It also transfers a bit of sidechain GAS, so this is ready for deployment. It also transfers a bit of sidechain GAS, so this
wallet can be used for FrostFS bootstrap. wallet can be used for FrostFS bootstrap.
#### Network maintenance #### Network maintenance
@ -75,7 +72,7 @@ credentials: # passwords for consensus node / alphabet wallets
- `force-new-epoch` increments FrostFS epoch number and executes new epoch - `force-new-epoch` increments FrostFS epoch number and executes new epoch
handlers in FrostFS nodes. handlers in FrostFS nodes.
- `refill-gas` transfers sidechain GAS to the specified wallet. - `refill-gas` transfers sidechain GAS to the specified wallet.
- `update-contracts` updates contracts to a new version. - `update-contracts` updates contracts to a new version.
@ -87,7 +84,7 @@ info. These commands **do not migrate actual objects**.
- `dump-containers` saves all containers and metadata registered in the container - `dump-containers` saves all containers and metadata registered in the container
contract to a file. contract to a file.
- `restore-containers` restores previously saved containers by their repeated registration in - `restore-containers` restores previously saved containers by their repeated registration in
the container contract. the container contract.
- `list-containers` output all containers ids. - `list-containers` output all containers ids.

View file

@ -2,7 +2,7 @@
This is a short guide on how to deploy a private FrostFS storage network on bare This is a short guide on how to deploy a private FrostFS storage network on bare
metal without docker images. This guide does not cover details on how to start metal without docker images. This guide does not cover details on how to start
consensus, Alphabet, or Storage nodes. This guide covers only `frostfs-adm` consensus, Alphabet, or Storage nodes. This guide covers only `frostfs-adm`
related configuration details. related configuration details.
## Prerequisites ## Prerequisites
@ -12,12 +12,13 @@ To follow this guide you need:
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases) utility (v0.25.1 at the moment), - latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases) utility (v0.25.1 at the moment),
- latest released version of compiled [frostfs-contract](https://github.com/TrueCloudLab/frostfs-contract/releases) (v0.11.0 at the moment). - latest released version of compiled [frostfs-contract](https://github.com/TrueCloudLab/frostfs-contract/releases) (v0.11.0 at the moment).
## Step 1: Prepare network configuration ## Step 1: Prepare network configuration
To start a network, you need a set of consensus nodes, the same number of To start a network, you need a set of consensus nodes, the same number of
Alphabet nodes and any number of Storage nodes. While the number of Storage Alphabet nodes and any number of Storage nodes. While the number of Storage
nodes can be scaled almost infinitely, the number of consensus and Alphabet nodes can be scaled almost infinitely, the number of consensus and Alphabet
nodes can't be changed so easily right now. Consider this before going any further. nodes can't be changed so easily right now. Consider this before going any further.
Note also that there is an upper limit on the number of alphabet nodes (currently 22).
It is easier to use`frostfs-adm` with a predefined configuration. First, create It is easier to use`frostfs-adm` with a predefined configuration. First, create
a network configuration file. In this example, there is going to be only one a network configuration file. In this example, there is going to be only one
@ -27,15 +28,13 @@ consensus / Alphabet node in the network.
$ frostfs-adm config init --path foo.network.yml $ frostfs-adm config init --path foo.network.yml
Initial config file saved to foo.network.yml Initial config file saved to foo.network.yml
$ cat foo.network.yml $ cat foo.network.yml
rpc-endpoint: https://neo.rpc.node:30333 rpc-endpoint: https://neo.rpc.node:30333
alphabet-wallets: /home/user/deploy/alphabet-wallets alphabet-wallets: /home/user/deploy/alphabet-wallets
network: network:
max_object_size: 67108864 max_object_size: 67108864
epoch_duration: 240 epoch_duration: 240
basic_income_rate: 0
fee: fee:
audit: 0
candidate: 0 candidate: 0
container: 0 container: 0
withdraw: 0 withdraw: 0
@ -43,17 +42,17 @@ credentials:
az: hunter2 az: hunter2
``` ```
For private installation, it is recommended to set all **fees** and **basic For private installation, it is recommended to set all **fees** and **basic
income rate** to 0. income rate** to 0.
As for **epoch duration**, consider consensus node block generation frequency. As for **epoch duration**, consider consensus node block generation frequency.
With default 15 seconds per block, 240 blocks are going to be a 1-hour epoch. With default 15 seconds per block, 240 blocks are going to be a 1-hour epoch.
For **max object size**, 67108864 (64 MiB) or 134217728 (128 MiB) should provide For **max object size**, 67108864 (64 MiB) or 134217728 (128 MiB) should provide
good chunk distribution in most cases. good chunk distribution in most cases.
With this config, generate wallets (private keys) of consensus nodes. The same With this config, generate wallets (private keys) of consensus nodes. The same
wallets will be used for Alphabet nodes. Make sure, that dir for alphabet wallets will be used for Alphabet nodes. Make sure, that dir for alphabet
wallets already exists. wallets already exists.
``` ```
@ -69,14 +68,14 @@ storage.
## Step 2: Launch consensus nodes ## Step 2: Launch consensus nodes
Configure blockchain nodes with the generated wallets from the previous step. Configure blockchain nodes with the generated wallets from the previous step.
Config examples can be found in Config examples can be found in
[neo-go repository](https://github.com/nspcc-dev/neo-go/tree/master/config). [neo-go repository](https://github.com/nspcc-dev/neo-go/tree/master/config).
Gather public keys from **all** generated wallets. We are interested in the first Gather public keys from **all** generated wallets. We are interested in the first
`simple signature contract` public key. `simple signature contract` public key.
``` ```
$ neo-go wallet dump-keys -w alphabet-wallets/az.json $ neo-go wallet dump-keys -w alphabet-wallets/az.json
NitdS4k4f1Hh5mbLJhAswBK3WC2gQgPN1o (simple signature contract): NitdS4k4f1Hh5mbLJhAswBK3WC2gQgPN1o (simple signature contract):
02c1cc85f9c856dbe2d02017349bcb7b4e5defa78b8056a09b3240ba2a8c078869 02c1cc85f9c856dbe2d02017349bcb7b4e5defa78b8056a09b3240ba2a8c078869
@ -87,10 +86,10 @@ NiMKabp3ddi3xShmLAXhTfbnuWb4cSJT6E (1 out of 1 multisig contract):
02c1cc85f9c856dbe2d02017349bcb7b4e5defa78b8056a09b3240ba2a8c078869 02c1cc85f9c856dbe2d02017349bcb7b4e5defa78b8056a09b3240ba2a8c078869
``` ```
Put the list of public keys into `ProtocolConfiguration.StandbyCommittee` Put the list of public keys into `ProtocolConfiguration.StandbyCommittee`
section. Specify the wallet path and the password in `ApplicationConfiguration.P2PNotary` section. Specify the wallet path and the password in `ApplicationConfiguration.P2PNotary`
and `ApplicationConfiguration.UnlockWallet` sections. If config includes and `ApplicationConfiguration.UnlockWallet` sections. If config includes
`ProtocolConfiguration.NativeActivations` section, add notary `ProtocolConfiguration.NativeActivations` section, add notary
contract `Notary: [0]`. contract `Notary: [0]`.
```yaml ```yaml
@ -121,7 +120,7 @@ and possible overload issues.
Use archive with compiled FrostFS contracts to initialize the sidechain. Use archive with compiled FrostFS contracts to initialize the sidechain.
``` ```
$ tar -xzvf frostfs-contract-v0.11.0.tar.gz $ tar -xzvf frostfs-contract-v0.11.0.tar.gz
$ ./frostfs-adm -c foo.network.yml morph init --contracts ./frostfs-contract-v0.11.0 $ ./frostfs-adm -c foo.network.yml morph init --contracts ./frostfs-contract-v0.11.0
Stage 1: transfer GAS to alphabet nodes. Stage 1: transfer GAS to alphabet nodes.
@ -141,20 +140,18 @@ Waiting for transactions to persist...
Stage 7: set addresses in NNS. Stage 7: set addresses in NNS.
Waiting for transactions to persist... Waiting for transactions to persist...
NNS: Set alphabet0.frostfs -> f692dfb4d43a15b464eb51a7041160fb29c44b6a NNS: Set alphabet0.frostfs -> f692dfb4d43a15b464eb51a7041160fb29c44b6a
NNS: Set audit.frostfs -> 7df847b993affb3852074345a7c2bd622171ee0d
NNS: Set balance.frostfs -> 103519b3067a66307080a66570c0491ee8f68879 NNS: Set balance.frostfs -> 103519b3067a66307080a66570c0491ee8f68879
NNS: Set container.frostfs -> cae60bdd689d185901e495352d0247752ce50846 NNS: Set container.frostfs -> cae60bdd689d185901e495352d0247752ce50846
NNS: Set frostfsid.frostfs -> c421fb60a3895865a8f24d197d6a80ef686041d2 NNS: Set frostfsid.frostfs -> c421fb60a3895865a8f24d197d6a80ef686041d2
NNS: Set netmap.frostfs -> 894eb854632f50fb124412ce7951ebc00763525e NNS: Set netmap.frostfs -> 894eb854632f50fb124412ce7951ebc00763525e
NNS: Set proxy.frostfs -> ac6e6fe4b373d0ca0ca4969d1e58fa0988724e7d NNS: Set proxy.frostfs -> ac6e6fe4b373d0ca0ca4969d1e58fa0988724e7d
NNS: Set reputation.frostfs -> 6eda57c9d93d990573646762d1fea327ce41191f
Waiting for transactions to persist... Waiting for transactions to persist...
``` ```
## Step 4: Launch Alphabet nodes ## Step 4: Launch Alphabet nodes
Configure Alphabet nodes with the wallets generated in step 1. For Configure Alphabet nodes with the wallets generated in step 1. For
`morph.validators` use a list of public keys from `morph.validators` use a list of public keys from
`ProtocolConfiguration.StandbyCommittee`. `ProtocolConfiguration.StandbyCommittee`.
```yaml ```yaml
@ -178,10 +175,10 @@ Generate a new wallet for a Storage node.
``` ```
$ frostfs-adm -c foo.network.yml morph generate-storage-wallet --storage-wallet ./sn01.json --initial-gas 10.0 $ frostfs-adm -c foo.network.yml morph generate-storage-wallet --storage-wallet ./sn01.json --initial-gas 10.0
New password > New password >
Waiting for transactions to persist... Waiting for transactions to persist...
$ neo-go wallet dump-keys -w sn01.json $ neo-go wallet dump-keys -w sn01.json
Ngr7p8Z9S22XDH6VkUG9oXobv8zZRAWwwv (simple signature contract): Ngr7p8Z9S22XDH6VkUG9oXobv8zZRAWwwv (simple signature contract):
0355eccb72cd46f09a3e5237eaa0f4949cceb5ecfa5a225bd3bb9fd021c4d75b85 0355eccb72cd46f09a3e5237eaa0f4949cceb5ecfa5a225bd3bb9fd021c4d75b85
``` ```
@ -205,7 +202,7 @@ Current epoch: 8, increase to 9.
Waiting for transactions to persist... Waiting for transactions to persist...
``` ```
--- ---
After that, FrostFS Storage is ready to work. You can access it directly or After that, FrostFS Storage is ready to work. You can access it directly or
with protocol gates. with protocol gates.

View file

@ -1,39 +0,0 @@
# FrostFS subnetwork creation
This is a short guide on how to create FrostFS subnetworks. This guide
considers that the sidechain and the inner ring (alphabet nodes) have already been
deployed and the sidechain contains a deployed `subnet` contract.
## Prerequisites
To follow this guide, you need:
- neo-go sidechain RPC endpoint;
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases);
- wallet with FrostFS account.
## Creation
```shell
$ frostfs-adm morph subnet create \
-r <side_chain_RPC_endpoint> \
-w </path/to/owner/wallet> \
--notary
Create subnet request sent successfully. ID: 4223489767.
```
**NOTE:** in notary-enabled environment you should have a sufficient
notary deposit (not expired, with enough GAS balance). Your subnet ID
will differ from the example.
The default account in the wallet that has been passed with `-w` flag is the owner
of the just created subnetwork.
You can check if your subnetwork was created successfully:
```shell
$ frostfs-adm morph subnet get \
-r <side_chain_RPC_endpoint> \
--subnet <subnet_ID>
Owner: NUc734PMJXiqa2J9jRtvskU3kCdyyuSN8Q
```
Your owner will differ from the example.

View file

@ -1,137 +0,0 @@
# Managing Subnetworks
This is a short guide on how to manage FrostFS subnetworks. This guide
considers that the sidechain and the inner ring (alphabet nodes) have already been
deployed, and the sidechain contains a deployed `subnet` contract.
## Prerequisites
- neo-go sidechain RPC endpoint;
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases);
- [created](subnetwork-creation.md) subnetwork;
- wallet with the account that owns the subnetwork;
- public key of the Storage Node;
- public keys of the node and client administrators;
- owner IDs of the FrostFS users.
## Add node administrator
Node administrators are accounts that can manage (add and delete nodes)
the whitelist of the nodes which can be included to a subnetwork. Only the subnet
owner is allowed to add and remove node administrators from the subnetwork.
```shell
$ frostfs-adm morph subnet admin add \
-r <side_chain_RPC_endpoint> \
-w </path/to/owner/wallet> \
--admin <HEX_admin_public_key> \
--subnet <subnet_ID>
Add admin request sent successfully.
```
## Add node
Adding a node to a subnetwork means that the node becomes able to service
containers that have been created in that subnetwork. Addition only changes
the list of the allowed nodes. Node is not required to be bootstrapped at the
moment of its inclusion.
```shell
$ frostfs-adm morph subnet node add \
-r <side_chain_RPC_endpoint> \
-w </path/to/node_admin/wallet> \
--node <HEX_node_public_key> \
--subnet <subnet_ID>
Add node request sent successfully.
```
**NOTE:** the owner of the subnetwork is also allowed to add nodes.
## Add client administrator
Client administrators are accounts that can manage (add and delete
nodes) the whitelist of the clients that can create containers in the
subnetwork. Only the subnet owner is allowed to add and remove client
administrators from the subnetwork.
```shell
$ frostfs-adm morph subnet admin add \
-r <side_chain_RPC_endpoint> \
-w </path/to/owner/wallet> \
--admin <HEX_admin_public_key> \
--subnet <subnet_ID> \
--client \
--group <group_ID>
Add admin request sent successfully.
```
**NOTE:** you do not need to create a group explicitly, it will be created
right after the first client admin is added. Group ID is a 4-byte
positive integer number.
## Add client
```shell
$ frostfs-adm morph subnet client add \
-r <side_chain_RPC_endpoint> \
-w </path/to/client_admin/wallet> \
--client <client_ownerID> \
--subnet <subnet_ID> \
--group <group_ID>
Add client request sent successfully.
```
**NOTE:** the owner of the subnetwork is also allowed to add clients. This is
the only one command that accepts `ownerID`, not the public key.
Administrator can manage only their group (a group where that administrator
has been added by the subnet owner).
# Bootstrapping Storage Node
After a subnetwork [is created](subnetwork-creation.md) and a node is included into it, the
node could be bootstrapped and service subnetwork containers.
For bootstrapping, you need to specify the ID of the subnetwork in the node's
configuration:
```yaml
...
node:
...
subnet:
entries: # list of IDs of subnets to enter in a text format of FrostFS API protocol (overrides corresponding attributes)
- <subnetwork_ID>
...
...
```
**NOTE:** specifying subnetwork that is denied for the node is not an error:
that configuration value would be ignored. You do not need to specify zero
(with 0 ID) subnetwork: its inclusion is implicit. On the contrary, to exclude
a node from the default zero subnetwork, you need to specify it explicitly:
```yaml
...
node:
...
subnet:
exit_zero: true # toggle entrance to zero subnet (overrides corresponding attribute and occurrence in `entries`)
...
...
```
# Creating container in non-zero subnetwork
Creating containers without using `--subnet` flag is equivalent to
creating container in the zero subnetwork.
To create a container in a private network, your wallet must be added to
the client whitelist by the client admins or the subnet owners:
```shell
$ frostfs-cli container create \
--policy 'REP 1' \
-w </path/to/wallet> \
-r s01.frostfs.devenv:8080 \
--subnet <subnet_ID>
```

View file

@ -7,7 +7,7 @@ import (
"path/filepath" "path/filepath"
"text/template" "text/template"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/cli/input" "github.com/nspcc-dev/neo-go/cli/input"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
@ -18,8 +18,6 @@ type configTemplate struct {
AlphabetDir string AlphabetDir string
MaxObjectSize int MaxObjectSize int
EpochDuration int EpochDuration int
BasicIncomeRate int
AuditFee int
CandidateFee int CandidateFee int
ContainerFee int ContainerFee int
ContainerAliasFee int ContainerAliasFee int
@ -33,10 +31,8 @@ alphabet-wallets: {{ .AlphabetDir}}
network: network:
max_object_size: {{ .MaxObjectSize}} max_object_size: {{ .MaxObjectSize}}
epoch_duration: {{ .EpochDuration}} epoch_duration: {{ .EpochDuration}}
basic_income_rate: {{ .BasicIncomeRate}}
homomorphic_hash_disabled: {{ .HomomorphicHashDisabled}} homomorphic_hash_disabled: {{ .HomomorphicHashDisabled}}
fee: fee:
audit: {{ .AuditFee}}
candidate: {{ .CandidateFee}} candidate: {{ .CandidateFee}}
container: {{ .ContainerFee}} container: {{ .ContainerFee}}
container_alias: {{ .ContainerAliasFee }} container_alias: {{ .ContainerAliasFee }}
@ -47,7 +43,7 @@ credentials:
{{.}}: password{{end}} {{.}}: password{{end}}
` `
func initConfig(cmd *cobra.Command, args []string) error { func initConfig(cmd *cobra.Command, _ []string) error {
configPath, err := readConfigPathFromArgs(cmd) configPath, err := readConfigPathFromArgs(cmd)
if err != nil { if err != nil {
return nil return nil
@ -111,9 +107,7 @@ func generateConfigExample(appDir string, credSize int) (string, error) {
Endpoint: "https://neo.rpc.node:30333", Endpoint: "https://neo.rpc.node:30333",
MaxObjectSize: 67108864, // 64 MiB MaxObjectSize: 67108864, // 64 MiB
EpochDuration: 240, // 1 hour with 15s per block EpochDuration: 240, // 1 hour with 15s per block
BasicIncomeRate: 1_0000_0000, // 0.0001 GAS per GiB (Fixed12)
HomomorphicHashDisabled: false, // object homomorphic hash is enabled HomomorphicHashDisabled: false, // object homomorphic hash is enabled
AuditFee: 1_0000, // 0.00000001 GAS per audit (Fixed12)
CandidateFee: 100_0000_0000, // 100.0 GAS (Fixed8) CandidateFee: 100_0000_0000, // 100.0 GAS (Fixed8)
ContainerFee: 1000, // 0.000000001 * 7 GAS per container (Fixed12) ContainerFee: 1000, // 0.000000001 * 7 GAS per container (Fixed12)
ContainerAliasFee: 500, // ContainerFee / 2 ContainerAliasFee: 500, // ContainerFee / 2

View file

@ -5,7 +5,7 @@ import (
"path/filepath" "path/filepath"
"testing" "testing"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/spf13/viper" "github.com/spf13/viper"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@ -28,8 +28,6 @@ func TestGenerateConfigExample(t *testing.T) {
require.Equal(t, filepath.Join(appDir, "alphabet-wallets"), v.GetString("alphabet-wallets")) require.Equal(t, filepath.Join(appDir, "alphabet-wallets"), v.GetString("alphabet-wallets"))
require.Equal(t, 67108864, v.GetInt("network.max_object_size")) require.Equal(t, 67108864, v.GetInt("network.max_object_size"))
require.Equal(t, 240, v.GetInt("network.epoch_duration")) require.Equal(t, 240, v.GetInt("network.epoch_duration"))
require.Equal(t, 100000000, v.GetInt("network.basic_income_rate"))
require.Equal(t, 10000, v.GetInt("network.fee.audit"))
require.Equal(t, 10000000000, v.GetInt("network.fee.candidate")) require.Equal(t, 10000000000, v.GetInt("network.fee.candidate"))
require.Equal(t, 1000, v.GetInt("network.fee.container")) require.Equal(t, 1000, v.GetInt("network.fee.container"))
require.Equal(t, 100000000, v.GetInt("network.fee.withdraw")) require.Equal(t, 100000000, v.GetInt("network.fee.withdraw"))

View file

@ -6,8 +6,8 @@ import (
"fmt" "fmt"
"math/big" "math/big"
"github.com/TrueCloudLab/frostfs-contract/nns" "git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
"github.com/TrueCloudLab/frostfs-sdk-go/netmap" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/nspcc-dev/neo-go/pkg/core/native/noderoles" "github.com/nspcc-dev/neo-go/pkg/core/native/noderoles"
"github.com/nspcc-dev/neo-go/pkg/core/state" "github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
@ -37,11 +37,6 @@ const (
dumpBalancesAlphabetFlag = "alphabet" dumpBalancesAlphabetFlag = "alphabet"
dumpBalancesProxyFlag = "proxy" dumpBalancesProxyFlag = "proxy"
dumpBalancesUseScriptHashFlag = "script-hash" dumpBalancesUseScriptHashFlag = "script-hash"
// notaryEnabled signifies whether contracts were deployed in a notary-enabled environment.
// The setting is here to simplify testing and building the command for testnet (notary currently disabled).
// It will be removed eventually.
notaryEnabled = true
) )
func dumpBalances(cmd *cobra.Command, _ []string) error { func dumpBalances(cmd *cobra.Command, _ []string) error {
@ -60,7 +55,7 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
inv := invoker.New(c, nil) inv := invoker.New(c, nil)
if !notaryEnabled || dumpStorage || dumpAlphabet || dumpProxy { if dumpStorage || dumpAlphabet || dumpProxy {
nnsCs, err = c.GetContractStateByID(1) nnsCs, err = c.GetContractStateByID(1)
if err != nil { if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err) return fmt.Errorf("can't get NNS contract info: %w", err)
@ -72,7 +67,7 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
} }
} }
irList, err := fetchIRNodes(c, nmHash, rolemgmt.Hash) irList, err := fetchIRNodes(c, rolemgmt.Hash)
if err != nil { if err != nil {
return err return err
} }
@ -83,120 +78,126 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
printBalances(cmd, "Inner ring nodes balances:", irList) printBalances(cmd, "Inner ring nodes balances:", irList)
if dumpStorage { if dumpStorage {
arr, err := unwrap.Array(inv.Call(nmHash, "netmap")) if err := printStorageNodeBalances(cmd, inv, nmHash); err != nil {
if err != nil {
return errors.New("can't fetch the list of storage nodes")
}
snList := make([]accBalancePair, len(arr))
for i := range arr {
node, ok := arr[i].Value().([]stackitem.Item)
if !ok || len(node) == 0 {
return errors.New("can't parse the list of storage nodes")
}
bs, err := node[0].TryBytes()
if err != nil {
return errors.New("can't parse the list of storage nodes")
}
var ni netmap.NodeInfo
if err := ni.Unmarshal(bs); err != nil {
return fmt.Errorf("can't parse the list of storage nodes: %w", err)
}
pub, err := keys.NewPublicKeyFromBytes(ni.PublicKey(), elliptic.P256())
if err != nil {
return fmt.Errorf("can't parse storage node public key: %w", err)
}
snList[i].scriptHash = pub.GetScriptHash()
}
if err := fetchBalances(inv, gas.Hash, snList); err != nil {
return err return err
} }
printBalances(cmd, "\nStorage node balances:", snList)
} }
if dumpProxy { if dumpProxy {
h, err := nnsResolveHash(inv, nnsCs.Hash, proxyContract+".frostfs") if err := printProxyContractBalance(cmd, inv, nnsCs.Hash); err != nil {
if err != nil {
return fmt.Errorf("can't get hash of the proxy contract: %w", err)
}
proxyList := []accBalancePair{{scriptHash: h}}
if err := fetchBalances(inv, gas.Hash, proxyList); err != nil {
return err return err
} }
printBalances(cmd, "\nProxy contract balance:", proxyList)
} }
if dumpAlphabet { if dumpAlphabet {
alphaList := make([]accBalancePair, len(irList)) if err := printAlphabetContractBalances(cmd, c, inv, len(irList), nnsCs.Hash); err != nil {
w := io.NewBufBinWriter()
for i := range alphaList {
emit.AppCall(w.BinWriter, nnsCs.Hash, "resolve", callflag.ReadOnly,
getAlphabetNNSDomain(i),
int64(nns.TXT))
}
if w.Err != nil {
panic(w.Err)
}
alphaRes, err := c.InvokeScript(w.Bytes(), nil)
if err != nil {
return fmt.Errorf("can't fetch info from NNS: %w", err)
}
for i := range alphaList {
h, err := parseNNSResolveResult(alphaRes.Stack[i])
if err != nil {
return fmt.Errorf("can't fetch the alphabet contract #%d hash: %w", i, err)
}
alphaList[i].scriptHash = h
}
if err := fetchBalances(inv, gas.Hash, alphaList); err != nil {
return err return err
} }
printBalances(cmd, "\nAlphabet contracts balances:", alphaList)
} }
return nil return nil
} }
func fetchIRNodes(c Client, nmHash, desigHash util.Uint160) ([]accBalancePair, error) { func printStorageNodeBalances(cmd *cobra.Command, inv *invoker.Invoker, nmHash util.Uint160) error {
var irList []accBalancePair arr, err := unwrap.Array(inv.Call(nmHash, "netmap"))
if err != nil {
return errors.New("can't fetch the list of storage nodes")
}
snList := make([]accBalancePair, len(arr))
for i := range arr {
node, ok := arr[i].Value().([]stackitem.Item)
if !ok || len(node) == 0 {
return errors.New("can't parse the list of storage nodes")
}
bs, err := node[0].TryBytes()
if err != nil {
return errors.New("can't parse the list of storage nodes")
}
var ni netmap.NodeInfo
if err := ni.Unmarshal(bs); err != nil {
return fmt.Errorf("can't parse the list of storage nodes: %w", err)
}
pub, err := keys.NewPublicKeyFromBytes(ni.PublicKey(), elliptic.P256())
if err != nil {
return fmt.Errorf("can't parse storage node public key: %w", err)
}
snList[i].scriptHash = pub.GetScriptHash()
}
if err := fetchBalances(inv, gas.Hash, snList); err != nil {
return err
}
printBalances(cmd, "\nStorage node balances:", snList)
return nil
}
func printProxyContractBalance(cmd *cobra.Command, inv *invoker.Invoker, nnsHash util.Uint160) error {
h, err := nnsResolveHash(inv, nnsHash, proxyContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get hash of the proxy contract: %w", err)
}
proxyList := []accBalancePair{{scriptHash: h}}
if err := fetchBalances(inv, gas.Hash, proxyList); err != nil {
return err
}
printBalances(cmd, "\nProxy contract balance:", proxyList)
return nil
}
func printAlphabetContractBalances(cmd *cobra.Command, c Client, inv *invoker.Invoker, count int, nnsHash util.Uint160) error {
alphaList := make([]accBalancePair, count)
w := io.NewBufBinWriter()
for i := range alphaList {
emit.AppCall(w.BinWriter, nnsHash, "resolve", callflag.ReadOnly,
getAlphabetNNSDomain(i),
int64(nns.TXT))
}
if w.Err != nil {
panic(w.Err)
}
alphaRes, err := c.InvokeScript(w.Bytes(), nil)
if err != nil {
return fmt.Errorf("can't fetch info from NNS: %w", err)
}
for i := range alphaList {
h, err := parseNNSResolveResult(alphaRes.Stack[i])
if err != nil {
return fmt.Errorf("can't fetch the alphabet contract #%d hash: %w", i, err)
}
alphaList[i].scriptHash = h
}
if err := fetchBalances(inv, gas.Hash, alphaList); err != nil {
return err
}
printBalances(cmd, "\nAlphabet contracts balances:", alphaList)
return nil
}
func fetchIRNodes(c Client, desigHash util.Uint160) ([]accBalancePair, error) {
inv := invoker.New(c, nil) inv := invoker.New(c, nil)
if notaryEnabled { height, err := c.GetBlockCount()
height, err := c.GetBlockCount() if err != nil {
if err != nil { return nil, fmt.Errorf("can't get block height: %w", err)
return nil, fmt.Errorf("can't get block height: %w", err) }
}
arr, err := getDesignatedByRole(inv, desigHash, noderoles.NeoFSAlphabet, height) arr, err := getDesignatedByRole(inv, desigHash, noderoles.NeoFSAlphabet, height)
if err != nil { if err != nil {
return nil, errors.New("can't fetch list of IR nodes from the netmap contract") return nil, errors.New("can't fetch list of IR nodes from the netmap contract")
} }
irList = make([]accBalancePair, len(arr)) irList := make([]accBalancePair, len(arr))
for i := range arr { for i := range arr {
irList[i].scriptHash = arr[i].GetScriptHash() irList[i].scriptHash = arr[i].GetScriptHash()
}
} else {
arr, err := unwrap.ArrayOfBytes(inv.Call(nmHash, "innerRingList"))
if err != nil {
return nil, errors.New("can't fetch list of IR nodes from the netmap contract")
}
irList = make([]accBalancePair, len(arr))
for i := range arr {
pub, err := keys.NewPublicKeyFromBytes(arr[i], elliptic.P256())
if err != nil {
return nil, fmt.Errorf("can't parse IR node public key: %w", err)
}
irList[i].scriptHash = pub.GetScriptHash()
}
} }
return irList, nil return irList, nil
} }

View file

@ -10,12 +10,12 @@ import (
"strings" "strings"
"text/tabwriter" "text/tabwriter"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/io" "github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker" "github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap" "github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag" "github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit" "github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
) )
@ -48,41 +48,24 @@ func dumpNetworkConfig(cmd *cobra.Command, _ []string) error {
buf := bytes.NewBuffer(nil) buf := bytes.NewBuffer(nil)
tw := tabwriter.NewWriter(buf, 0, 2, 2, ' ', 0) tw := tabwriter.NewWriter(buf, 0, 2, 2, ' ', 0)
for _, param := range arr { m, err := parseConfigFromNetmapContract(arr)
tuple, ok := param.Value().([]stackitem.Item) if err != nil {
if !ok || len(tuple) != 2 { return err
return errors.New("invalid ListConfig response from netmap contract") }
} for k, v := range m {
switch k {
k, err := tuple[0].TryBytes() case netmap.ContainerFeeConfig, netmap.ContainerAliasFeeConfig,
if err != nil { netmap.EpochDurationConfig, netmap.IrCandidateFeeConfig,
return errors.New("invalid config key from netmap contract") netmap.MaxObjectSizeConfig, netmap.WithdrawFeeConfig:
}
v, err := tuple[1].TryBytes()
if err != nil {
return invalidConfigValueErr(k)
}
switch string(k) {
case netmapAuditFeeKey, netmapBasicIncomeRateKey,
netmapContainerFeeKey, netmapContainerAliasFeeKey,
netmapEigenTrustIterationsKey,
netmapEpochKey, netmapInnerRingCandidateFeeKey,
netmapMaxObjectSizeKey, netmapWithdrawFeeKey:
nbuf := make([]byte, 8) nbuf := make([]byte, 8)
copy(nbuf[:], v) copy(nbuf[:], v)
n := binary.LittleEndian.Uint64(nbuf) n := binary.LittleEndian.Uint64(nbuf)
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%d (int)\n", k, n))) _, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%d (int)\n", k, n)))
case netmapEigenTrustAlphaKey: case netmap.HomomorphicHashingDisabledKey, netmap.MaintenanceModeAllowedConfig:
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%s (str)\n", k, v))) if len(v) == 0 || len(v) > 1 {
case netmapHomomorphicHashDisabledKey, netmapMaintenanceAllowedKey:
vBool, err := tuple[1].TryBool()
if err != nil {
return invalidConfigValueErr(k) return invalidConfigValueErr(k)
} }
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%t (bool)\n", k, v[0] == 1)))
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%t (bool)\n", k, vBool)))
default: default:
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%s (hex)\n", k, hex.EncodeToString(v)))) _, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%s (hex)\n", k, hex.EncodeToString(v))))
} }
@ -150,25 +133,14 @@ func parseConfigPair(kvStr string, force bool) (key string, val any, err error)
valRaw := v valRaw := v
switch key { switch key {
case netmapAuditFeeKey, netmapBasicIncomeRateKey, case netmap.ContainerFeeConfig, netmap.ContainerAliasFeeConfig,
netmapContainerFeeKey, netmapContainerAliasFeeKey, netmap.EpochDurationConfig, netmap.IrCandidateFeeConfig,
netmapEigenTrustIterationsKey, netmap.MaxObjectSizeConfig, netmap.WithdrawFeeConfig:
netmapEpochKey, netmapInnerRingCandidateFeeKey,
netmapMaxObjectSizeKey, netmapWithdrawFeeKey:
val, err = strconv.ParseInt(valRaw, 10, 64) val, err = strconv.ParseInt(valRaw, 10, 64)
if err != nil { if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as int: %w", key, valRaw, err) err = fmt.Errorf("could not parse %s's value '%s' as int: %w", key, valRaw, err)
} }
case netmapEigenTrustAlphaKey: case netmap.HomomorphicHashingDisabledKey, netmap.MaintenanceModeAllowedConfig:
// just check that it could
// be parsed correctly
_, err = strconv.ParseFloat(v, 64)
if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as float: %w", key, valRaw, err)
}
val = valRaw
case netmapHomomorphicHashDisabledKey, netmapMaintenanceAllowedKey:
val, err = strconv.ParseBool(valRaw) val, err = strconv.ParseBool(valRaw)
if err != nil { if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as bool: %w", key, valRaw, err) err = fmt.Errorf("could not parse %s's value '%s' as bool: %w", key, valRaw, err)
@ -187,6 +159,6 @@ func parseConfigPair(kvStr string, force bool) (key string, val any, err error)
return return
} }
func invalidConfigValueErr(key []byte) error { func invalidConfigValueErr(key string) error {
return fmt.Errorf("invalid %s config value from netmap contract", key) return fmt.Errorf("invalid %s config value from netmap contract", key)
} }

View file

@ -7,7 +7,7 @@ import (
"os" "os"
"sort" "sort"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/nspcc-dev/neo-go/pkg/crypto/hash" "github.com/nspcc-dev/neo-go/pkg/crypto/hash"
"github.com/nspcc-dev/neo-go/pkg/io" "github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker" "github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
@ -41,16 +41,28 @@ func getContainerContractHash(cmd *cobra.Command, inv *invoker.Invoker, c Client
return ch, nil return ch, nil
} }
func getContainersList(inv *invoker.Invoker, ch util.Uint160) ([][]byte, error) { func iterateContainerList(inv *invoker.Invoker, ch util.Uint160, f func([]byte) error) error {
res, err := inv.Call(ch, "list", "") sid, r, err := unwrap.SessionIterator(inv.Call(ch, "containersOf", ""))
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err) return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
} }
itm, err := unwrap.Item(res, err) // Nothing bad, except live session on the server, do not report to the user.
if _, ok := itm.(stackitem.Null); !ok { defer func() { _ = inv.TerminateSession(sid) }()
return unwrap.ArrayOfBytes(res, err)
items, err := inv.TraverseIterator(sid, &r, 0)
for err == nil && len(items) != 0 {
for j := range items {
b, err := items[j].TryBytes()
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if err := f(b); err != nil {
return err
}
}
items, err = inv.TraverseIterator(sid, &r, 0)
} }
return nil, nil return err
} }
func dumpContainers(cmd *cobra.Command, _ []string) error { func dumpContainers(cmd *cobra.Command, _ []string) error {
@ -71,56 +83,81 @@ func dumpContainers(cmd *cobra.Command, _ []string) error {
return fmt.Errorf("unable to get contaract hash: %w", err) return fmt.Errorf("unable to get contaract hash: %w", err)
} }
cids, err := getContainersList(inv, ch)
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
isOK, err := getCIDFilterFunc(cmd) isOK, err := getCIDFilterFunc(cmd)
if err != nil { if err != nil {
return err return err
} }
var containers []*Container f, err := os.OpenFile(filename, os.O_RDWR|os.O_TRUNC|os.O_CREATE, 0o660)
bw := io.NewBufBinWriter()
for _, id := range cids {
if !isOK(id) {
continue
}
bw.Reset()
emit.AppCall(bw.BinWriter, ch, "get", callflag.All, id)
emit.AppCall(bw.BinWriter, ch, "eACL", callflag.All, id)
res, err := inv.Run(bw.Bytes())
if err != nil {
return fmt.Errorf("can't get container info: %w", err)
}
if len(res.Stack) != 2 {
return fmt.Errorf("%w: expected 2 items on stack", errInvalidContainerResponse)
}
cnt := new(Container)
err = cnt.FromStackItem(res.Stack[0])
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
ea := new(EACL)
err = ea.FromStackItem(res.Stack[1])
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if len(ea.Value) != 0 {
cnt.EACL = ea
}
containers = append(containers, cnt)
}
out, err := json.Marshal(containers)
if err != nil { if err != nil {
return err return err
} }
return os.WriteFile(filename, out, 0o660) defer f.Close()
_, err = f.Write([]byte{'['})
if err != nil {
return err
}
written := 0
enc := json.NewEncoder(f)
bw := io.NewBufBinWriter()
iterErr := iterateContainerList(inv, ch, func(id []byte) error {
if !isOK(id) {
return nil
}
cnt, err := dumpSingleContainer(bw, ch, inv, id)
if err != nil {
return err
}
// Writing directly to the file is ok, because json.Encoder does no internal buffering.
if written != 0 {
_, err = f.Write([]byte{','})
if err != nil {
return err
}
}
written++
return enc.Encode(cnt)
})
if iterErr != nil {
return iterErr
}
_, err = f.Write([]byte{']'})
return err
}
func dumpSingleContainer(bw *io.BufBinWriter, ch util.Uint160, inv *invoker.Invoker, id []byte) (*Container, error) {
bw.Reset()
emit.AppCall(bw.BinWriter, ch, "get", callflag.All, id)
emit.AppCall(bw.BinWriter, ch, "eACL", callflag.All, id)
res, err := inv.Run(bw.Bytes())
if err != nil {
return nil, fmt.Errorf("can't get container info: %w", err)
}
if len(res.Stack) != 2 {
return nil, fmt.Errorf("%w: expected 2 items on stack", errInvalidContainerResponse)
}
cnt := new(Container)
err = cnt.FromStackItem(res.Stack[0])
if err != nil {
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
ea := new(EACL)
err = ea.FromStackItem(res.Stack[1])
if err != nil {
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if len(ea.Value) != 0 {
cnt.EACL = ea
}
return cnt, nil
} }
func listContainers(cmd *cobra.Command, _ []string) error { func listContainers(cmd *cobra.Command, _ []string) error {
@ -136,20 +173,15 @@ func listContainers(cmd *cobra.Command, _ []string) error {
return fmt.Errorf("unable to get contaract hash: %w", err) return fmt.Errorf("unable to get contaract hash: %w", err)
} }
cids, err := getContainersList(inv, ch) return iterateContainerList(inv, ch, func(id []byte) error {
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
for _, id := range cids {
var idCnr cid.ID var idCnr cid.ID
err = idCnr.Decode(id) err = idCnr.Decode(id)
if err != nil { if err != nil {
return fmt.Errorf("unable to decode container id: %w", err) return fmt.Errorf("unable to decode container id: %w", err)
} }
cmd.Println(idCnr) cmd.Println(idCnr)
} return nil
return nil })
} }
func restoreContainers(cmd *cobra.Command, _ []string) error { func restoreContainers(cmd *cobra.Command, _ []string) error {
@ -164,25 +196,14 @@ func restoreContainers(cmd *cobra.Command, _ []string) error {
} }
defer wCtx.close() defer wCtx.close()
nnsCs, err := wCtx.Client.GetContractStateByID(1) containers, err := parseContainers(filename)
if err != nil { if err != nil {
return fmt.Errorf("can't get NNS contract state: %w", err) return err
} }
ch, err := nnsResolveHash(wCtx.ReadOnlyInvoker, nnsCs.Hash, containerContract+".frostfs") ch, err := fetchContainerContractHash(wCtx)
if err != nil { if err != nil {
return fmt.Errorf("can't fetch container contract hash: %w", err) return err
}
data, err := os.ReadFile(filename)
if err != nil {
return fmt.Errorf("can't read dump file: %w", err)
}
var containers []Container
err = json.Unmarshal(data, &containers)
if err != nil {
return fmt.Errorf("can't parse dump file: %w", err)
} }
isOK, err := getCIDFilterFunc(cmd) isOK, err := getCIDFilterFunc(cmd)
@ -190,6 +211,15 @@ func restoreContainers(cmd *cobra.Command, _ []string) error {
return err return err
} }
err = restoreOrPutContainers(containers, isOK, cmd, wCtx, ch)
if err != nil {
return err
}
return wCtx.awaitTx()
}
func restoreOrPutContainers(containers []Container, isOK func([]byte) bool, cmd *cobra.Command, wCtx *initializeContext, ch util.Uint160) error {
bw := io.NewBufBinWriter() bw := io.NewBufBinWriter()
for _, cnt := range containers { for _, cnt := range containers {
hv := hash.Sha256(cnt.Value) hv := hash.Sha256(cnt.Value)
@ -197,33 +227,18 @@ func restoreContainers(cmd *cobra.Command, _ []string) error {
continue continue
} }
bw.Reset() bw.Reset()
emit.AppCall(bw.BinWriter, ch, "get", callflag.All, hv.BytesBE()) restored, err := isContainerRestored(cmd, wCtx, ch, bw, hv)
res, err := wCtx.Client.InvokeScript(bw.Bytes(), nil)
if err != nil { if err != nil {
return fmt.Errorf("can't check if container is already restored: %w", err) return err
} }
if len(res.Stack) == 0 { if restored {
return errors.New("empty stack")
}
old := new(Container)
if err := old.FromStackItem(res.Stack[0]); err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if len(old.Value) != 0 {
var id cid.ID
id.SetSHA256(hv)
cmd.Printf("Container %s is already deployed.\n", id)
continue continue
} }
bw.Reset() bw.Reset()
emit.AppCall(bw.BinWriter, ch, "put", callflag.All,
cnt.Value, cnt.Signature, cnt.PublicKey, cnt.Token) putContainer(bw, ch, cnt)
if ea := cnt.EACL; ea != nil {
emit.AppCall(bw.BinWriter, ch, "setEACL", callflag.All,
ea.Value, ea.Signature, ea.PublicKey, ea.Token)
}
if bw.Err != nil { if bw.Err != nil {
panic(bw.Err) panic(bw.Err)
} }
@ -232,8 +247,67 @@ func restoreContainers(cmd *cobra.Command, _ []string) error {
return err return err
} }
} }
return nil
}
return wCtx.awaitTx() func putContainer(bw *io.BufBinWriter, ch util.Uint160, cnt Container) {
emit.AppCall(bw.BinWriter, ch, "put", callflag.All,
cnt.Value, cnt.Signature, cnt.PublicKey, cnt.Token)
if ea := cnt.EACL; ea != nil {
emit.AppCall(bw.BinWriter, ch, "setEACL", callflag.All,
ea.Value, ea.Signature, ea.PublicKey, ea.Token)
}
}
func isContainerRestored(cmd *cobra.Command, wCtx *initializeContext, containerHash util.Uint160, bw *io.BufBinWriter, hashValue util.Uint256) (bool, error) {
emit.AppCall(bw.BinWriter, containerHash, "get", callflag.All, hashValue.BytesBE())
res, err := wCtx.Client.InvokeScript(bw.Bytes(), nil)
if err != nil {
return false, fmt.Errorf("can't check if container is already restored: %w", err)
}
if len(res.Stack) == 0 {
return false, errors.New("empty stack")
}
old := new(Container)
if err := old.FromStackItem(res.Stack[0]); err != nil {
return false, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if len(old.Value) != 0 {
var id cid.ID
id.SetSHA256(hashValue)
cmd.Printf("Container %s is already deployed.\n", id)
return true, nil
}
return false, nil
}
func parseContainers(filename string) ([]Container, error) {
data, err := os.ReadFile(filename)
if err != nil {
return nil, fmt.Errorf("can't read dump file: %w", err)
}
var containers []Container
err = json.Unmarshal(data, &containers)
if err != nil {
return nil, fmt.Errorf("can't parse dump file: %w", err)
}
return containers, nil
}
func fetchContainerContractHash(wCtx *initializeContext) (util.Uint160, error) {
nnsCs, err := wCtx.Client.GetContractStateByID(1)
if err != nil {
return util.Uint160{}, fmt.Errorf("can't get NNS contract state: %w", err)
}
ch, err := nnsResolveHash(wCtx.ReadOnlyInvoker, nnsCs.Hash, containerContract+".frostfs")
if err != nil {
return util.Uint160{}, fmt.Errorf("can't fetch container contract hash: %w", err)
}
return ch, nil
} }
// Container represents container struct in contract storage. // Container represents container struct in contract storage.

View file

@ -6,7 +6,7 @@ import (
"os" "os"
"strings" "strings"
"github.com/TrueCloudLab/frostfs-contract/nns" "git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
"github.com/nspcc-dev/neo-go/cli/cmdargs" "github.com/nspcc-dev/neo-go/cli/cmdargs"
"github.com/nspcc-dev/neo-go/pkg/core/state" "github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/encoding/address" "github.com/nspcc-dev/neo-go/pkg/encoding/address"
@ -100,80 +100,88 @@ func deployContractCmd(cmd *cobra.Command, args []string) error {
cs.Manifest.Name) cs.Manifest.Name)
} }
w := io.NewBufBinWriter() writer := io.NewBufBinWriter()
if err := emitDeploymentArguments(w.BinWriter, args); err != nil { if err := emitDeploymentArguments(writer.BinWriter, args); err != nil {
return err return err
} }
emit.Bytes(w.BinWriter, cs.RawManifest) emit.Bytes(writer.BinWriter, cs.RawManifest)
emit.Bytes(w.BinWriter, cs.RawNEF) emit.Bytes(writer.BinWriter, cs.RawNEF)
emit.Int(w.BinWriter, 3) emit.Int(writer.BinWriter, 3)
emit.Opcodes(w.BinWriter, opcode.PACK) emit.Opcodes(writer.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, callHash, method, callflag.All) emit.AppCallNoArgs(writer.BinWriter, callHash, method, callflag.All)
emit.Opcodes(w.BinWriter, opcode.DROP) // contract state on stack emit.Opcodes(writer.BinWriter, opcode.DROP) // contract state on stack
if !isUpdate { if !isUpdate {
bw := io.NewBufBinWriter() err := registerNNS(nnsCs, c, zone, domain, cs, writer)
emit.Instruction(bw.BinWriter, opcode.INITSSLOT, []byte{1})
emit.AppCall(bw.BinWriter, nnsCs.Hash, "getPrice", callflag.All)
emit.Opcodes(bw.BinWriter, opcode.STSFLD0)
emit.AppCall(bw.BinWriter, nnsCs.Hash, "setPrice", callflag.All, 1)
start := bw.Len()
needRecord := false
ok, err := c.nnsRootRegistered(nnsCs.Hash, zone)
if err != nil { if err != nil {
return err return err
} else if !ok {
needRecord = true
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
zone, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
domain, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
} else {
s, ok, err := c.nnsRegisterDomainScript(nnsCs.Hash, cs.Hash, domain)
if err != nil {
return err
}
needRecord = !ok
if len(s) != 0 {
bw.WriteBytes(s)
}
}
if needRecord {
emit.AppCall(bw.BinWriter, nnsCs.Hash, "deleteRecords", callflag.All, domain, int64(nns.TXT))
emit.AppCall(bw.BinWriter, nnsCs.Hash, "addRecord", callflag.All,
domain, int64(nns.TXT), address.Uint160ToString(cs.Hash))
}
if bw.Err != nil {
panic(fmt.Errorf("BUG: can't create deployment script: %w", w.Err))
} else if bw.Len() != start {
w.WriteBytes(bw.Bytes())
emit.Opcodes(w.BinWriter, opcode.LDSFLD0, opcode.PUSH1, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsCs.Hash, "setPrice", callflag.All)
if needRecord {
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
}
} }
} }
if w.Err != nil { if writer.Err != nil {
panic(fmt.Errorf("BUG: can't create deployment script: %w", w.Err)) panic(fmt.Errorf("BUG: can't create deployment script: %w", writer.Err))
} }
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil { if err := c.sendCommitteeTx(writer.Bytes(), false); err != nil {
return err return err
} }
return c.awaitTx() return c.awaitTx()
} }
func registerNNS(nnsCs *state.Contract, c *initializeContext, zone string, domain string, cs *contractState, writer *io.BufBinWriter) error {
bw := io.NewBufBinWriter()
emit.Instruction(bw.BinWriter, opcode.INITSSLOT, []byte{1})
emit.AppCall(bw.BinWriter, nnsCs.Hash, "getPrice", callflag.All)
emit.Opcodes(bw.BinWriter, opcode.STSFLD0)
emit.AppCall(bw.BinWriter, nnsCs.Hash, "setPrice", callflag.All, 1)
start := bw.Len()
needRecord := false
ok, err := c.nnsRootRegistered(nnsCs.Hash, zone)
if err != nil {
return err
} else if !ok {
needRecord = true
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
zone, c.CommitteeAcc.Contract.ScriptHash(),
frostfsOpsEmail, int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
domain, c.CommitteeAcc.Contract.ScriptHash(),
frostfsOpsEmail, int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
} else {
s, ok, err := c.nnsRegisterDomainScript(nnsCs.Hash, cs.Hash, domain)
if err != nil {
return err
}
needRecord = !ok
if len(s) != 0 {
bw.WriteBytes(s)
}
}
if needRecord {
emit.AppCall(bw.BinWriter, nnsCs.Hash, "deleteRecords", callflag.All, domain, int64(nns.TXT))
emit.AppCall(bw.BinWriter, nnsCs.Hash, "addRecord", callflag.All,
domain, int64(nns.TXT), address.Uint160ToString(cs.Hash))
}
if bw.Err != nil {
panic(fmt.Errorf("BUG: can't create deployment script: %w", writer.Err))
} else if bw.Len() != start {
writer.WriteBytes(bw.Bytes())
emit.Opcodes(writer.BinWriter, opcode.LDSFLD0, opcode.PUSH1, opcode.PACK)
emit.AppCallNoArgs(writer.BinWriter, nnsCs.Hash, "setPrice", callflag.All)
if needRecord {
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
}
}
return nil
}
func emitDeploymentArguments(w *io.BinWriter, args []string) error { func emitDeploymentArguments(w *io.BinWriter, args []string) error {
_, ps, err := cmdargs.ParseParams(args, true) _, ps, err := cmdargs.ParseParams(args, true)
if err != nil { if err != nil {

View file

@ -1,40 +0,0 @@
package morph
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"strings"
"github.com/google/go-github/v39/github"
"github.com/spf13/cobra"
)
func downloadContractsFromGithub(cmd *cobra.Command) (io.ReadCloser, error) {
gcl := github.NewClient(nil)
release, _, err := gcl.Repositories.GetLatestRelease(context.Background(), "nspcc-dev", "frostfs-contract")
if err != nil {
return nil, fmt.Errorf("can't fetch release info: %w", err)
}
cmd.Printf("Found %s (%s), downloading...\n", release.GetTagName(), release.GetName())
var url string
for _, a := range release.Assets {
if strings.HasPrefix(a.GetName(), "frostfs-contract") {
url = a.GetBrowserDownloadURL()
break
}
}
if url == "" {
return nil, errors.New("can't find contracts archive in release assets")
}
resp, err := http.Get(url)
if err != nil {
return nil, fmt.Errorf("can't fetch contracts archive: %w", err)
}
return resp.Body, nil
}

View file

@ -7,7 +7,8 @@ import (
"strings" "strings"
"text/tabwriter" "text/tabwriter"
"github.com/TrueCloudLab/frostfs-contract/nns" "git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
morphClient "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/io" "github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker" "github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap" "github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
@ -123,7 +124,7 @@ func dumpCustomZoneHashes(cmd *cobra.Command, nnsHash util.Uint160, zone string,
return return
} }
if !bytes.HasSuffix(bs, []byte(zone)) { if !bytes.HasSuffix(bs, []byte(zone)) || bytes.HasPrefix(bs, []byte(morphClient.NNSGroupKeyName)) {
// Related https://github.com/nspcc-dev/neofs-contract/issues/316. // Related https://github.com/nspcc-dev/neofs-contract/issues/316.
return return
} }

View file

@ -3,6 +3,7 @@ package morph
import ( import (
"errors" "errors"
"fmt" "fmt"
"strings"
"github.com/nspcc-dev/neo-go/pkg/io" "github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap" "github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
@ -13,7 +14,7 @@ import (
"github.com/spf13/viper" "github.com/spf13/viper"
) )
func forceNewEpochCmd(cmd *cobra.Command, args []string) error { func forceNewEpochCmd(cmd *cobra.Command, _ []string) error {
wCtx, err := newInitializeContext(cmd, viper.GetViper()) wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil { if err != nil {
return fmt.Errorf("can't to initialize context: %w", err) return fmt.Errorf("can't to initialize context: %w", err)
@ -38,7 +39,14 @@ func forceNewEpochCmd(cmd *cobra.Command, args []string) error {
return err return err
} }
return wCtx.awaitTx() if err := wCtx.awaitTx(); err != nil {
if strings.Contains(err.Error(), "invalid epoch") {
cmd.Println("Epoch has already ticked.")
return nil
}
return err
}
return nil
} }
func emitNewEpochCall(bw *io.BufBinWriter, wCtx *initializeContext, nmHash util.Uint160) error { func emitNewEpochCall(bw *io.BufBinWriter, wCtx *initializeContext, nmHash util.Uint160) error {

View file

@ -6,8 +6,8 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address" "github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn" "github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
@ -21,6 +21,7 @@ import (
"github.com/nspcc-dev/neo-go/pkg/wallet" "github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
"golang.org/x/sync/errgroup"
) )
const ( const (
@ -29,7 +30,7 @@ const (
consensusAccountName = "consensus" consensusAccountName = "consensus"
) )
func generateAlphabetCreds(cmd *cobra.Command, args []string) error { func generateAlphabetCreds(cmd *cobra.Command, _ []string) error {
// alphabet size is not part of the config // alphabet size is not part of the config
size, err := cmd.Flags().GetUint(alphabetSizeFlag) size, err := cmd.Flags().GetUint(alphabetSizeFlag)
if err != nil { if err != nil {
@ -38,6 +39,9 @@ func generateAlphabetCreds(cmd *cobra.Command, args []string) error {
if size == 0 { if size == 0 {
return errors.New("size must be > 0") return errors.New("size must be > 0")
} }
if size > maxAlphabetNodes {
return ErrTooManyAlphabetNodes
}
v := viper.GetViper() v := viper.GetViper()
walletDir := config.ResolveHomePath(viper.GetString(alphabetWalletsFlag)) walletDir := config.ResolveHomePath(viper.GetString(alphabetWalletsFlag))
@ -92,28 +96,31 @@ func initializeWallets(v *viper.Viper, walletDir string, size int) ([]string, er
pubs[i] = w.Accounts[0].PrivateKey().PublicKey() pubs[i] = w.Accounts[0].PrivateKey().PublicKey()
} }
var errG errgroup.Group
// Create committee account with N/2+1 multi-signature. // Create committee account with N/2+1 multi-signature.
majCount := smartcontract.GetMajorityHonestNodeCount(size) majCount := smartcontract.GetMajorityHonestNodeCount(size)
for i, w := range wallets {
if err := addMultisigAccount(w, majCount, committeeAccountName, passwords[i], pubs); err != nil {
return nil, fmt.Errorf("can't create committee account: %w", err)
}
}
// Create consensus account with 2*N/3+1 multi-signature. // Create consensus account with 2*N/3+1 multi-signature.
bftCount := smartcontract.GetDefaultHonestNodeCount(size) bftCount := smartcontract.GetDefaultHonestNodeCount(size)
for i, w := range wallets { for i := range wallets {
if err := addMultisigAccount(w, bftCount, consensusAccountName, passwords[i], pubs); err != nil { i := i
return nil, fmt.Errorf("can't create consensus account: %w", err) ps := pubs.Copy()
} errG.Go(func() error {
if err := addMultisigAccount(wallets[i], majCount, committeeAccountName, passwords[i], ps); err != nil {
return fmt.Errorf("can't create committee account: %w", err)
}
if err := addMultisigAccount(wallets[i], bftCount, consensusAccountName, passwords[i], ps); err != nil {
return fmt.Errorf("can't create consentus account: %w", err)
}
if err := wallets[i].SavePretty(); err != nil {
return fmt.Errorf("can't save wallet: %w", err)
}
return nil
})
} }
if err := errG.Wait(); err != nil {
for _, w := range wallets { return nil, err
if err := w.SavePretty(); err != nil {
return nil, fmt.Errorf("can't save wallet: %w", err)
}
} }
return passwords, nil return passwords, nil
} }

View file

@ -7,9 +7,10 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"strconv" "strconv"
"sync"
"testing" "testing"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/cli/input" "github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/smartcontract" "github.com/nspcc-dev/neo-go/pkg/smartcontract"
@ -71,24 +72,32 @@ func TestGenerateAlphabet(t *testing.T) {
buf.WriteString(testContractPassword + "\r") buf.WriteString(testContractPassword + "\r")
require.NoError(t, generateAlphabetCreds(generateAlphabetCmd, nil)) require.NoError(t, generateAlphabetCreds(generateAlphabetCmd, nil))
var wg sync.WaitGroup
for i := uint64(0); i < size; i++ { for i := uint64(0); i < size; i++ {
p := filepath.Join(walletDir, innerring.GlagoliticLetter(i).String()+".json") i := i
w, err := wallet.NewWalletFromFile(p) wg.Add(1)
require.NoError(t, err, "wallet doesn't exist") go func() {
require.Equal(t, 3, len(w.Accounts), "not all accounts were created") defer wg.Done()
for _, a := range w.Accounts { p := filepath.Join(walletDir, innerring.GlagoliticLetter(i).String()+".json")
err := a.Decrypt(strconv.FormatUint(i, 10), keys.NEP2ScryptParams()) w, err := wallet.NewWalletFromFile(p)
require.NoError(t, err, "can't decrypt account") require.NoError(t, err, "wallet doesn't exist")
switch a.Label { require.Equal(t, 3, len(w.Accounts), "not all accounts were created")
case consensusAccountName:
require.Equal(t, smartcontract.GetDefaultHonestNodeCount(size), len(a.Contract.Parameters)) for _, a := range w.Accounts {
case committeeAccountName: err := a.Decrypt(strconv.FormatUint(i, 10), keys.NEP2ScryptParams())
require.Equal(t, smartcontract.GetMajorityHonestNodeCount(size), len(a.Contract.Parameters)) require.NoError(t, err, "can't decrypt account")
default: switch a.Label {
require.Equal(t, singleAccountName, a.Label) case consensusAccountName:
require.Equal(t, smartcontract.GetDefaultHonestNodeCount(size), len(a.Contract.Parameters))
case committeeAccountName:
require.Equal(t, smartcontract.GetMajorityHonestNodeCount(size), len(a.Contract.Parameters))
default:
require.Equal(t, singleAccountName, a.Label)
}
} }
} }()
} }
wg.Wait()
t.Run("check contract group wallet", func(t *testing.T) { t.Run("check contract group wallet", func(t *testing.T) {
p := filepath.Join(walletDir, contractWalletFilename) p := filepath.Join(walletDir, contractWalletFilename)

View file

@ -6,7 +6,7 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/manifest" "github.com/nspcc-dev/neo-go/pkg/smartcontract/manifest"
"github.com/nspcc-dev/neo-go/pkg/util" "github.com/nspcc-dev/neo-go/pkg/util"

View file

@ -7,9 +7,9 @@ import (
"path/filepath" "path/filepath"
"time" "time"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
morphClient "github.com/TrueCloudLab/frostfs-node/pkg/morph/client" morphClient "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames" "github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/core/state" "github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction" "github.com/nspcc-dev/neo-go/pkg/core/transaction"
@ -23,6 +23,13 @@ import (
"github.com/spf13/viper" "github.com/spf13/viper"
) )
const (
// maxAlphabetNodes is the maximum number of candidates allowed, which is currently limited by the size
// of the invocation script.
// See: https://github.com/nspcc-dev/neo-go/blob/740488f7f35e367eaa99a71c0a609c315fe2b0fc/pkg/core/transaction/witness.go#L10
maxAlphabetNodes = 22
)
type cache struct { type cache struct {
nnsCs *state.Contract nnsCs *state.Contract
groupKey *keys.PublicKey groupKey *keys.PublicKey
@ -45,7 +52,9 @@ type initializeContext struct {
ContractPath string ContractPath string
} }
func initializeSideChainCmd(cmd *cobra.Command, args []string) error { var ErrTooManyAlphabetNodes = fmt.Errorf("too many alphabet nodes (maximum allowed is %d)", maxAlphabetNodes)
func initializeSideChainCmd(cmd *cobra.Command, _ []string) error {
initCtx, err := newInitializeContext(cmd, viper.GetViper()) initCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil { if err != nil {
return fmt.Errorf("initialization error: %w", err) return fmt.Errorf("initialization error: %w", err)
@ -91,11 +100,7 @@ func initializeSideChainCmd(cmd *cobra.Command, args []string) error {
} }
cmd.Println("Stage 7: set addresses in NNS.") cmd.Println("Stage 7: set addresses in NNS.")
if err := initCtx.setNNS(); err != nil { return initCtx.setNNS()
return err
}
return nil
} }
func (c *initializeContext) close() { func (c *initializeContext) close() {
@ -115,27 +120,21 @@ func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContex
return nil, err return nil, err
} }
if len(wallets) > maxAlphabetNodes {
return nil, ErrTooManyAlphabetNodes
}
needContracts := cmd.Name() == "update-contracts" || cmd.Name() == "init" needContracts := cmd.Name() == "update-contracts" || cmd.Name() == "init"
var w *wallet.Wallet var w *wallet.Wallet
if needContracts { w, err = getWallet(cmd, v, needContracts, walletDir)
w, err = openContractWallet(v, cmd, walletDir) if err != nil {
if err != nil { return nil, err
return nil, err
}
} }
var c Client c, err := createClient(cmd, v, wallets)
if v.GetString(localDumpFlag) != "" {
if v.GetString(endpointFlag) != "" {
return nil, fmt.Errorf("`%s` and `%s` flags are mutually exclusive", endpointFlag, localDumpFlag)
}
c, err = newLocalClient(cmd, v, wallets)
} else {
c, err = getN3Client(v)
}
if err != nil { if err != nil {
return nil, fmt.Errorf("can't create N3 client: %w", err) return nil, err
} }
committeeAcc, err := getWalletAccount(wallets[0], committeeAccountName) committeeAcc, err := getWalletAccount(wallets[0], committeeAccountName)
@ -148,35 +147,22 @@ func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContex
return nil, fmt.Errorf("can't find consensus account: %w", err) return nil, fmt.Errorf("can't find consensus account: %w", err)
} }
var ctrPath string if err := validateInit(cmd); err != nil {
if cmd.Name() == "init" { return nil, err
if viper.GetInt64(epochDurationInitFlag) <= 0 {
return nil, fmt.Errorf("epoch duration must be positive")
}
if viper.GetInt64(maxObjectSizeInitFlag) <= 0 {
return nil, fmt.Errorf("max object size must be positive")
}
} }
if needContracts { ctrPath, err := getContractsPath(cmd, needContracts)
ctrPath, err = cmd.Flags().GetString(contractsInitFlag) if err != nil {
if err != nil { return nil, err
return nil, fmt.Errorf("invalid contracts path: %w", err)
}
} }
if err := checkNotaryEnabled(c); err != nil { if err := checkNotaryEnabled(c); err != nil {
return nil, err return nil, err
} }
accounts := make([]*wallet.Account, len(wallets)) accounts, err := createWalletAccounts(wallets)
for i, w := range wallets { if err != nil {
acc, err := getWalletAccount(w, singleAccountName) return nil, err
if err != nil {
return nil, fmt.Errorf("wallet %s is invalid (no single account): %w", w.Path(), err)
}
accounts[i] = acc
} }
cliCtx, err := defaultClientContext(c, committeeAcc) cliCtx, err := defaultClientContext(c, committeeAcc)
@ -206,6 +192,69 @@ func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContex
return initCtx, nil return initCtx, nil
} }
func validateInit(cmd *cobra.Command) error {
if cmd.Name() != "init" {
return nil
}
if viper.GetInt64(epochDurationInitFlag) <= 0 {
return fmt.Errorf("epoch duration must be positive")
}
if viper.GetInt64(maxObjectSizeInitFlag) <= 0 {
return fmt.Errorf("max object size must be positive")
}
return nil
}
func createClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet) (Client, error) {
var c Client
var err error
if ldf := cmd.Flags().Lookup(localDumpFlag); ldf != nil && ldf.Changed {
if cmd.Flags().Changed(endpointFlag) {
return nil, fmt.Errorf("`%s` and `%s` flags are mutually exclusive", endpointFlag, localDumpFlag)
}
c, err = newLocalClient(cmd, v, wallets, ldf.Value.String())
} else {
c, err = getN3Client(v)
}
if err != nil {
return nil, fmt.Errorf("can't create N3 client: %w", err)
}
return c, nil
}
func getWallet(cmd *cobra.Command, v *viper.Viper, needContracts bool, walletDir string) (*wallet.Wallet, error) {
if !needContracts {
return nil, nil
}
return openContractWallet(v, cmd, walletDir)
}
func getContractsPath(cmd *cobra.Command, needContracts bool) (string, error) {
if !needContracts {
return "", nil
}
ctrPath, err := cmd.Flags().GetString(contractsInitFlag)
if err != nil {
return "", fmt.Errorf("invalid contracts path: %w", err)
}
return ctrPath, nil
}
func createWalletAccounts(wallets []*wallet.Wallet) ([]*wallet.Account, error) {
accounts := make([]*wallet.Account, len(wallets))
for i, w := range wallets {
acc, err := getWalletAccount(w, singleAccountName)
if err != nil {
return nil, fmt.Errorf("wallet %s is invalid (no single account): %w", w.Path(), err)
}
accounts[i] = acc
}
return accounts, nil
}
func openAlphabetWallets(v *viper.Viper, walletDir string) ([]*wallet.Wallet, error) { func openAlphabetWallets(v *viper.Viper, walletDir string) ([]*wallet.Wallet, error) {
walletFiles, err := os.ReadDir(walletDir) walletFiles, err := os.ReadDir(walletDir)
if err != nil { if err != nil {

View file

@ -12,10 +12,11 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"github.com/TrueCloudLab/frostfs-contract/common" "git.frostfs.info/TrueCloudLab/frostfs-contract/common"
"github.com/TrueCloudLab/frostfs-contract/nns" "git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
morphClient "github.com/TrueCloudLab/frostfs-node/pkg/morph/client" morphClient "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/core/state" "github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction" "github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/encoding/address" "github.com/nspcc-dev/neo-go/pkg/encoding/address"
@ -23,6 +24,7 @@ import (
"github.com/nspcc-dev/neo-go/pkg/rpcclient" "github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor" "github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/management" "github.com/nspcc-dev/neo-go/pkg/rpcclient/management"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract" "github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag" "github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/manifest" "github.com/nspcc-dev/neo-go/pkg/smartcontract/manifest"
@ -30,8 +32,8 @@ import (
"github.com/nspcc-dev/neo-go/pkg/util" "github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit" "github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode" "github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate" "github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/spf13/viper"
) )
const ( const (
@ -39,44 +41,20 @@ const (
frostfsContract = "frostfs" // not deployed in side-chain. frostfsContract = "frostfs" // not deployed in side-chain.
processingContract = "processing" // not deployed in side-chain. processingContract = "processing" // not deployed in side-chain.
alphabetContract = "alphabet" alphabetContract = "alphabet"
auditContract = "audit"
balanceContract = "balance" balanceContract = "balance"
containerContract = "container" containerContract = "container"
frostfsIDContract = "frostfsid" frostfsIDContract = "frostfsid"
netmapContract = "netmap" netmapContract = "netmap"
proxyContract = "proxy" proxyContract = "proxy"
reputationContract = "reputation"
subnetContract = "subnet"
)
const (
netmapEpochKey = "EpochDuration"
netmapMaxObjectSizeKey = "MaxObjectSize"
netmapAuditFeeKey = "AuditFee"
netmapContainerFeeKey = "ContainerFee"
netmapContainerAliasFeeKey = "ContainerAliasFee"
netmapEigenTrustIterationsKey = "EigenTrustIterations"
netmapEigenTrustAlphaKey = "EigenTrustAlpha"
netmapBasicIncomeRateKey = "BasicIncomeRate"
netmapInnerRingCandidateFeeKey = "InnerRingCandidateFee"
netmapWithdrawFeeKey = "WithdrawFee"
netmapHomomorphicHashDisabledKey = "HomomorphicHashingDisabled"
netmapMaintenanceAllowedKey = "MaintenanceModeAllowed"
defaultEigenTrustIterations = 4
defaultEigenTrustAlpha = "0.1"
) )
var ( var (
contractList = []string{ contractList = []string{
auditContract,
balanceContract, balanceContract,
containerContract, containerContract,
frostfsIDContract, frostfsIDContract,
netmapContract, netmapContract,
proxyContract, proxyContract,
reputationContract,
subnetContract,
} }
fullContractList = append([]string{ fullContractList = append([]string{
@ -85,6 +63,17 @@ var (
nnsContract, nnsContract,
alphabetContract, alphabetContract,
}, contractList...) }, contractList...)
netmapConfigKeys = []string{
netmap.EpochDurationConfig,
netmap.MaxObjectSizeConfig,
netmap.ContainerFeeConfig,
netmap.ContainerAliasFeeConfig,
netmap.IrCandidateFeeConfig,
netmap.WithdrawFeeConfig,
netmap.HomomorphicHashingDisabledKey,
netmap.MaintenanceModeAllowedConfig,
}
) )
type contractState struct { type contractState struct {
@ -167,8 +156,6 @@ func (c *initializeContext) updateContracts() error {
w := io2.NewBufBinWriter() w := io2.NewBufBinWriter()
var keysParam []any
// Update script size for a single-node committee is close to the maximum allowed size of 65535. // Update script size for a single-node committee is close to the maximum allowed size of 65535.
// Because of this we want to reuse alphabet contract NEF and manifest for different updates. // Because of this we want to reuse alphabet contract NEF and manifest for different updates.
// The generated script is as following. // The generated script is as following.
@ -182,42 +169,36 @@ func (c *initializeContext) updateContracts() error {
emit.Bytes(w.BinWriter, alphaCs.RawNEF) emit.Bytes(w.BinWriter, alphaCs.RawNEF)
emit.Opcodes(w.BinWriter, opcode.STSFLD0) emit.Opcodes(w.BinWriter, opcode.STSFLD0)
baseGroups := alphaCs.Manifest.Groups keysParam, err := c.deployAlphabetAccounts(nnsHash, w, alphaCs)
if err != nil {
// alphabet contracts should be deployed by individual nodes to get different hashes. return err
for i, acc := range c.Accounts {
ctrHash, err := nnsResolveHash(c.ReadOnlyInvoker, nnsHash, getAlphabetNNSDomain(i))
if err != nil {
return fmt.Errorf("can't resolve hash for contract update: %w", err)
}
keysParam = append(keysParam, acc.PrivateKey().PublicKey().Bytes())
params := c.getAlphabetDeployItems(i, len(c.Wallets))
emit.Array(w.BinWriter, params...)
alphaCs.Manifest.Groups = baseGroups
err = c.addManifestGroup(ctrHash, alphaCs)
if err != nil {
return fmt.Errorf("can't sign manifest group: %v", err)
}
emit.Bytes(w.BinWriter, alphaCs.RawManifest)
emit.Opcodes(w.BinWriter, opcode.LDSFLD0)
emit.Int(w.BinWriter, 3)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, ctrHash, updateMethodName, callflag.All)
}
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
if !strings.Contains(err.Error(), common.ErrAlreadyUpdated) {
return err
}
c.Command.Println("Alphabet contracts are already updated.")
} }
w.Reset() w.Reset()
if err = c.deployOrUpdateContracts(w, nnsHash, keysParam); err != nil {
return err
}
groupKey := c.ContractWallet.Accounts[0].PrivateKey().PublicKey()
_, _, err = c.emitUpdateNNSGroupScript(w, nnsHash, groupKey)
if err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", morphClient.NNSGroupKeyName, hex.EncodeToString(groupKey.Bytes()))
emit.Opcodes(w.BinWriter, opcode.LDSFLD0)
emit.Int(w.BinWriter, 1)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsHash, "setPrice", callflag.All)
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
return err
}
return c.awaitTx()
}
func (c *initializeContext) deployOrUpdateContracts(w *io2.BufBinWriter, nnsHash util.Uint160, keysParam []any) error {
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1}) emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
emit.AppCall(w.BinWriter, nnsHash, "getPrice", callflag.All) emit.AppCall(w.BinWriter, nnsHash, "getPrice", callflag.All)
emit.Opcodes(w.BinWriter, opcode.STSFLD0) emit.Opcodes(w.BinWriter, opcode.STSFLD0)
@ -247,7 +228,7 @@ func (c *initializeContext) updateContracts() error {
invokeHash = ctrHash invokeHash = ctrHash
} }
params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam)) params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam, updateMethodName))
res, err := c.CommitteeAct.MakeCall(invokeHash, method, params...) res, err := c.CommitteeAct.MakeCall(invokeHash, method, params...)
if err != nil { if err != nil {
if method != updateMethodName || !strings.Contains(err.Error(), common.ErrAlreadyUpdated) { if method != updateMethodName || !strings.Contains(err.Error(), common.ErrAlreadyUpdated) {
@ -277,23 +258,46 @@ func (c *initializeContext) updateContracts() error {
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE()) c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
} }
} }
return nil
}
groupKey := c.ContractWallet.Accounts[0].PrivateKey().PublicKey() func (c *initializeContext) deployAlphabetAccounts(nnsHash util.Uint160, w *io2.BufBinWriter, alphaCs *contractState) ([]any, error) {
_, _, err = c.emitUpdateNNSGroupScript(w, nnsHash, groupKey) var keysParam []any
if err != nil {
return err baseGroups := alphaCs.Manifest.Groups
// alphabet contracts should be deployed by individual nodes to get different hashes.
for i, acc := range c.Accounts {
ctrHash, err := nnsResolveHash(c.ReadOnlyInvoker, nnsHash, getAlphabetNNSDomain(i))
if err != nil {
return nil, fmt.Errorf("can't resolve hash for contract update: %w", err)
}
keysParam = append(keysParam, acc.PrivateKey().PublicKey().Bytes())
params := c.getAlphabetDeployItems(i, len(c.Wallets))
emit.Array(w.BinWriter, params...)
alphaCs.Manifest.Groups = baseGroups
err = c.addManifestGroup(ctrHash, alphaCs)
if err != nil {
return nil, fmt.Errorf("can't sign manifest group: %v", err)
}
emit.Bytes(w.BinWriter, alphaCs.RawManifest)
emit.Opcodes(w.BinWriter, opcode.LDSFLD0)
emit.Int(w.BinWriter, 3)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, ctrHash, updateMethodName, callflag.All)
} }
c.Command.Printf("NNS: Set %s -> %s\n", morphClient.NNSGroupKeyName, hex.EncodeToString(groupKey.Bytes()))
emit.Opcodes(w.BinWriter, opcode.LDSFLD0)
emit.Int(w.BinWriter, 1)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsHash, "setPrice", callflag.All)
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil { if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
return err if !strings.Contains(err.Error(), common.ErrAlreadyUpdated) {
return nil, err
}
c.Command.Println("Alphabet contracts are already updated.")
} }
return c.awaitTx()
return keysParam, nil
} }
func (c *initializeContext) deployContracts() error { func (c *initializeContext) deployContracts() error {
@ -347,7 +351,7 @@ func (c *initializeContext) deployContracts() error {
return fmt.Errorf("can't sign manifest group: %v", err) return fmt.Errorf("can't sign manifest group: %v", err)
} }
params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam)) params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam, deployMethodName))
res, err := c.CommitteeAct.MakeCall(management.Hash, deployMethodName, params...) res, err := c.CommitteeAct.MakeCall(management.Hash, deployMethodName, params...)
if err != nil { if err != nil {
return fmt.Errorf("can't deploy %s contract: %w", ctrName, err) return fmt.Errorf("can't deploy %s contract: %w", ctrName, err)
@ -393,11 +397,9 @@ func (c *initializeContext) readContracts(names []string) error {
} else { } else {
var r io.ReadCloser var r io.ReadCloser
if c.ContractPath == "" { if c.ContractPath == "" {
c.Command.Println("Contracts flag is missing, latest release will be fetched from Github.") return errors.New("contracts flag is missing")
r, err = downloadContractsFromGithub(c.Command)
} else {
r, err = os.Open(c.ContractPath)
} }
r, err = os.Open(c.ContractPath)
if err != nil { if err != nil {
return fmt.Errorf("can't open contracts archive: %w", err) return fmt.Errorf("can't open contracts archive: %w", err)
} }
@ -514,7 +516,7 @@ func getContractDeployParameters(cs *contractState, deployData []any) []any {
return []any{cs.RawNEF, cs.RawManifest, deployData} return []any{cs.RawNEF, cs.RawManifest, deployData}
} }
func (c *initializeContext) getContractDeployData(ctrName string, keysParam []any) []any { func (c *initializeContext) getContractDeployData(ctrName string, keysParam []any, method string) []any {
items := make([]any, 1, 6) items := make([]any, 1, 6)
items[0] = false // notaryDisabled is false items[0] = false // notaryDisabled is false
@ -527,8 +529,6 @@ func (c *initializeContext) getContractDeployData(ctrName string, keysParam []an
case processingContract: case processingContract:
items = append(items, c.Contracts[frostfsContract].Hash) items = append(items, c.Contracts[frostfsContract].Hash)
return items[1:] // no notary info return items[1:] // no notary info
case auditContract:
items = append(items, c.Contracts[netmapContract].Hash)
case balanceContract: case balanceContract:
items = append(items, items = append(items,
c.Contracts[netmapContract].Hash, c.Contracts[netmapContract].Hash,
@ -551,20 +551,31 @@ func (c *initializeContext) getContractDeployData(ctrName string, keysParam []an
c.Contracts[netmapContract].Hash, c.Contracts[netmapContract].Hash,
c.Contracts[containerContract].Hash) c.Contracts[containerContract].Hash)
case netmapContract: case netmapContract:
configParam := []any{ md := getDefaultNetmapContractConfigMap()
netmapEpochKey, viper.GetInt64(epochDurationInitFlag), if method == updateMethodName {
netmapMaxObjectSizeKey, viper.GetInt64(maxObjectSizeInitFlag), arr, err := c.getNetConfigFromNetmapContract()
netmapAuditFeeKey, viper.GetInt64(auditFeeInitFlag), if err != nil {
netmapContainerFeeKey, viper.GetInt64(containerFeeInitFlag), panic(err)
netmapContainerAliasFeeKey, viper.GetInt64(containerAliasFeeInitFlag), }
netmapEigenTrustIterationsKey, int64(defaultEigenTrustIterations), m, err := parseConfigFromNetmapContract(arr)
netmapEigenTrustAlphaKey, defaultEigenTrustAlpha, if err != nil {
netmapBasicIncomeRateKey, viper.GetInt64(incomeRateInitFlag), panic(err)
netmapInnerRingCandidateFeeKey, viper.GetInt64(candidateFeeInitFlag), }
netmapWithdrawFeeKey, viper.GetInt64(withdrawFeeInitFlag), for k, v := range m {
netmapHomomorphicHashDisabledKey, viper.GetBool(homomorphicHashDisabledInitFlag), for _, key := range netmapConfigKeys {
netmapMaintenanceAllowedKey, viper.GetBool(maintenanceModeAllowedInitFlag), if k == key {
md[k] = v
break
}
}
}
} }
var configParam []any
for k, v := range md {
configParam = append(configParam, k, v)
}
items = append(items, items = append(items,
c.Contracts[balanceContract].Hash, c.Contracts[balanceContract].Hash,
c.Contracts[containerContract].Hash, c.Contracts[containerContract].Hash,
@ -572,14 +583,28 @@ func (c *initializeContext) getContractDeployData(ctrName string, keysParam []an
configParam) configParam)
case proxyContract: case proxyContract:
items = nil items = nil
case reputationContract:
case subnetContract:
default: default:
panic(fmt.Sprintf("invalid contract name: %s", ctrName)) panic(fmt.Sprintf("invalid contract name: %s", ctrName))
} }
return items return items
} }
func (c *initializeContext) getNetConfigFromNetmapContract() ([]stackitem.Item, error) {
cs, err := c.Client.GetContractStateByID(1)
if err != nil {
return nil, fmt.Errorf("NNS is not yet deployed: %w", err)
}
nmHash, err := nnsResolveHash(c.ReadOnlyInvoker, cs.Hash, netmapContract+".frostfs")
if err != nil {
return nil, fmt.Errorf("can't get netmap contract hash: %w", err)
}
arr, err := unwrap.Array(c.ReadOnlyInvoker.Call(nmHash, "listConfig"))
if err != nil {
return nil, fmt.Errorf("can't fetch list of network config keys from the netmap contract")
}
return arr, err
}
func (c *initializeContext) getAlphabetDeployItems(i, n int) []any { func (c *initializeContext) getAlphabetDeployItems(i, n int) []any {
items := make([]any, 6) items := make([]any, 6)
items[0] = false items[0] = false

View file

@ -7,14 +7,15 @@ import (
"strconv" "strconv"
"time" "time"
"github.com/TrueCloudLab/frostfs-contract/nns" "git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
morphClient "github.com/TrueCloudLab/frostfs-node/pkg/morph/client" morphClient "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/core/state" "github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address" "github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/io" "github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient" "github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker" "github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
nnsClient "github.com/nspcc-dev/neo-go/pkg/rpcclient/nns"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap" "github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag" "github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util" "github.com/nspcc-dev/neo-go/pkg/util"
@ -26,6 +27,8 @@ import (
const defaultExpirationTime = 10 * 365 * 24 * time.Hour / time.Second const defaultExpirationTime = 10 * 365 * 24 * time.Hour / time.Second
const frostfsOpsEmail = "ops@frostfs.info"
func (c *initializeContext) setNNS() error { func (c *initializeContext) setNNS() error {
nnsCs, err := c.Client.GetContractStateByID(1) nnsCs, err := c.Client.GetContractStateByID(1)
if err != nil { if err != nil {
@ -39,7 +42,7 @@ func (c *initializeContext) setNNS() error {
bw := io.NewBufBinWriter() bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All, emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
"frostfs", c.CommitteeAcc.Contract.ScriptHash(), "frostfs", c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600)) frostfsOpsEmail, int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT) emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if err := c.sendCommitteeTx(bw.Bytes(), true); err != nil { if err := c.sendCommitteeTx(bw.Bytes(), true); err != nil {
return fmt.Errorf("can't add domain root to NNS: %w", err) return fmt.Errorf("can't add domain root to NNS: %w", err)
@ -82,13 +85,13 @@ func (c *initializeContext) setNNS() error {
func (c *initializeContext) updateNNSGroup(nnsHash util.Uint160, pub *keys.PublicKey) error { func (c *initializeContext) updateNNSGroup(nnsHash util.Uint160, pub *keys.PublicKey) error {
bw := io.NewBufBinWriter() bw := io.NewBufBinWriter()
needUpdate, needRegister, err := c.emitUpdateNNSGroupScript(bw, nnsHash, pub) keyAlreadyAdded, domainRegCodeEmitted, err := c.emitUpdateNNSGroupScript(bw, nnsHash, pub)
if !needUpdate || err != nil { if keyAlreadyAdded || err != nil {
return err return err
} }
script := bw.Bytes() script := bw.Bytes()
if needRegister { if domainRegCodeEmitted {
w := io.NewBufBinWriter() w := io.NewBufBinWriter()
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1}) emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
wrapRegisterScriptWithPrice(w, nnsHash, script) wrapRegisterScriptWithPrice(w, nnsHash, script)
@ -121,7 +124,7 @@ func (c *initializeContext) emitUpdateNNSGroupScript(bw *io.BufBinWriter, nnsHas
if isAvail { if isAvail {
emit.AppCall(bw.BinWriter, nnsHash, "register", callflag.All, emit.AppCall(bw.BinWriter, nnsHash, "register", callflag.All,
morphClient.NNSGroupKeyName, c.CommitteeAcc.Contract.ScriptHash(), morphClient.NNSGroupKeyName, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600)) frostfsOpsEmail, int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT) emit.Opcodes(bw.BinWriter, opcode.ASSERT)
} }
@ -169,7 +172,7 @@ func (c *initializeContext) nnsRegisterDomainScript(nnsHash, expectedHash util.U
bw := io.NewBufBinWriter() bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, nnsHash, "register", callflag.All, emit.AppCall(bw.BinWriter, nnsHash, "register", callflag.All,
domain, c.CommitteeAcc.Contract.ScriptHash(), domain, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600)) frostfsOpsEmail, int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT) emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if bw.Err != nil { if bw.Err != nil {
@ -228,20 +231,27 @@ func nnsResolve(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (stac
} }
func nnsResolveKey(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (*keys.PublicKey, error) { func nnsResolveKey(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (*keys.PublicKey, error) {
item, err := nnsResolve(inv, nnsHash, domain) res, err := nnsResolve(inv, nnsHash, domain)
if err != nil { if err != nil {
return nil, err return nil, err
} }
v, ok := item.Value().(stackitem.Null) if _, ok := res.Value().(stackitem.Null); ok {
if ok {
return nil, errors.New("NNS record is missing") return nil, errors.New("NNS record is missing")
} }
bs, err := v.TryBytes() arr, ok := res.Value().([]stackitem.Item)
if err != nil { if !ok {
return nil, errors.New("malformed response") return nil, errors.New("API of the NNS contract method `resolve` has changed")
} }
for i := range arr {
var bs []byte
bs, err = arr[i].TryBytes()
if err != nil {
continue
}
return keys.NewPublicKeyFromString(string(bs)) return keys.NewPublicKeyFromString(string(bs))
}
return nil, errors.New("no valid keys are found")
} }
// parseNNSResolveResult parses the result of resolving NNS record. // parseNNSResolveResult parses the result of resolving NNS record.
@ -277,9 +287,11 @@ func parseNNSResolveResult(res stackitem.Item) (util.Uint160, error) {
} }
func nnsIsAvailable(c Client, nnsHash util.Uint160, name string) (bool, error) { func nnsIsAvailable(c Client, nnsHash util.Uint160, name string) (bool, error) {
switch ct := c.(type) { switch c.(type) {
case *rpcclient.Client: case *rpcclient.Client:
return ct.NNSIsAvailable(nnsHash, name) inv := invoker.New(c, nil)
reader := nnsClient.NewReader(inv, nnsHash)
return reader.IsAvailable(name)
default: default:
b, err := unwrap.Bool(invokeFunction(c, nnsHash, "isAvailable", []any{name}, nil)) b, err := unwrap.Bool(invokeFunction(c, nnsHash, "isAvailable", []any{name}, nil))
if err != nil { if err != nil {

View file

@ -9,6 +9,7 @@ import (
"github.com/nspcc-dev/neo-go/pkg/core/transaction" "github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/io" "github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient" "github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/neo" "github.com/nspcc-dev/neo-go/pkg/rpcclient/neo"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap" "github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag" "github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
@ -18,33 +19,24 @@ import (
) )
// initialAlphabetNEOAmount represents the total amount of GAS distributed between alphabet nodes. // initialAlphabetNEOAmount represents the total amount of GAS distributed between alphabet nodes.
const initialAlphabetNEOAmount = native.NEOTotalSupply const (
initialAlphabetNEOAmount = native.NEOTotalSupply
func (c *initializeContext) registerCandidates() error { registerBatchSize = transaction.MaxAttributes - 1
neoHash := neo.Hash )
cc, err := unwrap.Array(c.ReadOnlyInvoker.Call(neoHash, "getCandidates"))
if err != nil {
return fmt.Errorf("`getCandidates`: %w", err)
}
if len(cc) > 0 {
c.Command.Println("Candidates are already registered.")
return nil
}
func (c *initializeContext) registerCandidateRange(start, end int) error {
regPrice, err := c.getCandidateRegisterPrice() regPrice, err := c.getCandidateRegisterPrice()
if err != nil { if err != nil {
return fmt.Errorf("can't fetch registration price: %w", err) return fmt.Errorf("can't fetch registration price: %w", err)
} }
w := io.NewBufBinWriter() w := io.NewBufBinWriter()
emit.AppCall(w.BinWriter, neoHash, "setRegisterPrice", callflag.States, 1) emit.AppCall(w.BinWriter, neo.Hash, "setRegisterPrice", callflag.States, 1)
for _, acc := range c.Accounts { for _, acc := range c.Accounts[start:end] {
emit.AppCall(w.BinWriter, neoHash, "registerCandidate", callflag.States, acc.PrivateKey().PublicKey().Bytes()) emit.AppCall(w.BinWriter, neo.Hash, "registerCandidate", callflag.States, acc.PrivateKey().PublicKey().Bytes())
emit.Opcodes(w.BinWriter, opcode.ASSERT) emit.Opcodes(w.BinWriter, opcode.ASSERT)
} }
emit.AppCall(w.BinWriter, neoHash, "setRegisterPrice", callflag.States, regPrice) emit.AppCall(w.BinWriter, neo.Hash, "setRegisterPrice", callflag.States, regPrice)
if w.Err != nil { if w.Err != nil {
panic(fmt.Sprintf("BUG: %v", w.Err)) panic(fmt.Sprintf("BUG: %v", w.Err))
} }
@ -53,14 +45,14 @@ func (c *initializeContext) registerCandidates() error {
Signer: c.getSigner(false, c.CommitteeAcc), Signer: c.getSigner(false, c.CommitteeAcc),
Account: c.CommitteeAcc, Account: c.CommitteeAcc,
}} }}
for i := range c.Accounts { for _, acc := range c.Accounts[start:end] {
signers = append(signers, rpcclient.SignerAccount{ signers = append(signers, rpcclient.SignerAccount{
Signer: transaction.Signer{ Signer: transaction.Signer{
Account: c.Accounts[i].Contract.ScriptHash(), Account: acc.Contract.ScriptHash(),
Scopes: transaction.CustomContracts, Scopes: transaction.CustomContracts,
AllowedContracts: []util.Uint160{neoHash}, AllowedContracts: []util.Uint160{neo.Hash},
}, },
Account: c.Accounts[i], Account: acc,
}) })
} }
@ -73,8 +65,8 @@ func (c *initializeContext) registerCandidates() error {
} }
network := c.CommitteeAct.GetNetwork() network := c.CommitteeAct.GetNetwork()
for i := range c.Accounts { for _, acc := range c.Accounts[start:end] {
if err := c.Accounts[i].SignTx(network, tx); err != nil { if err := acc.SignTx(network, tx); err != nil {
return fmt.Errorf("can't sign a transaction: %w", err) return fmt.Errorf("can't sign a transaction: %w", err)
} }
} }
@ -82,6 +74,39 @@ func (c *initializeContext) registerCandidates() error {
return c.sendTx(tx, c.Command, true) return c.sendTx(tx, c.Command, true)
} }
func (c *initializeContext) registerCandidates() error {
cc, err := unwrap.Array(c.ReadOnlyInvoker.Call(neo.Hash, "getCandidates"))
if err != nil {
return fmt.Errorf("`getCandidates`: %w", err)
}
need := len(c.Accounts)
have := len(cc)
if need == have {
c.Command.Println("Candidates are already registered.")
return nil
}
// Register candidates in batches in order to overcome the signers amount limit.
// See: https://github.com/nspcc-dev/neo-go/blob/master/pkg/core/transaction/transaction.go#L27
for i := 0; i < need; i += registerBatchSize {
start, end := i, i+registerBatchSize
if end > need {
end = need
}
// This check is sound because transactions are accepted/rejected atomically.
if have >= end {
continue
}
if err := c.registerCandidateRange(start, end); err != nil {
return fmt.Errorf("registering candidates %d..%d: %q", start, end-1, err)
}
}
return nil
}
func (c *initializeContext) transferNEOToAlphabetContracts() error { func (c *initializeContext) transferNEOToAlphabetContracts() error {
neoHash := neo.Hash neoHash := neo.Hash
@ -116,9 +141,11 @@ func (c *initializeContext) transferNEOFinished(neoHash util.Uint160) (bool, err
var errGetPriceInvalid = errors.New("`getRegisterPrice`: invalid response") var errGetPriceInvalid = errors.New("`getRegisterPrice`: invalid response")
func (c *initializeContext) getCandidateRegisterPrice() (int64, error) { func (c *initializeContext) getCandidateRegisterPrice() (int64, error) {
switch ct := c.Client.(type) { switch c.Client.(type) {
case *rpcclient.Client: case *rpcclient.Client:
return ct.GetCandidateRegisterPrice() inv := invoker.New(c.Client, nil)
reader := neo.NewReader(inv)
return reader.GetRegisterPrice()
default: default:
neoHash := neo.Hash neoHash := neo.Hash
res, err := invokeFunction(c.Client, neoHash, "getRegisterPrice", nil, nil) res, err := invokeFunction(c.Client, neoHash, "getRegisterPrice", nil, nil)

View file

@ -2,12 +2,14 @@ package morph
import ( import (
"encoding/hex" "encoding/hex"
"fmt"
"os" "os"
"path/filepath" "path/filepath"
"strconv" "strconv"
"testing" "testing"
"time"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/pkg/config" "github.com/nspcc-dev/neo-go/pkg/config"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/vm" "github.com/nspcc-dev/neo-go/pkg/vm"
@ -36,13 +38,22 @@ func TestInitialize(t *testing.T) {
t.Run("7 nodes", func(t *testing.T) { t.Run("7 nodes", func(t *testing.T) {
testInitialize(t, 7) testInitialize(t, 7)
}) })
t.Run("16 nodes", func(t *testing.T) {
testInitialize(t, 16)
})
t.Run("max nodes", func(t *testing.T) {
testInitialize(t, maxAlphabetNodes)
})
t.Run("too many nodes", func(t *testing.T) {
require.ErrorIs(t, generateTestData(t, t.TempDir(), maxAlphabetNodes+1), ErrTooManyAlphabetNodes)
})
} }
func testInitialize(t *testing.T, committeeSize int) { func testInitialize(t *testing.T, committeeSize int) {
testdataDir := t.TempDir() testdataDir := t.TempDir()
v := viper.GetViper() v := viper.GetViper()
generateTestData(t, testdataDir, committeeSize) require.NoError(t, generateTestData(t, testdataDir, committeeSize))
v.Set(protoConfigPath, filepath.Join(testdataDir, protoFileName)) v.Set(protoConfigPath, filepath.Join(testdataDir, protoFileName))
// Set to the path or remove the next statement to download from the network. // Set to the path or remove the next statement to download from the network.
@ -73,25 +84,33 @@ func testInitialize(t *testing.T, committeeSize int) {
}) })
} }
func generateTestData(t *testing.T, dir string, size int) { func generateTestData(t *testing.T, dir string, size int) error {
v := viper.GetViper() v := viper.GetViper()
v.Set(alphabetWalletsFlag, dir) v.Set(alphabetWalletsFlag, dir)
sizeStr := strconv.FormatUint(uint64(size), 10) sizeStr := strconv.FormatUint(uint64(size), 10)
require.NoError(t, generateAlphabetCmd.Flags().Set(alphabetSizeFlag, sizeStr)) if err := generateAlphabetCmd.Flags().Set(alphabetSizeFlag, sizeStr); err != nil {
return err
}
setTestCredentials(v, size) setTestCredentials(v, size)
require.NoError(t, generateAlphabetCreds(generateAlphabetCmd, nil)) if err := generateAlphabetCreds(generateAlphabetCmd, nil); err != nil {
return err
}
var pubs []string var pubs []string
for i := 0; i < size; i++ { for i := 0; i < size; i++ {
p := filepath.Join(dir, innerring.GlagoliticLetter(i).String()+".json") p := filepath.Join(dir, innerring.GlagoliticLetter(i).String()+".json")
w, err := wallet.NewWalletFromFile(p) w, err := wallet.NewWalletFromFile(p)
require.NoError(t, err, "wallet doesn't exist") if err != nil {
return fmt.Errorf("wallet doesn't exist: %w", err)
}
for _, acc := range w.Accounts { for _, acc := range w.Accounts {
if acc.Label == singleAccountName { if acc.Label == singleAccountName {
pub, ok := vm.ParseSignatureContract(acc.Contract.Script) pub, ok := vm.ParseSignatureContract(acc.Contract.Script)
require.True(t, ok) if !ok {
return fmt.Errorf("could not parse signature script for %s", acc.Address)
}
pubs = append(pubs, hex.EncodeToString(pub)) pubs = append(pubs, hex.EncodeToString(pub))
continue continue
} }
@ -100,17 +119,18 @@ func generateTestData(t *testing.T, dir string, size int) {
cfg := config.Config{} cfg := config.Config{}
cfg.ProtocolConfiguration.Magic = 12345 cfg.ProtocolConfiguration.Magic = 12345
cfg.ProtocolConfiguration.ValidatorsCount = size cfg.ProtocolConfiguration.ValidatorsCount = uint32(size)
cfg.ProtocolConfiguration.SecondsPerBlock = 1 cfg.ProtocolConfiguration.TimePerBlock = time.Second
cfg.ProtocolConfiguration.StandbyCommittee = pubs // sorted by glagolic letters cfg.ProtocolConfiguration.StandbyCommittee = pubs // sorted by glagolic letters
cfg.ProtocolConfiguration.P2PSigExtensions = true cfg.ProtocolConfiguration.P2PSigExtensions = true
cfg.ProtocolConfiguration.VerifyTransactions = true cfg.ProtocolConfiguration.VerifyTransactions = true
cfg.ProtocolConfiguration.VerifyBlocks = true
data, err := yaml.Marshal(cfg) data, err := yaml.Marshal(cfg)
require.NoError(t, err) if err != nil {
return err
}
protoPath := filepath.Join(dir, protoFileName) protoPath := filepath.Join(dir, protoFileName)
require.NoError(t, os.WriteFile(protoPath, data, os.ModePerm)) return os.WriteFile(protoPath, data, os.ModePerm)
} }
func setTestCredentials(v *viper.Viper, size int) { func setTestCredentials(v *viper.Viper, size int) {

View file

@ -1,65 +0,0 @@
package internal
import (
"fmt"
"strconv"
"google.golang.org/protobuf/proto"
)
// StringifySubnetClientGroupID returns string representation of SubnetClientGroupID using MarshalText.
// Returns a string with a message on error.
func StringifySubnetClientGroupID(id *SubnetClientGroupID) string {
text, err := id.MarshalText()
if err != nil {
return fmt.Sprintf("<invalid> %v", err)
}
return string(text)
}
// MarshalText encodes SubnetClientGroupID into text format according to FrostFS API V2 protocol:
// value in base-10 integer string format.
//
// It implements encoding.TextMarshaler.
func (x *SubnetClientGroupID) MarshalText() ([]byte, error) {
num := x.GetValue() // NPE safe, returns zero on nil
return []byte(strconv.FormatUint(uint64(num), 10)), nil
}
// UnmarshalText decodes the SubnetID from the text according to FrostFS API V2 protocol:
// should be base-10 integer string format with bitsize = 32.
//
// Returns strconv.ErrRange if integer overflows uint32.
//
// Must not be called on nil.
//
// Implements encoding.TextUnmarshaler.
func (x *SubnetClientGroupID) UnmarshalText(txt []byte) error {
num, err := strconv.ParseUint(string(txt), 10, 32)
if err != nil {
return fmt.Errorf("invalid numeric value: %w", err)
}
x.SetNumber(uint32(num))
return nil
}
// Marshal encodes the SubnetClientGroupID into a binary format of FrostFS API V2 protocol
// (Protocol Buffers with direct field order).
func (x *SubnetClientGroupID) Marshal() ([]byte, error) {
return proto.Marshal(x)
}
// Unmarshal decodes the SubnetClientGroupID from FrostFS API V2 binary format (see Marshal). Must not be called on nil.
func (x *SubnetClientGroupID) Unmarshal(data []byte) error {
return proto.Unmarshal(data, x)
}
// SetNumber sets SubnetClientGroupID value in uint32 format. Must not be called on nil.
// By default, number is 0.
func (x *SubnetClientGroupID) SetNumber(num uint32) {
x.Value = num
}

View file

@ -1,15 +0,0 @@
syntax = "proto3";
package neo.fs.v2.refs;
option go_package = "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/internal";
// Client group identifier in the FrostFS subnet.
//
// String representation of a value is base-10 integer.
//
// JSON representation is an object containing single `value` number field.
message SubnetClientGroupID {
// 4-byte integer identifier of the subnet client group.
fixed32 value = 1 [json_name = "value"];
}

View file

@ -51,7 +51,7 @@ type localClient struct {
maxGasInvoke int64 maxGasInvoke int64
} }
func newLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet) (*localClient, error) { func newLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet, dumpPath string) (*localClient, error) {
cfg, err := config.LoadFile(v.GetString(protoConfigPath)) cfg, err := config.LoadFile(v.GetString(protoConfigPath))
if err != nil { if err != nil {
return nil, err return nil, err
@ -62,7 +62,7 @@ func newLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet
return nil, err return nil, err
} }
m := smartcontract.GetDefaultHonestNodeCount(cfg.ProtocolConfiguration.ValidatorsCount) m := smartcontract.GetDefaultHonestNodeCount(int(cfg.ProtocolConfiguration.ValidatorsCount))
accounts := make([]*wallet.Account, len(wallets)) accounts := make([]*wallet.Account, len(wallets))
for i := range accounts { for i := range accounts {
accounts[i], err = getWalletAccount(wallets[i], consensusAccountName) accounts[i], err = getWalletAccount(wallets[i], consensusAccountName)
@ -87,7 +87,6 @@ func newLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet
go bc.Run() go bc.Run()
dumpPath := v.GetString(localDumpFlag)
if cmd.Name() != "init" { if cmd.Name() != "init" {
f, err := os.OpenFile(dumpPath, os.O_RDONLY, 0600) f, err := os.OpenFile(dumpPath, os.O_RDONLY, 0600)
if err != nil { if err != nil {
@ -248,7 +247,7 @@ func (l *localClient) GetVersion() (*result.Version, error) {
}, nil }, nil
} }
func (l *localClient) InvokeContractVerify(contract util.Uint160, params []smartcontract.Parameter, signers []transaction.Signer, witnesses ...transaction.Witness) (*result.Invoke, error) { func (l *localClient) InvokeContractVerify(util.Uint160, []smartcontract.Parameter, []transaction.Signer, ...transaction.Witness) (*result.Invoke, error) {
// not used by `morph init` command // not used by `morph init` command
panic("unexpected call") panic("unexpected call")
} }

View file

@ -1,9 +1,9 @@
package morph package morph
import ( import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/morph/client/netmap" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker" "github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"

View file

@ -0,0 +1,44 @@
package morph
import (
"errors"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/spf13/viper"
)
func getDefaultNetmapContractConfigMap() map[string]any {
m := make(map[string]any)
m[netmap.EpochDurationConfig] = viper.GetInt64(epochDurationInitFlag)
m[netmap.MaxObjectSizeConfig] = viper.GetInt64(maxObjectSizeInitFlag)
m[netmap.ContainerFeeConfig] = viper.GetInt64(containerFeeInitFlag)
m[netmap.ContainerAliasFeeConfig] = viper.GetInt64(containerAliasFeeInitFlag)
m[netmap.IrCandidateFeeConfig] = viper.GetInt64(candidateFeeInitFlag)
m[netmap.WithdrawFeeConfig] = viper.GetInt64(withdrawFeeInitFlag)
m[netmap.HomomorphicHashingDisabledKey] = viper.GetBool(homomorphicHashDisabledInitFlag)
m[netmap.MaintenanceModeAllowedConfig] = viper.GetBool(maintenanceModeAllowedInitFlag)
return m
}
func parseConfigFromNetmapContract(arr []stackitem.Item) (map[string][]byte, error) {
m := make(map[string][]byte, len(arr))
for _, param := range arr {
tuple, ok := param.Value().([]stackitem.Item)
if !ok || len(tuple) != 2 {
return nil, errors.New("invalid ListConfig response from netmap contract")
}
k, err := tuple[0].TryBytes()
if err != nil {
return nil, errors.New("invalid config key from netmap contract")
}
v, err := tuple[1].TryBytes()
if err != nil {
return nil, invalidConfigValueErr(string(k))
}
m[string(k)] = v
}
return m, nil
}

View file

@ -9,10 +9,12 @@ import (
"github.com/nspcc-dev/neo-go/pkg/core/transaction" "github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address" "github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor" "github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas" "github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/nep17" "github.com/nspcc-dev/neo-go/pkg/rpcclient/nep17"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/notary" "github.com/nspcc-dev/neo-go/pkg/rpcclient/notary"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/wallet" "github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
@ -23,16 +25,9 @@ import (
const defaultNotaryDepositLifetime = 5760 const defaultNotaryDepositLifetime = 5760
func depositNotary(cmd *cobra.Command, _ []string) error { func depositNotary(cmd *cobra.Command, _ []string) error {
p, err := cmd.Flags().GetString(storageWalletFlag) w, err := openWallet(cmd)
if err != nil { if err != nil {
return err return err
} else if p == "" {
return fmt.Errorf("missing wallet path (use '--%s <out.json>')", storageWalletFlag)
}
w, err := wallet.NewWalletFromFile(p)
if err != nil {
return fmt.Errorf("can't open wallet: %v", err)
} }
accHash := w.GetChangeAddress() accHash := w.GetChangeAddress()
@ -80,6 +75,10 @@ func depositNotary(cmd *cobra.Command, _ []string) error {
} }
} }
return transferGas(cmd, acc, accHash, gasAmount, till)
}
func transferGas(cmd *cobra.Command, acc *wallet.Account, accHash util.Uint160, gasAmount fixedn.Fixed8, till int64) error {
c, err := getN3Client(viper.GetViper()) c, err := getN3Client(viper.GetViper())
if err != nil { if err != nil {
return err return err
@ -119,3 +118,18 @@ func depositNotary(cmd *cobra.Command, _ []string) error {
return awaitTx(cmd, c, []hashVUBPair{{hash: txHash, vub: vub}}) return awaitTx(cmd, c, []hashVUBPair{{hash: txHash, vub: vub}})
} }
func openWallet(cmd *cobra.Command) (*wallet.Wallet, error) {
p, err := cmd.Flags().GetString(storageWalletFlag)
if err != nil {
return nil, err
} else if p == "" {
return nil, fmt.Errorf("missing wallet path (use '--%s <out.json>')", storageWalletFlag)
}
w, err := wallet.NewWalletFromFile(p)
if err != nil {
return nil, fmt.Errorf("can't open wallet: %v", err)
}
return w, nil
}

View file

@ -4,7 +4,7 @@ import (
"errors" "errors"
"fmt" "fmt"
netmapcontract "github.com/TrueCloudLab/frostfs-contract/netmap" netmapcontract "git.frostfs.info/TrueCloudLab/frostfs-contract/netmap"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/io" "github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag" "github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"

View file

@ -18,10 +18,6 @@ const (
maxObjectSizeCLIFlag = "max-object-size" maxObjectSizeCLIFlag = "max-object-size"
epochDurationInitFlag = "network.epoch_duration" epochDurationInitFlag = "network.epoch_duration"
epochDurationCLIFlag = "epoch-duration" epochDurationCLIFlag = "epoch-duration"
incomeRateInitFlag = "network.basic_income_rate"
incomeRateCLIFlag = "basic-income-rate"
auditFeeInitFlag = "network.fee.audit"
auditFeeCLIFlag = "audit-fee"
containerFeeInitFlag = "network.fee.container" containerFeeInitFlag = "network.fee.container"
containerAliasFeeInitFlag = "network.fee.container_alias" containerAliasFeeInitFlag = "network.fee.container_alias"
containerFeeCLIFlag = "container-fee" containerFeeCLIFlag = "container-fee"
@ -69,15 +65,12 @@ var (
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag)) _ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
_ = viper.BindPFlag(epochDurationInitFlag, cmd.Flags().Lookup(epochDurationCLIFlag)) _ = viper.BindPFlag(epochDurationInitFlag, cmd.Flags().Lookup(epochDurationCLIFlag))
_ = viper.BindPFlag(maxObjectSizeInitFlag, cmd.Flags().Lookup(maxObjectSizeCLIFlag)) _ = viper.BindPFlag(maxObjectSizeInitFlag, cmd.Flags().Lookup(maxObjectSizeCLIFlag))
_ = viper.BindPFlag(incomeRateInitFlag, cmd.Flags().Lookup(incomeRateCLIFlag))
_ = viper.BindPFlag(homomorphicHashDisabledInitFlag, cmd.Flags().Lookup(homomorphicHashDisabledCLIFlag)) _ = viper.BindPFlag(homomorphicHashDisabledInitFlag, cmd.Flags().Lookup(homomorphicHashDisabledCLIFlag))
_ = viper.BindPFlag(auditFeeInitFlag, cmd.Flags().Lookup(auditFeeCLIFlag))
_ = viper.BindPFlag(candidateFeeInitFlag, cmd.Flags().Lookup(candidateFeeCLIFlag)) _ = viper.BindPFlag(candidateFeeInitFlag, cmd.Flags().Lookup(candidateFeeCLIFlag))
_ = viper.BindPFlag(containerFeeInitFlag, cmd.Flags().Lookup(containerFeeCLIFlag)) _ = viper.BindPFlag(containerFeeInitFlag, cmd.Flags().Lookup(containerFeeCLIFlag))
_ = viper.BindPFlag(containerAliasFeeInitFlag, cmd.Flags().Lookup(containerAliasFeeCLIFlag)) _ = viper.BindPFlag(containerAliasFeeInitFlag, cmd.Flags().Lookup(containerAliasFeeCLIFlag))
_ = viper.BindPFlag(withdrawFeeInitFlag, cmd.Flags().Lookup(withdrawFeeCLIFlag)) _ = viper.BindPFlag(withdrawFeeInitFlag, cmd.Flags().Lookup(withdrawFeeCLIFlag))
_ = viper.BindPFlag(protoConfigPath, cmd.Flags().Lookup(protoConfigPath)) _ = viper.BindPFlag(protoConfigPath, cmd.Flags().Lookup(protoConfigPath))
_ = viper.BindPFlag(localDumpFlag, cmd.Flags().Lookup(localDumpFlag))
}, },
RunE: initializeSideChainCmd, RunE: initializeSideChainCmd,
} }
@ -239,14 +232,140 @@ var (
) )
func init() { func init() {
RootCmd.AddCommand(generateAlphabetCmd) initGenerateAlphabetCmd()
generateAlphabetCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir") initInitCmd()
generateAlphabetCmd.Flags().Uint(alphabetSizeFlag, 7, "Amount of alphabet wallets to generate") initDeployCmd()
initGenerateStorageCmd()
initForceNewEpochCmd()
initRemoveNodesCmd()
initSetPolicyCmd()
initDumpContractHashesCmd()
initDumpNetworkConfigCmd()
initSetConfigCmd()
initDumpBalancesCmd()
initUpdateContractsCmd()
initDumpContainersCmd()
initRestoreContainersCmd()
initListContainersCmd()
initRefillGasCmd()
initDepositoryNotaryCmd()
initNetmapCandidatesCmd()
}
func initNetmapCandidatesCmd() {
RootCmd.AddCommand(netmapCandidatesCmd)
netmapCandidatesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initDepositoryNotaryCmd() {
RootCmd.AddCommand(depositNotaryCmd)
depositNotaryCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
depositNotaryCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
depositNotaryCmd.Flags().String(walletAccountFlag, "", "Wallet account address")
depositNotaryCmd.Flags().String(refillGasAmountFlag, "", "Amount of GAS to deposit")
depositNotaryCmd.Flags().String(notaryDepositTillFlag, "", "Notary deposit duration in blocks")
}
func initRefillGasCmd() {
RootCmd.AddCommand(refillGasCmd)
refillGasCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
refillGasCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
refillGasCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
refillGasCmd.Flags().String(walletAddressFlag, "", "Address of wallet")
refillGasCmd.Flags().String(refillGasAmountFlag, "", "Additional amount of GAS to transfer")
refillGasCmd.MarkFlagsMutuallyExclusive(walletAddressFlag, storageWalletFlag)
}
func initListContainersCmd() {
RootCmd.AddCommand(listContainersCmd)
listContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
listContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
}
func initRestoreContainersCmd() {
RootCmd.AddCommand(restoreContainersCmd)
restoreContainersCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
restoreContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
restoreContainersCmd.Flags().String(containerDumpFlag, "", "File to restore containers from")
restoreContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to restore")
}
func initDumpContainersCmd() {
RootCmd.AddCommand(dumpContainersCmd)
dumpContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContainersCmd.Flags().String(containerDumpFlag, "", "File where to save dumped containers")
dumpContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
dumpContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to dump")
}
func initUpdateContractsCmd() {
RootCmd.AddCommand(updateContractsCmd)
updateContractsCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
updateContractsCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
updateContractsCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts")
_ = updateContractsCmd.MarkFlagRequired(contractsInitFlag)
}
func initDumpBalancesCmd() {
RootCmd.AddCommand(dumpBalancesCmd)
dumpBalancesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpBalancesCmd.Flags().BoolP(dumpBalancesStorageFlag, "s", false, "Dump balances of storage nodes from the current netmap")
dumpBalancesCmd.Flags().BoolP(dumpBalancesAlphabetFlag, "a", false, "Dump balances of alphabet contracts")
dumpBalancesCmd.Flags().BoolP(dumpBalancesProxyFlag, "p", false, "Dump balances of the proxy contract")
dumpBalancesCmd.Flags().Bool(dumpBalancesUseScriptHashFlag, false, "Use script-hash format for addresses")
}
func initSetConfigCmd() {
RootCmd.AddCommand(setConfig)
setConfig.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setConfig.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
setConfig.Flags().Bool(forceConfigSet, false, "Force setting not well-known configuration key")
}
func initDumpNetworkConfigCmd() {
RootCmd.AddCommand(dumpNetworkConfigCmd)
dumpNetworkConfigCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initDumpContractHashesCmd() {
RootCmd.AddCommand(dumpContractHashesCmd)
dumpContractHashesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContractHashesCmd.Flags().String(customZoneFlag, "", "Custom zone to search.")
}
func initSetPolicyCmd() {
RootCmd.AddCommand(setPolicy)
setPolicy.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setPolicy.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initRemoveNodesCmd() {
RootCmd.AddCommand(removeNodes)
removeNodes.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
removeNodes.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initForceNewEpochCmd() {
RootCmd.AddCommand(forceNewEpoch)
forceNewEpoch.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
forceNewEpoch.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initGenerateStorageCmd() {
RootCmd.AddCommand(generateStorageCmd)
generateStorageCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
generateStorageCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
generateStorageCmd.Flags().String(storageWalletFlag, "", "Path to new storage node wallet")
generateStorageCmd.Flags().String(storageGasCLIFlag, "", "Initial amount of GAS to transfer")
generateStorageCmd.Flags().StringP(storageWalletLabelFlag, "l", "", "Wallet label")
}
func initInitCmd() {
RootCmd.AddCommand(initCmd) RootCmd.AddCommand(initCmd)
initCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir") initCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
initCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint") initCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
initCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts (default fetched from latest github release)") initCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts")
_ = initCmd.MarkFlagRequired(contractsInitFlag)
initCmd.Flags().Uint(epochDurationCLIFlag, 240, "Amount of side chain blocks in one FrostFS epoch") initCmd.Flags().Uint(epochDurationCLIFlag, 240, "Amount of side chain blocks in one FrostFS epoch")
initCmd.Flags().Uint(maxObjectSizeCLIFlag, 67108864, "Max single object size in bytes") initCmd.Flags().Uint(maxObjectSizeCLIFlag, 67108864, "Max single object size in bytes")
initCmd.Flags().Bool(homomorphicHashDisabledCLIFlag, false, "Disable object homomorphic hashing") initCmd.Flags().Bool(homomorphicHashDisabledCLIFlag, false, "Disable object homomorphic hashing")
@ -255,85 +374,14 @@ func init() {
initCmd.Flags().Uint64(containerAliasFeeCLIFlag, 500, "Container alias fee") initCmd.Flags().Uint64(containerAliasFeeCLIFlag, 500, "Container alias fee")
initCmd.Flags().String(protoConfigPath, "", "Path to the consensus node configuration") initCmd.Flags().String(protoConfigPath, "", "Path to the consensus node configuration")
initCmd.Flags().String(localDumpFlag, "", "Path to the blocks dump file") initCmd.Flags().String(localDumpFlag, "", "Path to the blocks dump file")
}
RootCmd.AddCommand(deployCmd)
func initGenerateAlphabetCmd() {
RootCmd.AddCommand(generateStorageCmd) RootCmd.AddCommand(generateAlphabetCmd)
generateStorageCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir") generateAlphabetCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
generateStorageCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint") generateAlphabetCmd.Flags().Uint(alphabetSizeFlag, 7, "Amount of alphabet wallets to generate")
generateStorageCmd.Flags().String(storageWalletFlag, "", "Path to new storage node wallet") }
generateStorageCmd.Flags().String(storageGasCLIFlag, "", "Initial amount of GAS to transfer")
generateStorageCmd.Flags().StringP(storageWalletLabelFlag, "l", "", "Wallet label") func initDeployCmd() {
RootCmd.AddCommand(deployCmd)
RootCmd.AddCommand(forceNewEpoch)
forceNewEpoch.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
forceNewEpoch.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(removeNodes)
removeNodes.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
removeNodes.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(setPolicy)
setPolicy.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setPolicy.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(dumpContractHashesCmd)
dumpContractHashesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContractHashesCmd.Flags().String(customZoneFlag, "", "Custom zone to search.")
RootCmd.AddCommand(dumpNetworkConfigCmd)
dumpNetworkConfigCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(setConfig)
setConfig.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setConfig.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
setConfig.Flags().Bool(forceConfigSet, false, "Force setting not well-known configuration key")
RootCmd.AddCommand(dumpBalancesCmd)
dumpBalancesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpBalancesCmd.Flags().BoolP(dumpBalancesStorageFlag, "s", false, "Dump balances of storage nodes from the current netmap")
dumpBalancesCmd.Flags().BoolP(dumpBalancesAlphabetFlag, "a", false, "Dump balances of alphabet contracts")
dumpBalancesCmd.Flags().BoolP(dumpBalancesProxyFlag, "p", false, "Dump balances of the proxy contract")
dumpBalancesCmd.Flags().Bool(dumpBalancesUseScriptHashFlag, false, "Use script-hash format for addresses")
RootCmd.AddCommand(updateContractsCmd)
updateContractsCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
updateContractsCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
updateContractsCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts (default fetched from latest github release)")
RootCmd.AddCommand(dumpContainersCmd)
dumpContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContainersCmd.Flags().String(containerDumpFlag, "", "File where to save dumped containers")
dumpContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
dumpContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to dump")
RootCmd.AddCommand(restoreContainersCmd)
restoreContainersCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
restoreContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
restoreContainersCmd.Flags().String(containerDumpFlag, "", "File to restore containers from")
restoreContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to restore")
RootCmd.AddCommand(listContainersCmd)
listContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
listContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
RootCmd.AddCommand(refillGasCmd)
refillGasCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
refillGasCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
refillGasCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
refillGasCmd.Flags().String(walletAddressFlag, "", "Address of wallet")
refillGasCmd.Flags().String(refillGasAmountFlag, "", "Additional amount of GAS to transfer")
refillGasCmd.MarkFlagsMutuallyExclusive(walletAddressFlag, storageWalletFlag)
RootCmd.AddCommand(cmdSubnet)
RootCmd.AddCommand(depositNotaryCmd)
depositNotaryCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
depositNotaryCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
depositNotaryCmd.Flags().String(walletAccountFlag, "", "Wallet account address")
depositNotaryCmd.Flags().String(refillGasAmountFlag, "", "Amount of GAS to deposit")
depositNotaryCmd.Flags().String(notaryDepositTillFlag, "", "Notary deposit duration in blocks")
RootCmd.AddCommand(netmapCandidatesCmd)
netmapCandidatesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
} }

File diff suppressed because it is too large Load diff

View file

@ -3,14 +3,14 @@ package modules
import ( import (
"os" "os"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/storagecfg" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/storagecfg"
"github.com/TrueCloudLab/frostfs-node/misc" "git.frostfs.info/TrueCloudLab/frostfs-node/misc"
"github.com/TrueCloudLab/frostfs-node/pkg/util/autocomplete" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/autocomplete"
utilConfig "github.com/TrueCloudLab/frostfs-node/pkg/util/config" utilConfig "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/config"
"github.com/TrueCloudLab/frostfs-node/pkg/util/gendoc" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/gendoc"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
) )
@ -45,14 +45,14 @@ func init() {
rootCmd.AddCommand(storagecfg.RootCmd) rootCmd.AddCommand(storagecfg.RootCmd)
rootCmd.AddCommand(autocomplete.Command("frostfs-adm")) rootCmd.AddCommand(autocomplete.Command("frostfs-adm"))
rootCmd.AddCommand(gendoc.Command(rootCmd)) rootCmd.AddCommand(gendoc.Command(rootCmd, gendoc.Options{}))
} }
func Execute() error { func Execute() error {
return rootCmd.Execute() return rootCmd.Execute()
} }
func entryPoint(cmd *cobra.Command, args []string) error { func entryPoint(cmd *cobra.Command, _ []string) error {
printVersion, _ := cmd.Flags().GetBool("version") printVersion, _ := cmd.Flags().GetBool("version")
if printVersion { if printVersion {
cmd.Print(misc.BuildInfo("FrostFS Adm")) cmd.Print(misc.BuildInfo("FrostFS Adm"))

View file

@ -12,9 +12,6 @@ node:
- {{ .AnnouncedAddress }} - {{ .AnnouncedAddress }}
attribute_0: UN-LOCODE:{{ .Attribute.Locode }} attribute_0: UN-LOCODE:{{ .Attribute.Locode }}
relay: {{ .Relay }} # start Storage node in relay mode without bootstrapping into the Network map relay: {{ .Relay }} # start Storage node in relay mode without bootstrapping into the Network map
subnet:
exit_zero: false # toggle entrance to zero subnet (overrides corresponding attribute and occurrence in entries)
entries: [] # list of IDs of subnets to enter in a text format of FrostFS API protocol (overrides corresponding attributes)
grpc: grpc:
num: 1 # total number of listener endpoints num: 1 # total number of listener endpoints

View file

@ -16,7 +16,7 @@ import (
"text/template" "text/template"
"time" "time"
netutil "github.com/TrueCloudLab/frostfs-node/pkg/network" netutil "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network"
"github.com/chzyer/readline" "github.com/chzyer/readline"
"github.com/nspcc-dev/neo-go/cli/flags" "github.com/nspcc-dev/neo-go/cli/flags"
"github.com/nspcc-dev/neo-go/cli/input" "github.com/nspcc-dev/neo-go/cli/input"
@ -80,15 +80,7 @@ type config struct {
} }
func storageConfig(cmd *cobra.Command, args []string) { func storageConfig(cmd *cobra.Command, args []string) {
var outPath string outPath := getOutputPath(args)
if len(args) != 0 {
outPath = args[0]
} else {
outPath = getPath("File to write config at [./config.yml]: ")
if outPath == "" {
outPath = "./config.yml"
}
}
historyPath := filepath.Join(os.TempDir(), "frostfs-adm.history") historyPath := filepath.Join(os.TempDir(), "frostfs-adm.history")
readline.SetHistoryPath(historyPath) readline.SetHistoryPath(historyPath)
@ -103,14 +95,7 @@ func storageConfig(cmd *cobra.Command, args []string) {
w, err := wallet.NewWalletFromFile(c.Wallet.Path) w, err := wallet.NewWalletFromFile(c.Wallet.Path)
fatalOnErr(err) fatalOnErr(err)
c.Wallet.Account, _ = cmd.Flags().GetString(accountFlag) fillWalletAccount(cmd, &c, w)
if c.Wallet.Account == "" {
addr := address.Uint160ToString(w.GetChangeAddress())
c.Wallet.Account = getWalletAccount(w, fmt.Sprintf("Wallet account [%s]: ", addr))
if c.Wallet.Account == "" {
c.Wallet.Account = addr
}
}
accH, err := flags.ParseAddress(c.Wallet.Account) accH, err := flags.ParseAddress(c.Wallet.Account)
fatalOnErr(err) fatalOnErr(err)
@ -128,32 +113,51 @@ func storageConfig(cmd *cobra.Command, args []string) {
c.AuthorizedKeys = append(c.AuthorizedKeys, hex.EncodeToString(acc.PrivateKey().PublicKey().Bytes())) c.AuthorizedKeys = append(c.AuthorizedKeys, hex.EncodeToString(acc.PrivateKey().PublicKey().Bytes()))
var network string network := readNetwork(cmd)
for {
network = getString("Choose network [mainnet]/testnet: ")
switch network {
case "":
network = "mainnet"
case "testnet", "mainnet":
default:
cmd.Println(`Network must be either "mainnet" or "testnet"`)
continue
}
break
}
c.MorphRPC = n3config[network].MorphRPC c.MorphRPC = n3config[network].MorphRPC
depositGas(cmd, acc, network) depositGas(cmd, acc, network)
c.Attribute.Locode = getString("UN-LOCODE attribute in [XX YYY] format: ") c.Attribute.Locode = getString("UN-LOCODE attribute in [XX YYY] format: ")
endpoint := getDefaultEndpoint(cmd, &c)
c.Endpoint = getString(fmt.Sprintf("Listening address [%s]: ", endpoint))
if c.Endpoint == "" {
c.Endpoint = endpoint
}
c.ControlEndpoint = getString(fmt.Sprintf("Listening address (control endpoint) [%s]: ", defaultControlEndpoint))
if c.ControlEndpoint == "" {
c.ControlEndpoint = defaultControlEndpoint
}
c.TLSCert = getPath("TLS Certificate (optional): ")
if c.TLSCert != "" {
c.TLSKey = getPath("TLS Key: ")
}
c.Relay = getConfirmation(false, "Use node as a relay? yes/[no]: ")
if !c.Relay {
p := getPath("Path to the storage directory (all available storage will be used): ")
c.BlobstorPath = filepath.Join(p, "blob")
c.MetabasePath = filepath.Join(p, "meta")
}
out := applyTemplate(c)
fatalOnErr(os.WriteFile(outPath, out, 0644))
cmd.Println("Node is ready for work! Run `frostfs-node -config " + outPath + "`")
}
func getDefaultEndpoint(cmd *cobra.Command, c *config) string {
var addr, port string var addr, port string
for { for {
c.AnnouncedAddress = getString("Publicly announced address: ") c.AnnouncedAddress = getString("Publicly announced address: ")
validator := netutil.Address{} validator := netutil.Address{}
err := validator.FromString(c.AnnouncedAddress) err := validator.FromString(c.AnnouncedAddress)
if err != nil { if err != nil {
cmd.Println("Incorrect address format. See https://github.com/TrueCloudLab/frostfs-node/blob/master/pkg/network/address.go for details.") cmd.Println("Incorrect address format. See https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/network/address.go for details.")
continue continue
} }
uriAddr, err := url.Parse(validator.URIAddr()) uriAddr, err := url.Parse(validator.URIAddr())
@ -182,34 +186,46 @@ func storageConfig(cmd *cobra.Command, args []string) {
break break
} }
return net.JoinHostPort(defaultDataEndpoint, port)
}
defaultAddr := net.JoinHostPort(defaultDataEndpoint, port) func fillWalletAccount(cmd *cobra.Command, c *config, w *wallet.Wallet) {
c.Endpoint = getString(fmt.Sprintf("Listening address [%s]: ", defaultAddr)) c.Wallet.Account, _ = cmd.Flags().GetString(accountFlag)
if c.Endpoint == "" { if c.Wallet.Account == "" {
c.Endpoint = defaultAddr addr := address.Uint160ToString(w.GetChangeAddress())
c.Wallet.Account = getWalletAccount(w, fmt.Sprintf("Wallet account [%s]: ", addr))
if c.Wallet.Account == "" {
c.Wallet.Account = addr
}
} }
}
c.ControlEndpoint = getString(fmt.Sprintf("Listening address (control endpoint) [%s]: ", defaultControlEndpoint)) func readNetwork(cmd *cobra.Command) string {
if c.ControlEndpoint == "" { var network string
c.ControlEndpoint = defaultControlEndpoint for {
network = getString("Choose network [mainnet]/testnet: ")
switch network {
case "":
network = "mainnet"
case "testnet", "mainnet":
default:
cmd.Println(`Network must be either "mainnet" or "testnet"`)
continue
}
break
} }
return network
}
c.TLSCert = getPath("TLS Certificate (optional): ") func getOutputPath(args []string) string {
if c.TLSCert != "" { if len(args) != 0 {
c.TLSKey = getPath("TLS Key: ") return args[0]
} }
outPath := getPath("File to write config at [./config.yml]: ")
c.Relay = getConfirmation(false, "Use node as a relay? yes/[no]: ") if outPath == "" {
if !c.Relay { outPath = "./config.yml"
p := getPath("Path to the storage directory (all available storage will be used): ")
c.BlobstorPath = filepath.Join(p, "blob")
c.MetabasePath = filepath.Join(p, "meta")
} }
return outPath
out := applyTemplate(c)
fatalOnErr(os.WriteFile(outPath, out, 0644))
cmd.Println("Node is ready for work! Run `frostfs-node -config " + outPath + "`")
} }
func getWalletAccount(w *wallet.Wallet, prompt string) string { func getWalletAccount(w *wallet.Wallet, prompt string) string {

View file

@ -3,7 +3,7 @@ package main
import ( import (
"os" "os"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules"
) )
func main() { func main() {

View file

@ -9,16 +9,16 @@ duplicated header names or headers with empty values are considered invalid.
## Existing headers ## Existing headers
There are some "well-known" headers starting with `__FROSTFS__` prefix that There are some "well-known" headers starting with `__SYSTEM__` prefix that
affect system behaviour. For backward compatibility, the same set of affect system behaviour. For backward compatibility, the same set of
"well-known" headers may also use `__NEOFS__` prefix: "well-known" headers may also use `__NEOFS__` prefix:
* `__FROSTFS__NETMAP_EPOCH` - netmap epoch to use for object placement calculation. The `value` is string * `__SYSTEM__NETMAP_EPOCH` - netmap epoch to use for object placement calculation. The `value` is string
encoded `uint64` in decimal presentation. If set to '0' or omitted, the encoded `uint64` in decimal presentation. If set to '0' or omitted, the
current epoch only will be used. current epoch only will be used.
* `__FROSTFS__NETMAP_LOOKUP_DEPTH` - if object can't be found using current epoch's netmap, this header limits * `__SYSTEM__NETMAP_LOOKUP_DEPTH` - if object can't be found using current epoch's netmap, this header limits
how many past epochs the node can look up through. Depth is applied to a current epoch or the value how many past epochs the node can look up through. Depth is applied to a current epoch or the value
of `__FROSTFS__NETMAP_EPOCH` attribute. The `value` is string encoded `uint64` in decimal presentation. of `__SYSTEM__NETMAP_EPOCH` attribute. The `value` is string encoded `uint64` in decimal presentation.
If set to '0' or not set, only the current epoch is used. If set to '0' or not set, only the current epoch is used.
## `frostfs-cli` commands with `--xhdr` ## `frostfs-cli` commands with `--xhdr`
@ -26,9 +26,8 @@ If set to '0' or not set, only the current epoch is used.
List of commands with support of extended headers: List of commands with support of extended headers:
* `container list-objects` * `container list-objects`
* `object delete/get/hash/head/lock/put/range/search` * `object delete/get/hash/head/lock/put/range/search`
* `storagegroup delete/get/list/put`
Example: Example:
```shell ```shell
$ frostfs-cli object put -r s01.frostfs.devenv:8080 -w wallet.json --cid CID --file FILE --xhdr "__FROSTFS__NETMAP_EPOCH=777" $ frostfs-cli object put -r s01.frostfs.devenv:8080 -w wallet.json --cid CID --file FILE --xhdr "__SYSTEM__NETMAP_EPOCH=777"
``` ```

View file

@ -6,16 +6,19 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"sort"
"strings"
"github.com/TrueCloudLab/frostfs-sdk-go/accounting" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/accounting"
"github.com/TrueCloudLab/frostfs-sdk-go/client" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/checksum"
containerSDK "github.com/TrueCloudLab/frostfs-sdk-go/container" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id" containerSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/TrueCloudLab/frostfs-sdk-go/netmap" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/TrueCloudLab/frostfs-sdk-go/object" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id" objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
"github.com/TrueCloudLab/frostfs-sdk-go/version" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/version"
) )
// BalanceOfPrm groups parameters of BalanceOf operation. // BalanceOfPrm groups parameters of BalanceOf operation.
@ -37,8 +40,8 @@ func (x BalanceOfRes) Balance() accounting.Decimal {
// BalanceOf requests the current balance of a FrostFS user. // BalanceOf requests the current balance of a FrostFS user.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func BalanceOf(prm BalanceOfPrm) (res BalanceOfRes, err error) { func BalanceOf(ctx context.Context, prm BalanceOfPrm) (res BalanceOfRes, err error) {
res.cliRes, err = prm.cli.BalanceGet(context.Background(), prm.PrmBalanceGet) res.cliRes, err = prm.cli.BalanceGet(ctx, prm.PrmBalanceGet)
return return
} }
@ -62,16 +65,26 @@ func (x ListContainersRes) IDList() []cid.ID {
// ListContainers requests a list of FrostFS user's containers. // ListContainers requests a list of FrostFS user's containers.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func ListContainers(prm ListContainersPrm) (res ListContainersRes, err error) { func ListContainers(ctx context.Context, prm ListContainersPrm) (res ListContainersRes, err error) {
res.cliRes, err = prm.cli.ContainerList(context.Background(), prm.PrmContainerList) res.cliRes, err = prm.cli.ContainerList(ctx, prm.PrmContainerList)
return return
} }
// SortedIDList returns sorted list of identifiers of user's containers.
func (x ListContainersRes) SortedIDList() []cid.ID {
list := x.cliRes.Containers()
sort.Slice(list, func(i, j int) bool {
lhs, rhs := list[i].EncodeToString(), list[j].EncodeToString()
return strings.Compare(lhs, rhs) < 0
})
return list
}
// PutContainerPrm groups parameters of PutContainer operation. // PutContainerPrm groups parameters of PutContainer operation.
type PutContainerPrm struct { type PutContainerPrm struct {
commonPrm Client *client.Client
client.PrmContainerPut ClientParams client.PrmContainerPut
} }
// PutContainerRes groups the resulting values of PutContainer operation. // PutContainerRes groups the resulting values of PutContainer operation.
@ -92,8 +105,8 @@ func (x PutContainerRes) ID() cid.ID {
// Success can be verified by reading by identifier. // Success can be verified by reading by identifier.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func PutContainer(prm PutContainerPrm) (res PutContainerRes, err error) { func PutContainer(ctx context.Context, prm PutContainerPrm) (res PutContainerRes, err error) {
cliRes, err := prm.cli.ContainerPut(context.Background(), prm.PrmContainerPut) cliRes, err := prm.Client.ContainerPut(ctx, prm.ClientParams)
if err == nil { if err == nil {
res.cnr = cliRes.ID() res.cnr = cliRes.ID()
} }
@ -103,13 +116,15 @@ func PutContainer(prm PutContainerPrm) (res PutContainerRes, err error) {
// GetContainerPrm groups parameters of GetContainer operation. // GetContainerPrm groups parameters of GetContainer operation.
type GetContainerPrm struct { type GetContainerPrm struct {
commonPrm Client *client.Client
cliPrm client.PrmContainerGet ClientParams client.PrmContainerGet
} }
// SetContainer sets identifier of the container to be read. // SetContainer sets identifier of the container to be read.
//
// Deprecated: Use GetContainerPrm.ClientParams.ContainerID instead.
func (x *GetContainerPrm) SetContainer(id cid.ID) { func (x *GetContainerPrm) SetContainer(id cid.ID) {
x.cliPrm.SetContainer(id) x.ClientParams.ContainerID = &id
} }
// GetContainerRes groups the resulting values of GetContainer operation. // GetContainerRes groups the resulting values of GetContainer operation.
@ -125,20 +140,23 @@ func (x GetContainerRes) Container() containerSDK.Container {
// GetContainer reads a container from FrostFS by ID. // GetContainer reads a container from FrostFS by ID.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func GetContainer(prm GetContainerPrm) (res GetContainerRes, err error) { func GetContainer(ctx context.Context, prm GetContainerPrm) (res GetContainerRes, err error) {
res.cliRes, err = prm.cli.ContainerGet(context.Background(), prm.cliPrm) res.cliRes, err = prm.Client.ContainerGet(ctx, prm.ClientParams)
return return
} }
// IsACLExtendable checks if ACL of the container referenced by the given identifier // IsACLExtendable checks if ACL of the container referenced by the given identifier
// can be extended. Client connection MUST BE correctly established in advance. // can be extended. Client connection MUST BE correctly established in advance.
func IsACLExtendable(c *client.Client, cnr cid.ID) (bool, error) { func IsACLExtendable(ctx context.Context, c *client.Client, cnr cid.ID) (bool, error) {
var prm GetContainerPrm prm := GetContainerPrm{
prm.SetClient(c) Client: c,
prm.SetContainer(cnr) ClientParams: client.PrmContainerGet{
ContainerID: &cnr,
},
}
res, err := GetContainer(prm) res, err := GetContainer(ctx, prm)
if err != nil { if err != nil {
return false, fmt.Errorf("get container from the FrostFS: %w", err) return false, fmt.Errorf("get container from the FrostFS: %w", err)
} }
@ -148,8 +166,8 @@ func IsACLExtendable(c *client.Client, cnr cid.ID) (bool, error) {
// DeleteContainerPrm groups parameters of DeleteContainerPrm operation. // DeleteContainerPrm groups parameters of DeleteContainerPrm operation.
type DeleteContainerPrm struct { type DeleteContainerPrm struct {
commonPrm Client *client.Client
client.PrmContainerDelete ClientParams client.PrmContainerDelete
} }
// DeleteContainerRes groups the resulting values of DeleteContainer operation. // DeleteContainerRes groups the resulting values of DeleteContainer operation.
@ -163,16 +181,16 @@ type DeleteContainerRes struct{}
// Success can be verified by reading by identifier. // Success can be verified by reading by identifier.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func DeleteContainer(prm DeleteContainerPrm) (res DeleteContainerRes, err error) { func DeleteContainer(ctx context.Context, prm DeleteContainerPrm) (res DeleteContainerRes, err error) {
_, err = prm.cli.ContainerDelete(context.Background(), prm.PrmContainerDelete) _, err = prm.Client.ContainerDelete(ctx, prm.ClientParams)
return return
} }
// EACLPrm groups parameters of EACL operation. // EACLPrm groups parameters of EACL operation.
type EACLPrm struct { type EACLPrm struct {
commonPrm Client *client.Client
client.PrmContainerEACL ClientParams client.PrmContainerEACL
} }
// EACLRes groups the resulting values of EACL operation. // EACLRes groups the resulting values of EACL operation.
@ -188,16 +206,16 @@ func (x EACLRes) EACL() eacl.Table {
// EACL reads eACL table from FrostFS by container ID. // EACL reads eACL table from FrostFS by container ID.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func EACL(prm EACLPrm) (res EACLRes, err error) { func EACL(ctx context.Context, prm EACLPrm) (res EACLRes, err error) {
res.cliRes, err = prm.cli.ContainerEACL(context.Background(), prm.PrmContainerEACL) res.cliRes, err = prm.Client.ContainerEACL(ctx, prm.ClientParams)
return return
} }
// SetEACLPrm groups parameters of SetEACL operation. // SetEACLPrm groups parameters of SetEACL operation.
type SetEACLPrm struct { type SetEACLPrm struct {
commonPrm Client *client.Client
client.PrmContainerSetEACL ClientParams client.PrmContainerSetEACL
} }
// SetEACLRes groups the resulting values of SetEACL operation. // SetEACLRes groups the resulting values of SetEACL operation.
@ -211,16 +229,16 @@ type SetEACLRes struct{}
// Success can be verified by reading by container identifier. // Success can be verified by reading by container identifier.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func SetEACL(prm SetEACLPrm) (res SetEACLRes, err error) { func SetEACL(ctx context.Context, prm SetEACLPrm) (res SetEACLRes, err error) {
_, err = prm.cli.ContainerSetEACL(context.Background(), prm.PrmContainerSetEACL) _, err = prm.Client.ContainerSetEACL(ctx, prm.ClientParams)
return return
} }
// NetworkInfoPrm groups parameters of NetworkInfo operation. // NetworkInfoPrm groups parameters of NetworkInfo operation.
type NetworkInfoPrm struct { type NetworkInfoPrm struct {
commonPrm Client *client.Client
client.PrmNetworkInfo ClientParams client.PrmNetworkInfo
} }
// NetworkInfoRes groups the resulting values of NetworkInfo operation. // NetworkInfoRes groups the resulting values of NetworkInfo operation.
@ -236,16 +254,16 @@ func (x NetworkInfoRes) NetworkInfo() netmap.NetworkInfo {
// NetworkInfo reads information about the FrostFS network. // NetworkInfo reads information about the FrostFS network.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func NetworkInfo(prm NetworkInfoPrm) (res NetworkInfoRes, err error) { func NetworkInfo(ctx context.Context, prm NetworkInfoPrm) (res NetworkInfoRes, err error) {
res.cliRes, err = prm.cli.NetworkInfo(context.Background(), prm.PrmNetworkInfo) res.cliRes, err = prm.Client.NetworkInfo(ctx, prm.ClientParams)
return return
} }
// NodeInfoPrm groups parameters of NodeInfo operation. // NodeInfoPrm groups parameters of NodeInfo operation.
type NodeInfoPrm struct { type NodeInfoPrm struct {
commonPrm Client *client.Client
client.PrmEndpointInfo ClientParams client.PrmEndpointInfo
} }
// NodeInfoRes groups the resulting values of NodeInfo operation. // NodeInfoRes groups the resulting values of NodeInfo operation.
@ -266,8 +284,8 @@ func (x NodeInfoRes) LatestVersion() version.Version {
// NodeInfo requests information about the remote server from FrostFS netmap. // NodeInfo requests information about the remote server from FrostFS netmap.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func NodeInfo(prm NodeInfoPrm) (res NodeInfoRes, err error) { func NodeInfo(ctx context.Context, prm NodeInfoPrm) (res NodeInfoRes, err error) {
res.cliRes, err = prm.cli.EndpointInfo(context.Background(), prm.PrmEndpointInfo) res.cliRes, err = prm.Client.EndpointInfo(ctx, prm.ClientParams)
return return
} }
@ -290,8 +308,8 @@ func (x NetMapSnapshotRes) NetMap() netmap.NetMap {
// NetMapSnapshot requests current network view of the remote server. // NetMapSnapshot requests current network view of the remote server.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func NetMapSnapshot(prm NetMapSnapshotPrm) (res NetMapSnapshotRes, err error) { func NetMapSnapshot(ctx context.Context, prm NetMapSnapshotPrm) (res NetMapSnapshotRes, err error) {
res.cliRes, err = prm.cli.NetMapSnapshot(context.Background(), client.PrmNetMapSnapshot{}) res.cliRes, err = prm.cli.NetMapSnapshot(ctx, client.PrmNetMapSnapshot{})
return return
} }
@ -319,8 +337,8 @@ func (x CreateSessionRes) SessionKey() []byte {
// CreateSession opens a new unlimited session with the remote node. // CreateSession opens a new unlimited session with the remote node.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func CreateSession(prm CreateSessionPrm) (res CreateSessionRes, err error) { func CreateSession(ctx context.Context, prm CreateSessionPrm) (res CreateSessionRes, err error) {
res.cliRes, err = prm.cli.SessionCreate(context.Background(), prm.PrmSessionCreate) res.cliRes, err = prm.cli.SessionCreate(ctx, prm.PrmSessionCreate)
return return
} }
@ -329,15 +347,19 @@ func CreateSession(prm CreateSessionPrm) (res CreateSessionRes, err error) {
type PutObjectPrm struct { type PutObjectPrm struct {
commonObjectPrm commonObjectPrm
hdr *object.Object copyNum []uint32
hdr *objectSDK.Object
rdr io.Reader rdr io.Reader
headerCallback func(*object.Object) headerCallback func(*objectSDK.Object)
prepareLocally bool
} }
// SetHeader sets object header. // SetHeader sets object header.
func (x *PutObjectPrm) SetHeader(hdr *object.Object) { func (x *PutObjectPrm) SetHeader(hdr *objectSDK.Object) {
x.hdr = hdr x.hdr = hdr
} }
@ -348,10 +370,51 @@ func (x *PutObjectPrm) SetPayloadReader(rdr io.Reader) {
// SetHeaderCallback sets callback which is called on the object after the header is received // SetHeaderCallback sets callback which is called on the object after the header is received
// but before the payload is written. // but before the payload is written.
func (x *PutObjectPrm) SetHeaderCallback(f func(*object.Object)) { func (x *PutObjectPrm) SetHeaderCallback(f func(*objectSDK.Object)) {
x.headerCallback = f x.headerCallback = f
} }
// SetCopiesNumberByVectors sets ordered list of minimal required object copies numbers
// per placement vector.
func (x *PutObjectPrm) SetCopiesNumberByVectors(copiesNumbers []uint32) {
x.copyNum = copiesNumbers
}
// PrepareLocally generate object header on the client side.
// For big object - split locally too.
func (x *PutObjectPrm) PrepareLocally() {
x.prepareLocally = true
}
func (x *PutObjectPrm) convertToSDKPrm(ctx context.Context) (client.PrmObjectPutInit, error) {
var putPrm client.PrmObjectPutInit
if !x.prepareLocally && x.sessionToken != nil {
putPrm.WithinSession(*x.sessionToken)
}
if x.bearerToken != nil {
putPrm.WithBearerToken(*x.bearerToken)
}
if x.local {
putPrm.MarkLocal()
}
putPrm.WithXHeaders(x.xHeaders...)
putPrm.SetCopiesNumberByVectors(x.copyNum)
if x.prepareLocally {
res, err := x.cli.NetworkInfo(ctx, client.PrmNetworkInfo{})
if err != nil {
return client.PrmObjectPutInit{}, err
}
putPrm.WithObjectMaxSize(res.Info().MaxObjectSize())
putPrm.WithEpochSource(epochSource(res.Info().CurrentEpoch()))
putPrm.WithoutHomomorphicHash(res.Info().HomomorphicHashingDisabled())
}
return putPrm, nil
}
// PutObjectRes groups the resulting values of PutObject operation. // PutObjectRes groups the resulting values of PutObject operation.
type PutObjectRes struct { type PutObjectRes struct {
id oid.ID id oid.ID
@ -362,32 +425,26 @@ func (x PutObjectRes) ID() oid.ID {
return x.id return x.id
} }
type epochSource uint64
func (s epochSource) CurrentEpoch() uint64 {
return uint64(s)
}
// PutObject saves the object in FrostFS network. // PutObject saves the object in FrostFS network.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func PutObject(prm PutObjectPrm) (*PutObjectRes, error) { func PutObject(ctx context.Context, prm PutObjectPrm) (*PutObjectRes, error) {
var putPrm client.PrmObjectPutInit sdkPrm, err := prm.convertToSDKPrm(ctx)
if err != nil {
if prm.sessionToken != nil { return nil, fmt.Errorf("unable to create parameters of object put operation: %w", err)
putPrm.WithinSession(*prm.sessionToken)
} }
wrt, err := prm.cli.ObjectPutInit(ctx, sdkPrm)
if prm.bearerToken != nil {
putPrm.WithBearerToken(*prm.bearerToken)
}
if prm.local {
putPrm.MarkLocal()
}
putPrm.WithXHeaders(prm.xHeaders...)
wrt, err := prm.cli.ObjectPutInit(context.Background(), putPrm)
if err != nil { if err != nil {
return nil, fmt.Errorf("init object writing: %w", err) return nil, fmt.Errorf("init object writing: %w", err)
} }
if wrt.WriteHeader(*prm.hdr) { if wrt.WriteHeader(ctx, *prm.hdr) {
if prm.headerCallback != nil { if prm.headerCallback != nil {
prm.headerCallback(prm.hdr) prm.headerCallback(prm.hdr)
} }
@ -417,7 +474,7 @@ func PutObject(prm PutObjectPrm) (*PutObjectRes, error) {
for { for {
n, err = prm.rdr.Read(buf) n, err = prm.rdr.Read(buf)
if n > 0 { if n > 0 {
if !wrt.WritePayloadChunk(buf[:n]) { if !wrt.WritePayloadChunk(ctx, buf[:n]) {
break break
} }
@ -433,7 +490,7 @@ func PutObject(prm PutObjectPrm) (*PutObjectRes, error) {
} }
} }
cliRes, err := wrt.Close() cliRes, err := wrt.Close(ctx)
if err != nil { // here err already carries both status and client errors if err != nil { // here err already carries both status and client errors
return nil, fmt.Errorf("client failure: %w", err) return nil, fmt.Errorf("client failure: %w", err)
} }
@ -462,22 +519,19 @@ func (x DeleteObjectRes) Tombstone() oid.ID {
// DeleteObject marks an object to be removed from FrostFS through tombstone placement. // DeleteObject marks an object to be removed from FrostFS through tombstone placement.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func DeleteObject(prm DeleteObjectPrm) (*DeleteObjectRes, error) { func DeleteObject(ctx context.Context, prm DeleteObjectPrm) (*DeleteObjectRes, error) {
var delPrm client.PrmObjectDelete cnr := prm.objAddr.Container()
delPrm.FromContainer(prm.objAddr.Container()) obj := prm.objAddr.Object()
delPrm.ByID(prm.objAddr.Object())
if prm.sessionToken != nil { delPrm := client.PrmObjectDelete{
delPrm.WithinSession(*prm.sessionToken) XHeaders: prm.xHeaders,
ContainerID: &cnr,
ObjectID: &obj,
Session: prm.sessionToken,
BearerToken: prm.bearerToken,
} }
if prm.bearerToken != nil { cliRes, err := prm.cli.ObjectDelete(ctx, delPrm)
delPrm.WithBearerToken(*prm.bearerToken)
}
delPrm.WithXHeaders(prm.xHeaders...)
cliRes, err := prm.cli.ObjectDelete(context.Background(), delPrm)
if err != nil { if err != nil {
return nil, fmt.Errorf("remove object via client: %w", err) return nil, fmt.Errorf("remove object via client: %w", err)
} }
@ -493,22 +547,22 @@ type GetObjectPrm struct {
objectAddressPrm objectAddressPrm
rawPrm rawPrm
payloadWriterPrm payloadWriterPrm
headerCallback func(*object.Object) headerCallback func(*objectSDK.Object)
} }
// SetHeaderCallback sets callback which is called on the object after the header is received // SetHeaderCallback sets callback which is called on the object after the header is received
// but before the payload is written. // but before the payload is written.
func (p *GetObjectPrm) SetHeaderCallback(f func(*object.Object)) { func (p *GetObjectPrm) SetHeaderCallback(f func(*objectSDK.Object)) {
p.headerCallback = f p.headerCallback = f
} }
// GetObjectRes groups the resulting values of GetObject operation. // GetObjectRes groups the resulting values of GetObject operation.
type GetObjectRes struct { type GetObjectRes struct {
hdr *object.Object hdr *objectSDK.Object
} }
// Header returns the header of the request object. // Header returns the header of the request object.
func (x GetObjectRes) Header() *object.Object { func (x GetObjectRes) Header() *objectSDK.Object {
return x.hdr return x.hdr
} }
@ -518,35 +572,26 @@ func (x GetObjectRes) Header() *object.Object {
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
// For raw reading, returns *object.SplitInfoError error if object is virtual. // For raw reading, returns *object.SplitInfoError error if object is virtual.
func GetObject(prm GetObjectPrm) (*GetObjectRes, error) { func GetObject(ctx context.Context, prm GetObjectPrm) (*GetObjectRes, error) {
var getPrm client.PrmObjectGet cnr := prm.objAddr.Container()
getPrm.FromContainer(prm.objAddr.Container()) obj := prm.objAddr.Object()
getPrm.ByID(prm.objAddr.Object())
if prm.sessionToken != nil { getPrm := client.PrmObjectGet{
getPrm.WithinSession(*prm.sessionToken) XHeaders: prm.xHeaders,
BearerToken: prm.bearerToken,
Session: prm.sessionToken,
Raw: prm.raw,
Local: prm.local,
ContainerID: &cnr,
ObjectID: &obj,
} }
if prm.bearerToken != nil { rdr, err := prm.cli.ObjectGetInit(ctx, getPrm)
getPrm.WithBearerToken(*prm.bearerToken)
}
if prm.raw {
getPrm.MarkRaw()
}
if prm.local {
getPrm.MarkLocal()
}
getPrm.WithXHeaders(prm.xHeaders...)
rdr, err := prm.cli.ObjectGetInit(context.Background(), getPrm)
if err != nil { if err != nil {
return nil, fmt.Errorf("init object reading on client: %w", err) return nil, fmt.Errorf("init object reading on client: %w", err)
} }
var hdr object.Object var hdr objectSDK.Object
if !rdr.ReadHeader(&hdr) { if !rdr.ReadHeader(&hdr) {
_, err = rdr.Close() _, err = rdr.Close()
@ -582,11 +627,11 @@ func (x *HeadObjectPrm) SetMainOnlyFlag(v bool) {
// HeadObjectRes groups the resulting values of HeadObject operation. // HeadObjectRes groups the resulting values of HeadObject operation.
type HeadObjectRes struct { type HeadObjectRes struct {
hdr *object.Object hdr *objectSDK.Object
} }
// Header returns the requested object header. // Header returns the requested object header.
func (x HeadObjectRes) Header() *object.Object { func (x HeadObjectRes) Header() *objectSDK.Object {
return x.hdr return x.hdr
} }
@ -594,35 +639,26 @@ func (x HeadObjectRes) Header() *object.Object {
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
// For raw reading, returns *object.SplitInfoError error if object is virtual. // For raw reading, returns *object.SplitInfoError error if object is virtual.
func HeadObject(prm HeadObjectPrm) (*HeadObjectRes, error) { func HeadObject(ctx context.Context, prm HeadObjectPrm) (*HeadObjectRes, error) {
var cliPrm client.PrmObjectHead cnr := prm.objAddr.Container()
cliPrm.FromContainer(prm.objAddr.Container()) obj := prm.objAddr.Object()
cliPrm.ByID(prm.objAddr.Object())
if prm.sessionToken != nil { headPrm := client.PrmObjectHead{
cliPrm.WithinSession(*prm.sessionToken) XHeaders: prm.xHeaders,
BearerToken: prm.bearerToken,
Session: prm.sessionToken,
Raw: prm.raw,
Local: prm.local,
ContainerID: &cnr,
ObjectID: &obj,
} }
if prm.bearerToken != nil { res, err := prm.cli.ObjectHead(ctx, headPrm)
cliPrm.WithBearerToken(*prm.bearerToken)
}
if prm.raw {
cliPrm.MarkRaw()
}
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.WithXHeaders(prm.xHeaders...)
res, err := prm.cli.ObjectHead(context.Background(), cliPrm)
if err != nil { if err != nil {
return nil, fmt.Errorf("read object header via client: %w", err) return nil, fmt.Errorf("read object header via client: %w", err)
} }
var hdr object.Object var hdr objectSDK.Object
if !res.ReadHeader(&hdr) { if !res.ReadHeader(&hdr) {
return nil, fmt.Errorf("missing header in response") return nil, fmt.Errorf("missing header in response")
@ -638,11 +674,11 @@ type SearchObjectsPrm struct {
commonObjectPrm commonObjectPrm
containerIDPrm containerIDPrm
filters object.SearchFilters filters objectSDK.SearchFilters
} }
// SetFilters sets search filters. // SetFilters sets search filters.
func (x *SearchObjectsPrm) SetFilters(filters object.SearchFilters) { func (x *SearchObjectsPrm) SetFilters(filters objectSDK.SearchFilters) {
x.filters = filters x.filters = filters
} }
@ -659,7 +695,7 @@ func (x SearchObjectsRes) IDList() []oid.ID {
// SearchObjects selects objects from the container which match the filters. // SearchObjects selects objects from the container which match the filters.
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
func SearchObjects(prm SearchObjectsPrm) (*SearchObjectsRes, error) { func SearchObjects(ctx context.Context, prm SearchObjectsPrm) (*SearchObjectsRes, error) {
var cliPrm client.PrmObjectSearch var cliPrm client.PrmObjectSearch
cliPrm.InContainer(prm.cnrID) cliPrm.InContainer(prm.cnrID)
cliPrm.SetFilters(prm.filters) cliPrm.SetFilters(prm.filters)
@ -678,7 +714,7 @@ func SearchObjects(prm SearchObjectsPrm) (*SearchObjectsRes, error) {
cliPrm.WithXHeaders(prm.xHeaders...) cliPrm.WithXHeaders(prm.xHeaders...)
rdr, err := prm.cli.ObjectSearchInit(context.Background(), cliPrm) rdr, err := prm.cli.ObjectSearchInit(ctx, cliPrm)
if err != nil { if err != nil {
return nil, fmt.Errorf("init object search: %w", err) return nil, fmt.Errorf("init object search: %w", err)
} }
@ -703,6 +739,11 @@ func SearchObjects(prm SearchObjectsPrm) (*SearchObjectsRes, error) {
return nil, fmt.Errorf("read object list: %w", err) return nil, fmt.Errorf("read object list: %w", err)
} }
sort.Slice(list, func(i, j int) bool {
lhs, rhs := list[i].EncodeToString(), list[j].EncodeToString()
return strings.Compare(lhs, rhs) < 0
})
return &SearchObjectsRes{ return &SearchObjectsRes{
ids: list, ids: list,
}, nil }, nil
@ -715,7 +756,7 @@ type HashPayloadRangesPrm struct {
tz bool tz bool
rngs []*object.Range rngs []objectSDK.Range
salt []byte salt []byte
} }
@ -726,7 +767,7 @@ func (x *HashPayloadRangesPrm) TZ() {
} }
// SetRanges sets a list of payload ranges to hash. // SetRanges sets a list of payload ranges to hash.
func (x *HashPayloadRangesPrm) SetRanges(rngs []*object.Range) { func (x *HashPayloadRangesPrm) SetRanges(rngs []objectSDK.Range) {
x.rngs = rngs x.rngs = rngs
} }
@ -749,41 +790,27 @@ func (x HashPayloadRangesRes) HashList() [][]byte {
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
// Returns an error if number of received hashes differs with the number of requested ranges. // Returns an error if number of received hashes differs with the number of requested ranges.
func HashPayloadRanges(prm HashPayloadRangesPrm) (*HashPayloadRangesRes, error) { func HashPayloadRanges(ctx context.Context, prm HashPayloadRangesPrm) (*HashPayloadRangesRes, error) {
var cliPrm client.PrmObjectHash cs := checksum.SHA256
cliPrm.FromContainer(prm.objAddr.Container())
cliPrm.ByID(prm.objAddr.Object())
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.UseSalt(prm.salt)
rngs := make([]uint64, 2*len(prm.rngs))
for i := range prm.rngs {
rngs[2*i] = prm.rngs[i].GetOffset()
rngs[2*i+1] = prm.rngs[i].GetLength()
}
cliPrm.SetRangeList(rngs...)
if prm.tz { if prm.tz {
cliPrm.TillichZemorAlgo() cs = checksum.TZ
} }
if prm.sessionToken != nil { cnr := prm.objAddr.Container()
cliPrm.WithinSession(*prm.sessionToken) obj := prm.objAddr.Object()
cliPrm := client.PrmObjectHash{
ContainerID: &cnr,
ObjectID: &obj,
Local: prm.local,
Salt: prm.salt,
Ranges: prm.rngs,
ChecksumType: cs,
Session: prm.sessionToken,
BearerToken: prm.bearerToken,
XHeaders: prm.xHeaders,
} }
if prm.bearerToken != nil { res, err := prm.cli.ObjectHash(ctx, cliPrm)
cliPrm.WithBearerToken(*prm.bearerToken)
}
cliPrm.WithXHeaders(prm.xHeaders...)
res, err := prm.cli.ObjectHash(context.Background(), cliPrm)
if err != nil { if err != nil {
return nil, fmt.Errorf("read payload hashes via client: %w", err) return nil, fmt.Errorf("read payload hashes via client: %w", err)
} }
@ -800,11 +827,11 @@ type PayloadRangePrm struct {
rawPrm rawPrm
payloadWriterPrm payloadWriterPrm
rng *object.Range rng *objectSDK.Range
} }
// SetRange sets payload range to read. // SetRange sets payload range to read.
func (x *PayloadRangePrm) SetRange(rng *object.Range) { func (x *PayloadRangePrm) SetRange(rng *objectSDK.Range) {
x.rng = rng x.rng = rng
} }
@ -817,33 +844,23 @@ type PayloadRangeRes struct{}
// //
// Returns any error which prevented the operation from completing correctly in error return. // Returns any error which prevented the operation from completing correctly in error return.
// For raw reading, returns *object.SplitInfoError error if object is virtual. // For raw reading, returns *object.SplitInfoError error if object is virtual.
func PayloadRange(prm PayloadRangePrm) (*PayloadRangeRes, error) { func PayloadRange(ctx context.Context, prm PayloadRangePrm) (*PayloadRangeRes, error) {
var cliPrm client.PrmObjectRange cnr := prm.objAddr.Container()
cliPrm.FromContainer(prm.objAddr.Container()) obj := prm.objAddr.Object()
cliPrm.ByID(prm.objAddr.Object())
if prm.sessionToken != nil { rangePrm := client.PrmObjectRange{
cliPrm.WithinSession(*prm.sessionToken) XHeaders: prm.xHeaders,
BearerToken: prm.bearerToken,
Session: prm.sessionToken,
Raw: prm.raw,
Local: prm.local,
ContainerID: &cnr,
ObjectID: &obj,
Offset: prm.rng.GetOffset(),
Length: prm.rng.GetLength(),
} }
if prm.bearerToken != nil { rdr, err := prm.cli.ObjectRangeInit(ctx, rangePrm)
cliPrm.WithBearerToken(*prm.bearerToken)
}
if prm.raw {
cliPrm.MarkRaw()
}
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.SetOffset(prm.rng.GetOffset())
cliPrm.SetLength(prm.rng.GetLength())
cliPrm.WithXHeaders(prm.xHeaders...)
rdr, err := prm.cli.ObjectRangeInit(context.Background(), cliPrm)
if err != nil { if err != nil {
return nil, fmt.Errorf("init payload reading: %w", err) return nil, fmt.Errorf("init payload reading: %w", err)
} }
@ -877,12 +894,12 @@ type SyncContainerRes struct{}
// Interrupts on any writer error. // Interrupts on any writer error.
// //
// Panics if a container passed as a parameter is nil. // Panics if a container passed as a parameter is nil.
func SyncContainerSettings(prm SyncContainerPrm) (*SyncContainerRes, error) { func SyncContainerSettings(ctx context.Context, prm SyncContainerPrm) (*SyncContainerRes, error) {
if prm.c == nil { if prm.c == nil {
panic("sync container settings with the network: nil container") panic("sync container settings with the network: nil container")
} }
err := client.SyncContainerWithNetwork(context.Background(), prm.c, prm.cli) err := client.SyncContainerWithNetwork(ctx, prm.c, prm.cli)
if err != nil { if err != nil {
return nil, err return nil, err
} }

View file

@ -3,11 +3,11 @@ package internal
import ( import (
"io" "io"
"github.com/TrueCloudLab/frostfs-sdk-go/bearer" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
"github.com/TrueCloudLab/frostfs-sdk-go/client" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/TrueCloudLab/frostfs-sdk-go/session" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
) )
// here are small structures with public setters to share between parameter structures // here are small structures with public setters to share between parameter structures

View file

@ -8,13 +8,15 @@ import (
"errors" "errors"
"fmt" "fmt"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/network" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network"
"github.com/TrueCloudLab/frostfs-sdk-go/client" tracing "git.frostfs.info/TrueCloudLab/frostfs-observability/tracing/grpc"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
"google.golang.org/grpc"
) )
var errInvalidEndpoint = errors.New("provided RPC endpoint is incorrect") var errInvalidEndpoint = errors.New("provided RPC endpoint is incorrect")
@ -36,11 +38,11 @@ func getSDKClientByFlag(cmd *cobra.Command, key *ecdsa.PrivateKey, endpointFlag
if err != nil { if err != nil {
return nil, fmt.Errorf("%v: %w", errInvalidEndpoint, err) return nil, fmt.Errorf("%v: %w", errInvalidEndpoint, err)
} }
return GetSDKClient(cmd, key, addr) return GetSDKClient(cmd.Context(), cmd, key, addr)
} }
// GetSDKClient returns default frostfs-sdk-go client. // GetSDKClient returns default frostfs-sdk-go client.
func GetSDKClient(cmd *cobra.Command, key *ecdsa.PrivateKey, addr network.Address) (*client.Client, error) { func GetSDKClient(ctx context.Context, cmd *cobra.Command, key *ecdsa.PrivateKey, addr network.Address) (*client.Client, error) {
var ( var (
c client.Client c client.Client
prmInit client.PrmInit prmInit client.PrmInit
@ -48,7 +50,7 @@ func GetSDKClient(cmd *cobra.Command, key *ecdsa.PrivateKey, addr network.Addres
) )
prmInit.SetDefaultPrivateKey(*key) prmInit.SetDefaultPrivateKey(*key)
prmInit.ResolveNeoFSFailures() prmInit.ResolveFrostFSFailures()
prmDial.SetServerURI(addr.URIAddr()) prmDial.SetServerURI(addr.URIAddr())
if timeout := viper.GetDuration(commonflags.Timeout); timeout > 0 { if timeout := viper.GetDuration(commonflags.Timeout); timeout > 0 {
// In CLI we can only set a timeout for the whole operation. // In CLI we can only set a timeout for the whole operation.
@ -59,10 +61,13 @@ func GetSDKClient(cmd *cobra.Command, key *ecdsa.PrivateKey, addr network.Addres
common.PrintVerbose(cmd, "Set request timeout to %s.", timeout) common.PrintVerbose(cmd, "Set request timeout to %s.", timeout)
} }
prmDial.SetGRPCDialOptions(
grpc.WithChainUnaryInterceptor(tracing.NewUnaryClientInteceptor()),
grpc.WithChainStreamInterceptor(tracing.NewStreamClientInterceptor()))
c.Init(prmInit) c.Init(prmInit)
if err := c.Dial(prmDial); err != nil { if err := c.Dial(ctx, prmDial); err != nil {
return nil, fmt.Errorf("can't init SDK client: %w", err) return nil, fmt.Errorf("can't init SDK client: %w", err)
} }
@ -82,7 +87,7 @@ func GetCurrentEpoch(ctx context.Context, cmd *cobra.Command, endpoint string) (
return 0, fmt.Errorf("can't generate key to sign query: %w", err) return 0, fmt.Errorf("can't generate key to sign query: %w", err)
} }
c, err := GetSDKClient(cmd, key, addr) c, err := GetSDKClient(ctx, cmd, key, addr)
if err != nil { if err != nil {
return 0, err return 0, err
} }

View file

@ -4,10 +4,10 @@ import (
"errors" "errors"
"os" "os"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/core/version" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/core/version"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
versionSDK "github.com/TrueCloudLab/frostfs-sdk-go/version" versionSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/version"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )

View file

@ -6,8 +6,8 @@ import (
"fmt" "fmt"
"os" "os"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/bearer" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )

View file

@ -0,0 +1,62 @@
package common
import (
"context"
"sort"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/misc"
"git.frostfs.info/TrueCloudLab/frostfs-observability/tracing"
"github.com/spf13/cobra"
"go.opentelemetry.io/otel/trace"
)
type spanKey struct{}
// StopClientCommandSpan stops tracing span for the command and prints trace ID on the standard output.
func StopClientCommandSpan(cmd *cobra.Command, _ []string) {
span, ok := cmd.Context().Value(spanKey{}).(trace.Span)
if !ok {
return
}
span.End()
// Noop provider cannot fail on flush.
_ = tracing.Shutdown(cmd.Context())
cmd.PrintErrf("Trace ID: %s\n", span.SpanContext().TraceID())
}
// StartClientCommandSpan starts tracing span for the command.
func StartClientCommandSpan(cmd *cobra.Command) {
enableTracing, err := cmd.Flags().GetBool(commonflags.TracingFlag)
if err != nil || !enableTracing {
return
}
_, err = tracing.Setup(cmd.Context(), tracing.Config{
Enabled: true,
Exporter: tracing.NoOpExporter,
Service: "frostfs-cli",
Version: misc.Version,
})
commonCmd.ExitOnErr(cmd, "init tracing: %w", err)
var components sort.StringSlice
for c := cmd; c != nil; c = c.Parent() {
components = append(components, c.Name())
}
for i, j := 0, len(components)-1; i < j; {
components.Swap(i, j)
i++
j--
}
operation := strings.Join(components, ".")
ctx, span := tracing.StartSpanFromContext(cmd.Context(), operation)
ctx = context.WithValue(ctx, spanKey{}, span)
cmd.SetContext(ctx)
}

View file

@ -5,8 +5,8 @@ import (
"strconv" "strconv"
"time" "time"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-sdk-go/checksum" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/checksum"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
) )

View file

@ -47,6 +47,9 @@ const (
OIDFlag = "oid" OIDFlag = "oid"
OIDFlagUsage = "Object ID." OIDFlagUsage = "Object ID."
TracingFlag = "trace"
TracingFlagUsage = "Generate trace ID and print it."
) )
// Init adds common flags to the command: // Init adds common flags to the command:
@ -54,12 +57,14 @@ const (
// - WalletPath, // - WalletPath,
// - Account, // - Account,
// - RPC, // - RPC,
// - Tracing,
// - Timeout. // - Timeout.
func Init(cmd *cobra.Command) { func Init(cmd *cobra.Command) {
InitWithoutRPC(cmd) InitWithoutRPC(cmd)
ff := cmd.Flags() ff := cmd.Flags()
ff.StringP(RPC, RPCShorthand, RPCDefault, RPCUsage) ff.StringP(RPC, RPCShorthand, RPCDefault, RPCUsage)
ff.Bool(TracingFlag, false, TracingFlagUsage)
ff.DurationP(Timeout, TimeoutShorthand, TimeoutDefault, TimeoutUsage) ff.DurationP(Timeout, TimeoutShorthand, TimeoutDefault, TimeoutUsage)
} }

View file

@ -7,7 +7,7 @@ import (
"path/filepath" "path/filepath"
"testing" "testing"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/nspcc-dev/neo-go/cli/input" "github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/wallet" "github.com/nspcc-dev/neo-go/pkg/wallet"

View file

@ -6,8 +6,8 @@ import (
"fmt" "fmt"
"os" "os"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/wallet" "github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra" "github.com/spf13/cobra"

View file

@ -4,7 +4,7 @@ import (
"crypto/ecdsa" "crypto/ecdsa"
"errors" "errors"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/nspcc-dev/neo-go/cli/flags" "github.com/nspcc-dev/neo-go/cli/flags"
"github.com/nspcc-dev/neo-go/cli/input" "github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"

View file

@ -1,6 +1,6 @@
package main package main
import cmd "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules" import cmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules"
func main() { func main() {
cmd.Execute() cmd.Execute()

View file

@ -3,13 +3,13 @@ package accounting
import ( import (
"math/big" "math/big"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client" internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/util/precision" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/precision"
"github.com/TrueCloudLab/frostfs-sdk-go/accounting" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/accounting"
"github.com/TrueCloudLab/frostfs-sdk-go/user" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn" "github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
@ -41,7 +41,7 @@ var accountingBalanceCmd = &cobra.Command{
prm.SetClient(cli) prm.SetClient(cli)
prm.SetAccount(idUser) prm.SetAccount(idUser)
res, err := internalclient.BalanceOf(prm) res, err := internalclient.BalanceOf(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err) commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
// print to stdout // print to stdout

View file

@ -1,7 +1,8 @@
package accounting package accounting
import ( import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
) )
@ -17,7 +18,9 @@ var Cmd = &cobra.Command{
_ = viper.BindPFlag(commonflags.WalletPath, flags.Lookup(commonflags.WalletPath)) _ = viper.BindPFlag(commonflags.WalletPath, flags.Lookup(commonflags.WalletPath))
_ = viper.BindPFlag(commonflags.Account, flags.Lookup(commonflags.Account)) _ = viper.BindPFlag(commonflags.Account, flags.Lookup(commonflags.Account))
_ = viper.BindPFlag(commonflags.RPC, flags.Lookup(commonflags.RPC)) _ = viper.BindPFlag(commonflags.RPC, flags.Lookup(commonflags.RPC))
common.StartClientCommandSpan(cmd)
}, },
PersistentPostRun: common.StopClientCommandSpan,
} }
func init() { func init() {

View file

@ -1,9 +1,9 @@
package basic package basic
import ( import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/container/acl" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/acl"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )

View file

@ -6,11 +6,11 @@ import (
"os" "os"
"strings" "strings"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@ -26,18 +26,18 @@ Action is 'allow' or 'deny'.
Operation is an object service verb: 'get', 'head', 'put', 'search', 'delete', 'getrange', or 'getrangehash'. Operation is an object service verb: 'get', 'head', 'put', 'search', 'delete', 'getrange', or 'getrangehash'.
Filter consists of <typ>:<key><match><value> Filter consists of <typ>:<key><match><value>
Typ is 'obj' for object applied filter or 'req' for request applied filter. Typ is 'obj' for object applied filter or 'req' for request applied filter.
Key is a valid unicode string corresponding to object or request header key. Key is a valid unicode string corresponding to object or request header key.
Well-known system object headers start with '$Object:' prefix. Well-known system object headers start with '$Object:' prefix.
User defined headers start without prefix. User defined headers start without prefix.
Read more about filter keys at github.com/TrueCloudLab/frostfs-api/blob/master/proto-docs/acl.md#message-eaclrecordfilter Read more about filter keys at git.frostfs.info.com/TrueCloudLab/frostfs-api/src/branch/master/proto-docs/acl.md#message-eaclrecordfilter
Match is '=' for matching and '!=' for non-matching filter. Match is '=' for matching and '!=' for non-matching filter.
Value is a valid unicode string corresponding to object or request header value. Value is a valid unicode string corresponding to object or request header value.
Target is Target is
'user' for container owner, 'user' for container owner,
'system' for Storage nodes in container and Inner Ring nodes, 'system' for Storage nodes in container and Inner Ring nodes,
'others' for all other request senders, 'others' for all other request senders,
'pubkey:<key1>,<key2>,...' for exact request sender, where <key> is a hex-encoded 33-byte public key. 'pubkey:<key1>,<key2>,...' for exact request sender, where <key> is a hex-encoded 33-byte public key.
When both '--rule' and '--file' arguments are used, '--rule' records will be placed higher in resulting extended ACL table. When both '--rule' and '--file' arguments are used, '--rule' records will be placed higher in resulting extended ACL table.

View file

@ -3,8 +3,8 @@ package extended
import ( import (
"testing" "testing"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )

View file

@ -4,9 +4,9 @@ import (
"os" "os"
"strings" "strings"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )

View file

@ -1,8 +1,8 @@
package acl package acl
import ( import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/acl/basic" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/acl/basic"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/acl/extended" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/acl/extended"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )

View file

@ -7,13 +7,13 @@ import (
"os" "os"
"time" "time"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client" internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/bearer" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
eaclSDK "github.com/TrueCloudLab/frostfs-sdk-go/eacl" eaclSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/TrueCloudLab/frostfs-sdk-go/user" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@ -24,6 +24,7 @@ const (
ownerFlag = "owner" ownerFlag = "owner"
outFlag = "out" outFlag = "out"
jsonFlag = commonflags.JSON jsonFlag = commonflags.JSON
impersonateFlag = "impersonate"
) )
var createCmd = &cobra.Command{ var createCmd = &cobra.Command{
@ -39,19 +40,20 @@ is set to current epoch + n.
} }
func init() { func init() {
createCmd.Flags().StringP(eaclFlag, "e", "", "Path to the extended ACL table") createCmd.Flags().StringP(eaclFlag, "e", "", "Path to the extended ACL table (mutually exclusive with --impersonate flag)")
createCmd.Flags().StringP(issuedAtFlag, "i", "", "Epoch to issue token at") createCmd.Flags().StringP(issuedAtFlag, "i", "+0", "Epoch to issue token at")
createCmd.Flags().StringP(notValidBeforeFlag, "n", "", "Not valid before epoch") createCmd.Flags().StringP(notValidBeforeFlag, "n", "+0", "Not valid before epoch")
createCmd.Flags().StringP(commonflags.ExpireAt, "x", "", "The last active epoch for the token") createCmd.Flags().StringP(commonflags.ExpireAt, "x", "", "The last active epoch for the token")
createCmd.Flags().StringP(ownerFlag, "o", "", "Token owner") createCmd.Flags().StringP(ownerFlag, "o", "", "Token owner")
createCmd.Flags().String(outFlag, "", "File to write token to") createCmd.Flags().String(outFlag, "", "File to write token to")
createCmd.Flags().Bool(jsonFlag, false, "Output token in JSON") createCmd.Flags().Bool(jsonFlag, false, "Output token in JSON")
createCmd.Flags().Bool(impersonateFlag, false, "Mark token as impersonate to consider the token signer as the request owner (mutually exclusive with --eacl flag)")
createCmd.Flags().StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage) createCmd.Flags().StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage)
createCmd.MarkFlagsMutuallyExclusive(eaclFlag, impersonateFlag)
_ = cobra.MarkFlagFilename(createCmd.Flags(), eaclFlag) _ = cobra.MarkFlagFilename(createCmd.Flags(), eaclFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), issuedAtFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), notValidBeforeFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), commonflags.ExpireAt) _ = cobra.MarkFlagRequired(createCmd.Flags(), commonflags.ExpireAt)
_ = cobra.MarkFlagRequired(createCmd.Flags(), ownerFlag) _ = cobra.MarkFlagRequired(createCmd.Flags(), ownerFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), outFlag) _ = cobra.MarkFlagRequired(createCmd.Flags(), outFlag)
@ -68,10 +70,14 @@ func createToken(cmd *cobra.Command, _ []string) {
commonCmd.ExitOnErr(cmd, "can't parse --"+notValidBeforeFlag+" flag: %w", err) commonCmd.ExitOnErr(cmd, "can't parse --"+notValidBeforeFlag+" flag: %w", err)
if iatRelative || expRelative || nvbRelative { if iatRelative || expRelative || nvbRelative {
endpoint, _ := cmd.Flags().GetString(commonflags.RPC)
if len(endpoint) == 0 {
commonCmd.ExitOnErr(cmd, "can't fetch current epoch: %w", fmt.Errorf("'%s' flag value must be specified", commonflags.RPC))
}
ctx, cancel := context.WithTimeout(context.Background(), time.Second*30) ctx, cancel := context.WithTimeout(context.Background(), time.Second*30)
defer cancel() defer cancel()
endpoint, _ := cmd.Flags().GetString(commonflags.RPC)
currEpoch, err := internalclient.GetCurrentEpoch(ctx, cmd, endpoint) currEpoch, err := internalclient.GetCurrentEpoch(ctx, cmd, endpoint)
commonCmd.ExitOnErr(cmd, "can't fetch current epoch: %w", err) commonCmd.ExitOnErr(cmd, "can't fetch current epoch: %w", err)
@ -101,6 +107,9 @@ func createToken(cmd *cobra.Command, _ []string) {
b.SetIat(iat) b.SetIat(iat)
b.ForUser(ownerID) b.ForUser(ownerID)
impersonate, _ := cmd.Flags().GetBool(impersonateFlag)
b.SetImpersonate(impersonate)
eaclPath, _ := cmd.Flags().GetString(eaclFlag) eaclPath, _ := cmd.Flags().GetString(eaclFlag)
if eaclPath != "" { if eaclPath != "" {
table := eaclSDK.NewTable() table := eaclSDK.NewTable()

View file

@ -1,7 +1,7 @@
package cmd package cmd
import ( import (
"github.com/TrueCloudLab/frostfs-node/pkg/util/autocomplete" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/autocomplete"
) )
func init() { func init() {

View file

@ -7,17 +7,17 @@ import (
"strings" "strings"
"time" "time"
containerApi "github.com/TrueCloudLab/frostfs-api-go/v2/container" containerApi "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client" internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/container" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/TrueCloudLab/frostfs-sdk-go/container/acl" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
"github.com/TrueCloudLab/frostfs-sdk-go/netmap" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/acl"
subnetid "github.com/TrueCloudLab/frostfs-sdk-go/subnet/id" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/TrueCloudLab/frostfs-sdk-go/user" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@ -30,14 +30,13 @@ var (
containerNnsName string containerNnsName string
containerNnsZone string containerNnsZone string
containerNoTimestamp bool containerNoTimestamp bool
containerSubnet string
force bool force bool
) )
var createContainerCmd = &cobra.Command{ var createContainerCmd = &cobra.Command{
Use: "create", Use: "create",
Short: "Create new container", Short: "Create new container",
Long: `Create new container and register it in the FrostFS. Long: `Create new container and register it in the FrostFS.
It will be stored in sidechain when inner ring will accepts it.`, It will be stored in sidechain when inner ring will accepts it.`,
Run: func(cmd *cobra.Command, args []string) { Run: func(cmd *cobra.Command, args []string) {
placementPolicy, err := parseContainerPolicy(cmd, containerPolicy) placementPolicy, err := parseContainerPolicy(cmd, containerPolicy)
@ -50,7 +49,7 @@ It will be stored in sidechain when inner ring will accepts it.`,
var prm internalclient.NetMapSnapshotPrm var prm internalclient.NetMapSnapshotPrm
prm.SetClient(cli) prm.SetClient(cli)
resmap, err := internalclient.NetMapSnapshot(prm) resmap, err := internalclient.NetMapSnapshot(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "unable to get netmap snapshot to validate container placement, "+ commonCmd.ExitOnErr(cmd, "unable to get netmap snapshot to validate container placement, "+
"use --force option to skip this check: %w", err) "use --force option to skip this check: %w", err)
@ -70,15 +69,6 @@ It will be stored in sidechain when inner ring will accepts it.`,
} }
} }
if containerSubnet != "" {
var subnetID subnetid.ID
err = subnetID.DecodeString(containerSubnet)
commonCmd.ExitOnErr(cmd, "could not parse subnetID: %w", err)
placementPolicy.RestrictSubnet(subnetID)
}
var cnr container.Container var cnr container.Container
cnr.Init() cnr.Init()
@ -107,18 +97,18 @@ It will be stored in sidechain when inner ring will accepts it.`,
syncContainerPrm.SetClient(cli) syncContainerPrm.SetClient(cli)
syncContainerPrm.SetContainer(&cnr) syncContainerPrm.SetContainer(&cnr)
_, err = internalclient.SyncContainerSettings(syncContainerPrm) _, err = internalclient.SyncContainerSettings(cmd.Context(), syncContainerPrm)
commonCmd.ExitOnErr(cmd, "syncing container's settings rpc error: %w", err) commonCmd.ExitOnErr(cmd, "syncing container's settings rpc error: %w", err)
var putPrm internalclient.PutContainerPrm putPrm := internalclient.PutContainerPrm{
putPrm.SetClient(cli) Client: cli,
putPrm.SetContainer(cnr) ClientParams: client.PrmContainerPut{
Container: &cnr,
if tok != nil { Session: tok,
putPrm.WithinSession(*tok) },
} }
res, err := internalclient.PutContainer(putPrm) res, err := internalclient.PutContainer(cmd.Context(), putPrm)
commonCmd.ExitOnErr(cmd, "put container rpc error: %w", err) commonCmd.ExitOnErr(cmd, "put container rpc error: %w", err)
id := res.ID() id := res.ID()
@ -128,14 +118,17 @@ It will be stored in sidechain when inner ring will accepts it.`,
if containerAwait { if containerAwait {
cmd.Println("awaiting...") cmd.Println("awaiting...")
var getPrm internalclient.GetContainerPrm getPrm := internalclient.GetContainerPrm{
getPrm.SetClient(cli) Client: cli,
getPrm.SetContainer(id) ClientParams: client.PrmContainerGet{
ContainerID: &id,
},
}
for i := 0; i < awaitTimeout; i++ { for i := 0; i < awaitTimeout; i++ {
time.Sleep(1 * time.Second) time.Sleep(1 * time.Second)
_, err := internalclient.GetContainer(getPrm) _, err := internalclient.GetContainer(cmd.Context(), getPrm)
if err == nil { if err == nil {
cmd.Println("container has been persisted on sidechain") cmd.Println("container has been persisted on sidechain")
return return
@ -152,6 +145,7 @@ func initContainerCreateCmd() {
// Init common flags // Init common flags
flags.StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage) flags.StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage)
flags.Bool(commonflags.TracingFlag, false, commonflags.TracingFlagUsage)
flags.DurationP(commonflags.Timeout, commonflags.TimeoutShorthand, commonflags.TimeoutDefault, commonflags.TimeoutUsage) flags.DurationP(commonflags.Timeout, commonflags.TimeoutShorthand, commonflags.TimeoutDefault, commonflags.TimeoutUsage)
flags.StringP(commonflags.WalletPath, commonflags.WalletPathShorthand, commonflags.WalletPathDefault, commonflags.WalletPathUsage) flags.StringP(commonflags.WalletPath, commonflags.WalletPathShorthand, commonflags.WalletPathDefault, commonflags.WalletPathUsage)
flags.StringP(commonflags.Account, commonflags.AccountShorthand, commonflags.AccountDefault, commonflags.AccountUsage) flags.StringP(commonflags.Account, commonflags.AccountShorthand, commonflags.AccountDefault, commonflags.AccountUsage)
@ -166,7 +160,6 @@ func initContainerCreateCmd() {
flags.StringVar(&containerNnsName, "nns-name", "", "Container nns name attribute") flags.StringVar(&containerNnsName, "nns-name", "", "Container nns name attribute")
flags.StringVar(&containerNnsZone, "nns-zone", "", "Container nns zone attribute") flags.StringVar(&containerNnsZone, "nns-zone", "", "Container nns zone attribute")
flags.BoolVar(&containerNoTimestamp, "disable-timestamp", false, "Disable timestamp container attribute") flags.BoolVar(&containerNoTimestamp, "disable-timestamp", false, "Disable timestamp container attribute")
flags.StringVar(&containerSubnet, "subnet", "", "String representation of container subnetwork")
flags.BoolVarP(&force, commonflags.ForceFlag, commonflags.ForceFlagShorthand, false, flags.BoolVarP(&force, commonflags.ForceFlag, commonflags.ForceFlagShorthand, false,
"Skip placement validity check") "Skip placement validity check")
} }
@ -192,12 +185,12 @@ func parseContainerPolicy(cmd *cobra.Command, policyString string) (*netmap.Plac
return &result, nil return &result, nil
} }
if err = result.UnmarshalJSON([]byte(policyString)); err == nil { if err := result.UnmarshalJSON([]byte(policyString)); err == nil {
common.PrintVerbose(cmd, "Parsed JSON encoded policy") common.PrintVerbose(cmd, "Parsed JSON encoded policy")
return &result, nil return &result, nil
} }
return nil, errors.New("can't parse placement policy") return nil, fmt.Errorf("can't parse placement policy: %w", err)
} }
func parseAttributes(dst *container.Container, attributes []string) error { func parseAttributes(dst *container.Container, attributes []string) error {

View file

@ -4,20 +4,21 @@ import (
"fmt" "fmt"
"time" "time"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client" internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
objectSDK "github.com/TrueCloudLab/frostfs-sdk-go/object" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/TrueCloudLab/frostfs-sdk-go/user" objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
var deleteContainerCmd = &cobra.Command{ var deleteContainerCmd = &cobra.Command{
Use: "delete", Use: "delete",
Short: "Delete existing container", Short: "Delete existing container",
Long: `Delete existing container. Long: `Delete existing container.
Only owner of the container has a permission to remove container.`, Only owner of the container has a permission to remove container.`,
Run: func(cmd *cobra.Command, args []string) { Run: func(cmd *cobra.Command, args []string) {
id := parseContainerID(cmd) id := parseContainerID(cmd)
@ -30,11 +31,14 @@ Only owner of the container has a permission to remove container.`,
if force, _ := cmd.Flags().GetBool(commonflags.ForceFlag); !force { if force, _ := cmd.Flags().GetBool(commonflags.ForceFlag); !force {
common.PrintVerbose(cmd, "Reading the container to check ownership...") common.PrintVerbose(cmd, "Reading the container to check ownership...")
var getPrm internalclient.GetContainerPrm getPrm := internalclient.GetContainerPrm{
getPrm.SetClient(cli) Client: cli,
getPrm.SetContainer(id) ClientParams: client.PrmContainerGet{
ContainerID: &id,
},
}
resGet, err := internalclient.GetContainer(getPrm) resGet, err := internalclient.GetContainer(cmd.Context(), getPrm)
commonCmd.ExitOnErr(cmd, "can't get the container: %w", err) commonCmd.ExitOnErr(cmd, "can't get the container: %w", err)
owner := resGet.Container().Owner() owner := resGet.Container().Owner()
@ -72,26 +76,26 @@ Only owner of the container has a permission to remove container.`,
common.PrintVerbose(cmd, "Searching for LOCK objects...") common.PrintVerbose(cmd, "Searching for LOCK objects...")
res, err := internalclient.SearchObjects(searchPrm) res, err := internalclient.SearchObjects(cmd.Context(), searchPrm)
commonCmd.ExitOnErr(cmd, "can't search for LOCK objects: %w", err) commonCmd.ExitOnErr(cmd, "can't search for LOCK objects: %w", err)
if len(res.IDList()) != 0 { if len(res.IDList()) != 0 {
commonCmd.ExitOnErr(cmd, "", commonCmd.ExitOnErr(cmd, "",
fmt.Errorf("Container wasn't removed because LOCK objects were found.\n"+ fmt.Errorf("container wasn't removed because LOCK objects were found, "+
"Use --%s flag to remove anyway.", commonflags.ForceFlag)) "use --%s flag to remove anyway", commonflags.ForceFlag))
} }
} }
} }
var delPrm internalclient.DeleteContainerPrm delPrm := internalclient.DeleteContainerPrm{
delPrm.SetClient(cli) Client: cli,
delPrm.SetContainer(id) ClientParams: client.PrmContainerDelete{
ContainerID: &id,
if tok != nil { Session: tok,
delPrm.WithinSession(*tok) },
} }
_, err := internalclient.DeleteContainer(delPrm) _, err := internalclient.DeleteContainer(cmd.Context(), delPrm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err) commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
cmd.Println("container delete method invoked") cmd.Println("container delete method invoked")
@ -99,14 +103,17 @@ Only owner of the container has a permission to remove container.`,
if containerAwait { if containerAwait {
cmd.Println("awaiting...") cmd.Println("awaiting...")
var getPrm internalclient.GetContainerPrm getPrm := internalclient.GetContainerPrm{
getPrm.SetClient(cli) Client: cli,
getPrm.SetContainer(id) ClientParams: client.PrmContainerGet{
ContainerID: &id,
},
}
for i := 0; i < awaitTimeout; i++ { for i := 0; i < awaitTimeout; i++ {
time.Sleep(1 * time.Second) time.Sleep(1 * time.Second)
_, err := internalclient.GetContainer(getPrm) _, err := internalclient.GetContainer(cmd.Context(), getPrm)
if err != nil { if err != nil {
cmd.Println("container has been removed:", containerID) cmd.Println("container has been removed:", containerID)
return return
@ -124,6 +131,7 @@ func initContainerDeleteCmd() {
flags.StringP(commonflags.WalletPath, commonflags.WalletPathShorthand, commonflags.WalletPathDefault, commonflags.WalletPathUsage) flags.StringP(commonflags.WalletPath, commonflags.WalletPathShorthand, commonflags.WalletPathDefault, commonflags.WalletPathUsage)
flags.StringP(commonflags.Account, commonflags.AccountShorthand, commonflags.AccountDefault, commonflags.AccountUsage) flags.StringP(commonflags.Account, commonflags.AccountShorthand, commonflags.AccountDefault, commonflags.AccountUsage)
flags.StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage) flags.StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage)
flags.Bool(commonflags.TracingFlag, false, commonflags.TracingFlagUsage)
flags.StringVar(&containerID, commonflags.CIDFlag, "", commonflags.CIDFlagUsage) flags.StringVar(&containerID, commonflags.CIDFlag, "", commonflags.CIDFlagUsage)
flags.BoolVar(&containerAwait, "await", false, "Block execution until container is removed") flags.BoolVar(&containerAwait, "await", false, "Block execution until container is removed")

View file

@ -4,15 +4,16 @@ import (
"crypto/ecdsa" "crypto/ecdsa"
"os" "os"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client" internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/container" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/TrueCloudLab/frostfs-sdk-go/container/acl" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/acl"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@ -147,11 +148,14 @@ func getContainer(cmd *cobra.Command) (container.Container, *ecdsa.PrivateKey) {
pk = key.GetOrGenerate(cmd) pk = key.GetOrGenerate(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC) cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
var prm internalclient.GetContainerPrm prm := internalclient.GetContainerPrm{
prm.SetClient(cli) Client: cli,
prm.SetContainer(id) ClientParams: client.PrmContainerGet{
ContainerID: &id,
},
}
res, err := internalclient.GetContainer(prm) res, err := internalclient.GetContainer(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err) commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
cnr = res.Container() cnr = res.Container()

View file

@ -3,11 +3,12 @@ package container
import ( import (
"os" "os"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client" internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@ -20,11 +21,14 @@ var getExtendedACLCmd = &cobra.Command{
pk := key.GetOrGenerate(cmd) pk := key.GetOrGenerate(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC) cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
var eaclPrm internalclient.EACLPrm eaclPrm := internalclient.EACLPrm{
eaclPrm.SetClient(cli) Client: cli,
eaclPrm.SetContainer(id) ClientParams: client.PrmContainerEACL{
ContainerID: &id,
},
}
res, err := internalclient.EACL(eaclPrm) res, err := internalclient.EACL(cmd.Context(), eaclPrm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err) commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
eaclTable := res.EACL() eaclTable := res.EACL()

View file

@ -3,12 +3,13 @@ package container
import ( import (
"strings" "strings"
"github.com/TrueCloudLab/frostfs-api-go/v2/container" "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client" internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/user" containerSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@ -16,12 +17,14 @@ import (
const ( const (
flagListPrintAttr = "with-attr" flagListPrintAttr = "with-attr"
flagListContainerOwner = "owner" flagListContainerOwner = "owner"
flagListName = "name"
) )
// flag vars of list command. // flag vars of list command.
var ( var (
flagVarListPrintAttr bool flagVarListPrintAttr bool
flagVarListContainerOwner string flagVarListContainerOwner string
flagVarListName string
) )
var listContainersCmd = &cobra.Command{ var listContainersCmd = &cobra.Command{
@ -46,30 +49,42 @@ var listContainersCmd = &cobra.Command{
prm.SetClient(cli) prm.SetClient(cli)
prm.SetAccount(idUser) prm.SetAccount(idUser)
res, err := internalclient.ListContainers(prm) res, err := internalclient.ListContainers(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err) commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
var prmGet internalclient.GetContainerPrm prmGet := internalclient.GetContainerPrm{
prmGet.SetClient(cli) Client: cli,
}
list := res.IDList() containerIDs := res.SortedIDList()
for i := range list { for _, cnrID := range containerIDs {
cmd.Println(list[i].String()) if flagVarListName == "" && !flagVarListPrintAttr {
cmd.Println(cnrID.String())
continue
}
cnrID := cnrID
prmGet.ClientParams.ContainerID = &cnrID
res, err := internalclient.GetContainer(cmd.Context(), prmGet)
if err != nil {
cmd.Printf(" failed to read attributes: %v\n", err)
continue
}
cnr := res.Container()
if cnrName := containerSDK.Name(cnr); flagVarListName != "" && cnrName != flagVarListName {
continue
}
cmd.Println(cnrID.String())
if flagVarListPrintAttr { if flagVarListPrintAttr {
prmGet.SetContainer(list[i]) cnr.IterateAttributes(func(key, val string) {
if !strings.HasPrefix(key, container.SysAttributePrefix) && !strings.HasPrefix(key, container.SysAttributePrefixNeoFS) {
res, err := internalclient.GetContainer(prmGet) // FIXME(@cthulhu-rider): https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/issues/97
if err == nil { // Use dedicated method to skip system attributes.
res.Container().IterateAttributes(func(key, val string) { cmd.Printf(" %s: %s\n", key, val)
if !strings.HasPrefix(key, container.SysAttributePrefix) { }
// FIXME(@cthulhu-rider): neofs-sdk-go#314 use dedicated method to skip system attributes })
cmd.Printf(" %s: %s\n", key, val)
}
})
} else {
cmd.Printf(" failed to read attributes: %v\n", err)
}
} }
} }
}, },
@ -80,6 +95,9 @@ func initContainerListContainersCmd() {
flags := listContainersCmd.Flags() flags := listContainersCmd.Flags()
flags.StringVar(&flagVarListName, flagListName, "",
"List containers by the attribute name",
)
flags.StringVar(&flagVarListContainerOwner, flagListContainerOwner, "", flags.StringVar(&flagVarListContainerOwner, flagListContainerOwner, "",
"Owner of containers (omit to use owner from private key)", "Owner of containers (omit to use owner from private key)",
) )

View file

@ -3,14 +3,14 @@ package container
import ( import (
"strings" "strings"
v2object "github.com/TrueCloudLab/frostfs-api-go/v2/object" v2object "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client" internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
objectCli "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/object" objectCli "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/object"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/object" objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@ -31,7 +31,7 @@ var listContainerObjectsCmd = &cobra.Command{
Run: func(cmd *cobra.Command, args []string) { Run: func(cmd *cobra.Command, args []string) {
id := parseContainerID(cmd) id := parseContainerID(cmd)
filters := new(object.SearchFilters) filters := new(objectSDK.SearchFilters)
filters.AddRootFilter() // search only user created objects filters.AddRootFilter() // search only user created objects
cli := internalclient.GetSDKClientByFlag(cmd, key.GetOrGenerate(cmd), commonflags.RPC) cli := internalclient.GetSDKClientByFlag(cmd, key.GetOrGenerate(cmd), commonflags.RPC)
@ -51,7 +51,7 @@ var listContainerObjectsCmd = &cobra.Command{
prmSearch.SetContainerID(id) prmSearch.SetContainerID(id)
prmSearch.SetFilters(*filters) prmSearch.SetFilters(*filters)
res, err := internalclient.SearchObjects(prmSearch) res, err := internalclient.SearchObjects(cmd.Context(), prmSearch)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err) commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
objectIDs := res.IDList() objectIDs := res.IDList()
@ -65,13 +65,14 @@ var listContainerObjectsCmd = &cobra.Command{
addr.SetObject(objectIDs[i]) addr.SetObject(objectIDs[i])
prmHead.SetAddress(addr) prmHead.SetAddress(addr)
resHead, err := internalclient.HeadObject(prmHead) resHead, err := internalclient.HeadObject(cmd.Context(), prmHead)
if err == nil { if err == nil {
attrs := resHead.Header().Attributes() attrs := resHead.Header().Attributes()
for i := range attrs { for i := range attrs {
attrKey := attrs[i].Key() attrKey := attrs[i].Key()
if !strings.HasPrefix(attrKey, v2object.SysAttributePrefix) { if !strings.HasPrefix(attrKey, v2object.SysAttributePrefix) && !strings.HasPrefix(attrKey, v2object.SysAttributePrefixNeoFS) {
// FIXME(@cthulhu-rider): neofs-sdk-go#226 use dedicated method to skip system attributes // FIXME(@cthulhu-rider): https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/issues/97
// Use dedicated method to skip system attributes.
cmd.Printf(" %s: %s\n", attrKey, attrs[i].Value()) cmd.Printf(" %s: %s\n", attrKey, attrs[i].Value())
} }
} }

View file

@ -3,13 +3,13 @@ package container
import ( import (
"crypto/sha256" "crypto/sha256"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client" internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
containerAPI "github.com/TrueCloudLab/frostfs-sdk-go/container" containerAPI "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/TrueCloudLab/frostfs-sdk-go/netmap" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@ -31,7 +31,7 @@ var containerNodesCmd = &cobra.Command{
var prm internalclient.NetMapSnapshotPrm var prm internalclient.NetMapSnapshotPrm
prm.SetClient(cli) prm.SetClient(cli)
resmap, err := internalclient.NetMapSnapshot(prm) resmap, err := internalclient.NetMapSnapshot(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "unable to get netmap snapshot", err) commonCmd.ExitOnErr(cmd, "unable to get netmap snapshot", err)
var id cid.ID var id cid.ID

View file

@ -0,0 +1,233 @@
package container
import (
"bufio"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"os"
"strings"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
type policyPlaygroundREPL struct {
cmd *cobra.Command
args []string
nodes map[string]netmap.NodeInfo
}
func newPolicyPlaygroundREPL(cmd *cobra.Command, args []string) (*policyPlaygroundREPL, error) {
return &policyPlaygroundREPL{
cmd: cmd,
args: args,
nodes: map[string]netmap.NodeInfo{},
}, nil
}
func (repl *policyPlaygroundREPL) handleLs(args []string) error {
if len(args) > 0 {
return fmt.Errorf("too many arguments for command 'ls': got %d, want 0", len(args))
}
i := 1
for id, node := range repl.nodes {
var attrs []string
node.IterateAttributes(func(k, v string) {
attrs = append(attrs, fmt.Sprintf("%s:%q", k, v))
})
fmt.Printf("\t%2d: id=%s attrs={%v}\n", i, id, strings.Join(attrs, " "))
i++
}
return nil
}
func (repl *policyPlaygroundREPL) handleAdd(args []string) error {
if len(args) == 0 {
return fmt.Errorf("too few arguments for command 'add': got %d, want >0", len(args))
}
id := args[0]
key, err := hex.DecodeString(id)
if err != nil {
return fmt.Errorf("node id must be a hex string: got %q: %v", id, err)
}
node := repl.nodes[id]
node.SetPublicKey(key)
for _, attr := range args[1:] {
kv := strings.Split(attr, ":")
if len(kv) != 2 {
return fmt.Errorf("node attributes must be in the format 'KEY:VALUE': got %q", attr)
}
node.SetAttribute(kv[0], kv[1])
}
repl.nodes[id] = node
return nil
}
func (repl *policyPlaygroundREPL) handleLoad(args []string) error {
if len(args) != 1 {
return fmt.Errorf("too few arguments for command 'add': got %d, want 1", len(args))
}
jsonNetmap := map[string]map[string]string{}
b, err := os.ReadFile(args[0])
if err != nil {
return fmt.Errorf("reading netmap file %q: %v", args[0], err)
}
if err := json.Unmarshal(b, &jsonNetmap); err != nil {
return fmt.Errorf("decoding json netmap: %v", err)
}
repl.nodes = make(map[string]netmap.NodeInfo)
for id, attrs := range jsonNetmap {
key, err := hex.DecodeString(id)
if err != nil {
return fmt.Errorf("node id must be a hex string: got %q: %v", id, err)
}
node := repl.nodes[id]
node.SetPublicKey(key)
for k, v := range attrs {
node.SetAttribute(k, v)
}
repl.nodes[id] = node
}
return nil
}
func (repl *policyPlaygroundREPL) handleRemove(args []string) error {
if len(args) == 0 {
return fmt.Errorf("too few arguments for command 'remove': got %d, want >0", len(args))
}
id := args[0]
if _, exists := repl.nodes[id]; exists {
delete(repl.nodes, id)
return nil
}
return fmt.Errorf("node not found: id=%q", id)
}
func (repl *policyPlaygroundREPL) handleEval(args []string) error {
policyStr := strings.TrimSpace(strings.Join(args, " "))
var nodes [][]netmap.NodeInfo
nm := repl.netMap()
if strings.HasPrefix(policyStr, "CBF") || strings.HasPrefix(policyStr, "SELECT") || strings.HasPrefix(policyStr, "FILTER") {
// Assume that the input is a partial SELECT-FILTER expression.
// Full inline policies always start with UNIQUE or REP keywords,
// or different prefixes when it's the case of an external file.
sfExpr, err := netmap.DecodeSelectFilterString(policyStr)
if err != nil {
return fmt.Errorf("parsing select-filter expression: %v", err)
}
nodes, err = nm.SelectFilterNodes(sfExpr)
if err != nil {
return fmt.Errorf("building select-filter nodes: %v", err)
}
} else {
// Assume that the input is a full policy or input file otherwise.
placementPolicy, err := parseContainerPolicy(repl.cmd, policyStr)
if err != nil {
return fmt.Errorf("parsing placement policy: %v", err)
}
nodes, err = nm.ContainerNodes(*placementPolicy, nil)
if err != nil {
return fmt.Errorf("building container nodes: %v", err)
}
}
for i, ns := range nodes {
var ids []string
for _, node := range ns {
ids = append(ids, hex.EncodeToString(node.PublicKey()))
}
fmt.Printf("\t%2d: %v\n", i+1, ids)
}
return nil
}
func (repl *policyPlaygroundREPL) netMap() netmap.NetMap {
var nm netmap.NetMap
var nodes []netmap.NodeInfo
for _, node := range repl.nodes {
nodes = append(nodes, node)
}
nm.SetNodes(nodes)
return nm
}
func (repl *policyPlaygroundREPL) run() error {
if len(viper.GetString(commonflags.RPC)) > 0 {
key := key.GetOrGenerate(repl.cmd)
cli := internalclient.GetSDKClientByFlag(repl.cmd, key, commonflags.RPC)
var prm internalclient.NetMapSnapshotPrm
prm.SetClient(cli)
resp, err := internalclient.NetMapSnapshot(repl.cmd.Context(), prm)
commonCmd.ExitOnErr(repl.cmd, "unable to get netmap snapshot to populate initial netmap: %w", err)
for _, node := range resp.NetMap().Nodes() {
id := hex.EncodeToString(node.PublicKey())
repl.nodes[id] = node
}
}
cmdHandlers := map[string]func([]string) error{
"list": repl.handleLs,
"ls": repl.handleLs,
"add": repl.handleAdd,
"load": repl.handleLoad,
"remove": repl.handleRemove,
"rm": repl.handleRemove,
"eval": repl.handleEval,
}
for reader := bufio.NewReader(os.Stdin); ; {
fmt.Print("> ")
line, err := reader.ReadString('\n')
if err != nil {
if err == io.EOF {
return nil
}
return fmt.Errorf("reading line: %v", err)
}
parts := strings.Fields(line)
if len(parts) == 0 {
continue
}
cmd := parts[0]
handler, exists := cmdHandlers[cmd]
if exists {
if err := handler(parts[1:]); err != nil {
fmt.Printf("error: %v\n", err)
}
} else {
fmt.Printf("error: unknown command %q\n", cmd)
}
}
}
var policyPlaygroundCmd = &cobra.Command{
Use: "policy-playground",
Short: "A REPL for testing placement policies",
Long: `A REPL for testing placement policies.
If a wallet and endpoint is provided, the initial netmap data will be loaded from the snapshot of the node. Otherwise, an empty playground is created.`,
Run: func(cmd *cobra.Command, args []string) {
repl, err := newPolicyPlaygroundREPL(cmd, args)
commonCmd.ExitOnErr(cmd, "could not create policy playground: %w", err)
commonCmd.ExitOnErr(cmd, "policy playground failed: %w", repl.run())
},
}
func initContainerPolicyPlaygroundCmd() {
commonflags.Init(policyPlaygroundCmd)
}

View file

@ -1,7 +1,8 @@
package container package container
import ( import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@ -15,7 +16,9 @@ var Cmd = &cobra.Command{
// the viper before execution // the viper before execution
commonflags.Bind(cmd) commonflags.Bind(cmd)
commonflags.BindAPI(cmd) commonflags.BindAPI(cmd)
common.StartClientCommandSpan(cmd)
}, },
PersistentPostRun: common.StopClientCommandSpan,
} }
func init() { func init() {
@ -28,6 +31,7 @@ func init() {
getExtendedACLCmd, getExtendedACLCmd,
setExtendedACLCmd, setExtendedACLCmd,
containerNodesCmd, containerNodesCmd,
policyPlaygroundCmd,
} }
Cmd.AddCommand(containerChildCommand...) Cmd.AddCommand(containerChildCommand...)
@ -40,6 +44,7 @@ func init() {
initContainerGetEACLCmd() initContainerGetEACLCmd()
initContainerSetEACLCmd() initContainerSetEACLCmd()
initContainerNodesCmd() initContainerNodesCmd()
initContainerPolicyPlaygroundCmd()
for _, containerCommand := range containerChildCommand { for _, containerCommand := range containerChildCommand {
commonflags.InitAPI(containerCommand) commonflags.InitAPI(containerCommand)

View file

@ -5,11 +5,12 @@ import (
"errors" "errors"
"time" "time"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client" internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@ -38,7 +39,7 @@ Container ID in EACL table will be substituted with ID from the CLI.`,
if !flagVarsSetEACL.noPreCheck { if !flagVarsSetEACL.noPreCheck {
cmd.Println("Checking the ability to modify access rights in the container...") cmd.Println("Checking the ability to modify access rights in the container...")
extendable, err := internalclient.IsACLExtendable(cli, id) extendable, err := internalclient.IsACLExtendable(cmd.Context(), cli, id)
commonCmd.ExitOnErr(cmd, "Extensibility check failure: %w", err) commonCmd.ExitOnErr(cmd, "Extensibility check failure: %w", err)
if !extendable { if !extendable {
@ -48,15 +49,15 @@ Container ID in EACL table will be substituted with ID from the CLI.`,
cmd.Println("ACL extension is enabled in the container, continue processing.") cmd.Println("ACL extension is enabled in the container, continue processing.")
} }
var setEACLPrm internalclient.SetEACLPrm setEACLPrm := internalclient.SetEACLPrm{
setEACLPrm.SetClient(cli) Client: cli,
setEACLPrm.SetTable(*eaclTable) ClientParams: client.PrmContainerSetEACL{
Table: eaclTable,
if tok != nil { Session: tok,
setEACLPrm.WithinSession(*tok) },
} }
_, err := internalclient.SetEACL(setEACLPrm) _, err := internalclient.SetEACL(cmd.Context(), setEACLPrm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err) commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
if containerAwait { if containerAwait {
@ -65,14 +66,17 @@ Container ID in EACL table will be substituted with ID from the CLI.`,
cmd.Println("awaiting...") cmd.Println("awaiting...")
var getEACLPrm internalclient.EACLPrm getEACLPrm := internalclient.EACLPrm{
getEACLPrm.SetClient(cli) Client: cli,
getEACLPrm.SetContainer(id) ClientParams: client.PrmContainerEACL{
ContainerID: &id,
},
}
for i := 0; i < awaitTimeout; i++ { for i := 0; i < awaitTimeout; i++ {
time.Sleep(1 * time.Second) time.Sleep(1 * time.Second)
res, err := internalclient.EACL(getEACLPrm) res, err := internalclient.EACL(cmd.Context(), getEACLPrm)
if err == nil { if err == nil {
// compare binary values because EACL could have been set already // compare binary values because EACL could have been set already
table := res.EACL() table := res.EACL()

View file

@ -3,11 +3,11 @@ package container
import ( import (
"errors" "errors"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/TrueCloudLab/frostfs-sdk-go/session" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )

View file

@ -0,0 +1,53 @@
package control
import (
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
"github.com/spf13/cobra"
)
const (
concurrencyFlag = "concurrency"
removeDuplicatesFlag = "remove-duplicates"
)
var doctorCmd = &cobra.Command{
Use: "doctor",
Short: "Restructure node's storage",
Long: "Restructure node's storage",
Run: doctor,
}
func doctor(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd)
req := &control.DoctorRequest{Body: new(control.DoctorRequest_Body)}
req.Body.Concurrency, _ = cmd.Flags().GetUint32(concurrencyFlag)
req.Body.RemoveDuplicates, _ = cmd.Flags().GetBool(removeDuplicatesFlag)
signRequest(cmd, pk, req)
cli := getClient(cmd, pk)
var resp *control.DoctorResponse
var err error
err = cli.ExecRaw(func(client *client.Client) error {
resp, err = control.Doctor(client, req)
return err
})
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
verifyResponse(cmd, resp.GetSignature(), resp.GetBody())
cmd.Println("Operation has finished.")
}
func initControlDoctorCmd() {
initControlFlags(doctorCmd)
ff := doctorCmd.Flags()
ff.Uint32(concurrencyFlag, 0, "Number of parallel threads to use")
ff.Bool(removeDuplicatesFlag, false, "Remove duplicate objects")
}

View file

@ -1,10 +1,10 @@
package control package control
import ( import (
rawclient "github.com/TrueCloudLab/frostfs-api-go/v2/rpc/client" rawclient "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/services/control" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )

View file

@ -1,18 +1,21 @@
package control package control
import ( import (
"github.com/TrueCloudLab/frostfs-api-go/v2/rpc/client" "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/services/control" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
const ignoreErrorsFlag = "no-errors"
var evacuateShardCmd = &cobra.Command{ var evacuateShardCmd = &cobra.Command{
Use: "evacuate", Use: "evacuate",
Short: "Evacuate objects from shard", Short: "Evacuate objects from shard",
Long: "Evacuate objects from shard to other shards", Long: "Evacuate objects from shard to other shards",
Run: evacuateShard, Run: evacuateShard,
Deprecated: "use frostfs-cli control shards evacuation start",
} }
func evacuateShard(cmd *cobra.Command, _ []string) { func evacuateShard(cmd *cobra.Command, _ []string) {
@ -20,7 +23,7 @@ func evacuateShard(cmd *cobra.Command, _ []string) {
req := &control.EvacuateShardRequest{Body: new(control.EvacuateShardRequest_Body)} req := &control.EvacuateShardRequest{Body: new(control.EvacuateShardRequest_Body)}
req.Body.Shard_ID = getShardIDList(cmd) req.Body.Shard_ID = getShardIDList(cmd)
req.Body.IgnoreErrors, _ = cmd.Flags().GetBool(dumpIgnoreErrorsFlag) req.Body.IgnoreErrors, _ = cmd.Flags().GetBool(ignoreErrorsFlag)
signRequest(cmd, pk, req) signRequest(cmd, pk, req)
@ -47,7 +50,7 @@ func initControlEvacuateShardCmd() {
flags := evacuateShardCmd.Flags() flags := evacuateShardCmd.Flags()
flags.StringSlice(shardIDFlag, nil, "List of shard IDs in base58 encoding") flags.StringSlice(shardIDFlag, nil, "List of shard IDs in base58 encoding")
flags.Bool(shardAllFlag, false, "Process all shards") flags.Bool(shardAllFlag, false, "Process all shards")
flags.Bool(dumpIgnoreErrorsFlag, false, "Skip invalid/unreadable objects") flags.Bool(ignoreErrorsFlag, false, "Skip invalid/unreadable objects")
evacuateShardCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag) evacuateShardCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag)
} }

View file

@ -0,0 +1,315 @@
package control
import (
"crypto/ecdsa"
"fmt"
"strings"
"sync/atomic"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
clientSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
)
const (
awaitFlag = "await"
noProgressFlag = "no-progress"
)
var evacuationShardCmd = &cobra.Command{
Use: "evacuation",
Short: "Objects evacuation from shard",
Long: "Objects evacuation from shard to other shards",
}
var startEvacuationShardCmd = &cobra.Command{
Use: "start",
Short: "Start evacuate objects from shard",
Long: "Start evacuate objects from shard to other shards",
Run: startEvacuateShard,
}
var getEvacuationShardStatusCmd = &cobra.Command{
Use: "status",
Short: "Get evacuate objects from shard status",
Long: "Get evacuate objects from shard to other shards status",
Run: getEvacuateShardStatus,
}
var stopEvacuationShardCmd = &cobra.Command{
Use: "stop",
Short: "Stop running evacuate process",
Long: "Stop running evacuate process from shard to other shards",
Run: stopEvacuateShardStatus,
}
func startEvacuateShard(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd)
ignoreErrors, _ := cmd.Flags().GetBool(ignoreErrorsFlag)
req := &control.StartShardEvacuationRequest{
Body: &control.StartShardEvacuationRequest_Body{
Shard_ID: getShardIDList(cmd),
IgnoreErrors: ignoreErrors,
},
}
signRequest(cmd, pk, req)
cli := getClient(cmd, pk)
var resp *control.StartShardEvacuationResponse
var err error
err = cli.ExecRaw(func(client *client.Client) error {
resp, err = control.StartShardEvacuation(client, req)
return err
})
commonCmd.ExitOnErr(cmd, "Start evacuate shards failed, rpc error: %w", err)
verifyResponse(cmd, resp.GetSignature(), resp.GetBody())
cmd.Println("Shard evacuation has been successfully started.")
if awaitCompletion, _ := cmd.Flags().GetBool(awaitFlag); awaitCompletion {
noProgress, _ := cmd.Flags().GetBool(noProgressFlag)
waitEvacuateCompletion(cmd, pk, cli, !noProgress, true)
}
}
func getEvacuateShardStatus(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd)
req := &control.GetShardEvacuationStatusRequest{
Body: &control.GetShardEvacuationStatusRequest_Body{},
}
signRequest(cmd, pk, req)
cli := getClient(cmd, pk)
var resp *control.GetShardEvacuationStatusResponse
var err error
err = cli.ExecRaw(func(client *client.Client) error {
resp, err = control.GetShardEvacuationStatus(client, req)
return err
})
commonCmd.ExitOnErr(cmd, "Get evacuate shards status failed, rpc error: %w", err)
verifyResponse(cmd, resp.GetSignature(), resp.GetBody())
printStatus(cmd, resp)
}
func stopEvacuateShardStatus(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd)
req := &control.StopShardEvacuationRequest{
Body: &control.StopShardEvacuationRequest_Body{},
}
signRequest(cmd, pk, req)
cli := getClient(cmd, pk)
var resp *control.StopShardEvacuationResponse
var err error
err = cli.ExecRaw(func(client *client.Client) error {
resp, err = control.StopShardEvacuation(client, req)
return err
})
commonCmd.ExitOnErr(cmd, "Stop evacuate shards failed, rpc error: %w", err)
verifyResponse(cmd, resp.GetSignature(), resp.GetBody())
waitEvacuateCompletion(cmd, pk, cli, false, false)
cmd.Println("Evacuation stopped.")
}
func waitEvacuateCompletion(cmd *cobra.Command, pk *ecdsa.PrivateKey, cli *clientSDK.Client, printProgress, printCompleted bool) {
const statusPollingInterval = 1 * time.Second
const reportIntervalSeconds = 5
var resp *control.GetShardEvacuationStatusResponse
reportResponse := new(atomic.Pointer[control.GetShardEvacuationStatusResponse])
pollingCompleted := make(chan struct{})
progressReportCompleted := make(chan struct{})
go func() {
defer close(progressReportCompleted)
if !printProgress {
return
}
cmd.Printf("Progress will be reported every %d seconds.\n", reportIntervalSeconds)
for {
select {
case <-pollingCompleted:
return
case <-time.After(reportIntervalSeconds * time.Second):
r := reportResponse.Load()
if r == nil || r.GetBody().GetStatus() == control.GetShardEvacuationStatusResponse_Body_COMPLETED {
continue
}
printStatus(cmd, r)
}
}
}()
for {
req := &control.GetShardEvacuationStatusRequest{
Body: &control.GetShardEvacuationStatusRequest_Body{},
}
signRequest(cmd, pk, req)
var err error
err = cli.ExecRaw(func(client *client.Client) error {
resp, err = control.GetShardEvacuationStatus(client, req)
return err
})
reportResponse.Store(resp)
if err != nil {
commonCmd.ExitOnErr(cmd, "Failed to get evacuate status, rpc error: %w", err)
return
}
if resp.GetBody().GetStatus() != control.GetShardEvacuationStatusResponse_Body_RUNNING {
break
}
time.Sleep(statusPollingInterval)
}
close(pollingCompleted)
<-progressReportCompleted
if printCompleted {
printCompletedStatusMessage(cmd, resp)
}
}
func printCompletedStatusMessage(cmd *cobra.Command, resp *control.GetShardEvacuationStatusResponse) {
cmd.Println("Shard evacuation has been completed.")
sb := &strings.Builder{}
appendShardIDs(sb, resp)
appendCounts(sb, resp)
appendError(sb, resp)
appendStartedAt(sb, resp)
appendDuration(sb, resp)
cmd.Println(sb.String())
}
func printStatus(cmd *cobra.Command, resp *control.GetShardEvacuationStatusResponse) {
if resp.GetBody().GetStatus() == control.GetShardEvacuationStatusResponse_Body_EVACUATE_SHARD_STATUS_UNDEFINED {
cmd.Println("There is no running or completed evacuation.")
return
}
sb := &strings.Builder{}
appendShardIDs(sb, resp)
appendStatus(sb, resp)
appendCounts(sb, resp)
appendError(sb, resp)
appendStartedAt(sb, resp)
appendDuration(sb, resp)
appendEstimation(sb, resp)
cmd.Println(sb.String())
}
func appendEstimation(sb *strings.Builder, resp *control.GetShardEvacuationStatusResponse) {
if resp.GetBody().GetStatus() != control.GetShardEvacuationStatusResponse_Body_RUNNING ||
resp.GetBody().GetDuration() == nil ||
resp.GetBody().GetTotal() == 0 ||
resp.GetBody().GetEvacuated()+resp.GetBody().GetFailed() == 0 {
return
}
durationSeconds := float64(resp.GetBody().GetDuration().GetSeconds())
evacuated := float64(resp.GetBody().GetEvacuated() + resp.GetBody().GetFailed())
avgObjEvacuationTimeSeconds := durationSeconds / evacuated
objectsLeft := float64(resp.GetBody().GetTotal()) - evacuated
leftSeconds := avgObjEvacuationTimeSeconds * objectsLeft
leftMinutes := int(leftSeconds / 60)
sb.WriteString(fmt.Sprintf(" Estimated time left: %d minutes.", leftMinutes))
}
func appendDuration(sb *strings.Builder, resp *control.GetShardEvacuationStatusResponse) {
if resp.GetBody().GetDuration() != nil {
duration := time.Second * time.Duration(resp.GetBody().GetDuration().GetSeconds())
hour := int(duration.Seconds() / 3600)
minute := int(duration.Seconds()/60) % 60
second := int(duration.Seconds()) % 60
sb.WriteString(fmt.Sprintf(" Duration: %02d:%02d:%02d.", hour, minute, second))
}
}
func appendStartedAt(sb *strings.Builder, resp *control.GetShardEvacuationStatusResponse) {
if resp.GetBody().GetStartedAt() != nil {
startedAt := time.Unix(resp.GetBody().GetStartedAt().GetValue(), 0).UTC()
sb.WriteString(fmt.Sprintf(" Started at: %s UTC.", startedAt.Format(time.RFC3339)))
}
}
func appendError(sb *strings.Builder, resp *control.GetShardEvacuationStatusResponse) {
if len(resp.Body.GetErrorMessage()) > 0 {
sb.WriteString(fmt.Sprintf(" Error: %s.", resp.Body.GetErrorMessage()))
}
}
func appendStatus(sb *strings.Builder, resp *control.GetShardEvacuationStatusResponse) {
var status string
switch resp.GetBody().GetStatus() {
case control.GetShardEvacuationStatusResponse_Body_COMPLETED:
status = "completed"
case control.GetShardEvacuationStatusResponse_Body_RUNNING:
status = "running"
default:
status = "undefined"
}
sb.WriteString(fmt.Sprintf(" Status: %s.", status))
}
func appendShardIDs(sb *strings.Builder, resp *control.GetShardEvacuationStatusResponse) {
sb.WriteString("Shard IDs: ")
for idx, shardID := range resp.GetBody().GetShard_ID() {
shardIDStr := shard.NewIDFromBytes(shardID).String()
if idx > 0 {
sb.WriteString(", ")
}
sb.WriteString(shardIDStr)
if idx == len(resp.GetBody().GetShard_ID())-1 {
sb.WriteString(".")
}
}
}
func appendCounts(sb *strings.Builder, resp *control.GetShardEvacuationStatusResponse) {
sb.WriteString(fmt.Sprintf(" Evacuated %d object out of %d, failed to evacuate %d objects.",
resp.GetBody().GetEvacuated(),
resp.Body.GetTotal(),
resp.Body.GetFailed()))
}
func initControlEvacuationShardCmd() {
evacuationShardCmd.AddCommand(startEvacuationShardCmd)
evacuationShardCmd.AddCommand(getEvacuationShardStatusCmd)
evacuationShardCmd.AddCommand(stopEvacuationShardCmd)
initControlStartEvacuationShardCmd()
initControlFlags(getEvacuationShardStatusCmd)
initControlFlags(stopEvacuationShardCmd)
}
func initControlStartEvacuationShardCmd() {
initControlFlags(startEvacuationShardCmd)
flags := startEvacuationShardCmd.Flags()
flags.StringSlice(shardIDFlag, nil, "List of shard IDs in base58 encoding")
flags.Bool(shardAllFlag, false, "Process all shards")
flags.Bool(ignoreErrorsFlag, true, "Skip invalid/unreadable objects")
flags.Bool(awaitFlag, false, "Block execution until evacuation is completed")
flags.Bool(noProgressFlag, false, fmt.Sprintf("Print progress if %s provided", awaitFlag))
startEvacuationShardCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag)
}

Some files were not shown because too many files have changed in this diff Show more