Compare commits

...

733 commits

Author SHA1 Message Date
f2437f7ae9 [#734] shard: Fix Delete method
Due to the flushing data from the writecache to the storage
and simultaneous deletion, a partial deletion situation is possible.
So as a solution, deletion is allowed only when the object is in storage,
because object will be deleted from writecache by flush goroutine.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
f26233b47a [#734] metabase: Include UpdateStorageID in metrics and traces
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:18 +03:00
7e0c5a55de [#734] writecache: Fix flush
Now UpdateStorageID doesn't return error in case of logical error.
If object is in graveyard or GC market, it is still required to
update storage ID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-16 17:00:17 +03:00
d5c10612f4 [#735] policer: Register metrics
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-12 09:31:36 +03:00
3a997d1207 [#680] metrics: Initialize log metrics together with services
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
bf082348d4 [#680] metrics: Add step export-metrics in Makefile
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
994f48f8bb [#680] metrics: Export log and morph with script
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-11 17:08:03 +03:00
aca11d7474 [#735] policer: Allow to provide metrics from the outside
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-11 15:14:13 +03:00
5e229dc248 [#701] metrics: add metric to evaluate policer performance
Add processed objects counter in policerMetrics,
add policer field to NodeMetrics

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-10-09 19:02:08 +00:00
4caa934eea [#729] containersvc: Remove load announcement
IR code was removed in 8879c6ea.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-10-09 19:01:13 +00:00
d07afd803c [#726] writecache: Fix small object flush for Badger
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-06 11:32:50 +03:00
997ac7cd8d [#726] writecache: Fix small object flush for BBolt
Do not marshal object twice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-06 11:32:44 +03:00
bd5bf8b1a9 [#721] netmap: Drop already bootstraped check
Because of this check, under certain conditions,
the node could be removed from the network map,
although the node was functioning normally.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-05 11:47:06 +03:00
f3278d76a9 [#721] netmap: Send bootstrap at each epoch tick
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-10-05 11:46:56 +03:00
627b302745 [#709] node: Put in log info about listening endpoints
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-10-02 13:31:10 +00:00
a0a35ffbec [#702] node: Update SDK version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-29 18:41:48 +03:00
c1e4130020 [#146] node: Add trace_id to logs
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-09-27 11:05:27 +03:00
b8c3c2486d [#333] Sort containers by ID
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-25 11:30:36 +03:00
c14c9a023c [#333] Sort objects by ID in SearchObjects
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-25 11:30:36 +03:00
d9b93b12c1 [#333] Sort shards by shard_ID in cli output
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-25 10:04:29 +03:00
3889e829e6 [#667] writecache: Add logs for report error func in tests
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-09-14 17:02:54 +00:00
10570fc035 [#690] go.mod: Update contract and api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-14 14:50:30 +03:00
c6af4a3ec8 [#679] engine: Do not increase error counter on meta mismatch
It was introduced in 69e1e6ca to help node determine faulty shards.
However, the situation is possible in a real-life scenario:
1. Object O is evacuated from shard A to B.
2. Shard A is unmounted because of lower-level errors.
3. We now have object in meta on A and in blobstor on B. Technically we
   have it in meta on shard B too, but we still got the error if B goes
   to a degraded mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-14 10:39:18 +03:00
58239d1b2c [#683] cli: Add context to policy parsing errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-12 09:47:21 +00:00
3c76884182 [#682] cli: Unify array of ranges type
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-12 10:42:34 +03:00
f435ab1b26 [#682] go.mod: Update sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-12 10:42:34 +03:00
aa9f8dce3d [#677] client: Refactor PrmAnnounceSpace/EndpointInfo/NetworkInfo usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-09-08 09:42:28 +03:00
8a81af5a3b [#653] Add context parameter to Open functions
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2023-09-07 18:03:29 +03:00
a716db99db [#668] shard/test: Do not alter rootPath option
Supposedly, this was added to allow creating 2 different shards without
subtest. Now we use t.TempDir() everywhere, so this should not be a
problem.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
36759f8434 [#668] shard/test: Properly check event processing
See https://git.frostfs.info/TrueCloudLab/frostfs-node/actions/runs/1594/jobs/2

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
39879fa868 [#668] shard/test: Add dontRelease options
Most of the time we would like to close shard with minor exceptions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
c661ba1312 [#668] shard/test: Use sane defaults in the test constructor
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
268adb79cb [#668] shard/test: Simplify shard construction
newCustomShard() has many parameters but only the first is obligatory.
`enableWriteCache` is left as-is, because it directly affects the
functionality.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
429f941cda [#668] shard/test: Release shard in t.Cleanup()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
382eb8a485 [#668] shard/test: Disable GC where it is not needed
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
42696016de [#668] shard: Close stopChannel in GC
It is done once, but now we could read it from multiple places.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
bdecfbc1be [#668] shard/test: Move tests to the main package
Semantic patch (also, duplicate definitions are removed):
```
@@
var e identifier
@@
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"

-shard.e
+e
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
aa23c6a83a [#668] shard/test: Remove subtest from TestCounters
Otherwise, individual tests cannot be run.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
da8f384324 [#668] shard/test: Fix typo in existence
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-07 07:39:39 +00:00
aeeb8193d2 [#676] node: Fix header source creation when checking eacl
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-09-06 17:06:54 +03:00
d9de9e2bbb [#675] client: Refactor PrmObjectDelete usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-09-06 08:05:47 +00:00
88d50e4c77 [#656] policer: Add "bad" testcase
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-09-06 08:04:59 +00:00
054e3ef3d3 [#674] pre-commit: Update shellcheck-py
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-04 15:34:24 +03:00
a54b4472de [#674] network: Close connections on address updates
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-09-04 15:34:24 +03:00
7456c8556a [#536] blobovnicza: Add blobovniczatree DB cache
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:53:11 +03:00
c672f59ab8 [#536] blobovnicza: Drop cache
Each blobovnicza instance is opened
while is in use.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-09-01 13:51:26 +03:00
b9b86d2ec8 [#666] shard/test: Fix data race in metrics tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-31 08:39:42 +00:00
4dff9555f1 [#568] writecache: Improve flushing scheme for badger
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-30 17:22:28 +00:00
806cc13d9f [#658] client: Refactor PrmObjectGet/Head/Range usage
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-30 17:13:23 +00:00
1daef2ceeb [#660] writecache: Fix remaining addr2key uses
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-30 17:12:33 +00:00
fe5aa06a75 [#665] node: Bind length of copies number to number of replicas
Allow to use one digit in copies number array for backward compatibility.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-30 17:11:55 +00:00
91f3745b58 [#659] debian: Remove nspcc email
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-30 08:29:26 +00:00
7654847f79 [#659] adm: Remove nspcc.ru from the default email
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-30 08:29:26 +00:00
a724debb19 [#632] .forgejo: Print --version
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-29 12:41:45 +03:00
55b82e744b [#529] objectcore: Use common sender classifier
Use common sender classifier for ACL service and format validator.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:33:06 +03:00
ae81d6660a [#529] objectcore: Fix object content validation
There are old objects where the owner of the object
may not match the one who issued the token.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:33:06 +03:00
ab2614ec2d [#528] objectcore: Validate token issuer
Add token issuer against object owner validation.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-29 10:10:10 +03:00
4ea0df77d0 [#574] policer: Check if the container was really removed
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
554ff2c06b [#574] core: Extend Source interface with DeletionInfo method
* Introduce common method EverExisted
* Define DeletionInfo for struct that must implement Source
* Refactor tree srv

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-28 14:21:38 +00:00
9072772a09 [#649] shard/test: Increase GC remover interval
This was set in #348 to speed up tests.
It seems 100ms doesn't increase overall test time,
but it reduces the amount of logs by 100x factor.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
c4db8e7690 [#637] shard/test: Fix data race
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:10:25 +00:00
f8ba60aa0c [#648] objsvc/delete: Handle errors in Go style
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 09:45:35 +00:00
d2084ece41 [#648] objsvc/delete: Remove redundant logs
We never propagate delete requests to the container node, because
tombstone broadcast is done via PUT. No need to pollute logs.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 09:45:35 +00:00
40b556fc19 [#647] objsvc/search: Improve testing coverage
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
4db2cbc927 [#647] objsvc/search: Wrap in uniqueIDWriter during parameter setting
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
966ad22abf [#647] objsvc/search: Simplify error handling
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
56f841b022 [#647] objsvc/search: Remove TraverserGenerator wrapper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
ba58144de1 [#647] objsvc/search: Remove netmap.Source wrapper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-25 10:40:01 +03:00
c9e3c9956e [#643] objsvc/put: Unify extraBroadcastEnabled usage
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
facd3b2c4b [#643] objsvc/put: Unify placement iterators
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
3fcf56f2fb [#643] objsvc/put: Copy config to distributedTarget
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-24 11:03:17 +03:00
96e690883f [#638] Unify test loggers
In some places we have debug=false, in others debug=true.
Let's be consistent.

Semantic patch:
```
@@
@@
-test.NewLogger(..., false)
+test.NewLogger(..., true)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
322c1dc273 [#638] Use test.NewLogger() in tests
Semantic patch (restricted to **/*_test.go):
```
@@
@@
+import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger/test"
-import "go.uber.org/zap"
-import "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger"

-&logger.Logger{Logger: zap.L()}
+test.NewLogger(t, false)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
02b03d9c4f [#638] logger: Remove sampling from test loggers
Losing logs is always a bad idea, especially when we debug tests.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 11:21:05 +00:00
82cc453be9 [#xx] shard: Fix data race in metrics tests
Protect test metric store fields with a mutex. Probably, not every field
should be protected, but better safe than sorry.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-23 10:26:12 +00:00
238b8f10a0 [#630] cli: Fix SDK SetEACLPrm usage for PrmContainerSetEACL
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-22 14:25:39 +00:00
345a1a69a2 [#635] Use internal key type when deleting from badger wc
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-22 10:53:19 +03:00
dc3bc08c07 [#631] lens: Fix db type flag name
Typo from 1a0cb0f34a.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-21 17:18:05 +00:00
23be3eb627 [#574] tree: Check if container is really removed
* Use DeletionInfo method from morph client to check if
  the container has been really removed from neo-go

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-21 12:50:20 +03:00
42fb6fb372 [#574] morph: Add DeletionInfo method for morph client
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-21 12:49:06 +03:00
62c2ad4b22 [#626] logs: Remove autogenerated comments
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-21 11:15:06 +03:00
84ea075587 [#625] cli: Fix SDK EACLPrm usage for PrmContainerEACL
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-21 10:36:47 +03:00
354a92ea2c [#602] blobovnicza: Add leaf width implementation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-21 10:27:32 +03:00
d3904ec599 [#602] config: Add blobovnicza leaf width parameter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-21 10:27:32 +03:00
4d9a6c07fb [#618] core: Replace fmt.Sprintf with strconv.FormatUint
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-21 07:14:50 +00:00
a1f1d233cc [#618] linters: bump truecloudlab-linters to 0.0.2
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-21 07:14:50 +00:00
f2811f8585 [#602] metrics: Add blobovnicza items counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-18 13:01:27 +03:00
c4e1d8eb07 [#602] node: Fix blobovnicza typos
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-18 11:14:10 +03:00
10e63537b2 [#602] metrics: Rename blobovnicza size metric
`Size` is not size, but open db size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-17 19:17:35 +00:00
809e97626b [#602] blobovnicza: Fix size counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-17 19:17:35 +00:00
2e49d7ea7e [#602] blobovnicza: Init before using
Fix blobovnicza size: after restart size metric resets.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-17 19:17:35 +00:00
f7042c5a6f [#609] Replace zaptest.NewLogger() with zap.L()
Semantic patch:
```
@@
@@
-import "go.uber.org/zap/zaptest"
+import "go.uber.org/zap"

-zaptest.NewLogger(t)
+zap.L()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:51:10 +00:00
127c676786 [#607] *: Use keys.PublicKeys.Copy() where possible
Semantic patch:
```
@@
var dst identifier
var src identifier
var keys identifier
@@
 import keys "github.com/nspcc-dev/neo-go/pkg/crypto/keys"

-dst := make(keys.PublicKeys, len(src))
-copy(dst, src)
+dst := src.Copy()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
e604a3d749 [#607] *: Use zap.Stringer() where possible
Semantic patch:
```
@@
var f expression
var t expression
var a expression
@@
 f(
    ...,
-    zap.String(t, a.String()),
+    zap.Stringer(t, a),
    ...,
)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
a8de37c8a2 [#607] *: Remove redundant if on error returns
Semantic patch:
```
@@
@@
-if err != nil { return err }
-return nil
+return err
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 16:50:02 +00:00
5a51b78946 [#620] object: Send status response for server-side streams
Previously status responses were wrapped in the gRPC error and thus
couldn't be correctly handled on client.

Introduced in c2617baf63, thanks @ale64bit for having found.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-17 12:52:38 +00:00
6407bb5bd1 [#619] node: Fix object put when copies numbers contains only zeros
In this case object should placement according to replicas.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-17 11:31:34 +03:00
5335e7089e [#615] pilorama: Speedup TestForest_ApplyRandom()
Some of our pilorama tests fail on CI.
The reasons are not obvious, but one possible improvement
is using `WithNoSync` option for these. It should have much effect,
because we are writing on the tmpfs, but doesn't hurt anyway.

If I replace `t.TempDir()` with a local directory, test execution time
goes down from 5s (sync) to 0.4s (nosync), which is the same time as
with `t.TempDir()`. Maybe we have some strange CI configuration.

```
panic: test timed out after 10m0s
running tests:
	TestForest_ApplyRandom (8m22s)
	TestForest_ApplyRandom/bbolt (8m21s)
...
goroutine 170 [syscall]:
syscall.Syscall(0xc000100000?, 0xc00047b758?, 0x6aff9a?, 0xc00041c1b0?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/syscall_linux.go:69 +0x27
syscall.Fdatasync(0x9e35c0?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/zsyscall_linux_amd64.go:418 +0x2a
go.etcd.io/bbolt.fdatasync(0xc000189000?)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-16 19:09:08 +00:00
2efe9cc1be [#585] writecache: Fix DB counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
58c8722c81 [#585] fstree: Add optional file counter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
baad49990c [#585] fstree: Return logical error if object deleted
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
0c52186572 [#585] fstree: Remove unused method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
eec97d177e [#585] writecache: Count items periodically
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-16 14:47:44 +03:00
d15199c5d8 [#596] engine: Consider context errors as logical
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-16 10:39:41 +03:00
bc425b5bad [#608] pre-commit: Fix linter
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-15 07:42:46 +00:00
88b6755c5e [#598] Fix use-after-close bug in badger writecache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-15 07:04:06 +00:00
ae8be495c8 [#xx] Avoid manual management of files in tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-14 14:01:39 +03:00
376f03a445 [#598] Hold mode mutex when setting mode
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-14 07:12:27 +00:00
ad87493c41 [#8] Bump required go version to go1.20
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-11 15:51:06 +03:00
21800e9fcc [#162] core: Move literals to constants
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-11 15:48:42 +03:00
a5f51add25 [#162] ci: Add noliteral linter
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-08-11 15:48:42 +03:00
abdb0910cc [#600] adm/tests: Add missing WaitGroup.Add()
```
panic: sync: negative WaitGroup counter
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-11 09:50:07 +00:00
4d2af137e9 [#500] .forgejo: Add DCO action
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-11 09:47:41 +00:00
b44a8dd46c [#597] *: Fix linter warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-11 09:55:51 +03:00
20af34ecdb [#591] cli: fix SDK PrmContainerDelete usage for DeleteContainerPrm
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-10 16:40:35 +00:00
dd988a5912 [#592] adm: Allow to tick epoch concurrently
Epoch can be ticked by IR in the background. Because the only usecase
for this is to apply some changes, the right behaviour is ignoring an
error, not retrying.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:40:00 +00:00
32a9f51586 [#562] logs: Remove superfluous comment
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
e084c47bd6 [#248] innerring: Remove audit from tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
779da6ec35 [#248] morph: Remove audit contract name
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
2685b1d548 [#248] adm: Do not deploy audit contract
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
55ce4dc075 [#248] morph: Remove obsolete network parameters
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
c54fcb297d [#248] cli,node: Remove obsolete network parameters
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
26a78aa366 [#248] adm: Remove obsolete network parameters
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 16:38:43 +00:00
4ad0ebb32f [#565] Add metrics for current GRPC endpoint status
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-10 16:37:49 +00:00
c3e23a1448 [#578] gendoc: Allow to override flags
The command is used in multiple places across the whole FrostFS
ecosystem. While we want to have uniform interfaces everywhere,
sometimes we can't: already defined global flags can be harder to change
because of our obligations to the users. Cobra framework doesn't allow
conflicting flags (we can have global ones), so allow to override them.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
6bcba27757 [#578] gendoc: Remove flag shorthands
Make it harder to encounter conflicts in already existing commands.
Because the command is executed once, I don't think the usability is
worse.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
dca31e8888 [#578] gendoc: Add date to man pages
```
HISTORY
       8-Aug-2023 Auto generated by spf13/cobra
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
b02a1a34c1 [#578] gendoc: Allow to customize man pages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
082370583f [#578] gendoc: Replace NSPCC with FrostFS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 11:10:11 +00:00
34b5d90441 [#590] cli: fix SDK PrmContainerPut usage for PutContainerPrm
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com
2023-08-10 11:09:31 +00:00
6186329aec [#590] cli: fix SDK PrmContainerGet usage for GetContainerPrm
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-08-10 11:09:31 +00:00
c3c0574e3c [#589] cli: Add control shards evacuate command
It was accidentally removed in f7c0b50d70.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-10 09:06:06 +00:00
8f994163ee [#586] Fix writecache benchmarks and refactor hacky NeedsCompression
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-10 08:05:18 +00:00
023b90342c [#584] Disable compression in badger writecache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-10 10:14:06 +03:00
d641cba2fc [#587] Do not use math/rand.Read
Fix staticcheck warnings after go1.20 update.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-09 16:02:44 +03:00
33c11be0cf [#587] .golangci.yml: Fix pre-commit warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-09 16:02:44 +03:00
5b7e4a51b7 [#481] Update frostfs-sdk-go and error pointer receivers
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-09 10:26:53 +00:00
de3d1eb99c [#581] Bump required go version to go1.20
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-09 10:16:48 +00:00
ae322e9f73 [#576] Set SyncWrites for badger writecache by default
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-09 11:28:26 +03:00
8d589314b5 [#560] node: Fix Put in multi REP with intersecting sets of nodes
Once the node was processed it skipped, at the step of forming
result in case when all nodes skipped, because processed for
previous REP, service mark the whole request as incomplete.

Example of policies which are unblocked:
- REP 1 REP 1 CBF 1
- REP 4 IN X REP 4 IN Y
  CBF 4
  SELECT 2 FROM FX AS X SELECT 2 FROM FY AS Y
  FILTER Country EQ Russia OR Country EQ Sweden OR Country EQ Finland AS FY
  FILTER Price GE 0 AS FX

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-08 10:22:53 +00:00
7da4306e38 [#575] writecache: Fix log level for badger writecache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-08 08:16:36 +00:00
d3a52ec73a [#516] node: Send bootstrap request if attributes were updated
Consider following situation:
1. Epoch tick.
2. Update node attributes in the config.
3. Restart node.

Because we already sent bootstrap query after (1) and we are still
online, new bootstrap query won't be sent. This leads to the node
attributes being updated after another epoch.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-08 07:40:26 +00:00
0e697266c3 [#563] writecache: Fix metrics and bolt delete
Estimate cache size after delete objects to update metric.
Update counters on small object deletion.
Do not count bbolt DB file as FSTree object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-08-07 12:53:28 +00:00
5bbfebba2d [#570] Fix writecache type constant copy-pasta bug
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-07 12:06:44 +03:00
1a0cb0f34a [#421] Try using badger for the write-cache
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-07 08:16:57 +00:00
65c72f3e0b [#559] Remove manual path handling in fstree tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-08-03 10:03:41 +03:00
1e8b4b8a17 [#557] services: Regenerate stable marshalers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-02 13:37:40 +00:00
435a581b5e [#557] go.mod: Update sdk-go and api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-08-02 13:37:40 +00:00
93c46cfdf0 [#550] cli: make get-op-log meta pretty formatted
Close #550

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-02 10:27:15 +00:00
1b7b54ba89 [#540] cli: refactor client parameters usage
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-02 09:44:51 +00:00
b3695411d9 [#553] eacl: Fix bug with casting to ObjectAccessDenied error
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-08-02 07:22:48 +00:00
ec8a631d31 [#542] Update test for SizeInBytesSafe function
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-01 13:38:00 +00:00
9ca63ac8c3 [#542] Fix bug in SizeInBytesSafe function
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-08-01 13:38:00 +00:00
b8052c794e [#547] go.mod: Update api-go, sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
35dc64bd7b [#547] metabase: Fix datarace in tests
Quite an old one bf9e938a3b.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
05ac9e3637 [#547] objectsvc: Work with traversal struct from a single thread
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
7b0fdf0202 [#533] services: Assume API supports status codes
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-31 15:35:15 +00:00
ec8b4fdc48 [#541] writecache/test: Close writecache on exit
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-29 10:57:25 +00:00
ad5f527bd3 [#541] writecache/test: Remove initWC()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-29 10:57:25 +00:00
ea32913430 [#543] putsvc: Fix PutSingle implementation
Add Lock and Delete handlers to local PutSingle.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-28 12:09:41 +00:00
99bb488ebd [#539] getsvc: Write payload direct to out stream
To reduce memory allocations.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-27 17:02:08 +03:00
286242cad0 [#539] getsvc: Use buffer to assemble object
To reduce memory consumption.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-27 17:02:08 +03:00
32c77f3a23 [#537] node: Add runtime.memory_limit config parameter
This parameter allows to set soft memory limit for Go GC.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-25 17:27:09 +03:00
5ff82ff04f [#6] services/object: Simplify local/remote targets
We do not use the return result from Close() and we always execute both
methods in succession. It makes sense to unite them.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:39:12 +03:00
448b48287c [#6] services/util: Do not panic in sign function
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:39:12 +03:00
c2617baf63 [#6] services/util: Remove remaining stream wrappers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:39:09 +03:00
372160d048 [#6] services/util: Remove SignService.HandleUnaryRequest
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:36:08 +03:00
fef172c5b0 [#6] services/util: Simplify response.Service
It has only 1 parameter, which is obligatory.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-21 18:29:57 +03:00
5a4054eeb6 [#535] .forgejo: Add vulncheck workflow
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-20 09:52:03 +00:00
eed594431f [#335] treesvc: Add GetSubTree ordering unit test
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
af82c2865e [#335] treesvc: Fix inmemory unit tests and nil meta items
Bolt forest saves empty slice of items. Now inmemory forest
does it the same way.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
b4e72a2dfd [#335] treesvc: Sort nodes by Filename in GetSubTree
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-20 10:14:10 +03:00
94df541426 [#530] go.mod: Update frostfs-sdk-go and frostfs-api-go versions
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-07-19 16:15:21 +03:00
57e7fb5ccf [#521] cli: Add common aliases to policy playground commands
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-19 11:34:08 +00:00
24dffdac6f [#521] cli: Add netmap load command to policy playground
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-19 11:34:08 +00:00
a9d04ba86f [#244] Remove --local-dump from frostfs-adm config
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-18 13:51:25 +00:00
6429975584 [#511] docs: Remove GitHub mentions from CONTRIBUTING.md
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 13:24:02 +00:00
4680087711 [#527] Add support for select-filter expressions in policy playground
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-18 13:15:58 +00:00
f0355a453e [#463] policer: Remove capacity rebalance logic
Current implementation has some quirks. For example,
using only half of object.put.pool_size_remote threads
tells replicator that is node is 50% loaded,
but in reality we could be putting lot's of big objects.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
8b78db74bc [#463] replicator: Add tracing span
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
8966dd8e35 [#463] putsvc: Use PutSingle RPC for remote target
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:52:12 +00:00
b2487e8cc5 [#516] node: Do not bootstrap if node is online candidate
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-18 10:51:02 +00:00
3b66f98f27 [#519] Use Address.Equals in policer tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-17 10:45:07 +00:00
3e8de14e7d [#382] evacuate: Fix unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-17 10:24:05 +00:00
a0c7045f29 [#512] cli: Refactor UX for bearer create command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-17 11:45:27 +03:00
d8e37a827f [#497] config: Add examples and unit tests
Add examples and unit tests for tree.authorized_keys section.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-17 08:23:28 +00:00
486287c2f7 [#524] cli: Add impersonate flag for bearer token creation
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-17 08:20:50 +00:00
397131b0ea [#520] objectcore: Refactor format validator
Remove redundant FIXME.
Move error to consts.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-17 07:07:21 +00:00
11027945d8 [#479] writecache: Fix writecache fstree flush premature ctx cancel
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-14 10:25:52 +03:00
8a9fc2c372 [#510] treesvc: Rename tableFromBearer to useBearer
With impersonation, the old name is no longer descriptive.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-12 10:23:21 +00:00
b8bcfac531 [#510] treesvc: Fix panic in bearer token processing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-12 10:23:21 +00:00
24eb988897 [#294] deletesvc: Drop cast
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
c83e7c875f [#294] searchsvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
e8091101c7 [#294] searchsvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
ec9b738465 [#294] putsvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
800a685e84 [#294] putsvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
1420b8b9ea [#294] deletesvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
a476d8285a [#294] deletesvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
70a1081988 [#294] aclsvcv2: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
18d8898b00 [#294] aclsvc: Refactor service constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
61541eaec2 [#294] aclsvc: Refactor checker constructor
Pass required deps as args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-12 07:42:10 +00:00
7da284f3e8 [#509] go.mod: Update sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-11 17:38:00 +03:00
80481c015c [#509] putsvc: Omit AccessIdentifiers from iteratePlacement()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-11 17:38:00 +03:00
0754e6e654 [#390] cli: Fix bearer token reading for tree subcommands
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-07-11 14:11:30 +00:00
676a3efa79 [#390] cli: Make tree commands errors and messages more verbose
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-07-11 14:11:30 +00:00
c42db4e761 [#390] cli: Support more tree service API calls in CLI
* Support healthcheck API call
* Support move API call
* Support remove API call
* Support getSubtree API call
* Support getOpLog API call

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-07-11 14:11:30 +00:00
a65e26878b [#486] put service: Fix error typo
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
fcbf90d31b [#486] node: Add PutSingle implemetation
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
7b76527759 [#486] node: Add PutSingle wrappers
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
9be5d44a46 [#486] node: Update api-go and sdk-go versions
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 15:49:21 +03:00
040a623d39 [#476] cli: Fix object nodes command
Do not fail if client creation failed.
Use external addresses to create the client too.
Use public key as node ID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-10 09:58:31 +03:00
140d970a95 [#498] policer: Fix objectSDK import
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-07 18:52:40 +03:00
2310a5c7ba [#498] policer: Allow to set sleep duration between iterations
Speed up tests on CI.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-07 14:52:33 +00:00
e858479a74 [#498] policer: Explicitly Rewind() iterator after finish
Previously, we can continue to return `EndOfListing` infinitely.
Reflect iterator reuse via Rewind() method.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-07 14:52:33 +00:00
a0d51090a4 [#482] Enable staticcheck in foregjo actions
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-07 10:12:49 +03:00
033eaf77e1 [#496] node: Fix linter importas
Standardize the alias of the
import frostfs-sdk-go/object as objectSDK.

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-07-06 15:36:41 +03:00
4bbe9cc936 [#496] .golangci.yml: Add importas linter
Add importas linter

Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2023-07-06 15:35:23 +03:00
6eefe9747e treesvc: Do not provide credentials unless TLS is used
4f413fe86e was perfect, except it did the opposite of what we needed.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-07-06 08:04:28 +00:00
f354b8a270 [#473] Add tests for rem values of traverser
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-07-05 20:13:23 +03:00
c6df6c84ae [#473] Do not modify traverse plan when [0] copies number vector is provided
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-07-05 19:54:43 +03:00
b520a3049e [#483] cli: Fix object put cmd with flag --prepare-locally
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-04 18:58:26 +03:00
4c248d573e [#483] cli: Allow to split object on the client side
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-03 18:31:46 +03:00
90e9a85acc [#483] Update dependencies
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-07-03 15:44:59 +03:00
8d16d95376 [#484] morph: Add expired tx logging
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-07-03 08:03:25 +00:00
26acf5689e [#92] Ensure policer objects cannot be worked on concurrently
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
3223402c90 [#92] Embed policer's objectsInWork mutex
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
f9730f090d [#92] Refactor policer and add some unit tests
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-07-03 07:05:31 +00:00
0c5b025788 [#470] grpc: Increase message limits
For send message limit set to 2GiB, but there are custom
GET/GET RANGE limiters.
For receive message limit set to 256 MiB, but actual chunk size
will be managed by client.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-30 16:32:15 +03:00
f437ab8f15 [#197] object: Make Delete method return correct status
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-06-30 12:58:56 +00:00
d01c064674 [#455] Fix non-informative message when parse node attributes
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-28 15:02:29 +00:00
d0ab552a90 [#478] *: Fix funlen linter warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 15:01:49 +00:00
33d9ebbe7f [#478] .golangci.yml: Increase timeout
Make it enough for CI.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 15:01:49 +00:00
f91cfb36e6 [#478] .forgejo: Add build/test workflows
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 15:01:49 +00:00
8a4e250dae [#468] *: replace outdated TODO crypto-related links
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
4f413fe86e [#1] treesvc: Properly check for secure transport
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
cab51c8cbe [#1] metabase: Rename blindlyProcess()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
73a71a71b0 [#1] node: Use a proper validation of a substorage type
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
f4c71cea65 [#1] *: Replace outdated FIXME/TODO links
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
64e1383df6 [#1] Makefile: Remove image-storage-testnet target
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
13b53258b4 [#1] go.mod: Remove retract version
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-28 12:13:20 +00:00
0c866f62c5 [#473] placement: Fix backwards compatibility for copies_number
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-27 11:13:16 +03:00
43d263c3d5 [#428] linter: Fix unkeyed assignment
Thanks to gopls.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 13:29:39 +00:00
cac4ed93d6 [#428] engine: Add low_mem config parameter
Concurrent initialization in case of the metabase resync leads to
high memory consumption and potential OOM.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 13:29:39 +00:00
71a63b8e9c [#439] node: Update SDK-Go version
This fixed version: null header for complex object.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 07:56:32 +00:00
4bf345225c [#447] pilorama: Use named constant for the key size
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
b4ce0b0412 [#447] pilorama: Do not undo log for create ops
```
goos: linux
goarch: amd64
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
         │     old      │                 new                  │
         │    sec/op    │    sec/op     vs base                │
Create-8   36.48µ ± 11%   30.34µ ± 14%  -16.84% (p=0.000 n=10)

         │     old      │                 new                  │
         │     B/op     │     B/op      vs base                │
Create-8   43.01Ki ± 4%   37.78Ki ± 5%  -12.15% (p=0.000 n=10)

         │    old     │                new                 │
         │ allocs/op  │ allocs/op   vs base                │
Create-8   166.0 ± 3%   146.0 ± 3%  -12.05% (p=0.000 n=10)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
dd3874eff1 [#447] pilorama: Add benchmark for create ops
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-26 07:42:05 +00:00
72fedff7ad [#426] cli: Add object nodes command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-26 07:41:34 +00:00
ab489265b3 [#449] cli: Allow to exec tree get-by-path without bearer token
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-23 14:44:19 +03:00
71889234b7 [#449] tree: Allow reading requests signed by keys from allow list
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-23 14:44:19 +03:00
40eae22109 [#460] services/util: Remove CreateRequestStreamer
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:08:04 +03:00
a64dc9ad70 [#460] services/util: Remove HandleServerStreamRequest
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:08:01 +03:00
167a67f0b8 [#460] services/util: Remove HandleUnaryRequest
There is no need in a wrapper with many from-`interface{}` conversions.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:07:56 +03:00
785d81a68a [#460] services/object: Reduce distibutedTarget memory footprint
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-21 17:07:50 +03:00
4d48377cec [#459] blobovniczatree: Fix get error
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 16:44:53 +03:00
03aa210145 [#373] metrics: Move labels to consts
To unify label naming all lable keys and other consts are moved to
one file.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
b5d9f4a285 [#373] metrics: Add pilorama metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
e89fa110c7 [#373] metrics: Add metabase metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
af608da952 [#373] metrics: Add blobovnizca metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
56f320dd85 [#373] metrics: Add blobstor metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
16a142cd0c [#373] metrics: Add FSTree metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
d8ecc69d00 [#373] local storage: Pass parent ID
This is required to add shard ID as metric label.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
d5aaec1107 [#373] pilorama: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
059e9e88a2 [#373] metabase: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
f54cc0b607 [#373] blobstor: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
8318d90ad0 [#373] blobovniczatree: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
3ae3c8dfdb [#373] fstree: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
a8526d45e9 [#373] blobovnizca: Add missed/fix tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
028d4a8058 [#373] blobovnicza: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-21 15:13:26 +03:00
01a0c97760 [#453] engine: Set Disabled mode to deleted shard
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-20 12:04:07 +03:00
50caa388b0 [#303] ir: Use pub key when validate container deletion
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-20 08:02:48 +00:00
69df0d21c2 [#446] engine: Move to read-only on blobstor errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:53:32 +03:00
fe01781811 [#446] los: Wrap SSD errors in a separate type
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:53:31 +03:00
20b84f183a [#446] engine: Simplify logs for shard mode change
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-16 14:51:29 +03:00
69b788a90b [#424] metrics: Rename mode to mode_info
To follow best practice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 18:39:49 +03:00
26b305f82b [#424] morph: Fix cache metrics
Use separate morph cache metrics for node and IR

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 18:26:19 +03:00
4449006862 [#424] metrics: Use mode value as metric value for shard
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 18:26:19 +03:00
847732605c [#424] metrics: Rename counter to total
Suffix total should be used for a unit-less accumulating count.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
c348ae35b0 [#424] metrics: Drop embedded metrics
It was not obvious where metrics are used.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
1b364d8cf4 [#424] metrics: Refactor engine metrics
Use histogram vector to measure request duration.
Fix naming like in Prometheus best practice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
c8023a9c8d [#424] morphcache: Use labels for method duration
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:53:32 +03:00
85deb12f4d [#424] writecache: Drop metrics when close
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:52:40 +03:00
07f155ac77 [#424] metrics: Use labels for writecache methods and operations
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:52:40 +03:00
71bbeddb64 [#424] metrics: Drop unused arg
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-14 14:51:40 +03:00
fb8fee0c8e [#442] Use strconv.FormatBool instead of untyped conversion
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-14 11:08:36 +00:00
344d6b2ae1 [#xx] Fix invalid log metric namespace identifier
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-13 15:36:34 +03:00
4887f489a1 [#17] Add morph client metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-13 14:06:04 +03:00
90e9247b69 [#371] ir: Add morph cache metrics
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-06-13 10:10:13 +00:00
4f83ab0fb4 [#409] node: Do not sent initial bootstrap under maintenance
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:02:36 +00:00
e68384d4e3 [#409] node: Fetch last bootstrap info on startup
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:02:36 +00:00
898f0686b1 [#409] node: Log maintenance state on startup
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:02:36 +00:00
957a43a124 [#266] services/tree: Add sync check
Do not accept requests until initial sync is finished.
`Apply` is deliberately left out -- we don't want to miss anything new.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
e69a1e8482 [#266] services/tree: Return operation log up to some height
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
2541d319de [#266] pilorama: Allow to get current tree height
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 10:00:45 +00:00
0c40d98f7a [#375] Add log metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-13 09:52:45 +00:00
83d600ed77 [#424] node: Update api-go version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 07:19:55 +00:00
0400153b7d [#424] object: Do not store large slices in pool
Dynamically growing an unbounded buffers can cause a large
amount of memory to be pinned and never be freed.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 07:19:55 +00:00
41ab4d070e [#423] *: Use hrw.StringHash() where possible
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-13 07:18:25 +00:00
cd92d8a9e7 [#423] go.mod: Update sdk-go and hrw
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-13 07:18:25 +00:00
9bc1a25c07 [#434] metrics: Fix writecache duration
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-13 07:18:01 +00:00
508e2064eb [#438] cli: Drop tracing debug print
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-09 14:12:25 +03:00
5b75432ca2 [#431] metrics: Add missed label
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-07 13:51:36 +00:00
263c6fdc50 [#372] node: Add metrics for the error counter in the engine
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-06-07 13:04:47 +00:00
189a367ef2 [#390] frostfs-cli: Pass bearer token to Tree srv
* Add --bearer flag for "tree" subcommand

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-06-07 11:08:34 +03:00
a770b89fd8 [#422] adm: Fix ValidatorCount problems after neo-go update
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 14:48:51 +03:00
f8c1e0639d [#422] cli: Provide context to NetmapSnapshot()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 14:48:47 +03:00
f1f56ef871 [#406] cli: Add --trace flag
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 11:44:04 +00:00
96c9843591 [#406] cmd/common: Execute PersistentPostRun() on errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 11:44:04 +00:00
9562123c49 [#406] cli: Pass context to internal client
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 11:44:04 +00:00
63473d0806 [#417] go.mod: Update neo-go
Update to master, because after the API update notary signer is being
incorrectly formed.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-06-02 11:43:48 +00:00
2dd3fc8b7e [#xx] cli: Add policy-playground command
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-02 09:23:40 +00:00
55c28fd5f4 [#412] cache: Pass DialOptions to create new conn
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
dcdfb6ed41 [#412] node: Use observability interceptors
Use metrics and tracing interceptors.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
c09144ecf1 [#412] node: Replace metrics package
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
74578052f9 [#412] node: Replace tracing package
Use observability module.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
7e9a1f394a [#412] node: Update deps
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-06-01 13:23:11 +00:00
f7c4c07453 [#xx] Create innerring metrics before metrics usage
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-06-01 10:07:48 +03:00
4476a1dbaf [#413] morph/client: Fix govet warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-31 15:38:17 +03:00
dbf41391b5 [#401] engine: Extend evacuation logs
Add operation-tag to logger.
Log evacuation results.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 13:24:30 +03:00
3220c4df9f [#376] metrics: Add GC metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 10:22:12 +00:00
faca861451 [#411] Remove unnecessary pointers for sync objects
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-31 10:19:14 +00:00
f934abed8f [#402] doc: Update evacuation docs
Add estimated time left output to examples.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 10:18:44 +00:00
f64322576a [#402] cli: Add estimated evacuation time left
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-31 10:18:44 +00:00
ebcc8afbee [#374] Add inner-ring event metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-31 10:11:48 +00:00
8dcd06c587 [#394] node: Use Context in Blobovniczas.Iterate()
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-31 10:09:18 +00:00
a3e30062df [#325] node: Introduce unsafe_disable param to disable policer
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-05-31 10:09:05 +00:00
365a7ca0f4 [#366] node: Stop GC once termination signal received
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-29 09:35:08 +03:00
802168c0c6 [#364] node: Stop flushing big object when termination signal received
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-26 16:46:58 +03:00
9c54a24101 [#364] Fix trailing whitespace
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-26 16:46:58 +03:00
271a56c2ab [#395] metrics: Drop redundant metrics
HistogramVec already has labeled counter.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 13:39:37 +00:00
bc34fee6a7 [#370] Add tree service metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-26 13:39:12 +00:00
f2e5dead7e [#398] pilorama: Disallow applying same operations
1. In redo() we save the old state.
2. If we do redo() for the same operation twice, the old state will be
   overritten with the new one.
3. This in turn affects undo() and subsequent isAncestor() check.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-26 12:14:29 +00:00
5983617069 [#393] shard: Create tombstone source when reload
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 12:14:02 +00:00
20a489bdb5 [#393] gc: Use defer to mark handler done
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 12:14:02 +00:00
9119199f6e [#397] cli: Fix evacuation method names
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-26 11:28:30 +03:00
2613351008 [#387] gc: Cancel GC is change mode requested
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-25 09:38:16 +03:00
a1823423cf [#312] node: Add WC metrics info to CHANGELOG
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
2ce43935f9 [#312] metrics: Add writecache metrcis
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
d212d908b5 [#312] wc: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
4503a61997 [#312] wc: Delete unused Iterate method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 10:18:39 +00:00
53a1b15693 [#338] ir: Drop notaryless code from governance
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
0ab589dd52 [#338] ir: Always enable notary on sidechain
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
81718afb39 [#338] ir: Drop named named put fee
Named put fee value used only when notary disabled.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
656fd7f376 [#338] ir: Drop notaryless code from netmap
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
fb708b3a2d [#338] ir: Drop container notaryless code
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
03ab0ca30f [#338] adm: Drop notaryless code
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-24 08:44:47 +00:00
cf8531ccd7 [#381] go.mod: Update api-go and sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-23 08:18:01 +03:00
4b768fd115 [#381] *: Move to sync/atomic
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-23 08:18:01 +03:00
ff570847a4 [#381] go.mod: Update bbolt
Adopt new `ForEachBucket` function where possible.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-22 11:49:14 +03:00
7eb8fa6350 [#381] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-22 11:49:14 +03:00
731bf5d0ee [#380] node: go version up
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-22 08:48:14 +00:00
e3ad3c2965 [#283] cli: Move control healthcheck command under control ir
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-22 08:13:52 +00:00
35c9b6b26d [#314] writecache: remove objects right after they are flushed
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-19 09:37:05 +00:00
Pavel Karpy
bf79d06f03 [#314] writecache: Do not lose small objects on disk errors
Do return error if an object could not been stored on WC's disk.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-19 09:37:05 +00:00
Pavel Karpy
9e56592be3 [#314] writecache: Simplify background workers naming
Also, drop not used arg.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-19 09:37:05 +00:00
483fac03d6 [#329] docs: Add shard evacuation description
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
f7c0b50d70 [#329] cli: Add async evacuate commands
Add start, stop evacuate and evacuate status commands.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
e4889e06ba [#329] node: Make evacuate async
Now it's possible to run evacuate shard in async.
Also only one evacuate process can be in progress.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
100b1b5128 [#329] node: Add async evacuate proto methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-19 08:43:52 +00:00
0fc494637f [#367] node: Disable unit tests cache
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-18 15:14:55 +00:00
aabcd8d3e4 [#362] node: Cancel ctx in the same routine as for signal watcher
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-18 09:28:10 +03:00
df9e099fa7 [#352] Add explicit max number of alphabet nodes
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-17 15:01:26 +00:00
13a7a90101 [#355] Increase tree svc client cache size to test hypotheses
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-17 14:11:46 +03:00
6f47c75e43 [#125] ir: Set extra wallets on SIGHUP
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-16 12:47:43 +00:00
0624820909 [#336] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-16 12:45:13 +00:00
869fcbf591 [#332] gc: Fix expired complex object deletion
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-16 12:44:57 +00:00
ab07bad33d [#332] gc: Add complex object unit test
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-16 12:44:57 +00:00
8da6530f41 [#351] cli: Support copies number parameter in object put
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-16 08:22:56 +00:00
079b28fa0f [#341] Register candidates in separate transactions
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-15 12:46:12 +00:00
d4d921dcaf [#346] adm: fix race in wallet generation
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-15 11:50:20 +03:00
429a87e83b [#39] ir: Use defer for wg.Done()
Not important, but `exitOnErr` can alter control flow, let's be
explicit.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 17:49:47 +03:00
Pavel Karpy
f604d6bbdc [#39] ir: Add optional profilers
Includes `block` and `mutex` profiles configuration.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 17:49:47 +03:00
f989bc52be [#39] ir: Do not store config keys in httpComponent
Pprof will have specific options, it seems wrong to have them in a
generic struct.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 17:49:46 +03:00
61776033c2 [#39] ir: Do not reload services if they are disabled
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 15:49:04 +03:00
Pavel Karpy
14c35d776e [#39] node: Add optional profilers
Include settings for block and mutex profilers.
They are disabled by default, as in Go runtime itself.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 15:49:04 +03:00
a6ee7a3087 [#324] Add replicator metrics
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-12 11:00:17 +00:00
6055b18362 [#342] morph: Remove unused toStackParameter()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 13:24:49 +03:00
02c02974b3 [#309] locode: Parallelize DB generation
For v0.4.0 release:
Before:
```
Executed in  571.64 secs    fish           external
   usr time  283.07 secs  744.00 micros  283.07 secs
   sys time    8.41 secs  179.00 micros    8.41 secs
```

After:
```
Executed in   54.23 secs    fish           external
   usr time  418.65 secs    1.01 millis  418.65 secs
   sys time    0.61 secs    0.25 millis    0.60 secs
```
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:59:13 +00:00
147ae8728a [#309] go.mod: Update paulmach/orb to v0.9.1
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:59:13 +00:00
c62025c836 [#321] metabase/test: execute tests in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
c8c5f14e2e [#321] adm/test: Check wallet correctness in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
fe4082799a [#321] adm: Create multisig accounts in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
945454f60c [#321] engine/test: Execute tests in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
4578d00619 [#321] shard/test: Execute tests in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
d35e4c389f [#321] shard/test: Parallelize TestWriteCacheObjectLoss
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
969bfb603f [#321] shard/test: Parallelize TestShard_List
```
go test -count=1 -run TestShard_List -race .
Before: 2.492s
After:  0.109s
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:45:03 +00:00
47b0ec33c3 [#337] netmap: Remove unused events
Done in TrueCloudLab/frostfs-contract#16.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
b480df4985 [#337] container: Remove unused events
Done in TrueCloudLab/frostfs-contract#16.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
Anna Shaleva
bcdb0f330d [#337] morph: Completely remove fallbackTime from client cfg
It's unused and not needed, default fallback lifetime is set by Notary
actor.

Signed-off-by: Anna Shaleva <anna@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
Anna Shaleva
ddcc156ecc [#337] morph: Use Notary Actor for notary requests
Signed-off-by: Anna Shaleva <anna@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
35fdf6f315 [#337] morph: Move subscription logic to subscriber
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
Roman Khimov
a5f118a987 [#337] subscriber: Drop unused UnsubscribeForNotification
It's not really needed, closing the connection works fine when exiting and
normally the app doesn't need to unsubscribe at all.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-12 09:19:38 +00:00
800eb5e983 [#125] ir: Reconfigure pprof and metrics on SIGHUP
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-11 12:38:34 +03:00
a181c9e434 [#332] gc: Add additional logging
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 17:34:40 +03:00
90799497d3 [#114] Add remove-node IR control command
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-10 14:31:44 +00:00
ea10abb42a [#296] morph: Add parser unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 09:40:26 +03:00
1309622b20 [#296] morph: Add listener unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-10 09:40:26 +03:00
b2ffd7df53 [#291] object: Use PayloadSizeLimiter from SDK
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-05 19:07:06 +03:00
35ea207df6 [#291] object: Split validating target in two
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-05 19:07:06 +03:00
Pavel Karpy
ee58b390bb [#221] node: Allow using vector copies_number
Also, take into account that value in general (it was not used before at
all).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-05 16:07:13 +00:00
d02950ad63 [#330] morph: Fix linter issue
Revive became smarter after go 1.20.4 upgrade.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-05 18:02:38 +03:00
973af12854 [#327] tests: replace os.MkdirTemp with t.TempDir
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-05 11:24:51 +00:00
cedd07bbc8 [#304] morph: Iterate endpoints when create ws client in constructor
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-05-05 10:07:30 +03:00
Pavel Karpy
479c5a65e1 [#322] node: Fix tree svc panic
If a connection has not been established earlier, it stores `nil` in LRU
cache. Cache eviction tries to close every connection (even a `nil` one) and
panics but not crash the app because we are using pools.
That ugly bug also leads to a deadlock where `Unlock` is not called via
`defer` func (and that is the way I found it).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-05-04 19:48:51 +03:00
2f6757c828 [#311] ir: Fix data race in unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-04 16:59:15 +03:00
872fe90c40 [#317] pre-commit: Add go-mod-tidy hook
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-04 13:58:19 +00:00
d1661ae7dc [#308] Remove downloadContractsFromGithub from the frostfs-adm
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-05-04 14:08:42 +03:00
5f1af84587 [#297] go.mod: Update compress to v1.16.5
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 16:28:29 +03:00
a1b4ba9980 [#209] compression: Do not store uncompressible data
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-03 16:28:29 +03:00
529d0bc710 [#302] tree: Drop unused ctx
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 16:54:06 +03:00
eca5c210dd [#299] evacuate: Add context cancel checks
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 15:55:30 +03:00
b939e4e5c5 [#299] ir: Drop unused structs
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 14:05:35 +03:00
235fe84ea3 [#298] innerring: Fix broken tests
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-05-02 11:52:29 +03:00
d00b1c0d29 [#280] ir: Add netmap processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
fb5dcc15d2 [#280] ir: Add governance processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
31b4da225a [#280] ir: Add frostfs processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
5010b35466 [#280] ir: Add container processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
686f01bce5 [#280] ir: Add balance processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
e89fa7f69f [#280] ir: Add alphabet processor unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
53693071de [#280] ir: Add block timer unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
e70f808dc3 [#280] ir: Add parser unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
2dbe382b5f [#280] ir: Add state unit tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
500611f3c8 [#280] ir: Add indexer tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
3b64dffda2 [#280] ir: Add fee config tests
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-05-02 08:48:07 +00:00
22d47376a6 [#277] getsvc: Refactor errors
Move errors to separate files.
Use zap.Error for error logging.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
45438e7b06 [#277] getsvc: Rename and reorder code
Rename execCtx to request.
Move code to appropriate files.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
1440450606 [#277] getsvc: Drop cyclic struct dependency
Drop cyclic dependency between execCtx and Service.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
591c4e7d50 [#277] getsvc: Move headOnly to request params
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
265d2326a0 [#277] getsvc: Extract remote storage
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
30e1b62b67 [#277] getsvc: Fix service deps
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
8fc082b688 [#277] getsvc: Do not return status error
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-28 14:03:12 +00:00
3bac5a485d [#248] config: Remove audit-related parameters
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
f368ccbdf0 [#248] morph: Remove audit client
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
b2bc1fccbc [#248] logs: Remove unused messages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
a9c4ba62c3 [#248] metabase: Remove storage group bucket
Backwards compatible change, so no version increase.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
586f5986bc [#248] core: Remove storage group
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
f1ea8fec93 [#248] object_manager: Remove storage group
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
7f49f07255 [#248] services: Remove audit service
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
8879c6ea4a [#248] innerring: Remove audit and settlement code
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
8b2aae73c6 [#248] cli: Remove storagegroup commands
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 17:00:43 +03:00
f73ac6e02d [#290] control: Use generics for response wrappers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 10:57:24 +03:00
ff25521204 [#270] Add IR epoch tick control call
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-04-28 07:57:00 +00:00
58f1ba4b51 [#288] pilorama: Add missing operation in log
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
daa26f6e9b [#288] pilorama/test: Check operation order for TreeGetByPath()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
291f9e809a [#288] pilorama: Remove getMeta() wrapper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
0045f1bcd4 [#288] pilorama: Use more descriptive names for memory tree
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
f856ad7480 [#288] pilorama: Remove childMap from memory forest
Memory forest is here to check the correctness of boltdb optimized
implementation. Let's keep it simple.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-28 07:56:20 +00:00
ada081dfd5 [#19] node: Make policier read shards concurrently
* Introduce ListWithMultiCursor that simultaneously reads objects
  from different shards

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-27 11:11:47 +03:00
1f4061c0e2 [#285] blobonicza: Optimize upperPowerOfTwo()
The real reason is this:
```
pkg/local_object_storage/blobovnicza/sizes.go:36:69
    revive empty-block: this block is empty, you can remove it
```

Didn't want to make this function longer or to add `nolint`, thus this
change. To justify:
```
UpperBound/size=1-8          0.4924n ± 1%   0.2472n ± 2%  -49.80% (p=0.000 n=10)
UpperBound/size=1023-8       0.4936n ± 3%   0.2442n ± 1%  -50.52% (p=0.000 n=10)
UpperBound/size=66560-8      0.8201n ± 2%   0.2436n ± 1%  -70.29% (p=0.000 n=10)
UpperBound/size=41943040-8   6.6900n ± 5%   0.2432n ± 0%  -96.36% (p=0.000 n=10)
geomean                       1.075n        0.2446n       -77.24%
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:11 +03:00
dfe4ada838 [#285] lint: Resolve revive/if-return
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:08 +03:00
f07e2d4812 [#285] lint: Fix revive/unused-parameter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:04 +03:00
57718bd6b4 [#285] *: Update golangci-lint to v1.52.2
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-26 17:35:04 +03:00
14f83b8aa9 [#125] ir: Change log level on SIGHUP
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:54:19 +03:00
563780057d [#125] ir: Use internal/common/config for reading config
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
ef222e2487 [#125] cmd: Refactor internal/common/viper
Add `opts.WithViper`, set opts struct as private.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
e61aec4a7d [#125] node: Remove unused config.Prm
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
eb7be82e87 [#125] node: Avoid panic when reading config
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
d390f093e0 [#125] node: Move viper creation to internal/common/config
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
b2123bfd1a [#125] node: Add new option WithEnvPrefix
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
ee01275d25 [#125] node: Remove redundant env from config/internal
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-26 13:53:51 +03:00
9d01029733 [#166] node: Parallelize background tree service sync by batching
* Merge operations

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-26 10:17:56 +00:00
299b24b974 [#166] node: Parallelize background tree service sync by batching
* Concurrently dispatch TreeApply operations for batching in forest

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-26 10:17:56 +00:00
700f39c3f8 [#229] Update CHANGELOG.md
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-26 10:23:33 +03:00
Denis Kirillov
dce5924a89 [#229] services/tree: Use bearer owner as signer
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-26 10:23:33 +03:00
89530534a1 [#229] service/tree: Disable container owner check in tree service
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-04-26 10:23:33 +03:00
c04f6c5e59 [#229] acl: Allow Impersonate
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-04-26 10:23:33 +03:00
04be9415d9 [#231] node: Fix race condition in TTL cache
Use key locker to lock by key.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-25 09:54:12 +03:00
ddbc9e255f [#231] node: Invalidate container cache on PutSuccess event
For example: frostfs-cli creates container and makes polling
GetContainer requests. These requests go through container cache,
so not found error stores in container cache.
So container cache can contain not found error when PutSuccess event received.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-25 09:42:02 +03:00
c70306b324 [#164] scripts: Add metrics description exporter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00
6fef2726b8 [#164] metrics: Allow to export metrics description
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00
4ade5339da [#164] metrics: Rename metrics.go -> node.go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00
015d62425b [#164] metrics: Fill local registry explicitly
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-24 09:01:13 +00:00
Pavel Karpy
59822f7fb4 [#1248] node: Do not update cache twice
Do not request morph values on morph cache misses concurrently.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-21 09:45:53 +00:00
dc2e9d70c7 [#271] go.mod: Update prometheus to v1.15.0
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-20 19:30:26 +03:00
Pavel Karpy
09938a9841 Revert "[#262] meta: Do not return old expired objects"
This reverts commit 3d23b087

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 19:22:55 +03:00
Pavel Karpy
e9461686b8 [#274] wc: Resolve possible deadlock
If operation with WC are _fast enough_ (e.g. `Init` failed and `Close` is
called immediately) there is a race and a deadlock that do not allow finish
(and start, in fact) an initialization routine because of taken `modeMtx`
and also do not allow finish `Close` call because of awaiting initialization
finish. So do stop initialization _before_ any mutex is taken.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 19:22:54 +03:00
Pavel Karpy
6b6f33ed71 [#274] wc: Make wait groups work more explicit
Do not run routine that calls `wg.Done()` inside, it is hard to read.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 19:22:54 +03:00
Pavel Karpy
4f5f832137 [#268] notary_preparator: Actualize notary requests parsing
After 75d7891ca1
`neo-go` does claim that an empty invocation script is the only way to
fill missing signature for unsigned notary requests. The new notary actor
does it that way and, therefore, breaks notary request parsing by the
Alphabet because of skipping any request that is not filled with a dummy (64
zeros) invocation script. Support both way. The "Dummy" approach will be
dropped later.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-20 10:29:34 +03:00
6c90bb87f1 [#118] node: add ctx for unit tests for blobstor
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-19 18:09:33 +03:00
Pavel Karpy
3d23b08773 [#262] meta: Do not return old expired objects
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-19 13:37:38 +00:00
13c8afcb02 [#118] node: add unit concurrent tests for blobstor
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-19 10:22:50 +00:00
Pavel Karpy
20cd080323 [#255] write-cache: Fix init race condition
Do not use WC's internals in the initialization routines without mode
protection. WC should be able to change its mode even if the initialization
is not finished yet.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-19 09:29:07 +00:00
3d43b0f7f9 [#265] node: Fix after SDK & API-Go version up
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-18 12:09:19 +03:00
a358255c1b [#265] node: Drop unused
Resolve unused linter.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-18 12:09:16 +03:00
b447ff99aa [#265] node: Up SDK and API-Go versions
Fix tracing panic.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-18 12:09:12 +03:00
7b981bfe97 [#249] logs: Drop unused consts
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
f07d4158f5 [#249] node: Drop subnet from IR and morph
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
d757d881d0 [#249] node: Drop subnet from config
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
05c870f39a [#249] cli: Drop subnet support
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
160147b05d [#249] adm: Drop subnet support
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-17 09:46:02 +03:00
Pavel Karpy
262c9c2b93 [#256] blobovniczaTree: Make Exists test stable
Corrupt and request _the same_ file.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 16:46:24 +03:00
0b42a00a60 [#254] innerring: Remove unused TimersHandlers() method from processors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 15:34:15 +03:00
8466894fdf [#250] control: remove DumpShard and RestoreShard RPC
We have `Evacuate` with a cleaner interface.
Also, remove them from CLI and engine.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 12:28:49 +00:00
Pavel Karpy
070154d506 [#247] client: Drop reputation related RPCs
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:54:28 +03:00
Pavel Karpy
beabed788c [#247] network_config: Drop reputation
Drop the code that was expected to work with global reputation network
parameters.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:53:22 +03:00
Pavel Karpy
b453bb754c [#247] logs: Drop reputation log messages
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:44:14 +03:00
Pavel Karpy
8799138fcb [#247] morph: Drop reputation contract
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:44:14 +03:00
Pavel Karpy
960e3c219e [#247] config, doc: Drop reputation references
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:44:14 +03:00
Pavel Karpy
560f73ab7e [#247] node, ir: Drop reputation related code
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-14 14:44:14 +03:00
6121b541b5 [#242] treesvc: Add tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
d62c6e4ce6 [#242] node: Add tracing spans
Add tracing spans for PUT requests.
Add tracing spans for DELETE requests.
Add tracing spans for SELECT requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
200fc8b882 [#242] put: Pass context to relay function
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-14 10:25:53 +00:00
995db117d0 [#238] node: Read cfg from dir even if cfg file not set
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 10:19:10 +00:00
8d2f443868 [#238] Fix linter error
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 10:19:10 +00:00
41eb3129ae [#139] Refactor blobovnicza exist test to not use chmod
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-04-14 12:16:14 +03:00
adcfce39cf [#246] .gitattributes: Do not show diff for go.sum
When we update dependencies it can be rather big. However it is
generated automatically with `go mod tidy`, no need to review.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 06:02:04 +00:00
299b6a6938 [#100] adm: Use netmap constants from pkg
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 05:12:50 +00:00
0c6aeaaf18 [#100] adm: Take net settings into account during netmap contract update
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 05:12:50 +00:00
4496999e52 [#100] Fix CHANGELOG
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-04-14 05:12:50 +00:00
cffcc7745e [#240] logs: Factor out common service log messages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 05:06:09 +00:00
0e31c12e63 [#240] logs: Move log messages to constants
Drop duplicate entities.
Format entities.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-14 05:06:09 +00:00
Roman Khimov
d29b13454f [#239] morph/client: Simplify code interacting with magic numbers
It can't be uint64 in fact, but this error is buried deeply in the NetworkInfo
API structure, so we're not touching MagicNumber() for now.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
d686ab49e8 [#202] adm: Remove deprecated RPC client methods
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
Roman Khimov
f41ad9d419 [#239] morph/client: Recreate actor/wrappers in SetGroupSignerScope
That's the reason #2230 and #2263 were not detected earlier, we actually had
Global scope being used before reconnection to RPC node.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
Roman Khimov
be4df989e5 [#239] morph/client: Deduplicate signers in Client a bit
One signer in the cfg is enough.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
Roman Khimov
96b38f7e86 [#239] morph/client: Add CalledByEntry into the "grouped" scope
Fixes #2230, fixes #2263. CustomGroups are nice while we're only calling NeoFS
contracts, but it doesn't work at all for standard ones like GAS or Notary.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
2f1beddfd3 [#202] adm: Remove deprecated warnings in tests
`VerifyBlocks` is now `SkipBlockVerification` and is false by default.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-13 13:15:30 +00:00
01c0c90a86 [#113] cli: add "name" option for "get container" command
* Make get container command filter out the container by attribute name

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-12 18:25:37 +00:00
7d39fecc6a Release v0.36.0
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-12 16:57:02 +03:00
04727ce1d6 Revert "[#135] signature: Add tracing"
This reverts commit 5778980252.
2023-04-12 16:57:02 +03:00
08769f413f Revert "[#135] acl: Add tracing spans"
This reverts commit b2ca730547.
2023-04-12 16:54:13 +03:00
5d2affa5cd testutil: Fix linter warning
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-12 11:00:02 +03:00
5778980252 [#135] signature: Add tracing
Add tracing to verify request and sign response.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
b2ca730547 [#135] acl: Add tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
0920d848d0 [#135] get-object: Add tracing spans
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
5af9f58469 [#135] tracing: Add tracing to node gRPC endpoints
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
72565a91ef [#135] node: Update api-go version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-12 06:52:00 +00:00
c4865783fc [#236] blobstor/test: Prefill storage in parallel in read benchmark
`blobovniczatree` takes a really long time to prefill, because each
batch takes at least 10ms, so for 10k iterations we have at least 100s of
prefill.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:11 +03:00
6ad5c38225 [#236] testutil: Use random object id in RandObjGenerator
Before this commit it was like this:
```
BenchmarkSubstorageWritePerf/memstore-rand10-8            227425              4859 ns/op
BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8     --- FAIL: BenchmarkSubstorageWritePerf/fstree_nosync-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
BenchmarkSubstorageWritePerf/fstree-rand10-8            --- FAIL: BenchmarkSubstorageWritePerf/fstree-rand10-8
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
    perf_test.go:165: writing entry: file exists
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
c85a0bc866 [#236] blobstor/test: Reduce test descriptions
I tried to add 4 more tests and suddenly, it became harder to navigate in
code. Move directory creation in a common function.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 20:52:04 +03:00
6bf11f7cca [#230] CHANGELOG.md: Remove older entries
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-11 10:07:27 +00:00
2c07f831c7 [#223] node: Refactor cache usage
Drop excess type args.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
93eba19a8e [#223] objectsvc: Refactor split-tree traverse
Resolve funlen & gocognit linters for traverseSplitChain method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
2ed9fd3f94 [#223] objectsvc: Refactor request parameters
Resolve containedctx linter for commonPrm.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
ccf8463e69 [#223] controlsvc: Drop unnecessary nolint
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
ae86cda58c [#223] sg: Refactor storage group parameters
Resolve containedctx linter for SearchSGPrm and GetSGPrm  structs.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 14:16:34 +03:00
6016d78a45 [#223] core: Refactor object format validator
Resolve funlen linter for FormatValidator.ValidateContent method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 13:56:09 +03:00
02831d427b [#213] metrics: Refactor object metrics
Resolve funlen linter for newObjectServiceMetrics function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 09:43:24 +03:00
38ae71cc7d [#213] metrics: Refactor engine metrics
Resolve funlen linter for newEngineMetrics function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-10 09:43:24 +03:00
b689027d57 [#191] cli: Add control shards doctor command
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
0b9622c418 [#191] control: Add Doctor RPC
Doctor RPC performs complex operations on the storage engine.
Currently only duplicate removal is supported.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
dbc3811ff4 [#191] engine: Allow to remove redundant object copies
RemoveDuplicates() removes all duplicate object copies stored on
multiple shards. All shards are processed and the command tries to leave
a copy on the best shard according to HRW.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-07 17:25:50 +00:00
cb172e73a6 [#228] node: Use uber atomic package instead standard
Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 15:37:27 +00:00
c236b54a65 [#212] reputationsvc: Resolve funlen linter
Resolve funlen linter for Calculator.iterateDaughter method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
7ebbfa3358 [#212] reputationsvc: Resolve linters and rename
Resolved containedctx linters.
Renamed context structs and interfaces to more understandble names.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
469e8a6e59 [#212] reputationsvc: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:35:57 +00:00
e2f13d03d7 [#222] auditsvc: Refactor PoR audit
Resolve funlen linter for Context.checkStorageGroupPoR method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
e8d340287f [#222] auditsvc: Refactor audit task
Resolve containedctx linter. Cancel task by listen cancel.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
3dbff0a478 [#222] auditsvc: Resolve containedctx linter
Resolve containedctx linter for commonCommunicatorPrm type.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 17:29:13 +03:00
fe87735073 [#219] morph: Refactor notary preparator
Resolve funlen linter for Preparator.Prepare method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
d07e40d6fe [#219] morph: Refactor moprh event listener
Resolve funlen and gocognit linters for listener.listenLoop method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
775179f823 [#219] morph: Refactor notary invoke
Resolve funlen linter for Client.notaryInvoke method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
e815b19101 [#219] morph: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-07 15:43:20 +03:00
56282edf02 [#166] node: Parallelize background tree service sync
* Run sync task for nodes in parallel within errgroup worker pool

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-07 04:43:32 +00:00
9027695371 [#203] node: Add staticcheck target and pre-commit
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
4ec69cbbf8 [#203] control: Fix test
Fix shard compare loop.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
f32f61df87 [#203] pilorama: Refactor tests
Do not pass 0 as channel capacity.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
8908798f59 [#203] node: Resolve unused vars
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
bab11492ad [#203] node: Resolve never used errors
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
e21c5bea21 [#203] cli: Fix error message
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
68a2f36636 [#203] morph: Mark depracated methods
Skip staticcheck for depracated methods. Will be fixed soon.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
9e2df4b7c7 [#203] node: Fix double imports
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-06 16:33:36 +03:00
ab891517de [#116] node: Fix bug with extra generated files in TestReload
* Create testNewEngine in engineWithShards without default opts

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-05 18:49:34 +03:00
c58ab0c369 [#193] getsvc: Reduce private key requests
Get private key only once for request forwaring.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
89924071cd [#193] getsvc: Edit request forwarder signature
Pass context to forwarder direct, without closure.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
6c7b708a98 [#193] getsvc: Refactor get range params creation
Resolve funlen linter for toRangePrm function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
b0786d2e5c [#193] getsvc: Refactor get params creation
Resolve funlen linter for toPrm function.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
f889893216 [#193] getsvc: Refactor head param creation
Resolve funlen linter for toHeadPrm method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
91ead04fa4 [#193] getsvc: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
1bf21dbb47 [#193] getsvc: Resolve context linters
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:48 +00:00
206458c841 [#217] containersvc: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:00 +00:00
279261ace3 [#217] containersvc: Refactor route passing
Resolve containedctx for routeCtx.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:38:00 +00:00
6f7b6a8813 [#116] node: Improve shard/engine construction in tests
* Introduce testEngineWrapper that can be constructed with different options

Signed-off-by: Airat Arifullin a.arifullin@yadro.com
2023-04-05 14:36:40 +00:00
d6486d172e [#210] policier: Refactor nodes processing
Resolve funlen linter for processNodes method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
080be5cfcd [#210] policier: Refactor object placement
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
23575e1ac0 [#210] policier: Resolve contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-05 14:55:52 +03:00
9098d0eec0 [#211] engine: Unify shard mode checks for tree operations
All operations must ensure the shard is not in a degraded mode.
Write operations must also ensure the shard is not in a read-only mode.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-05 11:10:39 +00:00
760af6b912 [#211] fstree: Consider ENOSPC a logical error
We already do this for file writing, however directory creation can also
fail.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-05 11:10:39 +00:00
d85703a963 [#208] searchsvc: Refactor request forwarding
Resolve funlen & gocognit linters for toPrm method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 16:24:55 +03:00
0b38419fbf [#208] searchsvc: Resolve context linters
Resolve containedctx and contextcheck linters.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 16:24:50 +03:00
5f2a1531fe [#208] deletesvc: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 16:24:40 +03:00
4941926c9d [#207] aclsvc: Drop outdated tag
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:14 +00:00
585415fa92 [#207] aclsvc: Refactor send checker
Resolve funlen linter for putStreamBasicChecker.Send method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:14 +00:00
9ef790f782 [#207] aclsvc: Refactor object headers read
Resolve funlen linter for readObjectHeaders method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:13 +00:00
cd33a57f44 [#207] aclsvc: Refactor EACL check
Resolve funlen linter for CheckEACL method.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 13:22:13 +00:00
1f1aed87be [#188] metabase: Refactor object inhume
Resolve funlen linter for db.Inhume method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
5a66db80c5 [#188] engine: Refactor shard evacuation
Resolve funlen and gocognit linter for StorageEngine.Evacuate method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
456bc097f7 [#188] engine: Refactor get range from engine
Resolve funlen linter for StorageEngine.getRange method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
3010ca2649 [#188] engine: Refactor get object from engine
Resolve funlen linter for StorageEngine.get method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
0739c36a3b [#188] metabase: Refactor object put to metabase
Resolve funlen linter for db.put method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
8273a3dfb2 [#188] blobstor: Refactor blobstor test
Resolve funlen linter for TestIterate function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
594b5821ed [#188] blobstor: Refactor put data to blobovniczas
Resolve funlen linter for Blobovniczas.Put method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 14:50:43 +03:00
ee7468daa7 [#205] innerring: Provide alphabetState param to epochTimer
Fix NPE, introduced in f09ee27a.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-04 10:47:31 +00:00
49cc23e03c [#175] adm: pipeline container iteration
Do not accumulate everything in memory.
Also, CLI should be responsive.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-04 07:26:23 +00:00
e85e5382e4 [#175] adm: list containers using containersOf
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-04 07:26:23 +00:00
8e5a0dcf27 [#204] gc: Fix GC handlers start
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-04 06:48:27 +00:00
0948a280fa [#195] morph: use blocking unlimited pool for notifications
With non-blocking pool restricted by 10 in capacity, the probability of
dropping events is unexpectedly big. Notifications are an essential part of the FrostFS,
we should not drop anything, especially new epochs.
```
Mar 31 07:07:03 vedi neofs-ir[19164]: 2023-03-31T07:07:03.901Z        debug        subscriber/subscriber.go:154        new notification event from sidechain        {"name": "NewEpoch"}
Mar 31 07:07:03 vedi neofs-ir[19164]: 2023-03-31T07:07:03.901Z        warn        event/listener.go:248        listener worker pool drained        {"chain": "morph", "capacity": 10}
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-04-03 15:58:16 +00:00
ece6c820e7 [#199] putsvc: Refactor streamer initialization
Resolve funlen linter for initTarget method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
27bdddc48f [#199] putsvc: Refactor put object
Resolve containedctx linter for streamer and remote target

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
cecea8053a [#199] putsvc: Refactor streamer pool
Resolve staticcheck linter for putBytesPool

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
14d894178e [#199] putsvc: Refactor placement iterator
Resolve funlen linter for iteratePlacement method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-04-03 15:58:11 +00:00
Pavel Karpy
a69c6d1ec9 [#2272] morph: Do not subscribe to events without listening
It led to a neo-go dead-lock in the `subscriber` component. Subscribing to
notifications is the same RPC as any others, so it could also be blocked
forever if no async listening (reading the notification channel) routine
exists. If a number of subscriptions is big enough (or a caller is lucky
enough) subscribing loop might have not finished subscribing before the
first notification is received and then: subscribing RPC is blocked by
received notification (non)handling and listening notifications routine is
blocked by not finished subscription loop.
That commit starts listening notification channel _before_ any subscription
actions.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-04-03 08:42:41 +00:00
Pavel Karpy
2bdf7126b8 [#181] Update CHANGELOG
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:54:05 +03:00
Pavel Karpy
aa92f977ef [#181] ir: Do not pay for audit by non-alphabet nodes
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:53:37 +03:00
Pavel Karpy
f09ee27af9 [#181] ir: Do not process container estimations by non-alphabet nodes
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:53:09 +03:00
Pavel Karpy
db5321309d [#181] ir: Do not sync the Alphabet by non-alphabet nodes
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:52:16 +03:00
Pavel Karpy
44d5412e10 [#181] ir: Do not deposit notary GAS by non-alphabet nodes
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-31 12:51:12 +03:00
ed28ce24cd [#168] node: Refactor reputation service
Resolve funlen linter for initReputationService function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
1f929fdd57 [#168] node: Refactor netmap service init
Resolve funlen linter for initNetmapService function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
dd572825c7 [#168] node: Refactor object service init
Resolve funlen linter for initObjectService function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
56161d39b4 [#168] node: Refactor container services
Resolve containedctx for remoteLoadAnnounceWriter struct

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
28dc9e2190 [#168] node: Refactor container service init
Resolve funlen linter for initContainerService function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
dcd39f8fdd [#168] node: Refactor shard opts initialization
Resolve funlen linter for shardOpts method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
c94372e6f9 [#168] node: Refactor config initialization
Resolve funlen linter for initCfg function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
3bbb516528 [#168] node: Refactor node config read
Resolve funlen linter for readConfig method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
a7c79c773a [#168] node: Refactor node config
Resolve containedctx linter for cfg

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-31 09:32:59 +03:00
8426d25f4b [#185] ir: Resolve containedctx linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
c1cbe6ff2d [#185] ir: Refactor signature verification
Resolve funlen linter for verifySignature method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
aeb4bbc51e [#185] ir: Refactor alphabet update
Resolve funlen linter for processAlphabetSync method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
30c18d46cc [#185] ir: Refactor audit
Resolve funlen linter for processStartAudit method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
6be2688fb4 [#185] ir: Refactor gas emit
Resolve funlen linter for processEmit method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
4d160bd4ab [#185] ir: Refactor ir service creation
Resolve funlen linter for New function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
c8a6978563 [#185] ir: Refactor ir start
Resolve funlen linter for Server.Start method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
85fb9e77c4 [#185] ir: Resolve funlen linter
Resolve funlen linter for GlagoliticLetter.String method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 19:10:33 +00:00
7be5a0fd79 [#189] node: Fix bug with sync/atomic package
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-03-30 14:50:43 +00:00
ab32067152 [#183] gc: Fix drop expired locked complex objects
Do not delete bucket keys during iteration

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
9f0bce5c15 [#183] gc: Fix drop expired locked simple objects
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-30 15:33:42 +03:00
341fe1688f [#139] test: Add test storage implementation
This aims to reduce the usage of chmod hackery to induce or simulate
OS-related failures.

Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-29 14:28:49 +00:00
Pavel Karpy
e843e7f090 [#184] ir: Delete unused config reader param
Not used since notary environments do endless and non-configurable
deposits.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
Pavel Karpy
ba58a77f8c [#184] node, ir: Do not wait for already made notary deposits
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
Pavel Karpy
3646723ae3 [#184] morph: Make deposit logs INFO
We have already had and solved plenty of deposit issues and notary balance
is a really important thing. Deserves to be INFO even before the huge logs
severity refactor, happens on an app start only.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
Pavel Karpy
97e201993b [#184] *: Unify done contexts handling
If `ctx.Done()`, return `ctx.Err()` in every function that returns an error.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-29 10:34:54 +00:00
221203beeb [#180] node: Refactor panics in unit test
* Replace panics in unit tests by require.NoError and t.Fatalf

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2023-03-29 12:39:07 +03:00
91717d4b98 [#176] morph: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 18:49:55 +03:00
382ecae96a [#172] Use ContainersOf() for container list fetching
Previously we were limited by ~2048 containers because of
neo-go VM limits.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 14:32:55 +00:00
9e54646248 [#172] client/container: Support listing containers with containersOf
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 14:32:55 +00:00
bf7d80f44b [#172] morph/client: Support iterators via neo-go session API
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 14:32:55 +00:00
5bf1ec348f [#161] adm: Refactor storage-config command
Resolve funlen linter for storageConfig function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
93cb9e3a94 [#161] morph: Refactor invokeNotary method
Resolve funlen linter

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
512b72591a [#161] adm: Refactor subnet commands
Resolve funlen linter for init method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
e1b99dacad [#161] adm: Refactor netmap add/remove commands
Resolve funlen linter for manageSubnetAdmins function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
92f8810970 [#161] adm: Refactor commands initialization
Resolve funlen linter for init function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
b0fefcb21f [#161] adm: Refactor deposit-notary command
Resolve funlen linter for depositNotary function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
d62b11f5e7 [#161] adm: Refactor init context creation
Resolve funlen linter for newInitializeContext function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
ecbc211016 [#161] adm: Refactor update-contracts command
Resolve funlen linter for updateContracts method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
5f7d70c59c [#161] adm: Refactor deploy command
Resolve funlen linter for deployContractCmd function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
9534ade716 [#161] adm: Refactor restore-containers command
Resolve funlen linter for restoreContainers function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 12:42:37 +00:00
f2e880465e [#165] ir: Refactor config default init
Resolve funlen linter for defaultConfiguration function

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
04b3d9d068 [#165] cli: Drop redundant nolint comment
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
1d5f2dd681 [#165] cli: Refactor collectObjectRelatives
Resolve funlen linter

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
c2cf708e0e [#165] cli: Refactor put command
Resolve funlen linter for putObject method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
c78e9cc857 [#165] cli: Refactor get command
Resolve funlen linter for getObject method

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-27 07:43:56 +00:00
Pavel Karpy
3c7ed21f74 [#141] Add big object's parts to a lock object's body
That will prevent part/link object from being removed by both an external
`DELETE` call and the object expiration procedure.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-27 07:28:32 +00:00
a5ece7889d [#174] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-27 07:24:58 +00:00
Pavel Karpy
f3ff9fd251 [#73] morph: Rename vars that collide with package names
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-24 09:42:30 +00:00
Pavel Karpy
533e9f8b75 [#59] morph: Adopt updated neo-go client API for subs
It does not use deprecated methods anymore but also adds more code that
removes. Future refactor that will affect more components will optimize
usage of the updated API.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-24 09:42:30 +00:00
9ffa0d8fea [#171] Remove unit tests from pipeline
Unit tests are included as pre-commit hook. They are
triggered by the change of .go files.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-24 10:50:11 +03:00
d857ffeb2e [#171] Syncrhonize pre-commit settings across FrostFS repos
This change allows to use `[#xx]` placeholders for issue number.
This change adds go unit test run if .go files were changed.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-24 10:43:55 +03:00
aarifullin
34329d67ff [#86] node: Fix unit test and linter errors
Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 12:42:58 +03:00
9808dec591 [#86] node: Move testing utils to one package
Move testing utils from tests in local_object_storage package to
unified testutil package

Signed-off-by: Airat Arifullin <aarifullin@yadro.com>
2023-03-23 08:19:15 +00:00
342e571d89 [#159] Add handle __SYSTEM__ sys attributes
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-22 17:35:20 +03:00
f111704ceb [#158] Fix gitlint run in pre-commit under CI
Wrong stage was used.
It still may run gitlint against only latest commit in PR.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-22 08:58:34 +00:00
Pavel Karpy
da8da1c63a [#98] fstree: Do not fail iteration over just removed files
A directory is read and files are saved to a local variable. The iteration
over such files may lead to a non-existing files reading due to a normal SN
operation cycle and, therefore, may lead to a returning the OS error to a
caller. Skip just removed (or lost) files as the golang std library does in
similar situations:
5f1a0320b9/src/os/dir_unix.go (L128-L133).

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-22 07:26:27 +00:00
49234b915e [#155] search-service: Fix search with ST
Search should return only objects allowed in static session

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-22 07:21:09 +00:00
1637a3edce [#155] search-service: Add search with ST test
In case of session token (ST) with object IDs search should
return only objects allowed in static session

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-22 07:21:09 +00:00
cbc2efb1d6 [#156] object/get: Make toHeadPrm() pass gocognit
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
484ac502ca [#156] frostfs-adm: Make dumpBalances() pass linter checks
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
8014fdb21a [#156] metabase: Make freePotentialLocks() pass linter checks
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
fb13902db9 [#156] shard: Make refillMetabase() pass linter checks
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
3f6b962349 [#156] services/tree: Pass context to replicationWorker()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
5368c4207a [#156] services/tree: Split syncLoop() in functions
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
47e8c5bf23 [#156] pilorama: Remove CIDDescriptor from TreeApply()
Initially it was there to check whether an update is being initiated by
a proper node. It is now obsolete for 2 reasons:
1. Background synchronization fetches all operations from a single node.
2. There are a lot more problems with trust in the tree service, it is
   only used in controlled environments.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-22 07:14:18 +00:00
ec2c5d45b4 Add to Possible Solutions
Signed-off-by: Liza <e.chichindaeva@yadro.com>
2023-03-21 15:41:20 +03:00
7eb9e88f8f Add bug label; Delte label addition
Signed-off-by: Liza <e.chichindaeva@yadro.com>
2023-03-21 15:09:22 +03:00
9aeea0b974 [#153] ci: Minor pipeline fixes
- We can skip full pre-commit run
- On a very slow agent golangci run may take up to 10 minutes

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-21 09:33:45 +00:00
Pavel Karpy
9a4f40626c [#128] IR: Do not try to emit GAS to nobody
Fix sending GAS to an empty extra wallets receivers list. Also, send GAS to
extra wallets even if netmap is empty.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-21 08:59:46 +00:00
7a31988a36 [#145] docs: Add expired object collector params
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
5059dcc19d [#145] shard-gc: Delete expired objects after locks
GC deletes expired locks and objects sequentially. Expired locks and
objects are now being deleted concurrently in batches. Added a config
parameter that controls the number of concurrent workers and batch size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
6c4a1699ef [#145] shard-gc: Expired locked unit test
Added unit test that verifies that GC deletes expired
locked objects in one epoch.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 11:31:08 +03:00
Pavel Karpy
9cd8f7cea0 [#152] IR: Process empty basic incomes
If network is not configured for basic income earnings, do not distribute
GAS by the Alphabet nodes.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-21 06:59:33 +00:00
44b86bac5a [#148] linter: Add contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
481a1ca6f3 [#148] linter: Add gocognit linter
Code with high cognitive complexity is hard intuitively to understand

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
97c36ed3ec [#148] linter: Add funlen linter
Long functions are hard to understand and source of errors

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:54:41 +03:00
cc8ff015b4 [#148] linter: Add containedctx linter
Context has to be passed as an argument: https://pkg.go.dev/context

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:52:39 +03:00
2dc86058c3 [#148] memstore: Drop space line
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-21 09:52:39 +03:00
573d920821 [#149] Use custom image and kludges for node
Until #139 is fixed, we can't use root inside Docker container running
CI, but Woodpecker CI can't run non-root containers until they fix
https://github.com/woodpecker-ci/woodpecker/issues/1077, hence we use
temporary kludges with custom image and manual permissions in pipelines.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-20 18:37:37 +03:00
d64fb887ff [#149] Reorder pre-commit hooks
Minor changes to see what fails first

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-20 18:37:34 +03:00
392be818e5 [#149] Fix gitlint regex to match our policy
In our policy we mark commits not having a PR/Issue yet with a `[#XX]`
reference to be replaced after PR creation.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-20 18:37:27 +03:00
db3ccd2762 [#128] innerring: Add GAS pouring mechanism for a configurable list of wallets
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-03-20 12:50:05 +00:00
Pavel Karpy
abd21f8099 [#136] Revert "[#2260] services/object: Do not assemble object with TTL=1"
This reverts commit 2567f8020e. It assumes
that assembling logic could break some failover scenarios if request
forwarding is done. However, it also breaks requesting big objects via a
non-container node with TTL=2. Failover has been rechecked without that
commit and no problems were found. Any (if found) other bugs related to
the forwarding and object assembling must be solved more carefully.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-20 10:52:18 +03:00
Pavel Karpy
10c419adf0 [#67] node: Fix infinite recursion in SE's wrapper
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-17 18:47:59 +03:00
b70caa216b [#144] Don't copy cache inside Docker environment
May make docker builds not so clean.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-17 10:36:50 +03:00
Pavel Karpy
64bde68fb9 [#67] node: Accept expired locked objects
Allow replication of any (expired too) locked object. Information about
object locking is considered to be presented on the _container nodes_.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-16 16:22:19 +03:00
Pavel Karpy
f006f3b342 [#67] node: Make engine's IsLocked public
It will allow reusing that method in expiration checks.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-16 16:20:45 +03:00
22be532cbd object/put: Persist session token till the end of a session
Previously a token could've expired in the middle of an object.PUT
stream, leading to upload being interrupted. This is bad, because user
doesn't always now what is the right values for the session token
lifetime. More than that, setting it to a very high value will
eventually blow up the session token database.

In this commit we read the session token once and reuse it for the whole
stream duration.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-16 06:45:50 +00:00
724debfdcd [#81] node: Add basic read/write benchmarks for substorages
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-03-15 16:37:04 +00:00
b1c165a93b [#83] Fix shellcheck pre-commit action
Original shellcheck action requires Docker to run and it's not always
available, especially inside Docker containers. Replacing it with python
wrapper to simplify usage with Docker-based CI systems like Drone and
WoodpeckerCI.

Signed-off-by: Stanislav Bogatyrev <s.bogatyrev@yadro.com>
2023-03-15 12:24:41 +00:00
ac0a278a05 [#85] get-service: Drop unused assemble flag
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
b8e93d4c08 [#85] get-service: Use assembler to assemble LOB
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
07de839f18 [#85] get-service: Fix corrupted chain logic
Should return an error in case of a broken LOB reference chain.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
2886b1581b [#85] get-service: Add unit tests
Add unit tests to cover all assemble statements

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
8b9e40a848 [#85] get-service: Add assembler
Extract assemble logic to assembler

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-03-15 09:19:45 +03:00
b4582239bf [#130] adm: Fix adding of pub key for group.frostfs at init step
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-13 15:05:53 +03:00
4e244686cf [#83] Makefile fixes for pre-commit
Add make targets to simplify pre-commit setup for individual developers.

Signed-off-by: Stanislav Bogatyrev <realloc@realloc.spb.ru>
2023-03-13 11:39:18 +00:00
6cd806f998 [#82] services/tree: Save last synchronized height in a persistent storage
Remember the last synchronized height and use it after service restart.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 11:25:44 +00:00
3e6fd4c611 [#82] pilorama: Allow to store last sync height
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 11:25:44 +00:00
5ae4446280 [#50] ir: Add Health status
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-13 11:24:42 +00:00
5890cd4d7d [#50] ir: Fix config property name for prometheus
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-13 11:24:42 +00:00
365adb4ebd [#133] .github: Restore logo.svg
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 10:39:12 +03:00
bce5827f64 [#83] pre-commit: Add shellcheck hook
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:30 +00:00
05471d3827 [#83] util/autocomplete: Fix deprecated warning
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:30 +00:00
8226d49376 [#83] pre-commit: Add gitlint hook
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:30 +00:00
0893689c6a [#83] pre-commit: Add golangci-lint hook
Skip deprecated warning for now, adopting new neo-go API will be done in
another task.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:29 +00:00
a4931ea4c7 [#83] .github: Remove CODEOWNERS and actions
Issue templates are still supported by Gitea:
https://docs.gitea.io/en-us/issue-pull-request-templates/ .

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:29 +00:00
861e9ab59a [#83] pre-commit: Add initial configuration
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-13 07:07:29 +00:00
Leonard Lyubich
24a540caa8 [#132] cli/util: Fix basic ACL rendering
In previous implementation pretty-printer of basic ACL in NeoFS CLI had
mistakes:
 * F-bit was set to `Extendable()` property instead of its inversion
 * B-bits were set to `acl.RoleInnerRing` rights

Make `PrettyPrintTableBACL` to correctly render mentioned bits.

Signed-off-by: Leonard Lyubich <ctulhurider@gmail.com>
2023-03-10 14:55:45 +03:00
6226c3ba86 [#129] policer: Use safer defaults
If `processNodes` exits earlier for some reason, `needLocalCopy` could
be false.
See https://github.com/nspcc-dev/neofs-node/issues/2267

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-10 10:59:15 +00:00
f2250a316f [#129] tree: Do not remove tree if the netmap is empty
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-03-10 10:59:15 +00:00
9929dcf50b [#126] adm: Exclude group.frostfs key from output of the dump-hashes
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-09 18:29:36 +03:00
7486c02bbc [#88] adm: Fix method nnsResolveKey
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-09 15:41:28 +03:00
Pavel Karpy
f1f3c80dbf [#32] node: Init write-cache asynchronously
Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-09 11:07:33 +00:00
Pavel Karpy
381e363a8b [#32] node: Always close general components after testing
It will prevent test fails with `-race` flag on components that have
background processes and make some actions on test framework.

Signed-off-by: Pavel Karpy <p.karpy@yadro.com>
2023-03-09 11:07:33 +00:00
20de74a505 Rename package name
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 16:38:26 +03:00
79ba34714a [#79] cli: Fix panic when setting domain for container
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-03-01 11:11:11 +03:00
6d4f48f37a [TrueCloudLab#78] .github: Fix CODEOWNERS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-28 16:44:50 +03:00
e9f3c24229 [#65] Use strings.Cut instead of strings.Split* where possible
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-28 13:39:14 +03:00
88e3868f47 [#37] cli: Add nns-name and nns-zone for container create
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2023-02-28 13:37:23 +03:00
6925fb4c59 [TrueCloudLab/hrw#2] node: Use typed HRW methods
Update HRW lib and use typed HRW methods to sort shards and nodes

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-28 13:36:25 +03:00
c3a7039801 [TrueCloudLab/hrw#2] node: Optimize shard hash
Compute shard hash only once

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-28 13:36:25 +03:00
a1ab25b33e [#72] .github: Fix CODEOWNERS
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2023-02-22 16:29:58 +03:00
73bb590cb1 [#64] node: Use pool_size_local and separate pool for local puts
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-22 13:43:19 +03:00
7eaf159a8b [#63] adm: Fix contract wallet creation
Create contract wallet only by init and update-config command.

Close #63

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2023-02-22 10:08:37 +03:00
cb5468abb8 [#66] node: Replace interface{} with any
Signed-off-by: Alejandro Lopez <a.lopez@yadro.com>
2023-02-21 16:47:07 +03:00
1152 changed files with 38438 additions and 38977 deletions

View file

@ -1,4 +1,4 @@
FROM golang:1.18 as builder
FROM golang:1.21 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository

25
.docker/Dockerfile.ci Normal file
View file

@ -0,0 +1,25 @@
FROM golang:1.21
WORKDIR /tmp
# Install apt packages
RUN apt-get update && apt-get install --no-install-recommends -y \
pip \
&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
&& rm -rf /var/lib/apt/lists/*
# Dash → Bash
RUN echo "dash dash/sh boolean false" | debconf-set-selections
RUN DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
RUN useradd -u 1234 -d /home/ci -m ci
USER ci
ENV PATH="$PATH:/home/ci/.local/bin"
COPY .pre-commit-config.yaml .
RUN pip install "pre-commit==3.1.1" \
&& git init . \
&& pre-commit install-hooks \
&& rm -rf /tmp/*

View file

@ -1,4 +1,4 @@
FROM golang:1.18 as builder
FROM golang:1.21 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository

View file

@ -1,4 +1,4 @@
FROM golang:1.18 as builder
FROM golang:1.21 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository

View file

@ -1,4 +1,4 @@
FROM golang:1.18 as builder
FROM golang:1.21 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository

View file

@ -1,19 +0,0 @@
FROM golang:1.18 as builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository
WORKDIR /src
COPY . /src
RUN make bin/frostfs-node
# Executable image
FROM alpine AS frostfs-node
RUN apk add --no-cache bash
WORKDIR /
COPY --from=builder /src/bin/frostfs-node /bin/frostfs-node
COPY --from=builder /src/config/testnet/config.yml /config.yml
CMD ["frostfs-node", "--config", "/config.yml"]

View file

@ -6,3 +6,4 @@ Dockerfile
temp
.dockerignore
docker
.cache

View file

@ -0,0 +1,41 @@
name: Build
on: [pull_request]
jobs:
build:
name: Build Components
runs-on: ubuntu-latest
strategy:
matrix:
go_versions: [ '1.20', '1.21' ]
steps:
- uses: actions/checkout@v3
with:
# Allows to fetch all history for all branches and tags.
# Need this for proper versioning.
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '${{ matrix.go_versions }}'
- name: Build CLI
run: make bin/frostfs-cli
- run: bin/frostfs-cli --version
- name: Build NODE
run: make bin/frostfs-node
- name: Build IR
run: make bin/frostfs-ir
- name: Build ADM
run: make bin/frostfs-adm
- run: bin/frostfs-adm --version
- name: Build LENS
run: make bin/frostfs-lens
- run: bin/frostfs-lens --version

View file

@ -0,0 +1,21 @@
name: DCO action
on: [pull_request]
jobs:
dco:
name: DCO
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
- name: Run commit format checker
uses: https://git.frostfs.info/TrueCloudLab/dco-go@v2
with:
from: 'origin/${{ github.event.pull_request.base.ref }}'

View file

@ -0,0 +1,73 @@
name: Tests and linters
on: [pull_request]
jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
cache: true
- name: Install linters
run: make lint-install
- name: Run linters
run: make lint
tests:
name: Tests
runs-on: ubuntu-latest
strategy:
matrix:
go_versions: [ '1.20', '1.21' ]
fail-fast: false
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '${{ matrix.go_versions }}'
cache: true
- name: Run tests
run: make test
tests-race:
name: Tests with -race
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
cache: true
- name: Run tests
run: go test ./... -count=1 -race
staticcheck:
name: Staticcheck
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
cache: true
- name: Install staticcheck
run: make staticcheck-install
- name: Run staticcheck
run: make staticcheck-run

View file

@ -0,0 +1,22 @@
name: Vulncheck
on: [pull_request]
jobs:
vulncheck:
name: Vulncheck
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
- name: Install govulncheck
run: go install golang.org/x/vuln/cmd/govulncheck@latest
- name: Run govulncheck
run: govulncheck ./...

1
.gitattributes vendored
View file

@ -1,2 +1,3 @@
/**/*.pb.go -diff -merge
/**/*.pb.go linguist-generated=true
/go.sum -diff

1
.github/CODEOWNERS vendored
View file

@ -1 +0,0 @@
* @carpawell @fyrchik @acid-ant

View file

@ -2,7 +2,7 @@
name: Bug report
about: Create a report to help us improve
title: ''
labels: community, triage
labels: community, triage, bug
assignees: ''
---
@ -18,8 +18,11 @@ assignees: ''
If suggesting a change/improvement, explain the difference from current behavior -->
## Possible Solution
<!-- Not obligatory, but suggest a fix/reason for the bug,
or ideas how to implement the addition or change -->
<!-- Not obligatory
If no reason/fix/additions for the bug can be suggested,
uncomment the following phrase:
No fix can be suggested by a QA engineer. Further solutions shall be up to developers. -->
## Steps to Reproduce (for bugs)
<!-- Provide a link to a live example, or an unambiguous set of steps
@ -41,10 +44,3 @@ assignees: ''
* Version used:
* Server setup and configuration:
* Operating System and version (`uname -a`):
## Don't forget to add labels!
- component label (`frostfs-adm`, `frostfs-storage`, ...)
- `goodfirstissue`, `helpwanted` if needed
- does this issue belong to an epic?
- priority (`P0`-`P4`) if already triaged
- quarter label (`202XQY`) if possible

View file

@ -1,29 +0,0 @@
name: CHANGELOG check
on:
pull_request:
branches:
- master
- support/**
jobs:
build:
runs-on: ubuntu-latest
name: Check for updates
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Get changed CHANGELOG
id: changelog-diff
uses: tj-actions/changed-files@v29
with:
files: CHANGELOG.md
- name: Fail if changelog not updated
if: steps.changelog-diff.outputs.any_changed == 'false'
uses: actions/github-script@v3
with:
script: |
core.setFailed('CHANGELOG.md has not been updated')

View file

@ -1,37 +0,0 @@
name: Configuration check
on:
pull_request:
branches:
- master
- support/**
jobs:
build:
runs-on: ubuntu-latest
name: config-check
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Get changed config-related files
id: config-diff
uses: tj-actions/changed-files@v29
with:
files: |
config/**
cmd/neofs-node/config/**
- name: Get changed doc files
id: docs-diff
uses: tj-actions/changed-files@v29
with:
files: docs/**
- name: Fail if config files are changed but the documentation is not updated
if: steps.config-diff.outputs.any_changed == 'true' && steps.docs-diff.outputs.any_changed == 'false'
uses: actions/github-script@v3
with:
script: |
core.setFailed('Documentation has not been updated')

View file

@ -1,22 +0,0 @@
name: DCO check
on:
pull_request:
branches:
- master
- support/**
jobs:
commits_check_job:
runs-on: ubuntu-latest
name: Commits Check
steps:
- name: Get PR Commits
id: 'get-pr-commits'
uses: tim-actions/get-pr-commits@master
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: DCO Check
uses: tim-actions/dco@master
with:
commits: ${{ steps.get-pr-commits.outputs.commits }}

View file

@ -1,60 +0,0 @@
name: frostfs-node tests
on:
push:
branches:
- master
- support/**
paths-ignore:
- '*.md'
pull_request:
branches:
- master
- support/**
paths-ignore:
- '*.md'
jobs:
test:
runs-on: ubuntu-20.04
strategy:
matrix:
go: [ '1.18.x', '1.19.x' ]
steps:
- name: Setup go
uses: actions/setup-go@v3
with:
go-version: ${{ matrix.go }}
- name: Check out code
uses: actions/checkout@v3
- name: Cache go mod
uses: actions/cache@v3
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-go-${{ matrix.go }}-
- name: Run go test
run: go test -coverprofile=coverage.txt -covermode=atomic ./...
- name: Codecov
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
run: bash <(curl -s https://codecov.io/bash)
lint:
runs-on: ubuntu-20.04
steps:
- uses: actions/setup-go@v3
with:
go-version: 1.19
- uses: actions/checkout@v3
- name: golangci-lint
uses: golangci/golangci-lint-action@v3
with:
version: v1.50.0
args: --timeout=5m
only-new-issues: true

11
.gitlint Normal file
View file

@ -0,0 +1,11 @@
[general]
fail-without-commits=True
regex-style-search=True
contrib=CC1
[title-match-regex]
regex=^\[\#[0-9Xx]+\]\s
[ignore-by-title]
regex=^Release(.*)
ignore=title-match-regex

View file

@ -4,7 +4,7 @@
# options for analysis running
run:
# timeout for analysis, e.g. 30s, 5m, default is 1m
timeout: 5m
timeout: 20m
# include test files or not, default is true
tests: false
@ -24,6 +24,28 @@ linters-settings:
govet:
# report about shadowed variables
check-shadowing: false
staticcheck:
checks: ["all", "-SA1019"] # TODO Enable SA1019 after deprecated warning are fixed.
funlen:
lines: 80 # default 60
statements: 60 # default 40
gocognit:
min-complexity: 40 # default 30
importas:
no-unaliased: true
no-extra-aliases: false
alias:
pkg: git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object
alias: objectSDK
custom:
truecloudlab-linters:
path: bin/external_linters.so
original-url: git.frostfs.info/TrueCloudLab/linters.git
settings:
noliteral:
target-methods : ["reportFlushError", "reportError"]
disable-packages: ["codes", "err", "res","exec"]
constants-package: "git.frostfs.info/TrueCloudLab/frostfs-node/internal/logs"
linters:
enable:
@ -51,6 +73,11 @@ linters:
- predeclared
- reassign
- whitespace
- containedctx
- funlen
- gocognit
- contextcheck
- importas
- truecloudlab-linters
disable-all: true
fast: false

54
.pre-commit-config.yaml Normal file
View file

@ -0,0 +1,54 @@
ci:
autofix_prs: false
repos:
- repo: https://github.com/jorisroovers/gitlint
rev: v0.19.1
hooks:
- id: gitlint
stages: [commit-msg]
- id: gitlint-ci
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: check-added-large-files
- id: check-case-conflict
- id: check-executables-have-shebangs
- id: check-shebang-scripts-are-executable
- id: check-merge-conflict
- id: check-json
- id: check-xml
- id: check-yaml
- id: trailing-whitespace
args: [--markdown-linebreak-ext=md]
- id: end-of-file-fixer
exclude: ".key$"
- repo: https://github.com/shellcheck-py/shellcheck-py
rev: v0.9.0.5
hooks:
- id: shellcheck
- repo: local
hooks:
- id: make-lint
name: Run Make Lint
entry: make lint
language: system
pass_filenames: false
- repo: local
hooks:
- id: go-unit-tests
name: go unit tests
entry: make test
pass_filenames: false
types: [go]
language: system
- repo: https://github.com/TekWizely/pre-commit-golang
rev: v1.0.0-rc.1
hooks:
- id: go-staticcheck-repo-mod
- id: go-mod-tidy

View file

@ -0,0 +1,11 @@
pipeline:
# Kludge for non-root containers under WoodPecker
fix-ownership:
image: alpine:latest
commands: chown -R 1234:1234 .
pre-commit:
image: git.frostfs.info/truecloudlab/frostfs-ci:v0.36
commands:
- export HOME="$(getent passwd $(id -u) | cut '-d:' -f6)"
- pre-commit run --hook-stage manual

File diff suppressed because it is too large Load diff

View file

@ -3,8 +3,8 @@
First, thank you for contributing! We love and encourage pull requests from
everyone. Please follow the guidelines:
- Check the open [issues](https://github.com/TrueCloudLab/frostfs-node/issues) and
[pull requests](https://github.com/TrueCloudLab/frostfs-node/pulls) for existing
- Check the open [issues](https://git.frostfs.info/TrueCloudLab/frostfs-node/issues) and
[pull requests](https://git.frostfs.info/TrueCloudLab/frostfs-node/pulls) for existing
discussions.
- Open an issue first, to discuss a new feature or enhancement.
@ -27,19 +27,19 @@ Start by forking the `frostfs-node` repository, make changes in a branch and the
send a pull request. We encourage pull requests to discuss code changes. Here
are the steps in details:
### Set up your GitHub Repository
Fork [FrostFS node upstream](https://github.com/TrueCloudLab/frostfs-node/fork) source
### Set up your Forgejo repository
Fork [FrostFS node upstream](https://git.frostfs.info/TrueCloudLab/frostfs-node) source
repository to your own personal repository. Copy the URL of your fork (you will
need it for the `git clone` command below).
```sh
$ git clone https://github.com/TrueCloudLab/frostfs-node
$ git clone https://git.frostfs.info/TrueCloudLab/frostfs-node
```
### Set up git remote as ``upstream``
```sh
$ cd frostfs-node
$ git remote add upstream https://github.com/TrueCloudLab/frostfs-node
$ git remote add upstream https://git.frostfs.info/TrueCloudLab/frostfs-node
$ git fetch upstream
$ git merge upstream/master
...
@ -58,7 +58,7 @@ $ git checkout -b feature/123-something_awesome
After your code changes, make sure
- To add test cases for the new code.
- To run `make lint`
- To run `make lint` and `make staticcheck-run`
- To squash your commits into a single commit or a series of logically separated
commits run `git rebase -i`. It's okay to force update your pull request.
- To run `make test` and `make all` completes.
@ -89,8 +89,8 @@ $ git push origin feature/123-something_awesome
```
### Create a Pull Request
Pull requests can be created via GitHub. Refer to [this
document](https://help.github.com/articles/creating-a-pull-request/) for
Pull requests can be created via Forgejo. Refer to [this
document](https://docs.codeberg.org/collaborating/pull-requests-and-git-flow/) for
detailed steps on how to create a pull request. After a Pull Request gets peer
reviewed and approved, it will be merged.

59
Makefile Normal file → Executable file
View file

@ -7,8 +7,9 @@ VERSION ?= $(shell git describe --tags --dirty --match "v*" --always --abbrev=8
HUB_IMAGE ?= truecloudlab/frostfs
HUB_TAG ?= "$(shell echo ${VERSION} | sed 's/^v//')"
GO_VERSION ?= 1.19
LINT_VERSION ?= 1.50.0
GO_VERSION ?= 1.21
LINT_VERSION ?= 1.54.0
TRUECLOUDLAB_LINT_VERSION ?= 0.0.2
ARCH = amd64
BIN = bin
@ -25,8 +26,12 @@ PKG_VERSION ?= $(shell echo $(VERSION) | sed "s/^v//" | \
sed -E "s/(.*)-(g[a-fA-F0-9]{6,8})(.*)/\1\3~\2/" | \
sed "s/-/~/")-${OS_RELEASE}
OUTPUT_LINT_DIR ?= $(shell pwd)/bin
LINT_DIR = $(OUTPUT_LINT_DIR)/golangci-lint-$(LINT_VERSION)-v$(TRUECLOUDLAB_LINT_VERSION)
TMP_DIR := .cache
.PHONY: help all images dep clean fmts fmt imports test lint docker/lint
prepare-release debpackage
prepare-release debpackage pre-commit unpre-commit
# To build a specific binary, use it's name prefix with bin/ as a target
# For example `make bin/frostfs-node` will build only storage node binary
@ -65,12 +70,18 @@ dep:
CGO_ENABLED=0 \
go mod tidy -v && echo OK
# Build export-metrics
export-metrics: dep
@printf "⇒ Build export-metrics\n"
CGO_ENABLED=0 \
go build -v -trimpath -o bin/export-metrics ./scripts/export-metrics
# Regenerate proto files:
protoc:
@GOPRIVATE=github.com/TrueCloudLab go mod vendor
# Install specific version for protobuf lib
@go list -f '{{.Path}}/...@{{.Version}}' -m github.com/golang/protobuf | xargs go install -v
@GOBIN=$(abspath $(BIN)) go install -mod=mod -v github.com/TrueCloudLab/frostfs-api-go/v2/util/protogen
@GOBIN=$(abspath $(BIN)) go install -mod=mod -v git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/util/protogen
# Protoc generate
@for f in `find . -type f -name '*.proto' -not -path './vendor/*'`; do \
echo "⇒ Processing $$f "; \
@ -95,7 +106,7 @@ image-%:
-t $(HUB_IMAGE)-$*:$(HUB_TAG) .
# Build all Docker images
images: image-storage image-ir image-cli image-adm image-storage-testnet
images: image-storage image-ir image-cli image-adm
# Build dirty local Docker images
dirty-images: image-dirty-storage image-dirty-ir image-dirty-cli image-dirty-adm
@ -126,11 +137,36 @@ imports:
# Run Unit Test with go test
test:
@echo "⇒ Running go test"
@go test ./...
@go test ./... -count=1
pre-commit-run:
@pre-commit run -a --hook-stage manual
# Install linters
lint-install:
@mkdir -p $(TMP_DIR)
@rm -rf $(TMP_DIR)/linters
@git -c advice.detachedHead=false clone --branch v$(TRUECLOUDLAB_LINT_VERSION) https://git.frostfs.info/TrueCloudLab/linters.git $(TMP_DIR)/linters
@@make -C $(TMP_DIR)/linters lib CGO_ENABLED=1 OUT_DIR=$(OUTPUT_LINT_DIR)
@rm -rf $(TMP_DIR)/linters
@rmdir $(TMP_DIR) 2>/dev/null || true
@CGO_ENABLED=1 GOBIN=$(LINT_DIR) go install github.com/golangci/golangci-lint/cmd/golangci-lint@v$(LINT_VERSION)
# Run linters
lint:
@golangci-lint --timeout=5m run
@if [ ! -d "$(LINT_DIR)" ]; then \
echo "Run make lint-install"; \
exit 1; \
fi
$(LINT_DIR)/golangci-lint run
# Install staticcheck
staticcheck-install:
@go install honnef.co/go/tools/cmd/staticcheck@latest
# Run staticcheck
staticcheck-run:
@staticcheck ./...
# Run linters in Docker
docker/lint:
@ -140,10 +176,19 @@ docker/lint:
--env HOME=/src \
golangci/golangci-lint:v$(LINT_VERSION) bash -c 'cd /src/ && make lint'
# Activate pre-commit hooks
pre-commit:
pre-commit install -t pre-commit -t commit-msg
# Deactivate pre-commit hooks
unpre-commit:
pre-commit uninstall -t pre-commit -t commit-msg
# Print version
version:
@echo $(VERSION)
# Delete built artifacts
clean:
rm -rf vendor
rm -rf .cache

View file

@ -31,7 +31,7 @@ dApps directly from
code level. This way dApps are not limited to on-chain storage and can
manipulate large amounts of data without paying a prohibitive price.
FrostFS has a native [gRPC API](https://github.com/TrueCloudLab/frostfs-api) and has
FrostFS has a native [gRPC API](https://git.frostfs.info/TrueCloudLab/frostfs-api) and has
protocol gateways for popular protocols such as [AWS
S3](https://github.com/TrueCloudLab/frostfs-s3-gw),
[HTTP](https://github.com/TrueCloudLab/frostfs-http-gw),
@ -49,7 +49,7 @@ The latest version of frostfs-node works with frostfs-contract
# Building
To make all binaries you need Go 1.18+ and `make`:
To make all binaries you need Go 1.20+ and `make`:
```
make all
```

View file

@ -1 +1 @@
v0.35.0
v0.36.0

View file

@ -18,8 +18,7 @@ Build docker image with `make image-adm`.
At FrostFS private install deployment, frostfs-adm requires compiled FrostFS
contracts. Find them in the latest release of
[frostfs-contract repository](https://github.com/TrueCloudLab/frostfs-contract/releases).
[frostfs-contract repository](https://git.frostfs.info/TrueCloudLab/frostfs-contract/releases).
## Commands
@ -37,9 +36,7 @@ alphabet-wallets: /path # path to consensus node / alphabet wallets s
network:
max_object_size: 67108864 # max size of a single FrostFS object, bytes
epoch_duration: 240 # duration of a FrostFS epoch in blocks, consider block generation frequency in the sidechain
basic_income_rate: 0 # basic income rate, for private consider 0
fee:
audit: 0 # network audit fee, for private installation consider 0
candidate: 0 # inner ring candidate registration fee, for private installation consider 0
container: 0 # container creation fee, for private installation consider 0
container_alias: 0 # container nice-name registration fee, for private installation consider 0

View file

@ -18,6 +18,7 @@ To start a network, you need a set of consensus nodes, the same number of
Alphabet nodes and any number of Storage nodes. While the number of Storage
nodes can be scaled almost infinitely, the number of consensus and Alphabet
nodes can't be changed so easily right now. Consider this before going any further.
Note also that there is an upper limit on the number of alphabet nodes (currently 22).
It is easier to use`frostfs-adm` with a predefined configuration. First, create
a network configuration file. In this example, there is going to be only one
@ -33,9 +34,7 @@ alphabet-wallets: /home/user/deploy/alphabet-wallets
network:
max_object_size: 67108864
epoch_duration: 240
basic_income_rate: 0
fee:
audit: 0
candidate: 0
container: 0
withdraw: 0
@ -141,13 +140,11 @@ Waiting for transactions to persist...
Stage 7: set addresses in NNS.
Waiting for transactions to persist...
NNS: Set alphabet0.frostfs -> f692dfb4d43a15b464eb51a7041160fb29c44b6a
NNS: Set audit.frostfs -> 7df847b993affb3852074345a7c2bd622171ee0d
NNS: Set balance.frostfs -> 103519b3067a66307080a66570c0491ee8f68879
NNS: Set container.frostfs -> cae60bdd689d185901e495352d0247752ce50846
NNS: Set frostfsid.frostfs -> c421fb60a3895865a8f24d197d6a80ef686041d2
NNS: Set netmap.frostfs -> 894eb854632f50fb124412ce7951ebc00763525e
NNS: Set proxy.frostfs -> ac6e6fe4b373d0ca0ca4969d1e58fa0988724e7d
NNS: Set reputation.frostfs -> 6eda57c9d93d990573646762d1fea327ce41191f
Waiting for transactions to persist...
```

View file

@ -1,39 +0,0 @@
# FrostFS subnetwork creation
This is a short guide on how to create FrostFS subnetworks. This guide
considers that the sidechain and the inner ring (alphabet nodes) have already been
deployed and the sidechain contains a deployed `subnet` contract.
## Prerequisites
To follow this guide, you need:
- neo-go sidechain RPC endpoint;
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases);
- wallet with FrostFS account.
## Creation
```shell
$ frostfs-adm morph subnet create \
-r <side_chain_RPC_endpoint> \
-w </path/to/owner/wallet> \
--notary
Create subnet request sent successfully. ID: 4223489767.
```
**NOTE:** in notary-enabled environment you should have a sufficient
notary deposit (not expired, with enough GAS balance). Your subnet ID
will differ from the example.
The default account in the wallet that has been passed with `-w` flag is the owner
of the just created subnetwork.
You can check if your subnetwork was created successfully:
```shell
$ frostfs-adm morph subnet get \
-r <side_chain_RPC_endpoint> \
--subnet <subnet_ID>
Owner: NUc734PMJXiqa2J9jRtvskU3kCdyyuSN8Q
```
Your owner will differ from the example.

View file

@ -1,137 +0,0 @@
# Managing Subnetworks
This is a short guide on how to manage FrostFS subnetworks. This guide
considers that the sidechain and the inner ring (alphabet nodes) have already been
deployed, and the sidechain contains a deployed `subnet` contract.
## Prerequisites
- neo-go sidechain RPC endpoint;
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases);
- [created](subnetwork-creation.md) subnetwork;
- wallet with the account that owns the subnetwork;
- public key of the Storage Node;
- public keys of the node and client administrators;
- owner IDs of the FrostFS users.
## Add node administrator
Node administrators are accounts that can manage (add and delete nodes)
the whitelist of the nodes which can be included to a subnetwork. Only the subnet
owner is allowed to add and remove node administrators from the subnetwork.
```shell
$ frostfs-adm morph subnet admin add \
-r <side_chain_RPC_endpoint> \
-w </path/to/owner/wallet> \
--admin <HEX_admin_public_key> \
--subnet <subnet_ID>
Add admin request sent successfully.
```
## Add node
Adding a node to a subnetwork means that the node becomes able to service
containers that have been created in that subnetwork. Addition only changes
the list of the allowed nodes. Node is not required to be bootstrapped at the
moment of its inclusion.
```shell
$ frostfs-adm morph subnet node add \
-r <side_chain_RPC_endpoint> \
-w </path/to/node_admin/wallet> \
--node <HEX_node_public_key> \
--subnet <subnet_ID>
Add node request sent successfully.
```
**NOTE:** the owner of the subnetwork is also allowed to add nodes.
## Add client administrator
Client administrators are accounts that can manage (add and delete
nodes) the whitelist of the clients that can create containers in the
subnetwork. Only the subnet owner is allowed to add and remove client
administrators from the subnetwork.
```shell
$ frostfs-adm morph subnet admin add \
-r <side_chain_RPC_endpoint> \
-w </path/to/owner/wallet> \
--admin <HEX_admin_public_key> \
--subnet <subnet_ID> \
--client \
--group <group_ID>
Add admin request sent successfully.
```
**NOTE:** you do not need to create a group explicitly, it will be created
right after the first client admin is added. Group ID is a 4-byte
positive integer number.
## Add client
```shell
$ frostfs-adm morph subnet client add \
-r <side_chain_RPC_endpoint> \
-w </path/to/client_admin/wallet> \
--client <client_ownerID> \
--subnet <subnet_ID> \
--group <group_ID>
Add client request sent successfully.
```
**NOTE:** the owner of the subnetwork is also allowed to add clients. This is
the only one command that accepts `ownerID`, not the public key.
Administrator can manage only their group (a group where that administrator
has been added by the subnet owner).
# Bootstrapping Storage Node
After a subnetwork [is created](subnetwork-creation.md) and a node is included into it, the
node could be bootstrapped and service subnetwork containers.
For bootstrapping, you need to specify the ID of the subnetwork in the node's
configuration:
```yaml
...
node:
...
subnet:
entries: # list of IDs of subnets to enter in a text format of FrostFS API protocol (overrides corresponding attributes)
- <subnetwork_ID>
...
...
```
**NOTE:** specifying subnetwork that is denied for the node is not an error:
that configuration value would be ignored. You do not need to specify zero
(with 0 ID) subnetwork: its inclusion is implicit. On the contrary, to exclude
a node from the default zero subnetwork, you need to specify it explicitly:
```yaml
...
node:
...
subnet:
exit_zero: true # toggle entrance to zero subnet (overrides corresponding attribute and occurrence in `entries`)
...
...
```
# Creating container in non-zero subnetwork
Creating containers without using `--subnet` flag is equivalent to
creating container in the zero subnetwork.
To create a container in a private network, your wallet must be added to
the client whitelist by the client admins or the subnet owners:
```shell
$ frostfs-cli container create \
--policy 'REP 1' \
-w </path/to/wallet> \
-r s01.frostfs.devenv:8080 \
--subnet <subnet_ID>
```

View file

@ -7,7 +7,7 @@ import (
"path/filepath"
"text/template"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/spf13/cobra"
"github.com/spf13/viper"
@ -18,8 +18,6 @@ type configTemplate struct {
AlphabetDir string
MaxObjectSize int
EpochDuration int
BasicIncomeRate int
AuditFee int
CandidateFee int
ContainerFee int
ContainerAliasFee int
@ -33,10 +31,8 @@ alphabet-wallets: {{ .AlphabetDir}}
network:
max_object_size: {{ .MaxObjectSize}}
epoch_duration: {{ .EpochDuration}}
basic_income_rate: {{ .BasicIncomeRate}}
homomorphic_hash_disabled: {{ .HomomorphicHashDisabled}}
fee:
audit: {{ .AuditFee}}
candidate: {{ .CandidateFee}}
container: {{ .ContainerFee}}
container_alias: {{ .ContainerAliasFee }}
@ -47,7 +43,7 @@ credentials:
{{.}}: password{{end}}
`
func initConfig(cmd *cobra.Command, args []string) error {
func initConfig(cmd *cobra.Command, _ []string) error {
configPath, err := readConfigPathFromArgs(cmd)
if err != nil {
return nil
@ -111,9 +107,7 @@ func generateConfigExample(appDir string, credSize int) (string, error) {
Endpoint: "https://neo.rpc.node:30333",
MaxObjectSize: 67108864, // 64 MiB
EpochDuration: 240, // 1 hour with 15s per block
BasicIncomeRate: 1_0000_0000, // 0.0001 GAS per GiB (Fixed12)
HomomorphicHashDisabled: false, // object homomorphic hash is enabled
AuditFee: 1_0000, // 0.00000001 GAS per audit (Fixed12)
CandidateFee: 100_0000_0000, // 100.0 GAS (Fixed8)
ContainerFee: 1000, // 0.000000001 * 7 GAS per container (Fixed12)
ContainerAliasFee: 500, // ContainerFee / 2

View file

@ -5,7 +5,7 @@ import (
"path/filepath"
"testing"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
)
@ -28,8 +28,6 @@ func TestGenerateConfigExample(t *testing.T) {
require.Equal(t, filepath.Join(appDir, "alphabet-wallets"), v.GetString("alphabet-wallets"))
require.Equal(t, 67108864, v.GetInt("network.max_object_size"))
require.Equal(t, 240, v.GetInt("network.epoch_duration"))
require.Equal(t, 100000000, v.GetInt("network.basic_income_rate"))
require.Equal(t, 10000, v.GetInt("network.fee.audit"))
require.Equal(t, 10000000000, v.GetInt("network.fee.candidate"))
require.Equal(t, 1000, v.GetInt("network.fee.container"))
require.Equal(t, 100000000, v.GetInt("network.fee.withdraw"))

View file

@ -6,8 +6,8 @@ import (
"fmt"
"math/big"
"github.com/TrueCloudLab/frostfs-contract/nns"
"github.com/TrueCloudLab/frostfs-sdk-go/netmap"
"git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/nspcc-dev/neo-go/pkg/core/native/noderoles"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
@ -37,11 +37,6 @@ const (
dumpBalancesAlphabetFlag = "alphabet"
dumpBalancesProxyFlag = "proxy"
dumpBalancesUseScriptHashFlag = "script-hash"
// notaryEnabled signifies whether contracts were deployed in a notary-enabled environment.
// The setting is here to simplify testing and building the command for testnet (notary currently disabled).
// It will be removed eventually.
notaryEnabled = true
)
func dumpBalances(cmd *cobra.Command, _ []string) error {
@ -60,7 +55,7 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
inv := invoker.New(c, nil)
if !notaryEnabled || dumpStorage || dumpAlphabet || dumpProxy {
if dumpStorage || dumpAlphabet || dumpProxy {
nnsCs, err = c.GetContractStateByID(1)
if err != nil {
return fmt.Errorf("can't get NNS contract info: %w", err)
@ -72,7 +67,7 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
}
}
irList, err := fetchIRNodes(c, nmHash, rolemgmt.Hash)
irList, err := fetchIRNodes(c, rolemgmt.Hash)
if err != nil {
return err
}
@ -83,6 +78,27 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
printBalances(cmd, "Inner ring nodes balances:", irList)
if dumpStorage {
if err := printStorageNodeBalances(cmd, inv, nmHash); err != nil {
return err
}
}
if dumpProxy {
if err := printProxyContractBalance(cmd, inv, nnsCs.Hash); err != nil {
return err
}
}
if dumpAlphabet {
if err := printAlphabetContractBalances(cmd, c, inv, len(irList), nnsCs.Hash); err != nil {
return err
}
}
return nil
}
func printStorageNodeBalances(cmd *cobra.Command, inv *invoker.Invoker, nmHash util.Uint160) error {
arr, err := unwrap.Array(inv.Call(nmHash, "netmap"))
if err != nil {
return errors.New("can't fetch the list of storage nodes")
@ -112,11 +128,13 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
if err := fetchBalances(inv, gas.Hash, snList); err != nil {
return err
}
printBalances(cmd, "\nStorage node balances:", snList)
}
if dumpProxy {
h, err := nnsResolveHash(inv, nnsCs.Hash, proxyContract+".frostfs")
printBalances(cmd, "\nStorage node balances:", snList)
return nil
}
func printProxyContractBalance(cmd *cobra.Command, inv *invoker.Invoker, nnsHash util.Uint160) error {
h, err := nnsResolveHash(inv, nnsHash, proxyContract+".frostfs")
if err != nil {
return fmt.Errorf("can't get hash of the proxy contract: %w", err)
}
@ -125,15 +143,17 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
if err := fetchBalances(inv, gas.Hash, proxyList); err != nil {
return err
}
printBalances(cmd, "\nProxy contract balance:", proxyList)
}
if dumpAlphabet {
alphaList := make([]accBalancePair, len(irList))
printBalances(cmd, "\nProxy contract balance:", proxyList)
return nil
}
func printAlphabetContractBalances(cmd *cobra.Command, c Client, inv *invoker.Invoker, count int, nnsHash util.Uint160) error {
alphaList := make([]accBalancePair, count)
w := io.NewBufBinWriter()
for i := range alphaList {
emit.AppCall(w.BinWriter, nnsCs.Hash, "resolve", callflag.ReadOnly,
emit.AppCall(w.BinWriter, nnsHash, "resolve", callflag.ReadOnly,
getAlphabetNNSDomain(i),
int64(nns.TXT))
}
@ -157,18 +177,14 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
if err := fetchBalances(inv, gas.Hash, alphaList); err != nil {
return err
}
printBalances(cmd, "\nAlphabet contracts balances:", alphaList)
}
printBalances(cmd, "\nAlphabet contracts balances:", alphaList)
return nil
}
func fetchIRNodes(c Client, nmHash, desigHash util.Uint160) ([]accBalancePair, error) {
var irList []accBalancePair
func fetchIRNodes(c Client, desigHash util.Uint160) ([]accBalancePair, error) {
inv := invoker.New(c, nil)
if notaryEnabled {
height, err := c.GetBlockCount()
if err != nil {
return nil, fmt.Errorf("can't get block height: %w", err)
@ -179,25 +195,10 @@ func fetchIRNodes(c Client, nmHash, desigHash util.Uint160) ([]accBalancePair, e
return nil, errors.New("can't fetch list of IR nodes from the netmap contract")
}
irList = make([]accBalancePair, len(arr))
irList := make([]accBalancePair, len(arr))
for i := range arr {
irList[i].scriptHash = arr[i].GetScriptHash()
}
} else {
arr, err := unwrap.ArrayOfBytes(inv.Call(nmHash, "innerRingList"))
if err != nil {
return nil, errors.New("can't fetch list of IR nodes from the netmap contract")
}
irList = make([]accBalancePair, len(arr))
for i := range arr {
pub, err := keys.NewPublicKeyFromBytes(arr[i], elliptic.P256())
if err != nil {
return nil, fmt.Errorf("can't parse IR node public key: %w", err)
}
irList[i].scriptHash = pub.GetScriptHash()
}
}
return irList, nil
}

View file

@ -10,12 +10,12 @@ import (
"strings"
"text/tabwriter"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@ -48,41 +48,24 @@ func dumpNetworkConfig(cmd *cobra.Command, _ []string) error {
buf := bytes.NewBuffer(nil)
tw := tabwriter.NewWriter(buf, 0, 2, 2, ' ', 0)
for _, param := range arr {
tuple, ok := param.Value().([]stackitem.Item)
if !ok || len(tuple) != 2 {
return errors.New("invalid ListConfig response from netmap contract")
}
k, err := tuple[0].TryBytes()
m, err := parseConfigFromNetmapContract(arr)
if err != nil {
return errors.New("invalid config key from netmap contract")
return err
}
v, err := tuple[1].TryBytes()
if err != nil {
return invalidConfigValueErr(k)
}
switch string(k) {
case netmapAuditFeeKey, netmapBasicIncomeRateKey,
netmapContainerFeeKey, netmapContainerAliasFeeKey,
netmapEigenTrustIterationsKey,
netmapEpochKey, netmapInnerRingCandidateFeeKey,
netmapMaxObjectSizeKey, netmapWithdrawFeeKey:
for k, v := range m {
switch k {
case netmap.ContainerFeeConfig, netmap.ContainerAliasFeeConfig,
netmap.EpochDurationConfig, netmap.IrCandidateFeeConfig,
netmap.MaxObjectSizeConfig, netmap.WithdrawFeeConfig:
nbuf := make([]byte, 8)
copy(nbuf[:], v)
n := binary.LittleEndian.Uint64(nbuf)
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%d (int)\n", k, n)))
case netmapEigenTrustAlphaKey:
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%s (str)\n", k, v)))
case netmapHomomorphicHashDisabledKey, netmapMaintenanceAllowedKey:
vBool, err := tuple[1].TryBool()
if err != nil {
case netmap.HomomorphicHashingDisabledKey, netmap.MaintenanceModeAllowedConfig:
if len(v) == 0 || len(v) > 1 {
return invalidConfigValueErr(k)
}
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%t (bool)\n", k, vBool)))
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%t (bool)\n", k, v[0] == 1)))
default:
_, _ = tw.Write([]byte(fmt.Sprintf("%s:\t%s (hex)\n", k, hex.EncodeToString(v))))
}
@ -140,35 +123,24 @@ func setConfigCmd(cmd *cobra.Command, args []string) error {
return wCtx.awaitTx()
}
func parseConfigPair(kvStr string, force bool) (key string, val interface{}, err error) {
kv := strings.SplitN(kvStr, "=", 2)
if len(kv) != 2 {
func parseConfigPair(kvStr string, force bool) (key string, val any, err error) {
k, v, found := strings.Cut(kvStr, "=")
if !found {
return "", nil, fmt.Errorf("invalid parameter format: must be 'key=val', got: %s", kvStr)
}
key = kv[0]
valRaw := kv[1]
key = k
valRaw := v
switch key {
case netmapAuditFeeKey, netmapBasicIncomeRateKey,
netmapContainerFeeKey, netmapContainerAliasFeeKey,
netmapEigenTrustIterationsKey,
netmapEpochKey, netmapInnerRingCandidateFeeKey,
netmapMaxObjectSizeKey, netmapWithdrawFeeKey:
case netmap.ContainerFeeConfig, netmap.ContainerAliasFeeConfig,
netmap.EpochDurationConfig, netmap.IrCandidateFeeConfig,
netmap.MaxObjectSizeConfig, netmap.WithdrawFeeConfig:
val, err = strconv.ParseInt(valRaw, 10, 64)
if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as int: %w", key, valRaw, err)
}
case netmapEigenTrustAlphaKey:
// just check that it could
// be parsed correctly
_, err = strconv.ParseFloat(kv[1], 64)
if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as float: %w", key, valRaw, err)
}
val = valRaw
case netmapHomomorphicHashDisabledKey, netmapMaintenanceAllowedKey:
case netmap.HomomorphicHashingDisabledKey, netmap.MaintenanceModeAllowedConfig:
val, err = strconv.ParseBool(valRaw)
if err != nil {
err = fmt.Errorf("could not parse %s's value '%s' as bool: %w", key, valRaw, err)
@ -187,6 +159,6 @@ func parseConfigPair(kvStr string, force bool) (key string, val interface{}, err
return
}
func invalidConfigValueErr(key []byte) error {
func invalidConfigValueErr(key string) error {
return fmt.Errorf("invalid %s config value from netmap contract", key)
}

View file

@ -7,7 +7,7 @@ import (
"os"
"sort"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/nspcc-dev/neo-go/pkg/crypto/hash"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
@ -41,16 +41,28 @@ func getContainerContractHash(cmd *cobra.Command, inv *invoker.Invoker, c Client
return ch, nil
}
func getContainersList(inv *invoker.Invoker, ch util.Uint160) ([][]byte, error) {
res, err := inv.Call(ch, "list", "")
func iterateContainerList(inv *invoker.Invoker, ch util.Uint160, f func([]byte) error) error {
sid, r, err := unwrap.SessionIterator(inv.Call(ch, "containersOf", ""))
if err != nil {
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
itm, err := unwrap.Item(res, err)
if _, ok := itm.(stackitem.Null); !ok {
return unwrap.ArrayOfBytes(res, err)
// Nothing bad, except live session on the server, do not report to the user.
defer func() { _ = inv.TerminateSession(sid) }()
items, err := inv.TraverseIterator(sid, &r, 0)
for err == nil && len(items) != 0 {
for j := range items {
b, err := items[j].TryBytes()
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
return nil, nil
if err := f(b); err != nil {
return err
}
}
items, err = inv.TraverseIterator(sid, &r, 0)
}
return err
}
func dumpContainers(cmd *cobra.Command, _ []string) error {
@ -71,56 +83,81 @@ func dumpContainers(cmd *cobra.Command, _ []string) error {
return fmt.Errorf("unable to get contaract hash: %w", err)
}
cids, err := getContainersList(inv, ch)
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
isOK, err := getCIDFilterFunc(cmd)
if err != nil {
return err
}
var containers []*Container
bw := io.NewBufBinWriter()
for _, id := range cids {
if !isOK(id) {
continue
f, err := os.OpenFile(filename, os.O_RDWR|os.O_TRUNC|os.O_CREATE, 0o660)
if err != nil {
return err
}
defer f.Close()
_, err = f.Write([]byte{'['})
if err != nil {
return err
}
written := 0
enc := json.NewEncoder(f)
bw := io.NewBufBinWriter()
iterErr := iterateContainerList(inv, ch, func(id []byte) error {
if !isOK(id) {
return nil
}
cnt, err := dumpSingleContainer(bw, ch, inv, id)
if err != nil {
return err
}
// Writing directly to the file is ok, because json.Encoder does no internal buffering.
if written != 0 {
_, err = f.Write([]byte{','})
if err != nil {
return err
}
}
written++
return enc.Encode(cnt)
})
if iterErr != nil {
return iterErr
}
_, err = f.Write([]byte{']'})
return err
}
func dumpSingleContainer(bw *io.BufBinWriter, ch util.Uint160, inv *invoker.Invoker, id []byte) (*Container, error) {
bw.Reset()
emit.AppCall(bw.BinWriter, ch, "get", callflag.All, id)
emit.AppCall(bw.BinWriter, ch, "eACL", callflag.All, id)
res, err := inv.Run(bw.Bytes())
if err != nil {
return fmt.Errorf("can't get container info: %w", err)
return nil, fmt.Errorf("can't get container info: %w", err)
}
if len(res.Stack) != 2 {
return fmt.Errorf("%w: expected 2 items on stack", errInvalidContainerResponse)
return nil, fmt.Errorf("%w: expected 2 items on stack", errInvalidContainerResponse)
}
cnt := new(Container)
err = cnt.FromStackItem(res.Stack[0])
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
ea := new(EACL)
err = ea.FromStackItem(res.Stack[1])
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if len(ea.Value) != 0 {
cnt.EACL = ea
}
containers = append(containers, cnt)
}
out, err := json.Marshal(containers)
if err != nil {
return err
}
return os.WriteFile(filename, out, 0o660)
return cnt, nil
}
func listContainers(cmd *cobra.Command, _ []string) error {
@ -136,20 +173,15 @@ func listContainers(cmd *cobra.Command, _ []string) error {
return fmt.Errorf("unable to get contaract hash: %w", err)
}
cids, err := getContainersList(inv, ch)
if err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
for _, id := range cids {
return iterateContainerList(inv, ch, func(id []byte) error {
var idCnr cid.ID
err = idCnr.Decode(id)
if err != nil {
return fmt.Errorf("unable to decode container id: %w", err)
}
cmd.Println(idCnr)
}
return nil
})
}
func restoreContainers(cmd *cobra.Command, _ []string) error {
@ -164,25 +196,14 @@ func restoreContainers(cmd *cobra.Command, _ []string) error {
}
defer wCtx.close()
nnsCs, err := wCtx.Client.GetContractStateByID(1)
containers, err := parseContainers(filename)
if err != nil {
return fmt.Errorf("can't get NNS contract state: %w", err)
return err
}
ch, err := nnsResolveHash(wCtx.ReadOnlyInvoker, nnsCs.Hash, containerContract+".frostfs")
ch, err := fetchContainerContractHash(wCtx)
if err != nil {
return fmt.Errorf("can't fetch container contract hash: %w", err)
}
data, err := os.ReadFile(filename)
if err != nil {
return fmt.Errorf("can't read dump file: %w", err)
}
var containers []Container
err = json.Unmarshal(data, &containers)
if err != nil {
return fmt.Errorf("can't parse dump file: %w", err)
return err
}
isOK, err := getCIDFilterFunc(cmd)
@ -190,6 +211,15 @@ func restoreContainers(cmd *cobra.Command, _ []string) error {
return err
}
err = restoreOrPutContainers(containers, isOK, cmd, wCtx, ch)
if err != nil {
return err
}
return wCtx.awaitTx()
}
func restoreOrPutContainers(containers []Container, isOK func([]byte) bool, cmd *cobra.Command, wCtx *initializeContext, ch util.Uint160) error {
bw := io.NewBufBinWriter()
for _, cnt := range containers {
hv := hash.Sha256(cnt.Value)
@ -197,33 +227,18 @@ func restoreContainers(cmd *cobra.Command, _ []string) error {
continue
}
bw.Reset()
emit.AppCall(bw.BinWriter, ch, "get", callflag.All, hv.BytesBE())
res, err := wCtx.Client.InvokeScript(bw.Bytes(), nil)
restored, err := isContainerRestored(cmd, wCtx, ch, bw, hv)
if err != nil {
return fmt.Errorf("can't check if container is already restored: %w", err)
return err
}
if len(res.Stack) == 0 {
return errors.New("empty stack")
}
old := new(Container)
if err := old.FromStackItem(res.Stack[0]); err != nil {
return fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if len(old.Value) != 0 {
var id cid.ID
id.SetSHA256(hv)
cmd.Printf("Container %s is already deployed.\n", id)
if restored {
continue
}
bw.Reset()
emit.AppCall(bw.BinWriter, ch, "put", callflag.All,
cnt.Value, cnt.Signature, cnt.PublicKey, cnt.Token)
if ea := cnt.EACL; ea != nil {
emit.AppCall(bw.BinWriter, ch, "setEACL", callflag.All,
ea.Value, ea.Signature, ea.PublicKey, ea.Token)
}
putContainer(bw, ch, cnt)
if bw.Err != nil {
panic(bw.Err)
}
@ -232,8 +247,67 @@ func restoreContainers(cmd *cobra.Command, _ []string) error {
return err
}
}
return nil
}
return wCtx.awaitTx()
func putContainer(bw *io.BufBinWriter, ch util.Uint160, cnt Container) {
emit.AppCall(bw.BinWriter, ch, "put", callflag.All,
cnt.Value, cnt.Signature, cnt.PublicKey, cnt.Token)
if ea := cnt.EACL; ea != nil {
emit.AppCall(bw.BinWriter, ch, "setEACL", callflag.All,
ea.Value, ea.Signature, ea.PublicKey, ea.Token)
}
}
func isContainerRestored(cmd *cobra.Command, wCtx *initializeContext, containerHash util.Uint160, bw *io.BufBinWriter, hashValue util.Uint256) (bool, error) {
emit.AppCall(bw.BinWriter, containerHash, "get", callflag.All, hashValue.BytesBE())
res, err := wCtx.Client.InvokeScript(bw.Bytes(), nil)
if err != nil {
return false, fmt.Errorf("can't check if container is already restored: %w", err)
}
if len(res.Stack) == 0 {
return false, errors.New("empty stack")
}
old := new(Container)
if err := old.FromStackItem(res.Stack[0]); err != nil {
return false, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if len(old.Value) != 0 {
var id cid.ID
id.SetSHA256(hashValue)
cmd.Printf("Container %s is already deployed.\n", id)
return true, nil
}
return false, nil
}
func parseContainers(filename string) ([]Container, error) {
data, err := os.ReadFile(filename)
if err != nil {
return nil, fmt.Errorf("can't read dump file: %w", err)
}
var containers []Container
err = json.Unmarshal(data, &containers)
if err != nil {
return nil, fmt.Errorf("can't parse dump file: %w", err)
}
return containers, nil
}
func fetchContainerContractHash(wCtx *initializeContext) (util.Uint160, error) {
nnsCs, err := wCtx.Client.GetContractStateByID(1)
if err != nil {
return util.Uint160{}, fmt.Errorf("can't get NNS contract state: %w", err)
}
ch, err := nnsResolveHash(wCtx.ReadOnlyInvoker, nnsCs.Hash, containerContract+".frostfs")
if err != nil {
return util.Uint160{}, fmt.Errorf("can't fetch container contract hash: %w", err)
}
return ch, nil
}
// Container represents container struct in contract storage.

View file

@ -6,7 +6,7 @@ import (
"os"
"strings"
"github.com/TrueCloudLab/frostfs-contract/nns"
"git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
"github.com/nspcc-dev/neo-go/cli/cmdargs"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
@ -100,17 +100,34 @@ func deployContractCmd(cmd *cobra.Command, args []string) error {
cs.Manifest.Name)
}
w := io.NewBufBinWriter()
if err := emitDeploymentArguments(w.BinWriter, args); err != nil {
writer := io.NewBufBinWriter()
if err := emitDeploymentArguments(writer.BinWriter, args); err != nil {
return err
}
emit.Bytes(w.BinWriter, cs.RawManifest)
emit.Bytes(w.BinWriter, cs.RawNEF)
emit.Int(w.BinWriter, 3)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, callHash, method, callflag.All)
emit.Opcodes(w.BinWriter, opcode.DROP) // contract state on stack
emit.Bytes(writer.BinWriter, cs.RawManifest)
emit.Bytes(writer.BinWriter, cs.RawNEF)
emit.Int(writer.BinWriter, 3)
emit.Opcodes(writer.BinWriter, opcode.PACK)
emit.AppCallNoArgs(writer.BinWriter, callHash, method, callflag.All)
emit.Opcodes(writer.BinWriter, opcode.DROP) // contract state on stack
if !isUpdate {
err := registerNNS(nnsCs, c, zone, domain, cs, writer)
if err != nil {
return err
}
}
if writer.Err != nil {
panic(fmt.Errorf("BUG: can't create deployment script: %w", writer.Err))
}
if err := c.sendCommitteeTx(writer.Bytes(), false); err != nil {
return err
}
return c.awaitTx()
}
func registerNNS(nnsCs *state.Contract, c *initializeContext, zone string, domain string, cs *contractState, writer *io.BufBinWriter) error {
bw := io.NewBufBinWriter()
emit.Instruction(bw.BinWriter, opcode.INITSSLOT, []byte{1})
emit.AppCall(bw.BinWriter, nnsCs.Hash, "getPrice", callflag.All)
@ -128,12 +145,12 @@ func deployContractCmd(cmd *cobra.Command, args []string) error {
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
zone, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
frostfsOpsEmail, int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
domain, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
frostfsOpsEmail, int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
} else {
s, ok, err := c.nnsRegisterDomainScript(nnsCs.Hash, cs.Hash, domain)
@ -152,26 +169,17 @@ func deployContractCmd(cmd *cobra.Command, args []string) error {
}
if bw.Err != nil {
panic(fmt.Errorf("BUG: can't create deployment script: %w", w.Err))
panic(fmt.Errorf("BUG: can't create deployment script: %w", writer.Err))
} else if bw.Len() != start {
w.WriteBytes(bw.Bytes())
emit.Opcodes(w.BinWriter, opcode.LDSFLD0, opcode.PUSH1, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsCs.Hash, "setPrice", callflag.All)
writer.WriteBytes(bw.Bytes())
emit.Opcodes(writer.BinWriter, opcode.LDSFLD0, opcode.PUSH1, opcode.PACK)
emit.AppCallNoArgs(writer.BinWriter, nnsCs.Hash, "setPrice", callflag.All)
if needRecord {
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
}
}
}
if w.Err != nil {
panic(fmt.Errorf("BUG: can't create deployment script: %w", w.Err))
}
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
return err
}
return c.awaitTx()
return nil
}
func emitDeploymentArguments(w *io.BinWriter, args []string) error {

View file

@ -1,40 +0,0 @@
package morph
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"strings"
"github.com/google/go-github/v39/github"
"github.com/spf13/cobra"
)
func downloadContractsFromGithub(cmd *cobra.Command) (io.ReadCloser, error) {
gcl := github.NewClient(nil)
release, _, err := gcl.Repositories.GetLatestRelease(context.Background(), "nspcc-dev", "frostfs-contract")
if err != nil {
return nil, fmt.Errorf("can't fetch release info: %w", err)
}
cmd.Printf("Found %s (%s), downloading...\n", release.GetTagName(), release.GetName())
var url string
for _, a := range release.Assets {
if strings.HasPrefix(a.GetName(), "frostfs-contract") {
url = a.GetBrowserDownloadURL()
break
}
}
if url == "" {
return nil, errors.New("can't find contracts archive in release assets")
}
resp, err := http.Get(url)
if err != nil {
return nil, fmt.Errorf("can't fetch contracts archive: %w", err)
}
return resp.Body, nil
}

View file

@ -7,7 +7,8 @@ import (
"strings"
"text/tabwriter"
"github.com/TrueCloudLab/frostfs-contract/nns"
"git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
morphClient "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
@ -123,7 +124,7 @@ func dumpCustomZoneHashes(cmd *cobra.Command, nnsHash util.Uint160, zone string,
return
}
if !bytes.HasSuffix(bs, []byte(zone)) {
if !bytes.HasSuffix(bs, []byte(zone)) || bytes.HasPrefix(bs, []byte(morphClient.NNSGroupKeyName)) {
// Related https://github.com/nspcc-dev/neofs-contract/issues/316.
return
}

View file

@ -3,6 +3,7 @@ package morph
import (
"errors"
"fmt"
"strings"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
@ -13,7 +14,7 @@ import (
"github.com/spf13/viper"
)
func forceNewEpochCmd(cmd *cobra.Command, args []string) error {
func forceNewEpochCmd(cmd *cobra.Command, _ []string) error {
wCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return fmt.Errorf("can't to initialize context: %w", err)
@ -38,7 +39,14 @@ func forceNewEpochCmd(cmd *cobra.Command, args []string) error {
return err
}
return wCtx.awaitTx()
if err := wCtx.awaitTx(); err != nil {
if strings.Contains(err.Error(), "invalid epoch") {
cmd.Println("Epoch has already ticked.")
return nil
}
return err
}
return nil
}
func emitNewEpochCall(bw *io.BufBinWriter, wCtx *initializeContext, nmHash util.Uint160) error {

View file

@ -6,8 +6,8 @@ import (
"os"
"path/filepath"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
@ -21,6 +21,7 @@ import (
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"golang.org/x/sync/errgroup"
)
const (
@ -29,7 +30,7 @@ const (
consensusAccountName = "consensus"
)
func generateAlphabetCreds(cmd *cobra.Command, args []string) error {
func generateAlphabetCreds(cmd *cobra.Command, _ []string) error {
// alphabet size is not part of the config
size, err := cmd.Flags().GetUint(alphabetSizeFlag)
if err != nil {
@ -38,6 +39,9 @@ func generateAlphabetCreds(cmd *cobra.Command, args []string) error {
if size == 0 {
return errors.New("size must be > 0")
}
if size > maxAlphabetNodes {
return ErrTooManyAlphabetNodes
}
v := viper.GetViper()
walletDir := config.ResolveHomePath(viper.GetString(alphabetWalletsFlag))
@ -92,28 +96,31 @@ func initializeWallets(v *viper.Viper, walletDir string, size int) ([]string, er
pubs[i] = w.Accounts[0].PrivateKey().PublicKey()
}
var errG errgroup.Group
// Create committee account with N/2+1 multi-signature.
majCount := smartcontract.GetMajorityHonestNodeCount(size)
for i, w := range wallets {
if err := addMultisigAccount(w, majCount, committeeAccountName, passwords[i], pubs); err != nil {
return nil, fmt.Errorf("can't create committee account: %w", err)
}
}
// Create consensus account with 2*N/3+1 multi-signature.
bftCount := smartcontract.GetDefaultHonestNodeCount(size)
for i, w := range wallets {
if err := addMultisigAccount(w, bftCount, consensusAccountName, passwords[i], pubs); err != nil {
return nil, fmt.Errorf("can't create consensus account: %w", err)
for i := range wallets {
i := i
ps := pubs.Copy()
errG.Go(func() error {
if err := addMultisigAccount(wallets[i], majCount, committeeAccountName, passwords[i], ps); err != nil {
return fmt.Errorf("can't create committee account: %w", err)
}
if err := addMultisigAccount(wallets[i], bftCount, consensusAccountName, passwords[i], ps); err != nil {
return fmt.Errorf("can't create consentus account: %w", err)
}
for _, w := range wallets {
if err := w.SavePretty(); err != nil {
return nil, fmt.Errorf("can't save wallet: %w", err)
if err := wallets[i].SavePretty(); err != nil {
return fmt.Errorf("can't save wallet: %w", err)
}
return nil
})
}
if err := errG.Wait(); err != nil {
return nil, err
}
return passwords, nil
}

View file

@ -7,9 +7,10 @@ import (
"os"
"path/filepath"
"strconv"
"sync"
"testing"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
@ -71,11 +72,17 @@ func TestGenerateAlphabet(t *testing.T) {
buf.WriteString(testContractPassword + "\r")
require.NoError(t, generateAlphabetCreds(generateAlphabetCmd, nil))
var wg sync.WaitGroup
for i := uint64(0); i < size; i++ {
i := i
wg.Add(1)
go func() {
defer wg.Done()
p := filepath.Join(walletDir, innerring.GlagoliticLetter(i).String()+".json")
w, err := wallet.NewWalletFromFile(p)
require.NoError(t, err, "wallet doesn't exist")
require.Equal(t, 3, len(w.Accounts), "not all accounts were created")
for _, a := range w.Accounts {
err := a.Decrypt(strconv.FormatUint(i, 10), keys.NEP2ScryptParams())
require.NoError(t, err, "can't decrypt account")
@ -88,7 +95,9 @@ func TestGenerateAlphabet(t *testing.T) {
require.Equal(t, singleAccountName, a.Label)
}
}
}()
}
wg.Wait()
t.Run("check contract group wallet", func(t *testing.T) {
p := filepath.Join(walletDir, contractWalletFilename)

View file

@ -6,7 +6,7 @@ import (
"os"
"path/filepath"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/manifest"
"github.com/nspcc-dev/neo-go/pkg/util"

View file

@ -7,9 +7,9 @@ import (
"path/filepath"
"time"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
morphClient "github.com/TrueCloudLab/frostfs-node/pkg/morph/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
morphClient "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/core/native/nativenames"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
@ -23,6 +23,13 @@ import (
"github.com/spf13/viper"
)
const (
// maxAlphabetNodes is the maximum number of candidates allowed, which is currently limited by the size
// of the invocation script.
// See: https://github.com/nspcc-dev/neo-go/blob/740488f7f35e367eaa99a71c0a609c315fe2b0fc/pkg/core/transaction/witness.go#L10
maxAlphabetNodes = 22
)
type cache struct {
nnsCs *state.Contract
groupKey *keys.PublicKey
@ -45,7 +52,9 @@ type initializeContext struct {
ContractPath string
}
func initializeSideChainCmd(cmd *cobra.Command, args []string) error {
var ErrTooManyAlphabetNodes = fmt.Errorf("too many alphabet nodes (maximum allowed is %d)", maxAlphabetNodes)
func initializeSideChainCmd(cmd *cobra.Command, _ []string) error {
initCtx, err := newInitializeContext(cmd, viper.GetViper())
if err != nil {
return fmt.Errorf("initialization error: %w", err)
@ -91,11 +100,7 @@ func initializeSideChainCmd(cmd *cobra.Command, args []string) error {
}
cmd.Println("Stage 7: set addresses in NNS.")
if err := initCtx.setNNS(); err != nil {
return err
}
return nil
return initCtx.setNNS()
}
func (c *initializeContext) close() {
@ -115,25 +120,21 @@ func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContex
return nil, err
}
if len(wallets) > maxAlphabetNodes {
return nil, ErrTooManyAlphabetNodes
}
needContracts := cmd.Name() == "update-contracts" || cmd.Name() == "init"
var w *wallet.Wallet
if cmd.Name() != "deploy" {
w, err = openContractWallet(v, cmd, walletDir)
w, err = getWallet(cmd, v, needContracts, walletDir)
if err != nil {
return nil, err
}
}
var c Client
if v.GetString(localDumpFlag) != "" {
if v.GetString(endpointFlag) != "" {
return nil, fmt.Errorf("`%s` and `%s` flags are mutually exclusive", endpointFlag, localDumpFlag)
}
c, err = newLocalClient(cmd, v, wallets)
} else {
c, err = getN3Client(v)
}
c, err := createClient(cmd, v, wallets)
if err != nil {
return nil, fmt.Errorf("can't create N3 client: %w", err)
return nil, err
}
committeeAcc, err := getWalletAccount(wallets[0], committeeAccountName)
@ -146,36 +147,22 @@ func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContex
return nil, fmt.Errorf("can't find consensus account: %w", err)
}
var ctrPath string
if cmd.Name() == "init" {
if viper.GetInt64(epochDurationInitFlag) <= 0 {
return nil, fmt.Errorf("epoch duration must be positive")
if err := validateInit(cmd); err != nil {
return nil, err
}
if viper.GetInt64(maxObjectSizeInitFlag) <= 0 {
return nil, fmt.Errorf("max object size must be positive")
}
}
needContracts := cmd.Name() == "update-contracts" || cmd.Name() == "init"
if needContracts {
ctrPath, err = cmd.Flags().GetString(contractsInitFlag)
ctrPath, err := getContractsPath(cmd, needContracts)
if err != nil {
return nil, fmt.Errorf("invalid contracts path: %w", err)
}
return nil, err
}
if err := checkNotaryEnabled(c); err != nil {
return nil, err
}
accounts := make([]*wallet.Account, len(wallets))
for i, w := range wallets {
acc, err := getWalletAccount(w, singleAccountName)
accounts, err := createWalletAccounts(wallets)
if err != nil {
return nil, fmt.Errorf("wallet %s is invalid (no single account): %w", w.Path(), err)
}
accounts[i] = acc
return nil, err
}
cliCtx, err := defaultClientContext(c, committeeAcc)
@ -205,6 +192,69 @@ func newInitializeContext(cmd *cobra.Command, v *viper.Viper) (*initializeContex
return initCtx, nil
}
func validateInit(cmd *cobra.Command) error {
if cmd.Name() != "init" {
return nil
}
if viper.GetInt64(epochDurationInitFlag) <= 0 {
return fmt.Errorf("epoch duration must be positive")
}
if viper.GetInt64(maxObjectSizeInitFlag) <= 0 {
return fmt.Errorf("max object size must be positive")
}
return nil
}
func createClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet) (Client, error) {
var c Client
var err error
if ldf := cmd.Flags().Lookup(localDumpFlag); ldf != nil && ldf.Changed {
if cmd.Flags().Changed(endpointFlag) {
return nil, fmt.Errorf("`%s` and `%s` flags are mutually exclusive", endpointFlag, localDumpFlag)
}
c, err = newLocalClient(cmd, v, wallets, ldf.Value.String())
} else {
c, err = getN3Client(v)
}
if err != nil {
return nil, fmt.Errorf("can't create N3 client: %w", err)
}
return c, nil
}
func getWallet(cmd *cobra.Command, v *viper.Viper, needContracts bool, walletDir string) (*wallet.Wallet, error) {
if !needContracts {
return nil, nil
}
return openContractWallet(v, cmd, walletDir)
}
func getContractsPath(cmd *cobra.Command, needContracts bool) (string, error) {
if !needContracts {
return "", nil
}
ctrPath, err := cmd.Flags().GetString(contractsInitFlag)
if err != nil {
return "", fmt.Errorf("invalid contracts path: %w", err)
}
return ctrPath, nil
}
func createWalletAccounts(wallets []*wallet.Wallet) ([]*wallet.Account, error) {
accounts := make([]*wallet.Account, len(wallets))
for i, w := range wallets {
acc, err := getWalletAccount(w, singleAccountName)
if err != nil {
return nil, fmt.Errorf("wallet %s is invalid (no single account): %w", w.Path(), err)
}
accounts[i] = acc
}
return accounts, nil
}
func openAlphabetWallets(v *viper.Viper, walletDir string) ([]*wallet.Wallet, error) {
walletFiles, err := os.ReadDir(walletDir)
if err != nil {

View file

@ -12,10 +12,11 @@ import (
"path/filepath"
"strings"
"github.com/TrueCloudLab/frostfs-contract/common"
"github.com/TrueCloudLab/frostfs-contract/nns"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
morphClient "github.com/TrueCloudLab/frostfs-node/pkg/morph/client"
"git.frostfs.info/TrueCloudLab/frostfs-contract/common"
"git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
morphClient "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
@ -23,6 +24,7 @@ import (
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/management"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/manifest"
@ -30,8 +32,8 @@ import (
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/vm/emit"
"github.com/nspcc-dev/neo-go/pkg/vm/opcode"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/nspcc-dev/neo-go/pkg/vm/vmstate"
"github.com/spf13/viper"
)
const (
@ -39,44 +41,20 @@ const (
frostfsContract = "frostfs" // not deployed in side-chain.
processingContract = "processing" // not deployed in side-chain.
alphabetContract = "alphabet"
auditContract = "audit"
balanceContract = "balance"
containerContract = "container"
frostfsIDContract = "frostfsid"
netmapContract = "netmap"
proxyContract = "proxy"
reputationContract = "reputation"
subnetContract = "subnet"
)
const (
netmapEpochKey = "EpochDuration"
netmapMaxObjectSizeKey = "MaxObjectSize"
netmapAuditFeeKey = "AuditFee"
netmapContainerFeeKey = "ContainerFee"
netmapContainerAliasFeeKey = "ContainerAliasFee"
netmapEigenTrustIterationsKey = "EigenTrustIterations"
netmapEigenTrustAlphaKey = "EigenTrustAlpha"
netmapBasicIncomeRateKey = "BasicIncomeRate"
netmapInnerRingCandidateFeeKey = "InnerRingCandidateFee"
netmapWithdrawFeeKey = "WithdrawFee"
netmapHomomorphicHashDisabledKey = "HomomorphicHashingDisabled"
netmapMaintenanceAllowedKey = "MaintenanceModeAllowed"
defaultEigenTrustIterations = 4
defaultEigenTrustAlpha = "0.1"
)
var (
contractList = []string{
auditContract,
balanceContract,
containerContract,
frostfsIDContract,
netmapContract,
proxyContract,
reputationContract,
subnetContract,
}
fullContractList = append([]string{
@ -85,6 +63,17 @@ var (
nnsContract,
alphabetContract,
}, contractList...)
netmapConfigKeys = []string{
netmap.EpochDurationConfig,
netmap.MaxObjectSizeConfig,
netmap.ContainerFeeConfig,
netmap.ContainerAliasFeeConfig,
netmap.IrCandidateFeeConfig,
netmap.WithdrawFeeConfig,
netmap.HomomorphicHashingDisabledKey,
netmap.MaintenanceModeAllowedConfig,
}
)
type contractState struct {
@ -167,8 +156,6 @@ func (c *initializeContext) updateContracts() error {
w := io2.NewBufBinWriter()
var keysParam []interface{}
// Update script size for a single-node committee is close to the maximum allowed size of 65535.
// Because of this we want to reuse alphabet contract NEF and manifest for different updates.
// The generated script is as following.
@ -182,42 +169,36 @@ func (c *initializeContext) updateContracts() error {
emit.Bytes(w.BinWriter, alphaCs.RawNEF)
emit.Opcodes(w.BinWriter, opcode.STSFLD0)
baseGroups := alphaCs.Manifest.Groups
// alphabet contracts should be deployed by individual nodes to get different hashes.
for i, acc := range c.Accounts {
ctrHash, err := nnsResolveHash(c.ReadOnlyInvoker, nnsHash, getAlphabetNNSDomain(i))
keysParam, err := c.deployAlphabetAccounts(nnsHash, w, alphaCs)
if err != nil {
return fmt.Errorf("can't resolve hash for contract update: %w", err)
}
keysParam = append(keysParam, acc.PrivateKey().PublicKey().Bytes())
params := c.getAlphabetDeployItems(i, len(c.Wallets))
emit.Array(w.BinWriter, params...)
alphaCs.Manifest.Groups = baseGroups
err = c.addManifestGroup(ctrHash, alphaCs)
if err != nil {
return fmt.Errorf("can't sign manifest group: %v", err)
}
emit.Bytes(w.BinWriter, alphaCs.RawManifest)
emit.Opcodes(w.BinWriter, opcode.LDSFLD0)
emit.Int(w.BinWriter, 3)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, ctrHash, updateMethodName, callflag.All)
}
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
if !strings.Contains(err.Error(), common.ErrAlreadyUpdated) {
return err
}
c.Command.Println("Alphabet contracts are already updated.")
}
w.Reset()
if err = c.deployOrUpdateContracts(w, nnsHash, keysParam); err != nil {
return err
}
groupKey := c.ContractWallet.Accounts[0].PrivateKey().PublicKey()
_, _, err = c.emitUpdateNNSGroupScript(w, nnsHash, groupKey)
if err != nil {
return err
}
c.Command.Printf("NNS: Set %s -> %s\n", morphClient.NNSGroupKeyName, hex.EncodeToString(groupKey.Bytes()))
emit.Opcodes(w.BinWriter, opcode.LDSFLD0)
emit.Int(w.BinWriter, 1)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsHash, "setPrice", callflag.All)
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
return err
}
return c.awaitTx()
}
func (c *initializeContext) deployOrUpdateContracts(w *io2.BufBinWriter, nnsHash util.Uint160, keysParam []any) error {
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
emit.AppCall(w.BinWriter, nnsHash, "getPrice", callflag.All)
emit.Opcodes(w.BinWriter, opcode.STSFLD0)
@ -247,7 +228,7 @@ func (c *initializeContext) updateContracts() error {
invokeHash = ctrHash
}
params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam))
params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam, updateMethodName))
res, err := c.CommitteeAct.MakeCall(invokeHash, method, params...)
if err != nil {
if method != updateMethodName || !strings.Contains(err.Error(), common.ErrAlreadyUpdated) {
@ -277,29 +258,52 @@ func (c *initializeContext) updateContracts() error {
c.Command.Printf("NNS: Set %s -> %s\n", domain, cs.Hash.StringLE())
}
}
return nil
}
groupKey := c.ContractWallet.Accounts[0].PrivateKey().PublicKey()
_, _, err = c.emitUpdateNNSGroupScript(w, nnsHash, groupKey)
func (c *initializeContext) deployAlphabetAccounts(nnsHash util.Uint160, w *io2.BufBinWriter, alphaCs *contractState) ([]any, error) {
var keysParam []any
baseGroups := alphaCs.Manifest.Groups
// alphabet contracts should be deployed by individual nodes to get different hashes.
for i, acc := range c.Accounts {
ctrHash, err := nnsResolveHash(c.ReadOnlyInvoker, nnsHash, getAlphabetNNSDomain(i))
if err != nil {
return err
return nil, fmt.Errorf("can't resolve hash for contract update: %w", err)
}
c.Command.Printf("NNS: Set %s -> %s\n", morphClient.NNSGroupKeyName, hex.EncodeToString(groupKey.Bytes()))
keysParam = append(keysParam, acc.PrivateKey().PublicKey().Bytes())
params := c.getAlphabetDeployItems(i, len(c.Wallets))
emit.Array(w.BinWriter, params...)
alphaCs.Manifest.Groups = baseGroups
err = c.addManifestGroup(ctrHash, alphaCs)
if err != nil {
return nil, fmt.Errorf("can't sign manifest group: %v", err)
}
emit.Bytes(w.BinWriter, alphaCs.RawManifest)
emit.Opcodes(w.BinWriter, opcode.LDSFLD0)
emit.Int(w.BinWriter, 1)
emit.Int(w.BinWriter, 3)
emit.Opcodes(w.BinWriter, opcode.PACK)
emit.AppCallNoArgs(w.BinWriter, nnsHash, "setPrice", callflag.All)
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
return err
emit.AppCallNoArgs(w.BinWriter, ctrHash, updateMethodName, callflag.All)
}
return c.awaitTx()
if err := c.sendCommitteeTx(w.Bytes(), false); err != nil {
if !strings.Contains(err.Error(), common.ErrAlreadyUpdated) {
return nil, err
}
c.Command.Println("Alphabet contracts are already updated.")
}
return keysParam, nil
}
func (c *initializeContext) deployContracts() error {
alphaCs := c.getContract(alphabetContract)
var keysParam []interface{}
var keysParam []any
baseGroups := alphaCs.Manifest.Groups
@ -347,7 +351,7 @@ func (c *initializeContext) deployContracts() error {
return fmt.Errorf("can't sign manifest group: %v", err)
}
params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam))
params := getContractDeployParameters(cs, c.getContractDeployData(ctrName, keysParam, deployMethodName))
res, err := c.CommitteeAct.MakeCall(management.Hash, deployMethodName, params...)
if err != nil {
return fmt.Errorf("can't deploy %s contract: %w", ctrName, err)
@ -393,11 +397,9 @@ func (c *initializeContext) readContracts(names []string) error {
} else {
var r io.ReadCloser
if c.ContractPath == "" {
c.Command.Println("Contracts flag is missing, latest release will be fetched from Github.")
r, err = downloadContractsFromGithub(c.Command)
} else {
r, err = os.Open(c.ContractPath)
return errors.New("contracts flag is missing")
}
r, err = os.Open(c.ContractPath)
if err != nil {
return fmt.Errorf("can't open contracts archive: %w", err)
}
@ -510,12 +512,12 @@ func readContractsFromArchive(file io.Reader, names []string) (map[string]*contr
return m, nil
}
func getContractDeployParameters(cs *contractState, deployData []interface{}) []interface{} {
return []interface{}{cs.RawNEF, cs.RawManifest, deployData}
func getContractDeployParameters(cs *contractState, deployData []any) []any {
return []any{cs.RawNEF, cs.RawManifest, deployData}
}
func (c *initializeContext) getContractDeployData(ctrName string, keysParam []interface{}) []interface{} {
items := make([]interface{}, 1, 6)
func (c *initializeContext) getContractDeployData(ctrName string, keysParam []any, method string) []any {
items := make([]any, 1, 6)
items[0] = false // notaryDisabled is false
switch ctrName {
@ -527,8 +529,6 @@ func (c *initializeContext) getContractDeployData(ctrName string, keysParam []in
case processingContract:
items = append(items, c.Contracts[frostfsContract].Hash)
return items[1:] // no notary info
case auditContract:
items = append(items, c.Contracts[netmapContract].Hash)
case balanceContract:
items = append(items,
c.Contracts[netmapContract].Hash,
@ -551,20 +551,31 @@ func (c *initializeContext) getContractDeployData(ctrName string, keysParam []in
c.Contracts[netmapContract].Hash,
c.Contracts[containerContract].Hash)
case netmapContract:
configParam := []interface{}{
netmapEpochKey, viper.GetInt64(epochDurationInitFlag),
netmapMaxObjectSizeKey, viper.GetInt64(maxObjectSizeInitFlag),
netmapAuditFeeKey, viper.GetInt64(auditFeeInitFlag),
netmapContainerFeeKey, viper.GetInt64(containerFeeInitFlag),
netmapContainerAliasFeeKey, viper.GetInt64(containerAliasFeeInitFlag),
netmapEigenTrustIterationsKey, int64(defaultEigenTrustIterations),
netmapEigenTrustAlphaKey, defaultEigenTrustAlpha,
netmapBasicIncomeRateKey, viper.GetInt64(incomeRateInitFlag),
netmapInnerRingCandidateFeeKey, viper.GetInt64(candidateFeeInitFlag),
netmapWithdrawFeeKey, viper.GetInt64(withdrawFeeInitFlag),
netmapHomomorphicHashDisabledKey, viper.GetBool(homomorphicHashDisabledInitFlag),
netmapMaintenanceAllowedKey, viper.GetBool(maintenanceModeAllowedInitFlag),
md := getDefaultNetmapContractConfigMap()
if method == updateMethodName {
arr, err := c.getNetConfigFromNetmapContract()
if err != nil {
panic(err)
}
m, err := parseConfigFromNetmapContract(arr)
if err != nil {
panic(err)
}
for k, v := range m {
for _, key := range netmapConfigKeys {
if k == key {
md[k] = v
break
}
}
}
}
var configParam []any
for k, v := range md {
configParam = append(configParam, k, v)
}
items = append(items,
c.Contracts[balanceContract].Hash,
c.Contracts[containerContract].Hash,
@ -572,16 +583,30 @@ func (c *initializeContext) getContractDeployData(ctrName string, keysParam []in
configParam)
case proxyContract:
items = nil
case reputationContract:
case subnetContract:
default:
panic(fmt.Sprintf("invalid contract name: %s", ctrName))
}
return items
}
func (c *initializeContext) getAlphabetDeployItems(i, n int) []interface{} {
items := make([]interface{}, 6)
func (c *initializeContext) getNetConfigFromNetmapContract() ([]stackitem.Item, error) {
cs, err := c.Client.GetContractStateByID(1)
if err != nil {
return nil, fmt.Errorf("NNS is not yet deployed: %w", err)
}
nmHash, err := nnsResolveHash(c.ReadOnlyInvoker, cs.Hash, netmapContract+".frostfs")
if err != nil {
return nil, fmt.Errorf("can't get netmap contract hash: %w", err)
}
arr, err := unwrap.Array(c.ReadOnlyInvoker.Call(nmHash, "listConfig"))
if err != nil {
return nil, fmt.Errorf("can't fetch list of network config keys from the netmap contract")
}
return arr, err
}
func (c *initializeContext) getAlphabetDeployItems(i, n int) []any {
items := make([]any, 6)
items[0] = false
items[1] = c.Contracts[netmapContract].Hash
items[2] = c.Contracts[proxyContract].Hash

View file

@ -7,14 +7,15 @@ import (
"strconv"
"time"
"github.com/TrueCloudLab/frostfs-contract/nns"
morphClient "github.com/TrueCloudLab/frostfs-node/pkg/morph/client"
"git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
morphClient "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
nnsClient "github.com/nspcc-dev/neo-go/pkg/rpcclient/nns"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
"github.com/nspcc-dev/neo-go/pkg/util"
@ -26,6 +27,8 @@ import (
const defaultExpirationTime = 10 * 365 * 24 * time.Hour / time.Second
const frostfsOpsEmail = "ops@frostfs.info"
func (c *initializeContext) setNNS() error {
nnsCs, err := c.Client.GetContractStateByID(1)
if err != nil {
@ -39,7 +42,7 @@ func (c *initializeContext) setNNS() error {
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, nnsCs.Hash, "register", callflag.All,
"frostfs", c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
frostfsOpsEmail, int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if err := c.sendCommitteeTx(bw.Bytes(), true); err != nil {
return fmt.Errorf("can't add domain root to NNS: %w", err)
@ -82,13 +85,13 @@ func (c *initializeContext) setNNS() error {
func (c *initializeContext) updateNNSGroup(nnsHash util.Uint160, pub *keys.PublicKey) error {
bw := io.NewBufBinWriter()
needUpdate, needRegister, err := c.emitUpdateNNSGroupScript(bw, nnsHash, pub)
if !needUpdate || err != nil {
keyAlreadyAdded, domainRegCodeEmitted, err := c.emitUpdateNNSGroupScript(bw, nnsHash, pub)
if keyAlreadyAdded || err != nil {
return err
}
script := bw.Bytes()
if needRegister {
if domainRegCodeEmitted {
w := io.NewBufBinWriter()
emit.Instruction(w.BinWriter, opcode.INITSSLOT, []byte{1})
wrapRegisterScriptWithPrice(w, nnsHash, script)
@ -121,7 +124,7 @@ func (c *initializeContext) emitUpdateNNSGroupScript(bw *io.BufBinWriter, nnsHas
if isAvail {
emit.AppCall(bw.BinWriter, nnsHash, "register", callflag.All,
morphClient.NNSGroupKeyName, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
frostfsOpsEmail, int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
}
@ -169,7 +172,7 @@ func (c *initializeContext) nnsRegisterDomainScript(nnsHash, expectedHash util.U
bw := io.NewBufBinWriter()
emit.AppCall(bw.BinWriter, nnsHash, "register", callflag.All,
domain, c.CommitteeAcc.Contract.ScriptHash(),
"ops@nspcc.ru", int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
frostfsOpsEmail, int64(3600), int64(600), int64(defaultExpirationTime), int64(3600))
emit.Opcodes(bw.BinWriter, opcode.ASSERT)
if bw.Err != nil {
@ -228,20 +231,27 @@ func nnsResolve(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (stac
}
func nnsResolveKey(inv *invoker.Invoker, nnsHash util.Uint160, domain string) (*keys.PublicKey, error) {
item, err := nnsResolve(inv, nnsHash, domain)
res, err := nnsResolve(inv, nnsHash, domain)
if err != nil {
return nil, err
}
v, ok := item.Value().(stackitem.Null)
if ok {
if _, ok := res.Value().(stackitem.Null); ok {
return nil, errors.New("NNS record is missing")
}
bs, err := v.TryBytes()
arr, ok := res.Value().([]stackitem.Item)
if !ok {
return nil, errors.New("API of the NNS contract method `resolve` has changed")
}
for i := range arr {
var bs []byte
bs, err = arr[i].TryBytes()
if err != nil {
return nil, errors.New("malformed response")
continue
}
return keys.NewPublicKeyFromString(string(bs))
}
return nil, errors.New("no valid keys are found")
}
// parseNNSResolveResult parses the result of resolving NNS record.
@ -277,11 +287,13 @@ func parseNNSResolveResult(res stackitem.Item) (util.Uint160, error) {
}
func nnsIsAvailable(c Client, nnsHash util.Uint160, name string) (bool, error) {
switch ct := c.(type) {
switch c.(type) {
case *rpcclient.Client:
return ct.NNSIsAvailable(nnsHash, name)
inv := invoker.New(c, nil)
reader := nnsClient.NewReader(inv, nnsHash)
return reader.IsAvailable(name)
default:
b, err := unwrap.Bool(invokeFunction(c, nnsHash, "isAvailable", []interface{}{name}, nil))
b, err := unwrap.Bool(invokeFunction(c, nnsHash, "isAvailable", []any{name}, nil))
if err != nil {
return false, fmt.Errorf("`isAvailable`: invalid response: %w", err)
}

View file

@ -9,6 +9,7 @@ import (
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/neo"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"
@ -18,33 +19,24 @@ import (
)
// initialAlphabetNEOAmount represents the total amount of GAS distributed between alphabet nodes.
const initialAlphabetNEOAmount = native.NEOTotalSupply
func (c *initializeContext) registerCandidates() error {
neoHash := neo.Hash
cc, err := unwrap.Array(c.ReadOnlyInvoker.Call(neoHash, "getCandidates"))
if err != nil {
return fmt.Errorf("`getCandidates`: %w", err)
}
if len(cc) > 0 {
c.Command.Println("Candidates are already registered.")
return nil
}
const (
initialAlphabetNEOAmount = native.NEOTotalSupply
registerBatchSize = transaction.MaxAttributes - 1
)
func (c *initializeContext) registerCandidateRange(start, end int) error {
regPrice, err := c.getCandidateRegisterPrice()
if err != nil {
return fmt.Errorf("can't fetch registration price: %w", err)
}
w := io.NewBufBinWriter()
emit.AppCall(w.BinWriter, neoHash, "setRegisterPrice", callflag.States, 1)
for _, acc := range c.Accounts {
emit.AppCall(w.BinWriter, neoHash, "registerCandidate", callflag.States, acc.PrivateKey().PublicKey().Bytes())
emit.AppCall(w.BinWriter, neo.Hash, "setRegisterPrice", callflag.States, 1)
for _, acc := range c.Accounts[start:end] {
emit.AppCall(w.BinWriter, neo.Hash, "registerCandidate", callflag.States, acc.PrivateKey().PublicKey().Bytes())
emit.Opcodes(w.BinWriter, opcode.ASSERT)
}
emit.AppCall(w.BinWriter, neoHash, "setRegisterPrice", callflag.States, regPrice)
emit.AppCall(w.BinWriter, neo.Hash, "setRegisterPrice", callflag.States, regPrice)
if w.Err != nil {
panic(fmt.Sprintf("BUG: %v", w.Err))
}
@ -53,14 +45,14 @@ func (c *initializeContext) registerCandidates() error {
Signer: c.getSigner(false, c.CommitteeAcc),
Account: c.CommitteeAcc,
}}
for i := range c.Accounts {
for _, acc := range c.Accounts[start:end] {
signers = append(signers, rpcclient.SignerAccount{
Signer: transaction.Signer{
Account: c.Accounts[i].Contract.ScriptHash(),
Account: acc.Contract.ScriptHash(),
Scopes: transaction.CustomContracts,
AllowedContracts: []util.Uint160{neoHash},
AllowedContracts: []util.Uint160{neo.Hash},
},
Account: c.Accounts[i],
Account: acc,
})
}
@ -73,8 +65,8 @@ func (c *initializeContext) registerCandidates() error {
}
network := c.CommitteeAct.GetNetwork()
for i := range c.Accounts {
if err := c.Accounts[i].SignTx(network, tx); err != nil {
for _, acc := range c.Accounts[start:end] {
if err := acc.SignTx(network, tx); err != nil {
return fmt.Errorf("can't sign a transaction: %w", err)
}
}
@ -82,6 +74,39 @@ func (c *initializeContext) registerCandidates() error {
return c.sendTx(tx, c.Command, true)
}
func (c *initializeContext) registerCandidates() error {
cc, err := unwrap.Array(c.ReadOnlyInvoker.Call(neo.Hash, "getCandidates"))
if err != nil {
return fmt.Errorf("`getCandidates`: %w", err)
}
need := len(c.Accounts)
have := len(cc)
if need == have {
c.Command.Println("Candidates are already registered.")
return nil
}
// Register candidates in batches in order to overcome the signers amount limit.
// See: https://github.com/nspcc-dev/neo-go/blob/master/pkg/core/transaction/transaction.go#L27
for i := 0; i < need; i += registerBatchSize {
start, end := i, i+registerBatchSize
if end > need {
end = need
}
// This check is sound because transactions are accepted/rejected atomically.
if have >= end {
continue
}
if err := c.registerCandidateRange(start, end); err != nil {
return fmt.Errorf("registering candidates %d..%d: %q", start, end-1, err)
}
}
return nil
}
func (c *initializeContext) transferNEOToAlphabetContracts() error {
neoHash := neo.Hash
@ -116,9 +141,11 @@ func (c *initializeContext) transferNEOFinished(neoHash util.Uint160) (bool, err
var errGetPriceInvalid = errors.New("`getRegisterPrice`: invalid response")
func (c *initializeContext) getCandidateRegisterPrice() (int64, error) {
switch ct := c.Client.(type) {
switch c.Client.(type) {
case *rpcclient.Client:
return ct.GetCandidateRegisterPrice()
inv := invoker.New(c.Client, nil)
reader := neo.NewReader(inv)
return reader.GetRegisterPrice()
default:
neoHash := neo.Hash
res, err := invokeFunction(c.Client, neoHash, "getRegisterPrice", nil, nil)

View file

@ -16,7 +16,7 @@ func (c *initializeContext) setNotaryAndAlphabetNodes() error {
return err
}
var pubs []interface{}
var pubs []any
for _, acc := range c.Accounts {
pubs = append(pubs, acc.PrivateKey().PublicKey().Bytes())
}

View file

@ -2,12 +2,14 @@ package morph
import (
"encoding/hex"
"fmt"
"os"
"path/filepath"
"strconv"
"testing"
"time"
"github.com/TrueCloudLab/frostfs-node/pkg/innerring"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/pkg/config"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/vm"
@ -36,13 +38,22 @@ func TestInitialize(t *testing.T) {
t.Run("7 nodes", func(t *testing.T) {
testInitialize(t, 7)
})
t.Run("16 nodes", func(t *testing.T) {
testInitialize(t, 16)
})
t.Run("max nodes", func(t *testing.T) {
testInitialize(t, maxAlphabetNodes)
})
t.Run("too many nodes", func(t *testing.T) {
require.ErrorIs(t, generateTestData(t, t.TempDir(), maxAlphabetNodes+1), ErrTooManyAlphabetNodes)
})
}
func testInitialize(t *testing.T, committeeSize int) {
testdataDir := t.TempDir()
v := viper.GetViper()
generateTestData(t, testdataDir, committeeSize)
require.NoError(t, generateTestData(t, testdataDir, committeeSize))
v.Set(protoConfigPath, filepath.Join(testdataDir, protoFileName))
// Set to the path or remove the next statement to download from the network.
@ -73,25 +84,33 @@ func testInitialize(t *testing.T, committeeSize int) {
})
}
func generateTestData(t *testing.T, dir string, size int) {
func generateTestData(t *testing.T, dir string, size int) error {
v := viper.GetViper()
v.Set(alphabetWalletsFlag, dir)
sizeStr := strconv.FormatUint(uint64(size), 10)
require.NoError(t, generateAlphabetCmd.Flags().Set(alphabetSizeFlag, sizeStr))
if err := generateAlphabetCmd.Flags().Set(alphabetSizeFlag, sizeStr); err != nil {
return err
}
setTestCredentials(v, size)
require.NoError(t, generateAlphabetCreds(generateAlphabetCmd, nil))
if err := generateAlphabetCreds(generateAlphabetCmd, nil); err != nil {
return err
}
var pubs []string
for i := 0; i < size; i++ {
p := filepath.Join(dir, innerring.GlagoliticLetter(i).String()+".json")
w, err := wallet.NewWalletFromFile(p)
require.NoError(t, err, "wallet doesn't exist")
if err != nil {
return fmt.Errorf("wallet doesn't exist: %w", err)
}
for _, acc := range w.Accounts {
if acc.Label == singleAccountName {
pub, ok := vm.ParseSignatureContract(acc.Contract.Script)
require.True(t, ok)
if !ok {
return fmt.Errorf("could not parse signature script for %s", acc.Address)
}
pubs = append(pubs, hex.EncodeToString(pub))
continue
}
@ -100,17 +119,18 @@ func generateTestData(t *testing.T, dir string, size int) {
cfg := config.Config{}
cfg.ProtocolConfiguration.Magic = 12345
cfg.ProtocolConfiguration.ValidatorsCount = size
cfg.ProtocolConfiguration.SecondsPerBlock = 1
cfg.ProtocolConfiguration.ValidatorsCount = uint32(size)
cfg.ProtocolConfiguration.TimePerBlock = time.Second
cfg.ProtocolConfiguration.StandbyCommittee = pubs // sorted by glagolic letters
cfg.ProtocolConfiguration.P2PSigExtensions = true
cfg.ProtocolConfiguration.VerifyTransactions = true
cfg.ProtocolConfiguration.VerifyBlocks = true
data, err := yaml.Marshal(cfg)
require.NoError(t, err)
if err != nil {
return err
}
protoPath := filepath.Join(dir, protoFileName)
require.NoError(t, os.WriteFile(protoPath, data, os.ModePerm))
return os.WriteFile(protoPath, data, os.ModePerm)
}
func setTestCredentials(v *viper.Viper, size int) {

View file

@ -1,65 +0,0 @@
package internal
import (
"fmt"
"strconv"
"google.golang.org/protobuf/proto"
)
// StringifySubnetClientGroupID returns string representation of SubnetClientGroupID using MarshalText.
// Returns a string with a message on error.
func StringifySubnetClientGroupID(id *SubnetClientGroupID) string {
text, err := id.MarshalText()
if err != nil {
return fmt.Sprintf("<invalid> %v", err)
}
return string(text)
}
// MarshalText encodes SubnetClientGroupID into text format according to FrostFS API V2 protocol:
// value in base-10 integer string format.
//
// It implements encoding.TextMarshaler.
func (x *SubnetClientGroupID) MarshalText() ([]byte, error) {
num := x.GetValue() // NPE safe, returns zero on nil
return []byte(strconv.FormatUint(uint64(num), 10)), nil
}
// UnmarshalText decodes the SubnetID from the text according to FrostFS API V2 protocol:
// should be base-10 integer string format with bitsize = 32.
//
// Returns strconv.ErrRange if integer overflows uint32.
//
// Must not be called on nil.
//
// Implements encoding.TextUnmarshaler.
func (x *SubnetClientGroupID) UnmarshalText(txt []byte) error {
num, err := strconv.ParseUint(string(txt), 10, 32)
if err != nil {
return fmt.Errorf("invalid numeric value: %w", err)
}
x.SetNumber(uint32(num))
return nil
}
// Marshal encodes the SubnetClientGroupID into a binary format of FrostFS API V2 protocol
// (Protocol Buffers with direct field order).
func (x *SubnetClientGroupID) Marshal() ([]byte, error) {
return proto.Marshal(x)
}
// Unmarshal decodes the SubnetClientGroupID from FrostFS API V2 binary format (see Marshal). Must not be called on nil.
func (x *SubnetClientGroupID) Unmarshal(data []byte) error {
return proto.Unmarshal(data, x)
}
// SetNumber sets SubnetClientGroupID value in uint32 format. Must not be called on nil.
// By default, number is 0.
func (x *SubnetClientGroupID) SetNumber(num uint32) {
x.Value = num
}

View file

@ -1,15 +0,0 @@
syntax = "proto3";
package neo.fs.v2.refs;
option go_package = "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/internal";
// Client group identifier in the FrostFS subnet.
//
// String representation of a value is base-10 integer.
//
// JSON representation is an object containing single `value` number field.
message SubnetClientGroupID {
// 4-byte integer identifier of the subnet client group.
fixed32 value = 1 [json_name = "value"];
}

View file

@ -51,7 +51,7 @@ type localClient struct {
maxGasInvoke int64
}
func newLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet) (*localClient, error) {
func newLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet, dumpPath string) (*localClient, error) {
cfg, err := config.LoadFile(v.GetString(protoConfigPath))
if err != nil {
return nil, err
@ -62,7 +62,7 @@ func newLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet
return nil, err
}
m := smartcontract.GetDefaultHonestNodeCount(cfg.ProtocolConfiguration.ValidatorsCount)
m := smartcontract.GetDefaultHonestNodeCount(int(cfg.ProtocolConfiguration.ValidatorsCount))
accounts := make([]*wallet.Account, len(wallets))
for i := range accounts {
accounts[i], err = getWalletAccount(wallets[i], consensusAccountName)
@ -87,7 +87,6 @@ func newLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet
go bc.Run()
dumpPath := v.GetString(localDumpFlag)
if cmd.Name() != "init" {
f, err := os.OpenFile(dumpPath, os.O_RDONLY, 0600)
if err != nil {
@ -190,7 +189,7 @@ func (l *localClient) GetCommittee() (keys.PublicKeys, error) {
func (l *localClient) InvokeFunction(h util.Uint160, method string, sPrm []smartcontract.Parameter, ss []transaction.Signer) (*result.Invoke, error) {
var err error
pp := make([]interface{}, len(sPrm))
pp := make([]any, len(sPrm))
for i, p := range sPrm {
pp[i], err = smartcontract.ExpandParameterToEmitable(p)
if err != nil {
@ -248,7 +247,7 @@ func (l *localClient) GetVersion() (*result.Version, error) {
}, nil
}
func (l *localClient) InvokeContractVerify(contract util.Uint160, params []smartcontract.Parameter, signers []transaction.Signer, witnesses ...transaction.Witness) (*result.Invoke, error) {
func (l *localClient) InvokeContractVerify(util.Uint160, []smartcontract.Parameter, []transaction.Signer, ...transaction.Witness) (*result.Invoke, error) {
// not used by `morph init` command
panic("unexpected call")
}
@ -346,7 +345,7 @@ func getSigners(sender *wallet.Account, cosigners []rpcclient.SignerAccount) ([]
}
func (l *localClient) NEP17BalanceOf(h util.Uint160, acc util.Uint160) (int64, error) {
res, err := invokeFunction(l, h, "balanceOf", []interface{}{acc}, nil)
res, err := invokeFunction(l, h, "balanceOf", []any{acc}, nil)
if err != nil {
return 0, err
}
@ -451,7 +450,7 @@ func (l *localClient) putTransactions() error {
return l.bc.AddBlock(b)
}
func invokeFunction(c Client, h util.Uint160, method string, parameters []interface{}, signers []transaction.Signer) (*result.Invoke, error) {
func invokeFunction(c Client, h util.Uint160, method string, parameters []any, signers []transaction.Signer) (*result.Invoke, error) {
w := io.NewBufBinWriter()
emit.Array(w.BinWriter, parameters...)
emit.AppCallNoArgs(w.BinWriter, h, method, callflag.All)

View file

@ -1,9 +1,9 @@
package morph
import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/spf13/cobra"
"github.com/spf13/viper"

View file

@ -0,0 +1,44 @@
package morph
import (
"errors"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/netmap"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/spf13/viper"
)
func getDefaultNetmapContractConfigMap() map[string]any {
m := make(map[string]any)
m[netmap.EpochDurationConfig] = viper.GetInt64(epochDurationInitFlag)
m[netmap.MaxObjectSizeConfig] = viper.GetInt64(maxObjectSizeInitFlag)
m[netmap.ContainerFeeConfig] = viper.GetInt64(containerFeeInitFlag)
m[netmap.ContainerAliasFeeConfig] = viper.GetInt64(containerAliasFeeInitFlag)
m[netmap.IrCandidateFeeConfig] = viper.GetInt64(candidateFeeInitFlag)
m[netmap.WithdrawFeeConfig] = viper.GetInt64(withdrawFeeInitFlag)
m[netmap.HomomorphicHashingDisabledKey] = viper.GetBool(homomorphicHashDisabledInitFlag)
m[netmap.MaintenanceModeAllowedConfig] = viper.GetBool(maintenanceModeAllowedInitFlag)
return m
}
func parseConfigFromNetmapContract(arr []stackitem.Item) (map[string][]byte, error) {
m := make(map[string][]byte, len(arr))
for _, param := range arr {
tuple, ok := param.Value().([]stackitem.Item)
if !ok || len(tuple) != 2 {
return nil, errors.New("invalid ListConfig response from netmap contract")
}
k, err := tuple[0].TryBytes()
if err != nil {
return nil, errors.New("invalid config key from netmap contract")
}
v, err := tuple[1].TryBytes()
if err != nil {
return nil, invalidConfigValueErr(string(k))
}
m[string(k)] = v
}
return m, nil
}

View file

@ -9,10 +9,12 @@ import (
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/gas"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/nep17"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/notary"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"
"github.com/spf13/viper"
@ -23,16 +25,9 @@ import (
const defaultNotaryDepositLifetime = 5760
func depositNotary(cmd *cobra.Command, _ []string) error {
p, err := cmd.Flags().GetString(storageWalletFlag)
w, err := openWallet(cmd)
if err != nil {
return err
} else if p == "" {
return fmt.Errorf("missing wallet path (use '--%s <out.json>')", storageWalletFlag)
}
w, err := wallet.NewWalletFromFile(p)
if err != nil {
return fmt.Errorf("can't open wallet: %v", err)
}
accHash := w.GetChangeAddress()
@ -80,6 +75,10 @@ func depositNotary(cmd *cobra.Command, _ []string) error {
}
}
return transferGas(cmd, acc, accHash, gasAmount, till)
}
func transferGas(cmd *cobra.Command, acc *wallet.Account, accHash util.Uint160, gasAmount fixedn.Fixed8, till int64) error {
c, err := getN3Client(viper.GetViper())
if err != nil {
return err
@ -111,7 +110,7 @@ func depositNotary(cmd *cobra.Command, _ []string) error {
accHash,
notary.Hash,
big.NewInt(int64(gasAmount)),
[]interface{}{nil, int64(height) + till},
[]any{nil, int64(height) + till},
)
if err != nil {
return fmt.Errorf("could not send tx: %w", err)
@ -119,3 +118,18 @@ func depositNotary(cmd *cobra.Command, _ []string) error {
return awaitTx(cmd, c, []hashVUBPair{{hash: txHash, vub: vub}})
}
func openWallet(cmd *cobra.Command) (*wallet.Wallet, error) {
p, err := cmd.Flags().GetString(storageWalletFlag)
if err != nil {
return nil, err
} else if p == "" {
return nil, fmt.Errorf("missing wallet path (use '--%s <out.json>')", storageWalletFlag)
}
w, err := wallet.NewWalletFromFile(p)
if err != nil {
return nil, fmt.Errorf("can't open wallet: %v", err)
}
return w, nil
}

View file

@ -27,23 +27,23 @@ func setPolicyCmd(cmd *cobra.Command, args []string) error {
bw := io.NewBufBinWriter()
for i := range args {
kv := strings.SplitN(args[i], "=", 2)
if len(kv) != 2 {
k, v, found := strings.Cut(args[i], "=")
if !found {
return fmt.Errorf("invalid parameter format, must be Parameter=Value")
}
switch kv[0] {
switch k {
case execFeeParam, storagePriceParam, setFeeParam:
default:
return fmt.Errorf("parameter must be one of %s, %s and %s", execFeeParam, storagePriceParam, setFeeParam)
}
value, err := strconv.ParseUint(kv[1], 10, 32)
value, err := strconv.ParseUint(v, 10, 32)
if err != nil {
return fmt.Errorf("can't parse parameter value '%s': %w", args[1], err)
}
emit.AppCall(bw.BinWriter, policy.Hash, "set"+kv[0], callflag.All, int64(value))
emit.AppCall(bw.BinWriter, policy.Hash, "set"+k, callflag.All, int64(value))
}
if err := wCtx.sendCommitteeTx(bw.Bytes(), false); err != nil {

View file

@ -4,7 +4,7 @@ import (
"errors"
"fmt"
netmapcontract "github.com/TrueCloudLab/frostfs-contract/netmap"
netmapcontract "git.frostfs.info/TrueCloudLab/frostfs-contract/netmap"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/io"
"github.com/nspcc-dev/neo-go/pkg/smartcontract/callflag"

View file

@ -18,10 +18,6 @@ const (
maxObjectSizeCLIFlag = "max-object-size"
epochDurationInitFlag = "network.epoch_duration"
epochDurationCLIFlag = "epoch-duration"
incomeRateInitFlag = "network.basic_income_rate"
incomeRateCLIFlag = "basic-income-rate"
auditFeeInitFlag = "network.fee.audit"
auditFeeCLIFlag = "audit-fee"
containerFeeInitFlag = "network.fee.container"
containerAliasFeeInitFlag = "network.fee.container_alias"
containerFeeCLIFlag = "container-fee"
@ -69,15 +65,12 @@ var (
_ = viper.BindPFlag(endpointFlag, cmd.Flags().Lookup(endpointFlag))
_ = viper.BindPFlag(epochDurationInitFlag, cmd.Flags().Lookup(epochDurationCLIFlag))
_ = viper.BindPFlag(maxObjectSizeInitFlag, cmd.Flags().Lookup(maxObjectSizeCLIFlag))
_ = viper.BindPFlag(incomeRateInitFlag, cmd.Flags().Lookup(incomeRateCLIFlag))
_ = viper.BindPFlag(homomorphicHashDisabledInitFlag, cmd.Flags().Lookup(homomorphicHashDisabledCLIFlag))
_ = viper.BindPFlag(auditFeeInitFlag, cmd.Flags().Lookup(auditFeeCLIFlag))
_ = viper.BindPFlag(candidateFeeInitFlag, cmd.Flags().Lookup(candidateFeeCLIFlag))
_ = viper.BindPFlag(containerFeeInitFlag, cmd.Flags().Lookup(containerFeeCLIFlag))
_ = viper.BindPFlag(containerAliasFeeInitFlag, cmd.Flags().Lookup(containerAliasFeeCLIFlag))
_ = viper.BindPFlag(withdrawFeeInitFlag, cmd.Flags().Lookup(withdrawFeeCLIFlag))
_ = viper.BindPFlag(protoConfigPath, cmd.Flags().Lookup(protoConfigPath))
_ = viper.BindPFlag(localDumpFlag, cmd.Flags().Lookup(localDumpFlag))
},
RunE: initializeSideChainCmd,
}
@ -239,14 +232,140 @@ var (
)
func init() {
RootCmd.AddCommand(generateAlphabetCmd)
generateAlphabetCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
generateAlphabetCmd.Flags().Uint(alphabetSizeFlag, 7, "Amount of alphabet wallets to generate")
initGenerateAlphabetCmd()
initInitCmd()
initDeployCmd()
initGenerateStorageCmd()
initForceNewEpochCmd()
initRemoveNodesCmd()
initSetPolicyCmd()
initDumpContractHashesCmd()
initDumpNetworkConfigCmd()
initSetConfigCmd()
initDumpBalancesCmd()
initUpdateContractsCmd()
initDumpContainersCmd()
initRestoreContainersCmd()
initListContainersCmd()
initRefillGasCmd()
initDepositoryNotaryCmd()
initNetmapCandidatesCmd()
}
func initNetmapCandidatesCmd() {
RootCmd.AddCommand(netmapCandidatesCmd)
netmapCandidatesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initDepositoryNotaryCmd() {
RootCmd.AddCommand(depositNotaryCmd)
depositNotaryCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
depositNotaryCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
depositNotaryCmd.Flags().String(walletAccountFlag, "", "Wallet account address")
depositNotaryCmd.Flags().String(refillGasAmountFlag, "", "Amount of GAS to deposit")
depositNotaryCmd.Flags().String(notaryDepositTillFlag, "", "Notary deposit duration in blocks")
}
func initRefillGasCmd() {
RootCmd.AddCommand(refillGasCmd)
refillGasCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
refillGasCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
refillGasCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
refillGasCmd.Flags().String(walletAddressFlag, "", "Address of wallet")
refillGasCmd.Flags().String(refillGasAmountFlag, "", "Additional amount of GAS to transfer")
refillGasCmd.MarkFlagsMutuallyExclusive(walletAddressFlag, storageWalletFlag)
}
func initListContainersCmd() {
RootCmd.AddCommand(listContainersCmd)
listContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
listContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
}
func initRestoreContainersCmd() {
RootCmd.AddCommand(restoreContainersCmd)
restoreContainersCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
restoreContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
restoreContainersCmd.Flags().String(containerDumpFlag, "", "File to restore containers from")
restoreContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to restore")
}
func initDumpContainersCmd() {
RootCmd.AddCommand(dumpContainersCmd)
dumpContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContainersCmd.Flags().String(containerDumpFlag, "", "File where to save dumped containers")
dumpContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
dumpContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to dump")
}
func initUpdateContractsCmd() {
RootCmd.AddCommand(updateContractsCmd)
updateContractsCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
updateContractsCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
updateContractsCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts")
_ = updateContractsCmd.MarkFlagRequired(contractsInitFlag)
}
func initDumpBalancesCmd() {
RootCmd.AddCommand(dumpBalancesCmd)
dumpBalancesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpBalancesCmd.Flags().BoolP(dumpBalancesStorageFlag, "s", false, "Dump balances of storage nodes from the current netmap")
dumpBalancesCmd.Flags().BoolP(dumpBalancesAlphabetFlag, "a", false, "Dump balances of alphabet contracts")
dumpBalancesCmd.Flags().BoolP(dumpBalancesProxyFlag, "p", false, "Dump balances of the proxy contract")
dumpBalancesCmd.Flags().Bool(dumpBalancesUseScriptHashFlag, false, "Use script-hash format for addresses")
}
func initSetConfigCmd() {
RootCmd.AddCommand(setConfig)
setConfig.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setConfig.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
setConfig.Flags().Bool(forceConfigSet, false, "Force setting not well-known configuration key")
}
func initDumpNetworkConfigCmd() {
RootCmd.AddCommand(dumpNetworkConfigCmd)
dumpNetworkConfigCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initDumpContractHashesCmd() {
RootCmd.AddCommand(dumpContractHashesCmd)
dumpContractHashesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContractHashesCmd.Flags().String(customZoneFlag, "", "Custom zone to search.")
}
func initSetPolicyCmd() {
RootCmd.AddCommand(setPolicy)
setPolicy.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setPolicy.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initRemoveNodesCmd() {
RootCmd.AddCommand(removeNodes)
removeNodes.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
removeNodes.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initForceNewEpochCmd() {
RootCmd.AddCommand(forceNewEpoch)
forceNewEpoch.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
forceNewEpoch.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initGenerateStorageCmd() {
RootCmd.AddCommand(generateStorageCmd)
generateStorageCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
generateStorageCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
generateStorageCmd.Flags().String(storageWalletFlag, "", "Path to new storage node wallet")
generateStorageCmd.Flags().String(storageGasCLIFlag, "", "Initial amount of GAS to transfer")
generateStorageCmd.Flags().StringP(storageWalletLabelFlag, "l", "", "Wallet label")
}
func initInitCmd() {
RootCmd.AddCommand(initCmd)
initCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
initCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
initCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts (default fetched from latest github release)")
initCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts")
_ = initCmd.MarkFlagRequired(contractsInitFlag)
initCmd.Flags().Uint(epochDurationCLIFlag, 240, "Amount of side chain blocks in one FrostFS epoch")
initCmd.Flags().Uint(maxObjectSizeCLIFlag, 67108864, "Max single object size in bytes")
initCmd.Flags().Bool(homomorphicHashDisabledCLIFlag, false, "Disable object homomorphic hashing")
@ -255,85 +374,14 @@ func init() {
initCmd.Flags().Uint64(containerAliasFeeCLIFlag, 500, "Container alias fee")
initCmd.Flags().String(protoConfigPath, "", "Path to the consensus node configuration")
initCmd.Flags().String(localDumpFlag, "", "Path to the blocks dump file")
RootCmd.AddCommand(deployCmd)
RootCmd.AddCommand(generateStorageCmd)
generateStorageCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
generateStorageCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
generateStorageCmd.Flags().String(storageWalletFlag, "", "Path to new storage node wallet")
generateStorageCmd.Flags().String(storageGasCLIFlag, "", "Initial amount of GAS to transfer")
generateStorageCmd.Flags().StringP(storageWalletLabelFlag, "l", "", "Wallet label")
RootCmd.AddCommand(forceNewEpoch)
forceNewEpoch.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
forceNewEpoch.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(removeNodes)
removeNodes.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
removeNodes.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(setPolicy)
setPolicy.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setPolicy.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(dumpContractHashesCmd)
dumpContractHashesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContractHashesCmd.Flags().String(customZoneFlag, "", "Custom zone to search.")
RootCmd.AddCommand(dumpNetworkConfigCmd)
dumpNetworkConfigCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
RootCmd.AddCommand(setConfig)
setConfig.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
setConfig.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
setConfig.Flags().Bool(forceConfigSet, false, "Force setting not well-known configuration key")
RootCmd.AddCommand(dumpBalancesCmd)
dumpBalancesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpBalancesCmd.Flags().BoolP(dumpBalancesStorageFlag, "s", false, "Dump balances of storage nodes from the current netmap")
dumpBalancesCmd.Flags().BoolP(dumpBalancesAlphabetFlag, "a", false, "Dump balances of alphabet contracts")
dumpBalancesCmd.Flags().BoolP(dumpBalancesProxyFlag, "p", false, "Dump balances of the proxy contract")
dumpBalancesCmd.Flags().Bool(dumpBalancesUseScriptHashFlag, false, "Use script-hash format for addresses")
RootCmd.AddCommand(updateContractsCmd)
updateContractsCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
updateContractsCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
updateContractsCmd.Flags().String(contractsInitFlag, "", "Path to archive with compiled FrostFS contracts (default fetched from latest github release)")
RootCmd.AddCommand(dumpContainersCmd)
dumpContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
dumpContainersCmd.Flags().String(containerDumpFlag, "", "File where to save dumped containers")
dumpContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
dumpContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to dump")
RootCmd.AddCommand(restoreContainersCmd)
restoreContainersCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
restoreContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
restoreContainersCmd.Flags().String(containerDumpFlag, "", "File to restore containers from")
restoreContainersCmd.Flags().StringSlice(containerIDsFlag, nil, "Containers to restore")
RootCmd.AddCommand(listContainersCmd)
listContainersCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
listContainersCmd.Flags().String(containerContractFlag, "", "Container contract hash (for networks without NNS)")
RootCmd.AddCommand(refillGasCmd)
refillGasCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
refillGasCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
refillGasCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
refillGasCmd.Flags().String(walletAddressFlag, "", "Address of wallet")
refillGasCmd.Flags().String(refillGasAmountFlag, "", "Additional amount of GAS to transfer")
refillGasCmd.MarkFlagsMutuallyExclusive(walletAddressFlag, storageWalletFlag)
RootCmd.AddCommand(cmdSubnet)
RootCmd.AddCommand(depositNotaryCmd)
depositNotaryCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
depositNotaryCmd.Flags().String(storageWalletFlag, "", "Path to storage node wallet")
depositNotaryCmd.Flags().String(walletAccountFlag, "", "Wallet account address")
depositNotaryCmd.Flags().String(refillGasAmountFlag, "", "Amount of GAS to deposit")
depositNotaryCmd.Flags().String(notaryDepositTillFlag, "", "Notary deposit duration in blocks")
RootCmd.AddCommand(netmapCandidatesCmd)
netmapCandidatesCmd.Flags().StringP(endpointFlag, "r", "", "N3 RPC node endpoint")
}
func initGenerateAlphabetCmd() {
RootCmd.AddCommand(generateAlphabetCmd)
generateAlphabetCmd.Flags().String(alphabetWalletsFlag, "", "Path to alphabet wallets dir")
generateAlphabetCmd.Flags().Uint(alphabetSizeFlag, 7, "Amount of alphabet wallets to generate")
}
func initDeployCmd() {
RootCmd.AddCommand(deployCmd)
}

File diff suppressed because it is too large Load diff

View file

@ -3,14 +3,14 @@ package modules
import (
"os"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/storagecfg"
"github.com/TrueCloudLab/frostfs-node/misc"
"github.com/TrueCloudLab/frostfs-node/pkg/util/autocomplete"
utilConfig "github.com/TrueCloudLab/frostfs-node/pkg/util/config"
"github.com/TrueCloudLab/frostfs-node/pkg/util/gendoc"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/storagecfg"
"git.frostfs.info/TrueCloudLab/frostfs-node/misc"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/autocomplete"
utilConfig "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/gendoc"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@ -45,14 +45,14 @@ func init() {
rootCmd.AddCommand(storagecfg.RootCmd)
rootCmd.AddCommand(autocomplete.Command("frostfs-adm"))
rootCmd.AddCommand(gendoc.Command(rootCmd))
rootCmd.AddCommand(gendoc.Command(rootCmd, gendoc.Options{}))
}
func Execute() error {
return rootCmd.Execute()
}
func entryPoint(cmd *cobra.Command, args []string) error {
func entryPoint(cmd *cobra.Command, _ []string) error {
printVersion, _ := cmd.Flags().GetBool("version")
if printVersion {
cmd.Print(misc.BuildInfo("FrostFS Adm"))

View file

@ -12,9 +12,6 @@ node:
- {{ .AnnouncedAddress }}
attribute_0: UN-LOCODE:{{ .Attribute.Locode }}
relay: {{ .Relay }} # start Storage node in relay mode without bootstrapping into the Network map
subnet:
exit_zero: false # toggle entrance to zero subnet (overrides corresponding attribute and occurrence in entries)
entries: [] # list of IDs of subnets to enter in a text format of FrostFS API protocol (overrides corresponding attributes)
grpc:
num: 1 # total number of listener endpoints

View file

@ -16,7 +16,7 @@ import (
"text/template"
"time"
netutil "github.com/TrueCloudLab/frostfs-node/pkg/network"
netutil "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network"
"github.com/chzyer/readline"
"github.com/nspcc-dev/neo-go/cli/flags"
"github.com/nspcc-dev/neo-go/cli/input"
@ -80,15 +80,7 @@ type config struct {
}
func storageConfig(cmd *cobra.Command, args []string) {
var outPath string
if len(args) != 0 {
outPath = args[0]
} else {
outPath = getPath("File to write config at [./config.yml]: ")
if outPath == "" {
outPath = "./config.yml"
}
}
outPath := getOutputPath(args)
historyPath := filepath.Join(os.TempDir(), "frostfs-adm.history")
readline.SetHistoryPath(historyPath)
@ -103,14 +95,7 @@ func storageConfig(cmd *cobra.Command, args []string) {
w, err := wallet.NewWalletFromFile(c.Wallet.Path)
fatalOnErr(err)
c.Wallet.Account, _ = cmd.Flags().GetString(accountFlag)
if c.Wallet.Account == "" {
addr := address.Uint160ToString(w.GetChangeAddress())
c.Wallet.Account = getWalletAccount(w, fmt.Sprintf("Wallet account [%s]: ", addr))
if c.Wallet.Account == "" {
c.Wallet.Account = addr
}
}
fillWalletAccount(cmd, &c, w)
accH, err := flags.ParseAddress(c.Wallet.Account)
fatalOnErr(err)
@ -128,32 +113,51 @@ func storageConfig(cmd *cobra.Command, args []string) {
c.AuthorizedKeys = append(c.AuthorizedKeys, hex.EncodeToString(acc.PrivateKey().PublicKey().Bytes()))
var network string
for {
network = getString("Choose network [mainnet]/testnet: ")
switch network {
case "":
network = "mainnet"
case "testnet", "mainnet":
default:
cmd.Println(`Network must be either "mainnet" or "testnet"`)
continue
}
break
}
network := readNetwork(cmd)
c.MorphRPC = n3config[network].MorphRPC
depositGas(cmd, acc, network)
c.Attribute.Locode = getString("UN-LOCODE attribute in [XX YYY] format: ")
endpoint := getDefaultEndpoint(cmd, &c)
c.Endpoint = getString(fmt.Sprintf("Listening address [%s]: ", endpoint))
if c.Endpoint == "" {
c.Endpoint = endpoint
}
c.ControlEndpoint = getString(fmt.Sprintf("Listening address (control endpoint) [%s]: ", defaultControlEndpoint))
if c.ControlEndpoint == "" {
c.ControlEndpoint = defaultControlEndpoint
}
c.TLSCert = getPath("TLS Certificate (optional): ")
if c.TLSCert != "" {
c.TLSKey = getPath("TLS Key: ")
}
c.Relay = getConfirmation(false, "Use node as a relay? yes/[no]: ")
if !c.Relay {
p := getPath("Path to the storage directory (all available storage will be used): ")
c.BlobstorPath = filepath.Join(p, "blob")
c.MetabasePath = filepath.Join(p, "meta")
}
out := applyTemplate(c)
fatalOnErr(os.WriteFile(outPath, out, 0644))
cmd.Println("Node is ready for work! Run `frostfs-node -config " + outPath + "`")
}
func getDefaultEndpoint(cmd *cobra.Command, c *config) string {
var addr, port string
for {
c.AnnouncedAddress = getString("Publicly announced address: ")
validator := netutil.Address{}
err := validator.FromString(c.AnnouncedAddress)
if err != nil {
cmd.Println("Incorrect address format. See https://github.com/TrueCloudLab/frostfs-node/blob/master/pkg/network/address.go for details.")
cmd.Println("Incorrect address format. See https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/network/address.go for details.")
continue
}
uriAddr, err := url.Parse(validator.URIAddr())
@ -182,34 +186,46 @@ func storageConfig(cmd *cobra.Command, args []string) {
break
}
return net.JoinHostPort(defaultDataEndpoint, port)
}
defaultAddr := net.JoinHostPort(defaultDataEndpoint, port)
c.Endpoint = getString(fmt.Sprintf("Listening address [%s]: ", defaultAddr))
if c.Endpoint == "" {
c.Endpoint = defaultAddr
func fillWalletAccount(cmd *cobra.Command, c *config, w *wallet.Wallet) {
c.Wallet.Account, _ = cmd.Flags().GetString(accountFlag)
if c.Wallet.Account == "" {
addr := address.Uint160ToString(w.GetChangeAddress())
c.Wallet.Account = getWalletAccount(w, fmt.Sprintf("Wallet account [%s]: ", addr))
if c.Wallet.Account == "" {
c.Wallet.Account = addr
}
c.ControlEndpoint = getString(fmt.Sprintf("Listening address (control endpoint) [%s]: ", defaultControlEndpoint))
if c.ControlEndpoint == "" {
c.ControlEndpoint = defaultControlEndpoint
}
}
c.TLSCert = getPath("TLS Certificate (optional): ")
if c.TLSCert != "" {
c.TLSKey = getPath("TLS Key: ")
func readNetwork(cmd *cobra.Command) string {
var network string
for {
network = getString("Choose network [mainnet]/testnet: ")
switch network {
case "":
network = "mainnet"
case "testnet", "mainnet":
default:
cmd.Println(`Network must be either "mainnet" or "testnet"`)
continue
}
c.Relay = getConfirmation(false, "Use node as a relay? yes/[no]: ")
if !c.Relay {
p := getPath("Path to the storage directory (all available storage will be used): ")
c.BlobstorPath = filepath.Join(p, "blob")
c.MetabasePath = filepath.Join(p, "meta")
break
}
return network
}
out := applyTemplate(c)
fatalOnErr(os.WriteFile(outPath, out, 0644))
cmd.Println("Node is ready for work! Run `frostfs-node -config " + outPath + "`")
func getOutputPath(args []string) string {
if len(args) != 0 {
return args[0]
}
outPath := getPath("File to write config at [./config.yml]: ")
if outPath == "" {
outPath = "./config.yml"
}
return outPath
}
func getWalletAccount(w *wallet.Wallet, prompt string) string {

View file

@ -3,7 +3,7 @@ package main
import (
"os"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules"
)
func main() {

View file

@ -9,16 +9,16 @@ duplicated header names or headers with empty values are considered invalid.
## Existing headers
There are some "well-known" headers starting with `__FROSTFS__` prefix that
There are some "well-known" headers starting with `__SYSTEM__` prefix that
affect system behaviour. For backward compatibility, the same set of
"well-known" headers may also use `__NEOFS__` prefix:
* `__FROSTFS__NETMAP_EPOCH` - netmap epoch to use for object placement calculation. The `value` is string
* `__SYSTEM__NETMAP_EPOCH` - netmap epoch to use for object placement calculation. The `value` is string
encoded `uint64` in decimal presentation. If set to '0' or omitted, the
current epoch only will be used.
* `__FROSTFS__NETMAP_LOOKUP_DEPTH` - if object can't be found using current epoch's netmap, this header limits
* `__SYSTEM__NETMAP_LOOKUP_DEPTH` - if object can't be found using current epoch's netmap, this header limits
how many past epochs the node can look up through. Depth is applied to a current epoch or the value
of `__FROSTFS__NETMAP_EPOCH` attribute. The `value` is string encoded `uint64` in decimal presentation.
of `__SYSTEM__NETMAP_EPOCH` attribute. The `value` is string encoded `uint64` in decimal presentation.
If set to '0' or not set, only the current epoch is used.
## `frostfs-cli` commands with `--xhdr`
@ -26,9 +26,8 @@ If set to '0' or not set, only the current epoch is used.
List of commands with support of extended headers:
* `container list-objects`
* `object delete/get/hash/head/lock/put/range/search`
* `storagegroup delete/get/list/put`
Example:
```shell
$ frostfs-cli object put -r s01.frostfs.devenv:8080 -w wallet.json --cid CID --file FILE --xhdr "__FROSTFS__NETMAP_EPOCH=777"
$ frostfs-cli object put -r s01.frostfs.devenv:8080 -w wallet.json --cid CID --file FILE --xhdr "__SYSTEM__NETMAP_EPOCH=777"
```

View file

@ -6,16 +6,19 @@ import (
"errors"
"fmt"
"io"
"sort"
"strings"
"github.com/TrueCloudLab/frostfs-sdk-go/accounting"
"github.com/TrueCloudLab/frostfs-sdk-go/client"
containerSDK "github.com/TrueCloudLab/frostfs-sdk-go/container"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/TrueCloudLab/frostfs-sdk-go/object"
oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/TrueCloudLab/frostfs-sdk-go/version"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/accounting"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/checksum"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
containerSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/version"
)
// BalanceOfPrm groups parameters of BalanceOf operation.
@ -37,8 +40,8 @@ func (x BalanceOfRes) Balance() accounting.Decimal {
// BalanceOf requests the current balance of a FrostFS user.
//
// Returns any error which prevented the operation from completing correctly in error return.
func BalanceOf(prm BalanceOfPrm) (res BalanceOfRes, err error) {
res.cliRes, err = prm.cli.BalanceGet(context.Background(), prm.PrmBalanceGet)
func BalanceOf(ctx context.Context, prm BalanceOfPrm) (res BalanceOfRes, err error) {
res.cliRes, err = prm.cli.BalanceGet(ctx, prm.PrmBalanceGet)
return
}
@ -62,16 +65,26 @@ func (x ListContainersRes) IDList() []cid.ID {
// ListContainers requests a list of FrostFS user's containers.
//
// Returns any error which prevented the operation from completing correctly in error return.
func ListContainers(prm ListContainersPrm) (res ListContainersRes, err error) {
res.cliRes, err = prm.cli.ContainerList(context.Background(), prm.PrmContainerList)
func ListContainers(ctx context.Context, prm ListContainersPrm) (res ListContainersRes, err error) {
res.cliRes, err = prm.cli.ContainerList(ctx, prm.PrmContainerList)
return
}
// SortedIDList returns sorted list of identifiers of user's containers.
func (x ListContainersRes) SortedIDList() []cid.ID {
list := x.cliRes.Containers()
sort.Slice(list, func(i, j int) bool {
lhs, rhs := list[i].EncodeToString(), list[j].EncodeToString()
return strings.Compare(lhs, rhs) < 0
})
return list
}
// PutContainerPrm groups parameters of PutContainer operation.
type PutContainerPrm struct {
commonPrm
client.PrmContainerPut
Client *client.Client
ClientParams client.PrmContainerPut
}
// PutContainerRes groups the resulting values of PutContainer operation.
@ -92,8 +105,8 @@ func (x PutContainerRes) ID() cid.ID {
// Success can be verified by reading by identifier.
//
// Returns any error which prevented the operation from completing correctly in error return.
func PutContainer(prm PutContainerPrm) (res PutContainerRes, err error) {
cliRes, err := prm.cli.ContainerPut(context.Background(), prm.PrmContainerPut)
func PutContainer(ctx context.Context, prm PutContainerPrm) (res PutContainerRes, err error) {
cliRes, err := prm.Client.ContainerPut(ctx, prm.ClientParams)
if err == nil {
res.cnr = cliRes.ID()
}
@ -103,13 +116,15 @@ func PutContainer(prm PutContainerPrm) (res PutContainerRes, err error) {
// GetContainerPrm groups parameters of GetContainer operation.
type GetContainerPrm struct {
commonPrm
cliPrm client.PrmContainerGet
Client *client.Client
ClientParams client.PrmContainerGet
}
// SetContainer sets identifier of the container to be read.
//
// Deprecated: Use GetContainerPrm.ClientParams.ContainerID instead.
func (x *GetContainerPrm) SetContainer(id cid.ID) {
x.cliPrm.SetContainer(id)
x.ClientParams.ContainerID = &id
}
// GetContainerRes groups the resulting values of GetContainer operation.
@ -125,20 +140,23 @@ func (x GetContainerRes) Container() containerSDK.Container {
// GetContainer reads a container from FrostFS by ID.
//
// Returns any error which prevented the operation from completing correctly in error return.
func GetContainer(prm GetContainerPrm) (res GetContainerRes, err error) {
res.cliRes, err = prm.cli.ContainerGet(context.Background(), prm.cliPrm)
func GetContainer(ctx context.Context, prm GetContainerPrm) (res GetContainerRes, err error) {
res.cliRes, err = prm.Client.ContainerGet(ctx, prm.ClientParams)
return
}
// IsACLExtendable checks if ACL of the container referenced by the given identifier
// can be extended. Client connection MUST BE correctly established in advance.
func IsACLExtendable(c *client.Client, cnr cid.ID) (bool, error) {
var prm GetContainerPrm
prm.SetClient(c)
prm.SetContainer(cnr)
func IsACLExtendable(ctx context.Context, c *client.Client, cnr cid.ID) (bool, error) {
prm := GetContainerPrm{
Client: c,
ClientParams: client.PrmContainerGet{
ContainerID: &cnr,
},
}
res, err := GetContainer(prm)
res, err := GetContainer(ctx, prm)
if err != nil {
return false, fmt.Errorf("get container from the FrostFS: %w", err)
}
@ -148,8 +166,8 @@ func IsACLExtendable(c *client.Client, cnr cid.ID) (bool, error) {
// DeleteContainerPrm groups parameters of DeleteContainerPrm operation.
type DeleteContainerPrm struct {
commonPrm
client.PrmContainerDelete
Client *client.Client
ClientParams client.PrmContainerDelete
}
// DeleteContainerRes groups the resulting values of DeleteContainer operation.
@ -163,16 +181,16 @@ type DeleteContainerRes struct{}
// Success can be verified by reading by identifier.
//
// Returns any error which prevented the operation from completing correctly in error return.
func DeleteContainer(prm DeleteContainerPrm) (res DeleteContainerRes, err error) {
_, err = prm.cli.ContainerDelete(context.Background(), prm.PrmContainerDelete)
func DeleteContainer(ctx context.Context, prm DeleteContainerPrm) (res DeleteContainerRes, err error) {
_, err = prm.Client.ContainerDelete(ctx, prm.ClientParams)
return
}
// EACLPrm groups parameters of EACL operation.
type EACLPrm struct {
commonPrm
client.PrmContainerEACL
Client *client.Client
ClientParams client.PrmContainerEACL
}
// EACLRes groups the resulting values of EACL operation.
@ -188,16 +206,16 @@ func (x EACLRes) EACL() eacl.Table {
// EACL reads eACL table from FrostFS by container ID.
//
// Returns any error which prevented the operation from completing correctly in error return.
func EACL(prm EACLPrm) (res EACLRes, err error) {
res.cliRes, err = prm.cli.ContainerEACL(context.Background(), prm.PrmContainerEACL)
func EACL(ctx context.Context, prm EACLPrm) (res EACLRes, err error) {
res.cliRes, err = prm.Client.ContainerEACL(ctx, prm.ClientParams)
return
}
// SetEACLPrm groups parameters of SetEACL operation.
type SetEACLPrm struct {
commonPrm
client.PrmContainerSetEACL
Client *client.Client
ClientParams client.PrmContainerSetEACL
}
// SetEACLRes groups the resulting values of SetEACL operation.
@ -211,16 +229,16 @@ type SetEACLRes struct{}
// Success can be verified by reading by container identifier.
//
// Returns any error which prevented the operation from completing correctly in error return.
func SetEACL(prm SetEACLPrm) (res SetEACLRes, err error) {
_, err = prm.cli.ContainerSetEACL(context.Background(), prm.PrmContainerSetEACL)
func SetEACL(ctx context.Context, prm SetEACLPrm) (res SetEACLRes, err error) {
_, err = prm.Client.ContainerSetEACL(ctx, prm.ClientParams)
return
}
// NetworkInfoPrm groups parameters of NetworkInfo operation.
type NetworkInfoPrm struct {
commonPrm
client.PrmNetworkInfo
Client *client.Client
ClientParams client.PrmNetworkInfo
}
// NetworkInfoRes groups the resulting values of NetworkInfo operation.
@ -236,16 +254,16 @@ func (x NetworkInfoRes) NetworkInfo() netmap.NetworkInfo {
// NetworkInfo reads information about the FrostFS network.
//
// Returns any error which prevented the operation from completing correctly in error return.
func NetworkInfo(prm NetworkInfoPrm) (res NetworkInfoRes, err error) {
res.cliRes, err = prm.cli.NetworkInfo(context.Background(), prm.PrmNetworkInfo)
func NetworkInfo(ctx context.Context, prm NetworkInfoPrm) (res NetworkInfoRes, err error) {
res.cliRes, err = prm.Client.NetworkInfo(ctx, prm.ClientParams)
return
}
// NodeInfoPrm groups parameters of NodeInfo operation.
type NodeInfoPrm struct {
commonPrm
client.PrmEndpointInfo
Client *client.Client
ClientParams client.PrmEndpointInfo
}
// NodeInfoRes groups the resulting values of NodeInfo operation.
@ -266,8 +284,8 @@ func (x NodeInfoRes) LatestVersion() version.Version {
// NodeInfo requests information about the remote server from FrostFS netmap.
//
// Returns any error which prevented the operation from completing correctly in error return.
func NodeInfo(prm NodeInfoPrm) (res NodeInfoRes, err error) {
res.cliRes, err = prm.cli.EndpointInfo(context.Background(), prm.PrmEndpointInfo)
func NodeInfo(ctx context.Context, prm NodeInfoPrm) (res NodeInfoRes, err error) {
res.cliRes, err = prm.Client.EndpointInfo(ctx, prm.ClientParams)
return
}
@ -290,8 +308,8 @@ func (x NetMapSnapshotRes) NetMap() netmap.NetMap {
// NetMapSnapshot requests current network view of the remote server.
//
// Returns any error which prevented the operation from completing correctly in error return.
func NetMapSnapshot(prm NetMapSnapshotPrm) (res NetMapSnapshotRes, err error) {
res.cliRes, err = prm.cli.NetMapSnapshot(context.Background(), client.PrmNetMapSnapshot{})
func NetMapSnapshot(ctx context.Context, prm NetMapSnapshotPrm) (res NetMapSnapshotRes, err error) {
res.cliRes, err = prm.cli.NetMapSnapshot(ctx, client.PrmNetMapSnapshot{})
return
}
@ -319,8 +337,8 @@ func (x CreateSessionRes) SessionKey() []byte {
// CreateSession opens a new unlimited session with the remote node.
//
// Returns any error which prevented the operation from completing correctly in error return.
func CreateSession(prm CreateSessionPrm) (res CreateSessionRes, err error) {
res.cliRes, err = prm.cli.SessionCreate(context.Background(), prm.PrmSessionCreate)
func CreateSession(ctx context.Context, prm CreateSessionPrm) (res CreateSessionRes, err error) {
res.cliRes, err = prm.cli.SessionCreate(ctx, prm.PrmSessionCreate)
return
}
@ -329,15 +347,19 @@ func CreateSession(prm CreateSessionPrm) (res CreateSessionRes, err error) {
type PutObjectPrm struct {
commonObjectPrm
hdr *object.Object
copyNum []uint32
hdr *objectSDK.Object
rdr io.Reader
headerCallback func(*object.Object)
headerCallback func(*objectSDK.Object)
prepareLocally bool
}
// SetHeader sets object header.
func (x *PutObjectPrm) SetHeader(hdr *object.Object) {
func (x *PutObjectPrm) SetHeader(hdr *objectSDK.Object) {
x.hdr = hdr
}
@ -348,10 +370,51 @@ func (x *PutObjectPrm) SetPayloadReader(rdr io.Reader) {
// SetHeaderCallback sets callback which is called on the object after the header is received
// but before the payload is written.
func (x *PutObjectPrm) SetHeaderCallback(f func(*object.Object)) {
func (x *PutObjectPrm) SetHeaderCallback(f func(*objectSDK.Object)) {
x.headerCallback = f
}
// SetCopiesNumberByVectors sets ordered list of minimal required object copies numbers
// per placement vector.
func (x *PutObjectPrm) SetCopiesNumberByVectors(copiesNumbers []uint32) {
x.copyNum = copiesNumbers
}
// PrepareLocally generate object header on the client side.
// For big object - split locally too.
func (x *PutObjectPrm) PrepareLocally() {
x.prepareLocally = true
}
func (x *PutObjectPrm) convertToSDKPrm(ctx context.Context) (client.PrmObjectPutInit, error) {
var putPrm client.PrmObjectPutInit
if !x.prepareLocally && x.sessionToken != nil {
putPrm.WithinSession(*x.sessionToken)
}
if x.bearerToken != nil {
putPrm.WithBearerToken(*x.bearerToken)
}
if x.local {
putPrm.MarkLocal()
}
putPrm.WithXHeaders(x.xHeaders...)
putPrm.SetCopiesNumberByVectors(x.copyNum)
if x.prepareLocally {
res, err := x.cli.NetworkInfo(ctx, client.PrmNetworkInfo{})
if err != nil {
return client.PrmObjectPutInit{}, err
}
putPrm.WithObjectMaxSize(res.Info().MaxObjectSize())
putPrm.WithEpochSource(epochSource(res.Info().CurrentEpoch()))
putPrm.WithoutHomomorphicHash(res.Info().HomomorphicHashingDisabled())
}
return putPrm, nil
}
// PutObjectRes groups the resulting values of PutObject operation.
type PutObjectRes struct {
id oid.ID
@ -362,32 +425,26 @@ func (x PutObjectRes) ID() oid.ID {
return x.id
}
type epochSource uint64
func (s epochSource) CurrentEpoch() uint64 {
return uint64(s)
}
// PutObject saves the object in FrostFS network.
//
// Returns any error which prevented the operation from completing correctly in error return.
func PutObject(prm PutObjectPrm) (*PutObjectRes, error) {
var putPrm client.PrmObjectPutInit
if prm.sessionToken != nil {
putPrm.WithinSession(*prm.sessionToken)
func PutObject(ctx context.Context, prm PutObjectPrm) (*PutObjectRes, error) {
sdkPrm, err := prm.convertToSDKPrm(ctx)
if err != nil {
return nil, fmt.Errorf("unable to create parameters of object put operation: %w", err)
}
if prm.bearerToken != nil {
putPrm.WithBearerToken(*prm.bearerToken)
}
if prm.local {
putPrm.MarkLocal()
}
putPrm.WithXHeaders(prm.xHeaders...)
wrt, err := prm.cli.ObjectPutInit(context.Background(), putPrm)
wrt, err := prm.cli.ObjectPutInit(ctx, sdkPrm)
if err != nil {
return nil, fmt.Errorf("init object writing: %w", err)
}
if wrt.WriteHeader(*prm.hdr) {
if wrt.WriteHeader(ctx, *prm.hdr) {
if prm.headerCallback != nil {
prm.headerCallback(prm.hdr)
}
@ -417,7 +474,7 @@ func PutObject(prm PutObjectPrm) (*PutObjectRes, error) {
for {
n, err = prm.rdr.Read(buf)
if n > 0 {
if !wrt.WritePayloadChunk(buf[:n]) {
if !wrt.WritePayloadChunk(ctx, buf[:n]) {
break
}
@ -433,7 +490,7 @@ func PutObject(prm PutObjectPrm) (*PutObjectRes, error) {
}
}
cliRes, err := wrt.Close()
cliRes, err := wrt.Close(ctx)
if err != nil { // here err already carries both status and client errors
return nil, fmt.Errorf("client failure: %w", err)
}
@ -462,22 +519,19 @@ func (x DeleteObjectRes) Tombstone() oid.ID {
// DeleteObject marks an object to be removed from FrostFS through tombstone placement.
//
// Returns any error which prevented the operation from completing correctly in error return.
func DeleteObject(prm DeleteObjectPrm) (*DeleteObjectRes, error) {
var delPrm client.PrmObjectDelete
delPrm.FromContainer(prm.objAddr.Container())
delPrm.ByID(prm.objAddr.Object())
func DeleteObject(ctx context.Context, prm DeleteObjectPrm) (*DeleteObjectRes, error) {
cnr := prm.objAddr.Container()
obj := prm.objAddr.Object()
if prm.sessionToken != nil {
delPrm.WithinSession(*prm.sessionToken)
delPrm := client.PrmObjectDelete{
XHeaders: prm.xHeaders,
ContainerID: &cnr,
ObjectID: &obj,
Session: prm.sessionToken,
BearerToken: prm.bearerToken,
}
if prm.bearerToken != nil {
delPrm.WithBearerToken(*prm.bearerToken)
}
delPrm.WithXHeaders(prm.xHeaders...)
cliRes, err := prm.cli.ObjectDelete(context.Background(), delPrm)
cliRes, err := prm.cli.ObjectDelete(ctx, delPrm)
if err != nil {
return nil, fmt.Errorf("remove object via client: %w", err)
}
@ -493,22 +547,22 @@ type GetObjectPrm struct {
objectAddressPrm
rawPrm
payloadWriterPrm
headerCallback func(*object.Object)
headerCallback func(*objectSDK.Object)
}
// SetHeaderCallback sets callback which is called on the object after the header is received
// but before the payload is written.
func (p *GetObjectPrm) SetHeaderCallback(f func(*object.Object)) {
func (p *GetObjectPrm) SetHeaderCallback(f func(*objectSDK.Object)) {
p.headerCallback = f
}
// GetObjectRes groups the resulting values of GetObject operation.
type GetObjectRes struct {
hdr *object.Object
hdr *objectSDK.Object
}
// Header returns the header of the request object.
func (x GetObjectRes) Header() *object.Object {
func (x GetObjectRes) Header() *objectSDK.Object {
return x.hdr
}
@ -518,35 +572,26 @@ func (x GetObjectRes) Header() *object.Object {
//
// Returns any error which prevented the operation from completing correctly in error return.
// For raw reading, returns *object.SplitInfoError error if object is virtual.
func GetObject(prm GetObjectPrm) (*GetObjectRes, error) {
var getPrm client.PrmObjectGet
getPrm.FromContainer(prm.objAddr.Container())
getPrm.ByID(prm.objAddr.Object())
func GetObject(ctx context.Context, prm GetObjectPrm) (*GetObjectRes, error) {
cnr := prm.objAddr.Container()
obj := prm.objAddr.Object()
if prm.sessionToken != nil {
getPrm.WithinSession(*prm.sessionToken)
getPrm := client.PrmObjectGet{
XHeaders: prm.xHeaders,
BearerToken: prm.bearerToken,
Session: prm.sessionToken,
Raw: prm.raw,
Local: prm.local,
ContainerID: &cnr,
ObjectID: &obj,
}
if prm.bearerToken != nil {
getPrm.WithBearerToken(*prm.bearerToken)
}
if prm.raw {
getPrm.MarkRaw()
}
if prm.local {
getPrm.MarkLocal()
}
getPrm.WithXHeaders(prm.xHeaders...)
rdr, err := prm.cli.ObjectGetInit(context.Background(), getPrm)
rdr, err := prm.cli.ObjectGetInit(ctx, getPrm)
if err != nil {
return nil, fmt.Errorf("init object reading on client: %w", err)
}
var hdr object.Object
var hdr objectSDK.Object
if !rdr.ReadHeader(&hdr) {
_, err = rdr.Close()
@ -582,11 +627,11 @@ func (x *HeadObjectPrm) SetMainOnlyFlag(v bool) {
// HeadObjectRes groups the resulting values of HeadObject operation.
type HeadObjectRes struct {
hdr *object.Object
hdr *objectSDK.Object
}
// Header returns the requested object header.
func (x HeadObjectRes) Header() *object.Object {
func (x HeadObjectRes) Header() *objectSDK.Object {
return x.hdr
}
@ -594,35 +639,26 @@ func (x HeadObjectRes) Header() *object.Object {
//
// Returns any error which prevented the operation from completing correctly in error return.
// For raw reading, returns *object.SplitInfoError error if object is virtual.
func HeadObject(prm HeadObjectPrm) (*HeadObjectRes, error) {
var cliPrm client.PrmObjectHead
cliPrm.FromContainer(prm.objAddr.Container())
cliPrm.ByID(prm.objAddr.Object())
func HeadObject(ctx context.Context, prm HeadObjectPrm) (*HeadObjectRes, error) {
cnr := prm.objAddr.Container()
obj := prm.objAddr.Object()
if prm.sessionToken != nil {
cliPrm.WithinSession(*prm.sessionToken)
headPrm := client.PrmObjectHead{
XHeaders: prm.xHeaders,
BearerToken: prm.bearerToken,
Session: prm.sessionToken,
Raw: prm.raw,
Local: prm.local,
ContainerID: &cnr,
ObjectID: &obj,
}
if prm.bearerToken != nil {
cliPrm.WithBearerToken(*prm.bearerToken)
}
if prm.raw {
cliPrm.MarkRaw()
}
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.WithXHeaders(prm.xHeaders...)
res, err := prm.cli.ObjectHead(context.Background(), cliPrm)
res, err := prm.cli.ObjectHead(ctx, headPrm)
if err != nil {
return nil, fmt.Errorf("read object header via client: %w", err)
}
var hdr object.Object
var hdr objectSDK.Object
if !res.ReadHeader(&hdr) {
return nil, fmt.Errorf("missing header in response")
@ -638,11 +674,11 @@ type SearchObjectsPrm struct {
commonObjectPrm
containerIDPrm
filters object.SearchFilters
filters objectSDK.SearchFilters
}
// SetFilters sets search filters.
func (x *SearchObjectsPrm) SetFilters(filters object.SearchFilters) {
func (x *SearchObjectsPrm) SetFilters(filters objectSDK.SearchFilters) {
x.filters = filters
}
@ -659,7 +695,7 @@ func (x SearchObjectsRes) IDList() []oid.ID {
// SearchObjects selects objects from the container which match the filters.
//
// Returns any error which prevented the operation from completing correctly in error return.
func SearchObjects(prm SearchObjectsPrm) (*SearchObjectsRes, error) {
func SearchObjects(ctx context.Context, prm SearchObjectsPrm) (*SearchObjectsRes, error) {
var cliPrm client.PrmObjectSearch
cliPrm.InContainer(prm.cnrID)
cliPrm.SetFilters(prm.filters)
@ -678,7 +714,7 @@ func SearchObjects(prm SearchObjectsPrm) (*SearchObjectsRes, error) {
cliPrm.WithXHeaders(prm.xHeaders...)
rdr, err := prm.cli.ObjectSearchInit(context.Background(), cliPrm)
rdr, err := prm.cli.ObjectSearchInit(ctx, cliPrm)
if err != nil {
return nil, fmt.Errorf("init object search: %w", err)
}
@ -703,6 +739,11 @@ func SearchObjects(prm SearchObjectsPrm) (*SearchObjectsRes, error) {
return nil, fmt.Errorf("read object list: %w", err)
}
sort.Slice(list, func(i, j int) bool {
lhs, rhs := list[i].EncodeToString(), list[j].EncodeToString()
return strings.Compare(lhs, rhs) < 0
})
return &SearchObjectsRes{
ids: list,
}, nil
@ -715,7 +756,7 @@ type HashPayloadRangesPrm struct {
tz bool
rngs []*object.Range
rngs []objectSDK.Range
salt []byte
}
@ -726,7 +767,7 @@ func (x *HashPayloadRangesPrm) TZ() {
}
// SetRanges sets a list of payload ranges to hash.
func (x *HashPayloadRangesPrm) SetRanges(rngs []*object.Range) {
func (x *HashPayloadRangesPrm) SetRanges(rngs []objectSDK.Range) {
x.rngs = rngs
}
@ -749,41 +790,27 @@ func (x HashPayloadRangesRes) HashList() [][]byte {
//
// Returns any error which prevented the operation from completing correctly in error return.
// Returns an error if number of received hashes differs with the number of requested ranges.
func HashPayloadRanges(prm HashPayloadRangesPrm) (*HashPayloadRangesRes, error) {
var cliPrm client.PrmObjectHash
cliPrm.FromContainer(prm.objAddr.Container())
cliPrm.ByID(prm.objAddr.Object())
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.UseSalt(prm.salt)
rngs := make([]uint64, 2*len(prm.rngs))
for i := range prm.rngs {
rngs[2*i] = prm.rngs[i].GetOffset()
rngs[2*i+1] = prm.rngs[i].GetLength()
}
cliPrm.SetRangeList(rngs...)
func HashPayloadRanges(ctx context.Context, prm HashPayloadRangesPrm) (*HashPayloadRangesRes, error) {
cs := checksum.SHA256
if prm.tz {
cliPrm.TillichZemorAlgo()
cs = checksum.TZ
}
if prm.sessionToken != nil {
cliPrm.WithinSession(*prm.sessionToken)
cnr := prm.objAddr.Container()
obj := prm.objAddr.Object()
cliPrm := client.PrmObjectHash{
ContainerID: &cnr,
ObjectID: &obj,
Local: prm.local,
Salt: prm.salt,
Ranges: prm.rngs,
ChecksumType: cs,
Session: prm.sessionToken,
BearerToken: prm.bearerToken,
XHeaders: prm.xHeaders,
}
if prm.bearerToken != nil {
cliPrm.WithBearerToken(*prm.bearerToken)
}
cliPrm.WithXHeaders(prm.xHeaders...)
res, err := prm.cli.ObjectHash(context.Background(), cliPrm)
res, err := prm.cli.ObjectHash(ctx, cliPrm)
if err != nil {
return nil, fmt.Errorf("read payload hashes via client: %w", err)
}
@ -800,11 +827,11 @@ type PayloadRangePrm struct {
rawPrm
payloadWriterPrm
rng *object.Range
rng *objectSDK.Range
}
// SetRange sets payload range to read.
func (x *PayloadRangePrm) SetRange(rng *object.Range) {
func (x *PayloadRangePrm) SetRange(rng *objectSDK.Range) {
x.rng = rng
}
@ -817,33 +844,23 @@ type PayloadRangeRes struct{}
//
// Returns any error which prevented the operation from completing correctly in error return.
// For raw reading, returns *object.SplitInfoError error if object is virtual.
func PayloadRange(prm PayloadRangePrm) (*PayloadRangeRes, error) {
var cliPrm client.PrmObjectRange
cliPrm.FromContainer(prm.objAddr.Container())
cliPrm.ByID(prm.objAddr.Object())
func PayloadRange(ctx context.Context, prm PayloadRangePrm) (*PayloadRangeRes, error) {
cnr := prm.objAddr.Container()
obj := prm.objAddr.Object()
if prm.sessionToken != nil {
cliPrm.WithinSession(*prm.sessionToken)
rangePrm := client.PrmObjectRange{
XHeaders: prm.xHeaders,
BearerToken: prm.bearerToken,
Session: prm.sessionToken,
Raw: prm.raw,
Local: prm.local,
ContainerID: &cnr,
ObjectID: &obj,
Offset: prm.rng.GetOffset(),
Length: prm.rng.GetLength(),
}
if prm.bearerToken != nil {
cliPrm.WithBearerToken(*prm.bearerToken)
}
if prm.raw {
cliPrm.MarkRaw()
}
if prm.local {
cliPrm.MarkLocal()
}
cliPrm.SetOffset(prm.rng.GetOffset())
cliPrm.SetLength(prm.rng.GetLength())
cliPrm.WithXHeaders(prm.xHeaders...)
rdr, err := prm.cli.ObjectRangeInit(context.Background(), cliPrm)
rdr, err := prm.cli.ObjectRangeInit(ctx, rangePrm)
if err != nil {
return nil, fmt.Errorf("init payload reading: %w", err)
}
@ -877,12 +894,12 @@ type SyncContainerRes struct{}
// Interrupts on any writer error.
//
// Panics if a container passed as a parameter is nil.
func SyncContainerSettings(prm SyncContainerPrm) (*SyncContainerRes, error) {
func SyncContainerSettings(ctx context.Context, prm SyncContainerPrm) (*SyncContainerRes, error) {
if prm.c == nil {
panic("sync container settings with the network: nil container")
}
err := client.SyncContainerWithNetwork(context.Background(), prm.c, prm.cli)
err := client.SyncContainerWithNetwork(ctx, prm.c, prm.cli)
if err != nil {
return nil, err
}

View file

@ -3,11 +3,11 @@ package internal
import (
"io"
"github.com/TrueCloudLab/frostfs-sdk-go/bearer"
"github.com/TrueCloudLab/frostfs-sdk-go/client"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/TrueCloudLab/frostfs-sdk-go/session"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
)
// here are small structures with public setters to share between parameter structures

View file

@ -8,13 +8,15 @@ import (
"errors"
"fmt"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/network"
"github.com/TrueCloudLab/frostfs-sdk-go/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network"
tracing "git.frostfs.info/TrueCloudLab/frostfs-observability/tracing/grpc"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"google.golang.org/grpc"
)
var errInvalidEndpoint = errors.New("provided RPC endpoint is incorrect")
@ -36,11 +38,11 @@ func getSDKClientByFlag(cmd *cobra.Command, key *ecdsa.PrivateKey, endpointFlag
if err != nil {
return nil, fmt.Errorf("%v: %w", errInvalidEndpoint, err)
}
return GetSDKClient(cmd, key, addr)
return GetSDKClient(cmd.Context(), cmd, key, addr)
}
// GetSDKClient returns default frostfs-sdk-go client.
func GetSDKClient(cmd *cobra.Command, key *ecdsa.PrivateKey, addr network.Address) (*client.Client, error) {
func GetSDKClient(ctx context.Context, cmd *cobra.Command, key *ecdsa.PrivateKey, addr network.Address) (*client.Client, error) {
var (
c client.Client
prmInit client.PrmInit
@ -48,7 +50,7 @@ func GetSDKClient(cmd *cobra.Command, key *ecdsa.PrivateKey, addr network.Addres
)
prmInit.SetDefaultPrivateKey(*key)
prmInit.ResolveNeoFSFailures()
prmInit.ResolveFrostFSFailures()
prmDial.SetServerURI(addr.URIAddr())
if timeout := viper.GetDuration(commonflags.Timeout); timeout > 0 {
// In CLI we can only set a timeout for the whole operation.
@ -59,10 +61,13 @@ func GetSDKClient(cmd *cobra.Command, key *ecdsa.PrivateKey, addr network.Addres
common.PrintVerbose(cmd, "Set request timeout to %s.", timeout)
}
prmDial.SetGRPCDialOptions(
grpc.WithChainUnaryInterceptor(tracing.NewUnaryClientInteceptor()),
grpc.WithChainStreamInterceptor(tracing.NewStreamClientInterceptor()))
c.Init(prmInit)
if err := c.Dial(prmDial); err != nil {
if err := c.Dial(ctx, prmDial); err != nil {
return nil, fmt.Errorf("can't init SDK client: %w", err)
}
@ -82,7 +87,7 @@ func GetCurrentEpoch(ctx context.Context, cmd *cobra.Command, endpoint string) (
return 0, fmt.Errorf("can't generate key to sign query: %w", err)
}
c, err := GetSDKClient(cmd, key, addr)
c, err := GetSDKClient(ctx, cmd, key, addr)
if err != nil {
return 0, err
}

View file

@ -4,10 +4,10 @@ import (
"errors"
"os"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/core/version"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
versionSDK "github.com/TrueCloudLab/frostfs-sdk-go/version"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/core/version"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
versionSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/version"
"github.com/spf13/cobra"
)

View file

@ -6,8 +6,8 @@ import (
"fmt"
"os"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/bearer"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
"github.com/spf13/cobra"
)

View file

@ -0,0 +1,62 @@
package common
import (
"context"
"sort"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/misc"
"git.frostfs.info/TrueCloudLab/frostfs-observability/tracing"
"github.com/spf13/cobra"
"go.opentelemetry.io/otel/trace"
)
type spanKey struct{}
// StopClientCommandSpan stops tracing span for the command and prints trace ID on the standard output.
func StopClientCommandSpan(cmd *cobra.Command, _ []string) {
span, ok := cmd.Context().Value(spanKey{}).(trace.Span)
if !ok {
return
}
span.End()
// Noop provider cannot fail on flush.
_ = tracing.Shutdown(cmd.Context())
cmd.PrintErrf("Trace ID: %s\n", span.SpanContext().TraceID())
}
// StartClientCommandSpan starts tracing span for the command.
func StartClientCommandSpan(cmd *cobra.Command) {
enableTracing, err := cmd.Flags().GetBool(commonflags.TracingFlag)
if err != nil || !enableTracing {
return
}
_, err = tracing.Setup(cmd.Context(), tracing.Config{
Enabled: true,
Exporter: tracing.NoOpExporter,
Service: "frostfs-cli",
Version: misc.Version,
})
commonCmd.ExitOnErr(cmd, "init tracing: %w", err)
var components sort.StringSlice
for c := cmd; c != nil; c = c.Parent() {
components = append(components, c.Name())
}
for i, j := 0, len(components)-1; i < j; {
components.Swap(i, j)
i++
j--
}
operation := strings.Join(components, ".")
ctx, span := tracing.StartSpanFromContext(cmd.Context(), operation)
ctx = context.WithValue(ctx, spanKey{}, span)
cmd.SetContext(ctx)
}

View file

@ -5,14 +5,14 @@ import (
"strconv"
"time"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-sdk-go/checksum"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/checksum"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
// PrintVerbose prints to the stdout if the commonflags.Verbose flag is on.
func PrintVerbose(cmd *cobra.Command, format string, a ...interface{}) {
func PrintVerbose(cmd *cobra.Command, format string, a ...any) {
if viper.GetBool(commonflags.Verbose) {
cmd.Printf(format+"\n", a...)
}

View file

@ -47,6 +47,9 @@ const (
OIDFlag = "oid"
OIDFlagUsage = "Object ID."
TracingFlag = "trace"
TracingFlagUsage = "Generate trace ID and print it."
)
// Init adds common flags to the command:
@ -54,12 +57,14 @@ const (
// - WalletPath,
// - Account,
// - RPC,
// - Tracing,
// - Timeout.
func Init(cmd *cobra.Command) {
InitWithoutRPC(cmd)
ff := cmd.Flags()
ff.StringP(RPC, RPCShorthand, RPCDefault, RPCUsage)
ff.Bool(TracingFlag, false, TracingFlagUsage)
ff.DurationP(Timeout, TimeoutShorthand, TimeoutDefault, TimeoutUsage)
}

View file

@ -7,7 +7,7 @@ import (
"path/filepath"
"testing"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/wallet"

View file

@ -6,8 +6,8 @@ import (
"fmt"
"os"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/cobra"

View file

@ -4,7 +4,7 @@ import (
"crypto/ecdsa"
"errors"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/nspcc-dev/neo-go/cli/flags"
"github.com/nspcc-dev/neo-go/cli/input"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"

View file

@ -1,6 +1,6 @@
package main
import cmd "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules"
import cmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules"
func main() {
cmd.Execute()

View file

@ -3,13 +3,13 @@ package accounting
import (
"math/big"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/util/precision"
"github.com/TrueCloudLab/frostfs-sdk-go/accounting"
"github.com/TrueCloudLab/frostfs-sdk-go/user"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/precision"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/accounting"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/spf13/cobra"
"github.com/spf13/viper"
@ -41,7 +41,7 @@ var accountingBalanceCmd = &cobra.Command{
prm.SetClient(cli)
prm.SetAccount(idUser)
res, err := internalclient.BalanceOf(prm)
res, err := internalclient.BalanceOf(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
// print to stdout

View file

@ -1,7 +1,8 @@
package accounting
import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@ -17,7 +18,9 @@ var Cmd = &cobra.Command{
_ = viper.BindPFlag(commonflags.WalletPath, flags.Lookup(commonflags.WalletPath))
_ = viper.BindPFlag(commonflags.Account, flags.Lookup(commonflags.Account))
_ = viper.BindPFlag(commonflags.RPC, flags.Lookup(commonflags.RPC))
common.StartClientCommandSpan(cmd)
},
PersistentPostRun: common.StopClientCommandSpan,
}
func init() {

View file

@ -1,9 +1,9 @@
package basic
import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/container/acl"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/acl"
"github.com/spf13/cobra"
)

View file

@ -6,11 +6,11 @@ import (
"os"
"strings"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/spf13/cobra"
)
@ -30,7 +30,7 @@ Filter consists of <typ>:<key><match><value>
Key is a valid unicode string corresponding to object or request header key.
Well-known system object headers start with '$Object:' prefix.
User defined headers start without prefix.
Read more about filter keys at github.com/TrueCloudLab/frostfs-api/blob/master/proto-docs/acl.md#message-eaclrecordfilter
Read more about filter keys at git.frostfs.info.com/TrueCloudLab/frostfs-api/src/branch/master/proto-docs/acl.md#message-eaclrecordfilter
Match is '=' for matching and '!=' for non-matching filter.
Value is a valid unicode string corresponding to object or request header value.

View file

@ -3,8 +3,8 @@ package extended
import (
"testing"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/stretchr/testify/require"
)

View file

@ -4,9 +4,9 @@ import (
"os"
"strings"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/spf13/cobra"
)

View file

@ -1,8 +1,8 @@
package acl
import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/acl/basic"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/acl/extended"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/acl/basic"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/acl/extended"
"github.com/spf13/cobra"
)

View file

@ -7,13 +7,13 @@ import (
"os"
"time"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/bearer"
eaclSDK "github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/TrueCloudLab/frostfs-sdk-go/user"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
eaclSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra"
)
@ -24,6 +24,7 @@ const (
ownerFlag = "owner"
outFlag = "out"
jsonFlag = commonflags.JSON
impersonateFlag = "impersonate"
)
var createCmd = &cobra.Command{
@ -39,19 +40,20 @@ is set to current epoch + n.
}
func init() {
createCmd.Flags().StringP(eaclFlag, "e", "", "Path to the extended ACL table")
createCmd.Flags().StringP(issuedAtFlag, "i", "", "Epoch to issue token at")
createCmd.Flags().StringP(notValidBeforeFlag, "n", "", "Not valid before epoch")
createCmd.Flags().StringP(eaclFlag, "e", "", "Path to the extended ACL table (mutually exclusive with --impersonate flag)")
createCmd.Flags().StringP(issuedAtFlag, "i", "+0", "Epoch to issue token at")
createCmd.Flags().StringP(notValidBeforeFlag, "n", "+0", "Not valid before epoch")
createCmd.Flags().StringP(commonflags.ExpireAt, "x", "", "The last active epoch for the token")
createCmd.Flags().StringP(ownerFlag, "o", "", "Token owner")
createCmd.Flags().String(outFlag, "", "File to write token to")
createCmd.Flags().Bool(jsonFlag, false, "Output token in JSON")
createCmd.Flags().Bool(impersonateFlag, false, "Mark token as impersonate to consider the token signer as the request owner (mutually exclusive with --eacl flag)")
createCmd.Flags().StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage)
createCmd.MarkFlagsMutuallyExclusive(eaclFlag, impersonateFlag)
_ = cobra.MarkFlagFilename(createCmd.Flags(), eaclFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), issuedAtFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), notValidBeforeFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), commonflags.ExpireAt)
_ = cobra.MarkFlagRequired(createCmd.Flags(), ownerFlag)
_ = cobra.MarkFlagRequired(createCmd.Flags(), outFlag)
@ -68,10 +70,14 @@ func createToken(cmd *cobra.Command, _ []string) {
commonCmd.ExitOnErr(cmd, "can't parse --"+notValidBeforeFlag+" flag: %w", err)
if iatRelative || expRelative || nvbRelative {
endpoint, _ := cmd.Flags().GetString(commonflags.RPC)
if len(endpoint) == 0 {
commonCmd.ExitOnErr(cmd, "can't fetch current epoch: %w", fmt.Errorf("'%s' flag value must be specified", commonflags.RPC))
}
ctx, cancel := context.WithTimeout(context.Background(), time.Second*30)
defer cancel()
endpoint, _ := cmd.Flags().GetString(commonflags.RPC)
currEpoch, err := internalclient.GetCurrentEpoch(ctx, cmd, endpoint)
commonCmd.ExitOnErr(cmd, "can't fetch current epoch: %w", err)
@ -101,6 +107,9 @@ func createToken(cmd *cobra.Command, _ []string) {
b.SetIat(iat)
b.ForUser(ownerID)
impersonate, _ := cmd.Flags().GetBool(impersonateFlag)
b.SetImpersonate(impersonate)
eaclPath, _ := cmd.Flags().GetString(eaclFlag)
if eaclPath != "" {
table := eaclSDK.NewTable()

View file

@ -1,7 +1,7 @@
package cmd
import (
"github.com/TrueCloudLab/frostfs-node/pkg/util/autocomplete"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/autocomplete"
)
func init() {

View file

@ -7,16 +7,17 @@ import (
"strings"
"time"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/container"
"github.com/TrueCloudLab/frostfs-sdk-go/container/acl"
"github.com/TrueCloudLab/frostfs-sdk-go/netmap"
subnetid "github.com/TrueCloudLab/frostfs-sdk-go/subnet/id"
"github.com/TrueCloudLab/frostfs-sdk-go/user"
containerApi "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/acl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra"
)
@ -26,8 +27,9 @@ var (
containerAttributes []string
containerAwait bool
containerName string
containerNnsName string
containerNnsZone string
containerNoTimestamp bool
containerSubnet string
force bool
)
@ -47,7 +49,7 @@ It will be stored in sidechain when inner ring will accepts it.`,
var prm internalclient.NetMapSnapshotPrm
prm.SetClient(cli)
resmap, err := internalclient.NetMapSnapshot(prm)
resmap, err := internalclient.NetMapSnapshot(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "unable to get netmap snapshot to validate container placement, "+
"use --force option to skip this check: %w", err)
@ -67,15 +69,6 @@ It will be stored in sidechain when inner ring will accepts it.`,
}
}
if containerSubnet != "" {
var subnetID subnetid.ID
err = subnetID.DecodeString(containerSubnet)
commonCmd.ExitOnErr(cmd, "could not parse subnetID: %w", err)
placementPolicy.RestrictSubnet(subnetID)
}
var cnr container.Container
cnr.Init()
@ -104,18 +97,18 @@ It will be stored in sidechain when inner ring will accepts it.`,
syncContainerPrm.SetClient(cli)
syncContainerPrm.SetContainer(&cnr)
_, err = internalclient.SyncContainerSettings(syncContainerPrm)
_, err = internalclient.SyncContainerSettings(cmd.Context(), syncContainerPrm)
commonCmd.ExitOnErr(cmd, "syncing container's settings rpc error: %w", err)
var putPrm internalclient.PutContainerPrm
putPrm.SetClient(cli)
putPrm.SetContainer(cnr)
if tok != nil {
putPrm.WithinSession(*tok)
putPrm := internalclient.PutContainerPrm{
Client: cli,
ClientParams: client.PrmContainerPut{
Container: &cnr,
Session: tok,
},
}
res, err := internalclient.PutContainer(putPrm)
res, err := internalclient.PutContainer(cmd.Context(), putPrm)
commonCmd.ExitOnErr(cmd, "put container rpc error: %w", err)
id := res.ID()
@ -125,14 +118,17 @@ It will be stored in sidechain when inner ring will accepts it.`,
if containerAwait {
cmd.Println("awaiting...")
var getPrm internalclient.GetContainerPrm
getPrm.SetClient(cli)
getPrm.SetContainer(id)
getPrm := internalclient.GetContainerPrm{
Client: cli,
ClientParams: client.PrmContainerGet{
ContainerID: &id,
},
}
for i := 0; i < awaitTimeout; i++ {
time.Sleep(1 * time.Second)
_, err := internalclient.GetContainer(getPrm)
_, err := internalclient.GetContainer(cmd.Context(), getPrm)
if err == nil {
cmd.Println("container has been persisted on sidechain")
return
@ -149,6 +145,7 @@ func initContainerCreateCmd() {
// Init common flags
flags.StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage)
flags.Bool(commonflags.TracingFlag, false, commonflags.TracingFlagUsage)
flags.DurationP(commonflags.Timeout, commonflags.TimeoutShorthand, commonflags.TimeoutDefault, commonflags.TimeoutUsage)
flags.StringP(commonflags.WalletPath, commonflags.WalletPathShorthand, commonflags.WalletPathDefault, commonflags.WalletPathUsage)
flags.StringP(commonflags.Account, commonflags.AccountShorthand, commonflags.AccountDefault, commonflags.AccountUsage)
@ -160,8 +157,9 @@ func initContainerCreateCmd() {
flags.StringSliceVarP(&containerAttributes, "attributes", "a", nil, "Comma separated pairs of container attributes in form of Key1=Value1,Key2=Value2")
flags.BoolVar(&containerAwait, "await", false, "Block execution until container is persisted")
flags.StringVar(&containerName, "name", "", "Container name attribute")
flags.StringVar(&containerNnsName, "nns-name", "", "Container nns name attribute")
flags.StringVar(&containerNnsZone, "nns-zone", "", "Container nns zone attribute")
flags.BoolVar(&containerNoTimestamp, "disable-timestamp", false, "Disable timestamp container attribute")
flags.StringVar(&containerSubnet, "subnet", "", "String representation of container subnetwork")
flags.BoolVarP(&force, commonflags.ForceFlag, commonflags.ForceFlagShorthand, false,
"Skip placement validity check")
}
@ -187,22 +185,22 @@ func parseContainerPolicy(cmd *cobra.Command, policyString string) (*netmap.Plac
return &result, nil
}
if err = result.UnmarshalJSON([]byte(policyString)); err == nil {
if err := result.UnmarshalJSON([]byte(policyString)); err == nil {
common.PrintVerbose(cmd, "Parsed JSON encoded policy")
return &result, nil
}
return nil, errors.New("can't parse placement policy")
return nil, fmt.Errorf("can't parse placement policy: %w", err)
}
func parseAttributes(dst *container.Container, attributes []string) error {
for i := range attributes {
kvPair := strings.Split(attributes[i], attributeDelimiter)
if len(kvPair) != 2 {
k, v, found := strings.Cut(attributes[i], attributeDelimiter)
if !found {
return errors.New("invalid container attribute")
}
dst.SetAttribute(kvPair[0], kvPair[1])
dst.SetAttribute(k, v)
}
if !containerNoTimestamp {
@ -213,5 +211,12 @@ func parseAttributes(dst *container.Container, attributes []string) error {
container.SetName(dst, containerName)
}
if containerNnsName != "" {
dst.SetAttribute(containerApi.SysAttributeName, containerNnsName)
}
if containerNnsZone != "" {
dst.SetAttribute(containerApi.SysAttributeZone, containerNnsZone)
}
return nil
}

View file

@ -4,13 +4,14 @@ import (
"fmt"
"time"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
objectSDK "github.com/TrueCloudLab/frostfs-sdk-go/object"
"github.com/TrueCloudLab/frostfs-sdk-go/user"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra"
)
@ -30,11 +31,14 @@ Only owner of the container has a permission to remove container.`,
if force, _ := cmd.Flags().GetBool(commonflags.ForceFlag); !force {
common.PrintVerbose(cmd, "Reading the container to check ownership...")
var getPrm internalclient.GetContainerPrm
getPrm.SetClient(cli)
getPrm.SetContainer(id)
getPrm := internalclient.GetContainerPrm{
Client: cli,
ClientParams: client.PrmContainerGet{
ContainerID: &id,
},
}
resGet, err := internalclient.GetContainer(getPrm)
resGet, err := internalclient.GetContainer(cmd.Context(), getPrm)
commonCmd.ExitOnErr(cmd, "can't get the container: %w", err)
owner := resGet.Container().Owner()
@ -72,26 +76,26 @@ Only owner of the container has a permission to remove container.`,
common.PrintVerbose(cmd, "Searching for LOCK objects...")
res, err := internalclient.SearchObjects(searchPrm)
res, err := internalclient.SearchObjects(cmd.Context(), searchPrm)
commonCmd.ExitOnErr(cmd, "can't search for LOCK objects: %w", err)
if len(res.IDList()) != 0 {
commonCmd.ExitOnErr(cmd, "",
fmt.Errorf("Container wasn't removed because LOCK objects were found.\n"+
"Use --%s flag to remove anyway.", commonflags.ForceFlag))
fmt.Errorf("container wasn't removed because LOCK objects were found, "+
"use --%s flag to remove anyway", commonflags.ForceFlag))
}
}
}
var delPrm internalclient.DeleteContainerPrm
delPrm.SetClient(cli)
delPrm.SetContainer(id)
if tok != nil {
delPrm.WithinSession(*tok)
delPrm := internalclient.DeleteContainerPrm{
Client: cli,
ClientParams: client.PrmContainerDelete{
ContainerID: &id,
Session: tok,
},
}
_, err := internalclient.DeleteContainer(delPrm)
_, err := internalclient.DeleteContainer(cmd.Context(), delPrm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
cmd.Println("container delete method invoked")
@ -99,14 +103,17 @@ Only owner of the container has a permission to remove container.`,
if containerAwait {
cmd.Println("awaiting...")
var getPrm internalclient.GetContainerPrm
getPrm.SetClient(cli)
getPrm.SetContainer(id)
getPrm := internalclient.GetContainerPrm{
Client: cli,
ClientParams: client.PrmContainerGet{
ContainerID: &id,
},
}
for i := 0; i < awaitTimeout; i++ {
time.Sleep(1 * time.Second)
_, err := internalclient.GetContainer(getPrm)
_, err := internalclient.GetContainer(cmd.Context(), getPrm)
if err != nil {
cmd.Println("container has been removed:", containerID)
return
@ -124,6 +131,7 @@ func initContainerDeleteCmd() {
flags.StringP(commonflags.WalletPath, commonflags.WalletPathShorthand, commonflags.WalletPathDefault, commonflags.WalletPathUsage)
flags.StringP(commonflags.Account, commonflags.AccountShorthand, commonflags.AccountDefault, commonflags.AccountUsage)
flags.StringP(commonflags.RPC, commonflags.RPCShorthand, commonflags.RPCDefault, commonflags.RPCUsage)
flags.Bool(commonflags.TracingFlag, false, commonflags.TracingFlagUsage)
flags.StringVar(&containerID, commonflags.CIDFlag, "", commonflags.CIDFlagUsage)
flags.BoolVar(&containerAwait, "await", false, "Block execution until container is removed")

View file

@ -4,15 +4,16 @@ import (
"crypto/ecdsa"
"os"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/container"
"github.com/TrueCloudLab/frostfs-sdk-go/container/acl"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/acl"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/spf13/cobra"
)
@ -147,11 +148,14 @@ func getContainer(cmd *cobra.Command) (container.Container, *ecdsa.PrivateKey) {
pk = key.GetOrGenerate(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
var prm internalclient.GetContainerPrm
prm.SetClient(cli)
prm.SetContainer(id)
prm := internalclient.GetContainerPrm{
Client: cli,
ClientParams: client.PrmContainerGet{
ContainerID: &id,
},
}
res, err := internalclient.GetContainer(prm)
res, err := internalclient.GetContainer(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
cnr = res.Container()

View file

@ -3,11 +3,12 @@ package container
import (
"os"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
)
@ -20,11 +21,14 @@ var getExtendedACLCmd = &cobra.Command{
pk := key.GetOrGenerate(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
var eaclPrm internalclient.EACLPrm
eaclPrm.SetClient(cli)
eaclPrm.SetContainer(id)
eaclPrm := internalclient.EACLPrm{
Client: cli,
ClientParams: client.PrmContainerEACL{
ContainerID: &id,
},
}
res, err := internalclient.EACL(eaclPrm)
res, err := internalclient.EACL(cmd.Context(), eaclPrm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
eaclTable := res.EACL()

View file

@ -3,12 +3,13 @@ package container
import (
"strings"
"github.com/TrueCloudLab/frostfs-api-go/v2/container"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/user"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
containerSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra"
)
@ -16,12 +17,14 @@ import (
const (
flagListPrintAttr = "with-attr"
flagListContainerOwner = "owner"
flagListName = "name"
)
// flag vars of list command.
var (
flagVarListPrintAttr bool
flagVarListContainerOwner string
flagVarListName string
)
var listContainersCmd = &cobra.Command{
@ -46,30 +49,42 @@ var listContainersCmd = &cobra.Command{
prm.SetClient(cli)
prm.SetAccount(idUser)
res, err := internalclient.ListContainers(prm)
res, err := internalclient.ListContainers(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
var prmGet internalclient.GetContainerPrm
prmGet.SetClient(cli)
prmGet := internalclient.GetContainerPrm{
Client: cli,
}
list := res.IDList()
for i := range list {
cmd.Println(list[i].String())
containerIDs := res.SortedIDList()
for _, cnrID := range containerIDs {
if flagVarListName == "" && !flagVarListPrintAttr {
cmd.Println(cnrID.String())
continue
}
cnrID := cnrID
prmGet.ClientParams.ContainerID = &cnrID
res, err := internalclient.GetContainer(cmd.Context(), prmGet)
if err != nil {
cmd.Printf(" failed to read attributes: %v\n", err)
continue
}
cnr := res.Container()
if cnrName := containerSDK.Name(cnr); flagVarListName != "" && cnrName != flagVarListName {
continue
}
cmd.Println(cnrID.String())
if flagVarListPrintAttr {
prmGet.SetContainer(list[i])
res, err := internalclient.GetContainer(prmGet)
if err == nil {
res.Container().IterateAttributes(func(key, val string) {
if !strings.HasPrefix(key, container.SysAttributePrefix) {
// FIXME(@cthulhu-rider): neofs-sdk-go#314 use dedicated method to skip system attributes
cnr.IterateAttributes(func(key, val string) {
if !strings.HasPrefix(key, container.SysAttributePrefix) && !strings.HasPrefix(key, container.SysAttributePrefixNeoFS) {
// FIXME(@cthulhu-rider): https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/issues/97
// Use dedicated method to skip system attributes.
cmd.Printf(" %s: %s\n", key, val)
}
})
} else {
cmd.Printf(" failed to read attributes: %v\n", err)
}
}
}
},
@ -80,6 +95,9 @@ func initContainerListContainersCmd() {
flags := listContainersCmd.Flags()
flags.StringVar(&flagVarListName, flagListName, "",
"List containers by the attribute name",
)
flags.StringVar(&flagVarListContainerOwner, flagListContainerOwner, "",
"Owner of containers (omit to use owner from private key)",
)

View file

@ -3,14 +3,14 @@ package container
import (
"strings"
v2object "github.com/TrueCloudLab/frostfs-api-go/v2/object"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
objectCli "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/object"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-sdk-go/object"
oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
v2object "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
objectCli "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/object"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/spf13/cobra"
)
@ -31,7 +31,7 @@ var listContainerObjectsCmd = &cobra.Command{
Run: func(cmd *cobra.Command, args []string) {
id := parseContainerID(cmd)
filters := new(object.SearchFilters)
filters := new(objectSDK.SearchFilters)
filters.AddRootFilter() // search only user created objects
cli := internalclient.GetSDKClientByFlag(cmd, key.GetOrGenerate(cmd), commonflags.RPC)
@ -51,7 +51,7 @@ var listContainerObjectsCmd = &cobra.Command{
prmSearch.SetContainerID(id)
prmSearch.SetFilters(*filters)
res, err := internalclient.SearchObjects(prmSearch)
res, err := internalclient.SearchObjects(cmd.Context(), prmSearch)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
objectIDs := res.IDList()
@ -65,13 +65,14 @@ var listContainerObjectsCmd = &cobra.Command{
addr.SetObject(objectIDs[i])
prmHead.SetAddress(addr)
resHead, err := internalclient.HeadObject(prmHead)
resHead, err := internalclient.HeadObject(cmd.Context(), prmHead)
if err == nil {
attrs := resHead.Header().Attributes()
for i := range attrs {
attrKey := attrs[i].Key()
if !strings.HasPrefix(attrKey, v2object.SysAttributePrefix) {
// FIXME(@cthulhu-rider): neofs-sdk-go#226 use dedicated method to skip system attributes
if !strings.HasPrefix(attrKey, v2object.SysAttributePrefix) && !strings.HasPrefix(attrKey, v2object.SysAttributePrefixNeoFS) {
// FIXME(@cthulhu-rider): https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/issues/97
// Use dedicated method to skip system attributes.
cmd.Printf(" %s: %s\n", attrKey, attrs[i].Value())
}
}

View file

@ -3,13 +3,13 @@ package container
import (
"crypto/sha256"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
containerAPI "github.com/TrueCloudLab/frostfs-sdk-go/container"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/TrueCloudLab/frostfs-sdk-go/netmap"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
containerAPI "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/spf13/cobra"
)
@ -31,7 +31,7 @@ var containerNodesCmd = &cobra.Command{
var prm internalclient.NetMapSnapshotPrm
prm.SetClient(cli)
resmap, err := internalclient.NetMapSnapshot(prm)
resmap, err := internalclient.NetMapSnapshot(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "unable to get netmap snapshot", err)
var id cid.ID

View file

@ -0,0 +1,233 @@
package container
import (
"bufio"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"os"
"strings"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
type policyPlaygroundREPL struct {
cmd *cobra.Command
args []string
nodes map[string]netmap.NodeInfo
}
func newPolicyPlaygroundREPL(cmd *cobra.Command, args []string) (*policyPlaygroundREPL, error) {
return &policyPlaygroundREPL{
cmd: cmd,
args: args,
nodes: map[string]netmap.NodeInfo{},
}, nil
}
func (repl *policyPlaygroundREPL) handleLs(args []string) error {
if len(args) > 0 {
return fmt.Errorf("too many arguments for command 'ls': got %d, want 0", len(args))
}
i := 1
for id, node := range repl.nodes {
var attrs []string
node.IterateAttributes(func(k, v string) {
attrs = append(attrs, fmt.Sprintf("%s:%q", k, v))
})
fmt.Printf("\t%2d: id=%s attrs={%v}\n", i, id, strings.Join(attrs, " "))
i++
}
return nil
}
func (repl *policyPlaygroundREPL) handleAdd(args []string) error {
if len(args) == 0 {
return fmt.Errorf("too few arguments for command 'add': got %d, want >0", len(args))
}
id := args[0]
key, err := hex.DecodeString(id)
if err != nil {
return fmt.Errorf("node id must be a hex string: got %q: %v", id, err)
}
node := repl.nodes[id]
node.SetPublicKey(key)
for _, attr := range args[1:] {
kv := strings.Split(attr, ":")
if len(kv) != 2 {
return fmt.Errorf("node attributes must be in the format 'KEY:VALUE': got %q", attr)
}
node.SetAttribute(kv[0], kv[1])
}
repl.nodes[id] = node
return nil
}
func (repl *policyPlaygroundREPL) handleLoad(args []string) error {
if len(args) != 1 {
return fmt.Errorf("too few arguments for command 'add': got %d, want 1", len(args))
}
jsonNetmap := map[string]map[string]string{}
b, err := os.ReadFile(args[0])
if err != nil {
return fmt.Errorf("reading netmap file %q: %v", args[0], err)
}
if err := json.Unmarshal(b, &jsonNetmap); err != nil {
return fmt.Errorf("decoding json netmap: %v", err)
}
repl.nodes = make(map[string]netmap.NodeInfo)
for id, attrs := range jsonNetmap {
key, err := hex.DecodeString(id)
if err != nil {
return fmt.Errorf("node id must be a hex string: got %q: %v", id, err)
}
node := repl.nodes[id]
node.SetPublicKey(key)
for k, v := range attrs {
node.SetAttribute(k, v)
}
repl.nodes[id] = node
}
return nil
}
func (repl *policyPlaygroundREPL) handleRemove(args []string) error {
if len(args) == 0 {
return fmt.Errorf("too few arguments for command 'remove': got %d, want >0", len(args))
}
id := args[0]
if _, exists := repl.nodes[id]; exists {
delete(repl.nodes, id)
return nil
}
return fmt.Errorf("node not found: id=%q", id)
}
func (repl *policyPlaygroundREPL) handleEval(args []string) error {
policyStr := strings.TrimSpace(strings.Join(args, " "))
var nodes [][]netmap.NodeInfo
nm := repl.netMap()
if strings.HasPrefix(policyStr, "CBF") || strings.HasPrefix(policyStr, "SELECT") || strings.HasPrefix(policyStr, "FILTER") {
// Assume that the input is a partial SELECT-FILTER expression.
// Full inline policies always start with UNIQUE or REP keywords,
// or different prefixes when it's the case of an external file.
sfExpr, err := netmap.DecodeSelectFilterString(policyStr)
if err != nil {
return fmt.Errorf("parsing select-filter expression: %v", err)
}
nodes, err = nm.SelectFilterNodes(sfExpr)
if err != nil {
return fmt.Errorf("building select-filter nodes: %v", err)
}
} else {
// Assume that the input is a full policy or input file otherwise.
placementPolicy, err := parseContainerPolicy(repl.cmd, policyStr)
if err != nil {
return fmt.Errorf("parsing placement policy: %v", err)
}
nodes, err = nm.ContainerNodes(*placementPolicy, nil)
if err != nil {
return fmt.Errorf("building container nodes: %v", err)
}
}
for i, ns := range nodes {
var ids []string
for _, node := range ns {
ids = append(ids, hex.EncodeToString(node.PublicKey()))
}
fmt.Printf("\t%2d: %v\n", i+1, ids)
}
return nil
}
func (repl *policyPlaygroundREPL) netMap() netmap.NetMap {
var nm netmap.NetMap
var nodes []netmap.NodeInfo
for _, node := range repl.nodes {
nodes = append(nodes, node)
}
nm.SetNodes(nodes)
return nm
}
func (repl *policyPlaygroundREPL) run() error {
if len(viper.GetString(commonflags.RPC)) > 0 {
key := key.GetOrGenerate(repl.cmd)
cli := internalclient.GetSDKClientByFlag(repl.cmd, key, commonflags.RPC)
var prm internalclient.NetMapSnapshotPrm
prm.SetClient(cli)
resp, err := internalclient.NetMapSnapshot(repl.cmd.Context(), prm)
commonCmd.ExitOnErr(repl.cmd, "unable to get netmap snapshot to populate initial netmap: %w", err)
for _, node := range resp.NetMap().Nodes() {
id := hex.EncodeToString(node.PublicKey())
repl.nodes[id] = node
}
}
cmdHandlers := map[string]func([]string) error{
"list": repl.handleLs,
"ls": repl.handleLs,
"add": repl.handleAdd,
"load": repl.handleLoad,
"remove": repl.handleRemove,
"rm": repl.handleRemove,
"eval": repl.handleEval,
}
for reader := bufio.NewReader(os.Stdin); ; {
fmt.Print("> ")
line, err := reader.ReadString('\n')
if err != nil {
if err == io.EOF {
return nil
}
return fmt.Errorf("reading line: %v", err)
}
parts := strings.Fields(line)
if len(parts) == 0 {
continue
}
cmd := parts[0]
handler, exists := cmdHandlers[cmd]
if exists {
if err := handler(parts[1:]); err != nil {
fmt.Printf("error: %v\n", err)
}
} else {
fmt.Printf("error: unknown command %q\n", cmd)
}
}
}
var policyPlaygroundCmd = &cobra.Command{
Use: "policy-playground",
Short: "A REPL for testing placement policies",
Long: `A REPL for testing placement policies.
If a wallet and endpoint is provided, the initial netmap data will be loaded from the snapshot of the node. Otherwise, an empty playground is created.`,
Run: func(cmd *cobra.Command, args []string) {
repl, err := newPolicyPlaygroundREPL(cmd, args)
commonCmd.ExitOnErr(cmd, "could not create policy playground: %w", err)
commonCmd.ExitOnErr(cmd, "policy playground failed: %w", repl.run())
},
}
func initContainerPolicyPlaygroundCmd() {
commonflags.Init(policyPlaygroundCmd)
}

View file

@ -1,7 +1,8 @@
package container
import (
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/spf13/cobra"
)
@ -15,7 +16,9 @@ var Cmd = &cobra.Command{
// the viper before execution
commonflags.Bind(cmd)
commonflags.BindAPI(cmd)
common.StartClientCommandSpan(cmd)
},
PersistentPostRun: common.StopClientCommandSpan,
}
func init() {
@ -28,6 +31,7 @@ func init() {
getExtendedACLCmd,
setExtendedACLCmd,
containerNodesCmd,
policyPlaygroundCmd,
}
Cmd.AddCommand(containerChildCommand...)
@ -40,6 +44,7 @@ func init() {
initContainerGetEACLCmd()
initContainerSetEACLCmd()
initContainerNodesCmd()
initContainerPolicyPlaygroundCmd()
for _, containerCommand := range containerChildCommand {
commonflags.InitAPI(containerCommand)

View file

@ -5,11 +5,12 @@ import (
"errors"
"time"
internalclient "github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
)
@ -38,7 +39,7 @@ Container ID in EACL table will be substituted with ID from the CLI.`,
if !flagVarsSetEACL.noPreCheck {
cmd.Println("Checking the ability to modify access rights in the container...")
extendable, err := internalclient.IsACLExtendable(cli, id)
extendable, err := internalclient.IsACLExtendable(cmd.Context(), cli, id)
commonCmd.ExitOnErr(cmd, "Extensibility check failure: %w", err)
if !extendable {
@ -48,15 +49,15 @@ Container ID in EACL table will be substituted with ID from the CLI.`,
cmd.Println("ACL extension is enabled in the container, continue processing.")
}
var setEACLPrm internalclient.SetEACLPrm
setEACLPrm.SetClient(cli)
setEACLPrm.SetTable(*eaclTable)
if tok != nil {
setEACLPrm.WithinSession(*tok)
setEACLPrm := internalclient.SetEACLPrm{
Client: cli,
ClientParams: client.PrmContainerSetEACL{
Table: eaclTable,
Session: tok,
},
}
_, err := internalclient.SetEACL(setEACLPrm)
_, err := internalclient.SetEACL(cmd.Context(), setEACLPrm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
if containerAwait {
@ -65,14 +66,17 @@ Container ID in EACL table will be substituted with ID from the CLI.`,
cmd.Println("awaiting...")
var getEACLPrm internalclient.EACLPrm
getEACLPrm.SetClient(cli)
getEACLPrm.SetContainer(id)
getEACLPrm := internalclient.EACLPrm{
Client: cli,
ClientParams: client.PrmContainerEACL{
ContainerID: &id,
},
}
for i := 0; i < awaitTimeout; i++ {
time.Sleep(1 * time.Second)
res, err := internalclient.EACL(getEACLPrm)
res, err := internalclient.EACL(cmd.Context(), getEACLPrm)
if err == nil {
// compare binary values because EACL could have been set already
table := res.EACL()

View file

@ -3,11 +3,11 @@ package container
import (
"errors"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/TrueCloudLab/frostfs-sdk-go/session"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
"github.com/spf13/cobra"
)

View file

@ -0,0 +1,53 @@
package control
import (
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
"github.com/spf13/cobra"
)
const (
concurrencyFlag = "concurrency"
removeDuplicatesFlag = "remove-duplicates"
)
var doctorCmd = &cobra.Command{
Use: "doctor",
Short: "Restructure node's storage",
Long: "Restructure node's storage",
Run: doctor,
}
func doctor(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd)
req := &control.DoctorRequest{Body: new(control.DoctorRequest_Body)}
req.Body.Concurrency, _ = cmd.Flags().GetUint32(concurrencyFlag)
req.Body.RemoveDuplicates, _ = cmd.Flags().GetBool(removeDuplicatesFlag)
signRequest(cmd, pk, req)
cli := getClient(cmd, pk)
var resp *control.DoctorResponse
var err error
err = cli.ExecRaw(func(client *client.Client) error {
resp, err = control.Doctor(client, req)
return err
})
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
verifyResponse(cmd, resp.GetSignature(), resp.GetBody())
cmd.Println("Operation has finished.")
}
func initControlDoctorCmd() {
initControlFlags(doctorCmd)
ff := doctorCmd.Flags()
ff.Uint32(concurrencyFlag, 0, "Number of parallel threads to use")
ff.Bool(removeDuplicatesFlag, false, "Remove duplicate objects")
}

View file

@ -1,10 +1,10 @@
package control
import (
rawclient "github.com/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/services/control"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
"github.com/spf13/cobra"
)

View file

@ -1,18 +1,21 @@
package control
import (
"github.com/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"github.com/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "github.com/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/TrueCloudLab/frostfs-node/pkg/services/control"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
"github.com/spf13/cobra"
)
const ignoreErrorsFlag = "no-errors"
var evacuateShardCmd = &cobra.Command{
Use: "evacuate",
Short: "Evacuate objects from shard",
Long: "Evacuate objects from shard to other shards",
Run: evacuateShard,
Deprecated: "use frostfs-cli control shards evacuation start",
}
func evacuateShard(cmd *cobra.Command, _ []string) {
@ -20,7 +23,7 @@ func evacuateShard(cmd *cobra.Command, _ []string) {
req := &control.EvacuateShardRequest{Body: new(control.EvacuateShardRequest_Body)}
req.Body.Shard_ID = getShardIDList(cmd)
req.Body.IgnoreErrors, _ = cmd.Flags().GetBool(dumpIgnoreErrorsFlag)
req.Body.IgnoreErrors, _ = cmd.Flags().GetBool(ignoreErrorsFlag)
signRequest(cmd, pk, req)
@ -47,7 +50,7 @@ func initControlEvacuateShardCmd() {
flags := evacuateShardCmd.Flags()
flags.StringSlice(shardIDFlag, nil, "List of shard IDs in base58 encoding")
flags.Bool(shardAllFlag, false, "Process all shards")
flags.Bool(dumpIgnoreErrorsFlag, false, "Skip invalid/unreadable objects")
flags.Bool(ignoreErrorsFlag, false, "Skip invalid/unreadable objects")
evacuateShardCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag)
}

Some files were not shown because too many files have changed in this diff Show more