Compare commits

...

392 commits

Author SHA1 Message Date
148d68933b [#1573] node: Simplify bootstrapWithState()
After #1382 we have no need to use lambdas.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-20 08:17:05 +00:00
51ee132ea3
[#1342] network/cache: Add node address to error multiClient
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-12-18 19:27:35 +03:00
226dd25dd0 [#1568] pilorama: Replace "containerID" with "container ID" in the error message
It is "container ID" in every other place.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-18 15:52:26 +00:00
bd0197eaa8 [#1568] storage: Remove "could not/can't/failed to" from error messages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-18 15:52:26 +00:00
e44b84c18c
[#1569] cli: Remove unnecessary variable after refactoring
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-18 10:17:04 +03:00
bed49e6ace
[#1569] cli: Make --range flag required in object hash
Previously, `object head` was used if no range was provided.
This is wrong on multiple levels:
1. We print an error if the checksum is missing in header,
   even though taking hash is possible.
2. We silently ignore --salt parameter.
3. `--range` is required for Object.RANGEHASH RPC, custom logic for one
   specific usecase has no value.

So we make it required and make CLI command follow more closely
the FrostFS API.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-18 10:17:04 +03:00
df05057ed4 [#1452] container: Add ListStream method
* Added new method for listing containers to container service.
  It opens stream and sends containers in batches.

* Added TransportSplitter wrapper around ExecutionService to
  split container ID list read from contract in parts that are
  smaller than grpc max message size. Batch size can be changed
  in node configuration file (as in example config file).

* Changed `container list` implementaion in cli: now ListStream
  is called by default. Old List is called only if ListStream
  is not implemented.

* Changed `internalclient.ListContainersPrm`.`Account` to
  `OwnerID` since `client.PrmContainerList`.`Account` was
  renamed to `OwnerID` in sdk.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-12-17 16:22:43 +03:00
b6c8ebf493 [#1453] container: Replace sort.Slice with slices.SortFunc
* Replaced `sort.Slice` with `slices.SortFunc` in
  `ListContainersRes.SortedIDList()` as it is a bit faster,
  according to 15102e6dfd.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-12-17 13:33:43 +03:00
6e82661c35 [#1563] tree: Wrap only ChainRouterError erros with ObjectAccessDenied
* Such wrapping helps to differentiate logical check errors and server internal
  errors.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-12-16 15:16:07 +03:00
1a091ea7bb [#1563] object: Wrap only ChainRouterError erros with ObjectAccessDenied
* Such wrapping helps to differentiate logical check errors and server internal
  errors.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-12-16 15:15:25 +03:00
7ac3542714 [#1563] ape: Introduce ChainRouterError error type
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-12-16 15:12:30 +03:00
f0c43c8d80
[#1502] Use zap.Error for logging errors
Use `zap.Error` instead of `zap.String` for logging errors: change all expressions like
`zap.String("error", err.Error())` or `zap.String("err", err.Error())` to `zap.Error(err)`.
Leave similar expressions with other messages unchanged, for example,
`zap.String("last_error", lastErr.Error())` or `zap.String("reason", ctx.Err().Error())`.

This change was made by applying the following patch:
```diff
@@
var err expression
@@
-zap.String("error", err.Error())
+zap.Error(err)

@@
var err expression
@@
-zap.String("err", err.Error())
+zap.Error(err)
```

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-16 11:13:42 +03:00
8ba9f31fca
[#1510] metabase/test: Fix BenchmarkListWithCursor
- Fix misplaced `(*DB).Close` (broken after 47dcfa20f3)
- Use `errors.Is` for error checking (broken after fcdbf5e509)

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-13 13:19:15 +03:00
2af3409d39
[#1510] metabase/test: Fix BenchmarkGet
Fix misplaced `(*DB).Close` (broken after 47dcfa20f3)

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-13 13:18:43 +03:00
d165ac042c
[#1558] morph/client: Reuse notary rpcclient wrapper
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-12 15:30:12 +03:00
7151c71d51
[#1558] morph/client: Remove "could not"/"can't"/"failed to" from error messages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-12 15:30:12 +03:00
91d9dc2676
[#1558] morph/event: Remove "could not" from error messages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-12 15:30:12 +03:00
7853dbc315 [#1557] morph/event: Remove embedded structs from scriptHashWithValue
Also, make them public, because otherwise `unused` linter complains.
```
pkg/morph/event/utils.go:25:2  unused  field `typ` is unused
```
This complain is wrong, though: we _use_ `typ` field because the whole
struct is used as a map key.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-12 11:55:09 +00:00
3821645085
[#1555] engine: Refactor (*StorageEngine).GetLocks
Refactored after renaming the method to replace the confusing `locked`
variable with `locks`.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-11 15:06:38 +03:00
72470d6b48
[#1555] local_object_storage: Rename method GetLocked -> GetLocks
Renamed to better reflect the method's purpose of returning locks
for the specified object.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-11 15:06:37 +03:00
e9837bbcf9 [#1554] morph/event: Remove unused AlphabetUpdate event
Refs TrueCloudLab/frostfs-contract#138.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-11 12:01:17 +00:00
a641c91594 [#1550] Add CODEOWNERS
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2024-12-11 10:34:57 +00:00
b1614a284d [#1546] morph/event: Export NotificationHandlerInfo fields
Hiding them achieves nothing, as the struct has no methods and is not
used concurrently.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-11 07:39:49 +00:00
d0ce835fbf [#1546] morph/event: Merge notification parser and handlers
They are decoupled, but it is an error to have a handler without a
corresponding parser. Register them together on the code level and get
rid of unreachable code.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-11 07:39:49 +00:00
dfa51048a8 [#1546] morph/event: Remove "is started" checks from event handler registrar
This codepath hides possible bugs in code.
All initialization function should run before init stage.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-11 07:39:49 +00:00
670305a721 [#1546] morph/event: Remove nil checks from event handler registrar
This codepath hides possible bugs in code.
We would rather panic then silently fail.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-11 07:39:49 +00:00
1f6cf57e30 [#1548] metabase: Check if EC parent is removed or expired
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-12-11 07:26:33 +00:00
386a12eea4 [#1548] engine: Rename parent -> ecParent
Parent could mean split parent or EC parent. In this case it is EC parent only.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-12-11 07:26:33 +00:00
15139d80c9 [#1548] policer: Do not replicate EC chunk if object already removed
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-12-11 07:26:33 +00:00
41da27dad5
[#1549] engine: Drop Async flag from evacuation parameters
Now it is only async evacuation.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-12-10 17:00:00 +03:00
ac0511d214
[#1549] controlSvc: Drop deprecated EvacuateShard rpc
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-12-10 16:59:52 +03:00
7e542906ef [#1539] go.mod: Bump frostfs-sdk-go version
* Also fix placement unit-test in object manager

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-12-06 15:29:37 +03:00
d1bc4351c3
[#1545] morph/event: Simplify frostfs contract event parsing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-06 14:00:23 +03:00
1c12f23b84 [#1541] morph/event: Simplify netmap contract event parsing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-06 10:26:39 +00:00
a353d45742 [#1541] morph/event: Simplify container contract event parsing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-06 10:26:39 +00:00
d5c46d812a [#1541] go.mod: Update frostfs-contract
New version contains more idiomatic types in the auto-generated code.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-06 10:26:39 +00:00
d5d5ce2074 [#1541] morph/event: Simplify balance contract event parsing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-06 10:26:39 +00:00
7df3520d48 [#1540] getSvc: Drop redundant returns
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-12-05 12:39:49 +00:00
5fe78e51d1 [#1540] getSvc: Do not log context canceled errors during EC assemble
Those errors are fired when it is enough chunks retrieved and error group
cancels other requests.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-12-05 12:39:49 +00:00
84b4051b4d
[#1538] morph/container: Make opts struct similar to that of other contracts
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-04 15:30:58 +03:00
6a51086030
[#1538] morph/client: Remove TryNotary() option from side-chain contracts
The notary is always enabled and this option does always work.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-04 15:30:58 +03:00
5c3b2d95ba
[#1538] node: Assume notary is enabled
Notaryless environments are not tested at all since a while.
We use neo-go only and it has notary contract enabled.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-04 15:30:58 +03:00
2d5d4093be
[#1537] morph: Use (user.ID).ScriptHash() where possible
Pick up changes from TrueCloudLab/frostfs-sdk-go#198.

gopatch:
```
@@
var user expression
@@
-address.StringToUint160(user.EncodeToString())
+user.ScriptHash()
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-04 13:25:44 +03:00
e3487d5af5 [#1535] morph: Unify test invoke error messages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-04 09:50:20 +00:00
e37dcdf88b [#1535] morph/netmap: Unify error messages for config retrieval
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-04 09:50:20 +00:00
6c679d1535 [#1535] morph: Unify client creation error messages
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-04 09:50:20 +00:00
281d65435e
[#1450] engine: Group object by shard before Inhume
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/engine
cpu: 12th Gen Intel(R) Core(TM) i5-1235U
                                 │   old.txt    │              new.txt                │
                                 │    sec/op    │   sec/op     vs base                │
InhumeMultipart/objects=1-12        11.42m ± 1%   10.71m ± 0%   -6.27% (p=0.000 n=10)
InhumeMultipart/objects=10-12       113.5m ± 0%   100.9m ± 3%  -11.08% (p=0.000 n=10)
InhumeMultipart/objects=100-12     1135.4m ± 1%   681.3m ± 2%  -40.00% (p=0.000 n=10)
InhumeMultipart/objects=1000-12     11.358 ± 0%    1.089 ± 1%  -90.41% (p=0.000 n=10)
InhumeMultipart/objects=10000-12   113.251 ± 0%    1.645 ± 1%  -98.55% (p=0.000 n=10)
geomean                              1.136        265.5m       -76.63%
```

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-04 10:09:00 +03:00
b348b20289
[#1450] engine: Add benchmark for Inhume operation
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-04 10:08:34 +03:00
748edd1999
[#1450] engine: Return shard-level error if object is expired on inhume
Since we have errors defined on the shard-level, it looks strage that we
check an error againt the shard-level error `ErrLockObjectRemoval`, but
then return the metabase-level error. Let's return the same shard-level
error instead.

Since we have errors defined on the shard-level

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-04 10:06:57 +03:00
47dfd8840c [#1532] node: Allow to omit metabase.path if shard is disabled
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-12-04 03:30:19 +00:00
432042c534
[#1527] engine: Add tests for handling expired objects on inhume and lock
Currently, it's allowed to inhume or lock an expired object.
Consider the following scenario:

1) An user inhumes or locks an object
2) The object expires
3) GC hasn't yet deleted the object
4) The node loses the associated tombstone or lock
5) Another node replicates tombstone or lock to the first node

In this case, the second node succeeds, which is the desired behavior.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-03 12:29:45 +03:00
9cabca9dfe
[#1527] engine/test: Move default metabase options to separate function
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-02 16:13:37 +03:00
60feed3b5f
[#1527] engine/test: Allow to specify current epoch in epochState
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-12-02 15:37:25 +03:00
635a292ae4 [#1528] cli: Keep order for required nodes in the result of object nodes
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-12-02 13:58:24 +03:00
edfa3f4825 [#1528] node: Keep order for equal elements when sort priority metrics
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-12-02 13:58:19 +03:00
e0ac3a583f [#1523] metabase: Remove (*DB).IterateCoveredByTombstones
Remove this method because it isn't used anywhere since 7799f8e4c.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-11-29 10:49:24 +00:00
00c608c05e [#1524] tree: Make check APE error get wrapped to api status
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-29 10:48:16 +00:00
bba1892fa1 [#1524] ape: Make APE checker return error without status
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-29 10:48:16 +00:00
01acec708f
[#1525] pilorama: Use AppendUint* helpers from stdlib
gopatch:
```
@@
var slice, e expression
@@
+import "encoding/binary"

-append(slice, byte(e), byte(e >> 8))
+binary.LittleEndian.AppendUint16(slice, e)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-28 09:40:20 +03:00
aac65001e5 [#1522] adm/frostfsid: Remove unreachable condition
SendConsensusTx() modifies SendTxs field, if it is not the case, there
is a bug in code.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-26 08:13:35 +00:00
1170370753 [#1522] adm/helper: Rename createSingleAccounts() -> getSingleAccounts()
It doesn't create any accounts, purely finds them in the wallet.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-26 08:13:35 +00:00
9e275d44c8 [#1522] adm/helper: Unexport DefaultClientContext()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-26 08:13:35 +00:00
2469e0c683 [#1522] adm/helper: Remove NewActor() helper
It is used once, it is used only internally and it is single-statement.
I see no justification in having it as a separate function.
It introduces confusion, because we also have NewLocalActor().

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-26 08:13:35 +00:00
a6ef4ab524 [#1522] adm/helper: Rename GetN3Client() -> NewRemoteClient()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-26 08:13:35 +00:00
49959c4166 [#1522] adm/helper: Unexport GetFrostfsIDAdmin()
It is used in `helper` package only, besides unit-tests.
Move unit-tests to the same package, where they belong.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-26 08:13:35 +00:00
61ee1b5610 [#1522] adm: Simplify LocalClient.SendRawTransaction()
The old code was there before Copy() method was introduced.
It was also supposed to check errors, however, they are already checked
server-side.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-26 08:13:35 +00:00
b10c954377 [#1522] adm: Split NewLocalClient() into functions
No functional changes.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-26 08:13:35 +00:00
1605391628 [#1522] adm/helper: Simplify Client interface
Just reuse `actor.RPCActor`. No functional changes.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-26 08:13:35 +00:00
b1766e47c7 [#1522] adm/helper: Remove unused GetCommittee() method from the Client interface
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-26 08:13:35 +00:00
caa4253249 [#1522] adm: Remove unnecessary variable declaration
It is better to have small scope.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-26 08:13:35 +00:00
7eac5fb18b
Release v0.44.0
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-25 14:41:36 +03:00
0e5524dac7 [#1515] adm: Print address in base58 format in morph ape get-admin
Signed-off-by: George Bartolomey <george@bh4.ru>
2024-11-25 10:38:05 +00:00
3ebd560f42 [#1519] cli: Make descriptive help for--rule option
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-25 07:21:05 +00:00
1ed7ab75fb [#1517] cli: Print the reason of ape manager error
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-22 15:24:32 +03:00
99f9e59de9 [#1514] adm: Remove --alphabet-wallets flag from readonly commands
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-21 14:37:34 +00:00
256f96e252 [#1514] adm/nns: Rename getRPCClient() to nnsWriter()
Make it more specific and similar to nnsReader().

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-21 14:37:34 +00:00
e5ea95c045 [#1514] adm/nns: Do not return hash from getRPCClient()
It was unused and we employ better abstractions now.
gopatch:
```
@@
var a, b expression
@@
-a, b, _ := getRPCClient(...)
+a, b := getRPCClient(...)
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-21 14:37:34 +00:00
9073e555db [#1514] adm/nns: Do not create actor for readonly commands
`nns get-records` and `nns tokens` command do not need to sign anything,
so remove useless actor and use invoker directly.
`NewLocalActor()` is only used in `ape` and `nns` packages. `ape`
package seem to use it correctly, only when alphabet wallets are
provided, so no changes there.
Also, remove --alphabet-wallets flag from commands that do not need it.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-21 14:37:34 +00:00
2771fdb8c7 [#1514] adm/nns: Use nns.GetAllRecords() wrapper
It was not possible previously, because GetAllRecords() was not declared
safe in frostfs-contract.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-21 14:37:34 +00:00
efa4ce00b8 [#1514] go.mod: Update frostfs-contract to v0.21.0-rc.3
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-21 14:37:34 +00:00
f12f04199e [#1516] traverser: Check for placement vector out of range
Placement vector may contain fewer nodes count than it required by policy
due to the outage of the one of the node.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-11-21 14:18:55 +03:00
2e2c62147d
[#1513] adm: Move ProtoConfigPath from constants to commonflags package
Refs #932

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-21 09:39:06 +03:00
6ae8667fb4
[#1509] .forgejo: Run actions on push to master
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-20 11:42:12 +03:00
49a4e727fd [#1507] timer/test: Use const for constants
Make it easy to see what the test is about.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-20 08:36:25 +00:00
2e974f734c [#1507] timer/test: Improve test coverage
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-20 08:36:25 +00:00
3042490340 [#1507] timer: Remove unused OnDelta() method
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-20 08:36:25 +00:00
a339b52a60 [#1501] adm: Refactor APE-chains managing subcommands
* Use `cmd/internal/common/ape` parser commands within `ape`
  subcommands
* Use flag names from `cmd/internal/common/ape

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-20 07:58:32 +00:00
4ab4ed6f96 [#1501] cli: Refactor bearer subcommand
* Use `cmd/internal/common/ape` parser commands within `generate-ape-override`
  subcommand
* Use flag names from `cmd/internal/common/ape`

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-20 07:58:32 +00:00
3b1364e4cf [#1501] cli: Refactor ape-manager subcommands
* Refactor ape-manager subcommands
* Use `cmd/internal/common/ape` parser commands within ape-manager subcommands
* Use flag names from `cmd/internal/common/ape`

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-20 07:58:32 +00:00
daff77b273 [#1501] cli: Refactor local override managing subcommands
* Refactor local override managing subcommands
* Use `cmd/internal/common/ape` parser commands within local
  override subcommands
* Use flag names from `cmd/internal/common/ape`

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-20 07:58:32 +00:00
7a7ee71a4d [#1501] cmd: Introduce common APE-chain parser commands
* Introduce common parsing commands to use them in `frostfs-cli`
  and `frostfs-adm` APE-related subcommands
* Introduce common flags for these parsing commands

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-20 07:58:32 +00:00
ffe9906266 [#1501] cli: Move APE-chain parser methods to pkg/util
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-20 07:58:32 +00:00
ae31ef3602 [#1501] cli: Move PrintHumanReadableAPEChain to a common package
* Both `frostfs-cli` and `frostfs-adm` APE-related subcommands use
  `PrintHumanReadableAPEChain` to print a parsed APE-chain. So, it's
  more correct to have it in a common package over `frostfs-cli` and
  `frostfs-adm` folders.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-20 07:58:32 +00:00
e2cb0640f1 [#1501] util: Move eACL-to-APE converter to pkg/util
* `ConvertEACLToAPE` is useful method which couldn't be imported
  out of frostfs-node so far as it has been in `internal`
* Since `ConvertEACLToAPE` and related structures and unit-tests
  are placed in `pkg/util`

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-20 07:58:32 +00:00
9f4ce600ac
[#1505] adm: Allow to manage additional keys in frostfsid
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-19 16:25:16 +03:00
d82f0d1926
[#1496] node/control: Await until SetNetmapStatus() persists
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-15 16:36:07 +03:00
acd5babd86
[#1496] morph: Merge InvokeRes and WaitParams
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-15 16:36:07 +03:00
b65874d1c3
[#1496] morph: Return InvokeRes from all invoke*() methods
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-15 16:36:07 +03:00
69c63006da
[#1496] morph: Move tx waiter to morph package
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-15 16:36:07 +03:00
d77a218f7c [#1493] metabase: Merge Inhume() and DropGraves() for tombstones
DropGraves() is only used to drop gravemarks after a tombstone
removal. Thus, it makes sense to do Inhume() and DropGraves() in one
transaction. It has less overhead and no unexpected problems in case
of sudden power failure.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-14 06:47:04 +00:00
44df67492f [#1493] metabase: Split inhumeTx() into 2 functions
No functional changes.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-14 06:47:04 +00:00
1e6f132b4e [#1493] metabase: Pass InhumePrm by value
Unify with the other code, no functional changes.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-14 06:47:04 +00:00
6dc0dc6691 [#1493] shard: Take mode mutex in HandleExpiredTombstones()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-14 06:47:04 +00:00
f7cb6b4d87 [#1482] Makefile: Update golangci-lint
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-11-13 15:01:41 +00:00
7fc6101bec
[#1491] engine/test: Rework engine test utils
- Remove `testNewShard` and `setInitializedShards` because they
violated the default engine workflow. The correct workflow is:
first use `New()`, followed by `Open()`, and then `Init()`. As a
result, adding new logic to `(*StorageEngine).Init` caused several
tests to fail with a panic when attempting to access uninitialized
resources. Now, all engines created with the test utils must be
initialized manually. The new helper method `prepare` can be used
for that purpose.
- Additionally, `setInitializedShards` hardcoded the shard worker
pool size, which prevented it from being configured in tests and
benchmarks. This has been fixed as well.
- Ensure engine initialization is done wherever it was missing.
- Refactor `setShardsNumOpts`, `setShardsNumAdditionalOpts`, and
`setShardsNum`. Make them all depend on `setShardsNumOpts`.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-11-13 14:42:53 +03:00
7ef36749d0
[#1491] engine/test: Move BenchmarkExists to exists_test.go
Move `BenchmarkExists` from `engine_test.go` to `exists_test.go`
for better organization and clarity.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-11-13 14:09:29 +03:00
c6066d6ee4
[#1491] engine/test: Use more suitable testing utils here and there
Use `setShardsNum` instead of `setInitializedShards` wherever possible.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-11-13 14:09:29 +03:00
612b34d570
[#1437] logger: Add caller skip to log original caller position
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:12 +03:00
7429553266
[#1437] node: Fix contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:10 +03:00
6921a89061
[#1437] ir: Fix contextcheck linters
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:10 +03:00
16598553d9
[#1437] shard: Fix contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:09 +03:00
c139892117
[#1437] ir: Fix contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:09 +03:00
62b5181618
[#1437] blobovnicza: Fix contextcheck linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:08 +03:00
6db46257c0
[#1437] node: Use ctx for logging
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-13 10:36:07 +03:00
c16dae8b4d
[#1437] logger: Use context to log trace id
Signed-off-by: Dmitrii Stepanov
2024-11-13 10:36:07 +03:00
fd004add00 [#1492] metabase: Fix import formatting
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-13 07:30:25 +00:00
8ed7a676d5 [#1492] metabase: Ensure Unmarshal() is called on a cloned slice
The slice returned from bucket.Get() is only valid during the tx
lifetime. Cloning it is not necessary everywhere, but better safe than
sorry.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-13 07:30:25 +00:00
b451de94c8 [#1492] metabase: Fix typo in objData
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-13 07:30:25 +00:00
f1556e3c42
[#1488] Makefile: Drop all containers created on env-up
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-12 17:24:44 +03:00
e122ff6013
[#1488] dev: Add Jaeger image and enable tracing on debug
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-12 17:24:44 +03:00
e2658c7519
[#1488] tracing: KV attributes for spans from config
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-12 17:24:43 +03:00
c00f4bab18
[#1488] go.mod: Bump observability version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-12 17:24:43 +03:00
46fef276b4 [#1449] tree: Log tree sync with Info level
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-12 12:11:07 +00:00
9bd05e94c8 [#1449] tree: Add ApplyBatch method
Concurrent Apply can lead to child node applies before parent, so
undo/redo operations will perform. This leads to performance degradation
in case of tree with many sublevels.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-12 12:11:07 +00:00
16830033f8 [#1483] cli: Remove --basic-acl flag
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-12 12:10:51 +00:00
1cf51a8079 [#1483] cli/docs: Remove set-eacl mention
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-12 12:10:51 +00:00
3324c26fd8 [#1483] morph: Remove container.GetEACL()
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-12 12:10:51 +00:00
a692298533 [#1483] node: Remove eACL cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-12 12:10:51 +00:00
be2753de00 [#1490] docs: Update description for object.get.priority
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-11-12 13:35:13 +03:00
b543569c3f [#1486] node: Introduce dual service support
* Register GRPC services for both neo.fs.v2 and frost.fs namespaces
* Use this temporary solution until all nodes are updated

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-12 08:43:47 +00:00
80f8a8fd3a
[#1396] cli/playground: Refactor
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-11-12 11:09:27 +03:00
2f3bc6eb84
[#1396] cli/playground: Improve terminal control key handling
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-11-12 11:07:48 +03:00
8a57c78f5f
[#1484] engine: Fix engine metrics
1. Add forgotten metrics for client requests
2. Include execIfNotBlocked into metrics

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-11 12:59:20 +03:00
ad01fb958a [#1474] Stop using obsolete .github directory
This commit is a part of multi-repo cleanup effort:
TrueCloudLab/frostfs-infra#136

Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2024-11-11 06:44:13 +00:00
d336f2d487 [#1393] adm: Make NewLocalActor receive accout name
* Some RPC-clients for contracts require different wallet account types.
  Since, `Policy` contract gets `consensus` accounts while `NNS` gets
  `committee` accounts.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-08 13:57:51 +00:00
c82c753e9f [#1480] objsvc: Remove useless stream wrappers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-08 12:01:14 +00:00
f666898e5d [#1480] objsvc: Remove EACL checks
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-08 12:01:14 +00:00
b1a31281e4 [#1480] ape: Remove SoftAPECheck flag
Previous release was EACL-compatible.
Starting from now all EACL should've been migrated to APE chains.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-08 12:01:14 +00:00
764450d04a [#1479] tree: Regenerate service protobufs
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-08 10:43:44 +03:00
755cae3f19 [#1479] control: Regenerate protobufs for service
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-08 10:43:44 +03:00
9b13a18aac [#1479] go.mod: Bump frostfs-sdk-go version
* Update version within go.mod;
* Fix deprecated frostfs-api-go/v2 package and use frostfs-sdk-go/api
  instead.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-11-08 10:43:19 +03:00
ef64930fef
[#1477] ape: Fix EC chunk test
Initially, this test was a check that only the container node can
assemble an EC object. But the implementation of this test was wrong.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-07 16:00:20 +03:00
c8fb154151
[#1475] Remove container estimation code
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-06 15:32:33 +03:00
7edec9193c [#1451] placement: Return copy of slice from container nodes cache
Nodes from cache could be changed by traverser, if no objectID specified.
So it is required to return copy of cache's slice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
3cf6ea745d [#1451] ec: Check all parts are saved
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
9a77527f46 [#1451] ape: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
5b1ba8e23d [#1451] ape: Perform strict APE checks for EC parts
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
9902965ff4 [#1451] writer: Sign EC parts with node's private key
As EC put request may be processed only by container node, so sign requests
with current node private to not to perform APE checks.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
33ad753302 [#1473] policer: Add tracing span
To filter HEAD requests from policer.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:15:11 +00:00
15102e6dfd [#1471] Replace sort.Slice in some places
`slices.SortFunc` doesn't use reflection and is a bit faster.
I have done some micro-benchmarks for `[]NodeInfo`:
```
$ benchstat -col "/func" out
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/pilorama
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
       │ sort.Slice  │           slices.SortFunc           │
       │   sec/op    │   sec/op     vs base                │
Sort-8   2.130µ ± 2%   1.253µ ± 2%  -41.20% (p=0.000 n=10)
```

Haven't included them, though, as they I don't see them being used a
lot.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-06 08:07:32 +00:00
17ec84151b
[#532] cli: Respect XDG base directory spec
XDG base directory specification defines where various files
should be looked by an application. Hopefully, this makes `frostfs-cli`
more predictable and pleasant to work with. Luckily for us, golang already
has everything we need in the stdlib. This commit also gets rid of
`github.com/mitchellh/go-homedir` dependency.

Close #532
Refs #1455

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-05 16:27:18 +03:00
6c45a17af6
[#1467] node: Break notary deposit wait after VUB
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-01 16:42:02 +03:00
d19ab43500
[#1462] node: Add off-cpu profiler
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-31 11:32:13 +03:00
5bcf81d1cc
[#1466] Remove woodpecker CI
We use forgejo actions now.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-31 10:07:33 +03:00
c2effcc61c [#1465] Makefile: Update golangci-lint, fix warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-31 06:39:59 +00:00
2285cfc36f
[#1464] frostfsid: Cache subject not found error
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-30 18:27:33 +03:00
e74d05c03f
[#1464] frostfsid: Add cache metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-30 18:27:32 +03:00
48862e0e63 [#1459] .golanci.yml: Add tenv linter, fix issues
Refs #1309

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-30 15:18:22 +00:00
89892d9754 [#1459] cli: Simplify slice append
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-30 15:18:22 +00:00
7ac0852364 [#1459] .golangci.yml: Add intrange linter, fix issues
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-30 15:18:22 +00:00
d28a5d2d7a
[#1448] container/ape: Ignore an error when getting a role
When getting a role in the APE checker for the container services,
an error may be returned if network maps of the previous two epochs
don't have enough nodes to fulfil a container placement policy.
It's a logical error, so we should ignore it.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-30 12:51:57 +03:00
87ac3c5279 [#1458] object: Make patch not set key before target construction
* `SignRequestPrivateKey` field should be initialized either within
  `newUntrustedTarget` or within `newTrustedTarget`. Otherwise, all
  requests are signed by local node key that makes impossible to perform
  patch on non-container node.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-10-29 15:20:28 +00:00
d5ee6d3039
[#1456] morph: Use DialerSource interface instead of internal struct
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-29 17:34:14 +03:00
433aab12bb
[#1455] cli: Handle missing home directory
go-homedir library incorrectly handles some of the errors
that could occur. It is archived, so no PR, but let's fix it on our
side. The scenario in case: executing command in an empty environment.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-29 16:37:55 +03:00
81f4cdbb91 [#1439] object: Sort nodes by priority metrics to compute GET request
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-29 08:05:09 +00:00
3cd7d23f10 [#1439] node: Reduce usage of netmapAPI.NodeInfo
Remove outdated code from `netmap` service.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-29 08:05:09 +00:00
012af5cc38 [#1406] tree: Add unit-tests for ape check
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-10-29 08:04:23 +00:00
eb5336d5ff [#1406] tree: Use delete verb instead put for Remove
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-10-29 08:04:23 +00:00
bc8d79ddf9
[#1447] services/tree: Move relaying code to a separate function
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-24 10:01:03 +03:00
29708b78d7 [#1442] cli/tree: Enchance error message if rpc-endpoint isn't defined
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-23 13:05:17 +00:00
b9284604d9 [#1442] cli/tree: Allow to specify rpc-endpoint with config file
We have several ways to specify the `rpc-endpoint`: with a flag,
with a single config file or multiple files. Before, the `rpc-endpoint`
flag was marked as required. Because `cobra` checked the required flag
presence first, it prevented specifying `rpc-endpoint` with a config file.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-23 13:05:17 +00:00
65a4320c75 [#1441] services/tree: Use grpc.WaitForReady option when creating client
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-23 11:45:44 +00:00
9a260c2e64 [#1441] network/cache: Use grpc.WaitForReady option when creating client
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-23 11:45:44 +00:00
6f798b9c4b [#1441] cli: Use grpc.WaitForReady while initializing SDK client
Before, when the target RPC server was unavailable, requests made
by CLI didn't wait for a timeout specified by the `--timeout` option
if the timeout was more than 20 seconds. It's because of the gRPC
default backoff strategy. Adding this option fixes that behavior.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-23 11:45:44 +00:00
e515dd4582
[#1444] config: Fix data race on morph component init
It could be called for every shard on metabase resync concurrently and
it is possible to get state with initialized client but not initialized
contract hashes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-23 10:41:36 +03:00
8b6ec57c61 [#1440] sdnotify: Fix status for RELOADING
Before:
```
RELOADING=1
MONOTONIC_USEC=17951246687
STATUS=RELOADING=1
MONOTONIC_USEC=17951246687
```
After:
```
RELOADING=1
MONOTONIC_USEC=17951246687
STATUS=RELOADING
```

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-21 14:25:08 +03:00
ed13387c0e
[#1438] .docker: Use go1.23 for builders
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-18 15:20:37 +03:00
5afea62ec0
[#1438] debian: Remove package scripts
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-18 15:20:37 +03:00
c0a2f20eee [#1422] multinet: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
2d064d0bd8 [#1422] morph: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
ef38420623 [#1422] ir: Add dialer source
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
f7caef355b [#1422] node: Use dialer source for morph
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
fbdfd503e4 [#1422] morph: Add dialer source support
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
67798bb50e [#1422] mod: Bump neoneo-go version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
5b653aa65f [#1422] morph: Drop single client as not used
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
e314f328c4 [#1422] tree: Use dialer source for tree service connections
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
6c96cc2af6 [#1422] node: Use dialer source for SDK cache
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
74db735265 [#1422] node: Add dialer source to config
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
3304afa9d1 [#1422] config: Add multinet config
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
b42bcdc6fa
[#1433] services/object: Put object before auxiliary info
Consider the following operations ordering:
1. Inhume(with tombstone A) --> add tombstone mark for an object
2. --> new epoch arives
3. --> GCMark is added for a tombstone A, because it is unavailable
4. Put(A) --> return error, because the object already has a GCMark

It is possible, and I have successfully reproduced it with a test on the
shard level. However, the error is related to the specific
_ordering_ of operations with engine. And triggering race-conditions like
this is only possible on a shard level currently, so no tests are
written.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-17 14:43:13 +03:00
b0c5def2d9
[#1433] shard/test: Use WithDisabledGC() option where possible
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-17 14:38:07 +03:00
90f3669399 [#1342] network/cache: Add node address to error multiClient
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 14:05:33 +00:00
07ce40e119 [#1430] adm/morph: Add NNS address display in 'deploy'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 13:56:37 +00:00
41038b2ec0 [#1431] node: Fix 'empty slice declaration using a literal'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
d83879d4b8 [#1431] node: Fix comment format
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
f6582081a4 [#1431] obj_storage/metabase: Delete unused variable
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
00b1cecfb7 [#1431] obj_storage/shard: Fix visibility of 'newMetricStore'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
63466d71b2 [#1431] engine: Delete unused constants
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
d53732f663 [#1431] engine: Delete always false condition
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
3012286452 [#1431] metabase: Fix unreachable code
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
714ff784fa [#1431] objsvc: Use specific values in message about address mismatch
This makes troubleshooting failed operations much easier

Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2024-10-15 07:34:03 +00:00
d2a59b2de8
[#1429] lens/explorer: Fix locked object records display text
Display texts for a locked object and a list of it lockers were mistakenly swapped.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-14 15:58:14 +03:00
acd6eb1815 [#1427] object: Fix Put for EC object when node unavailable
There might be situation when context canceled earlier than traverser move to another part of the nodes.
To avoid this, need to wait for the result from concurrent put at each traverser iteration.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-11 16:53:30 +03:00
42bf03e5cc
[#1411] adm/nns: Add 'delRecord'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-11 11:33:56 +03:00
5992ee901a
[#1411] go.mod: Bump frostfs-contract to v0.20.0
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-11 11:33:36 +03:00
dfb00083d0
[#1426] go.mod: Update sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-10 14:58:46 +03:00
1134760271
[#1425] services/tree: Remove eACL mentions from bearer token parsing errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:49 +03:00
02bb7159a5
[#1425] services/tree: Remove eACL processing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:48 +03:00
94302235d0
[#1425] adm: Remove eACL fetching from dump-containers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:48 +03:00
cc5360a578
[#1425] morph/event: Rename eacl_test.go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:48 +03:00
4190fba86d
[#1425] Remove SetEACL-related code
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:48 +03:00
936ebbb8e5 [#1423] metabase: Hide BucketName form upper levels
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-08 18:41:16 +03:00
c065d55ca3
[#1412] metabase: Drop logging inside transaction
This could lead to hang the db.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:52 +03:00
fe9f664b57
[#1412] metabase: Drop empty user attribute buckets on upgrade
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:52 +03:00
87f4b934d1
[#1412] metabase: Run bucket drop steps on upgrade concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:52 +03:00
8093e145b3
[#1412] adm: Resolve container type by metabase upgrade
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:51 +03:00
3da168f8cf
[#1412] shard: Resolve container is indexed on metabase resync
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:51 +03:00
4572fa4874
[#1412] searchSvc: Check container is indexed
For non S3 containers it is expected to use attributes index for some
attributes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:50 +03:00
1efa64ee72
[#1412] metabase: Add search by indexed attributes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:50 +03:00
be744ae3e6
[#1412] metabase: Index attributes for indexed containers
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:49 +03:00
1b520f7973
[#1412] engine: Add IsIndexedContainer flag
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:49 +03:00
899cd55c27
[#1412] engine: PutPrm refactoring
Use fields instead of methods.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:48 +03:00
0c49bca19c [#1415] lens/explorer: Add timeout for opening database
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-08 07:39:31 +00:00
5fbb2657ca
[#1419] mod: Bump sdk-go version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 10:04:11 +03:00
a5de74a249 [#1418] go.mod: Update api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-07 12:55:28 +00:00
fc032838c0 [#1215] blobstor/test: Cover iteration behaviour
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-07 12:44:27 +00:00
2f710d8f94 [#1414] metabase: Check parameter for CountAliveObjectsInBucket
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-04 17:11:15 +03:00
4dc9a1b300 [#1413] engine: Remove error counting methods from Shard
All error counting and hangling logic is present on the engine level.
Currently, we pass engine metrics with shard ID metric to shard, then
export 3 methods to manipulate these metrics.
In this commits all methods are removed and error counter is tracked on
the engine level exlusively.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-04 15:10:17 +03:00
963faa615a [#1413] engine: Cleanup shard error reporting
- `reportShardErrorBackground()` no longer differs from
  `reportShardError()`, reflect this in its name;
- reuse common pieces of code to make it simpler.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-04 15:10:17 +03:00
9a87acb87a [#1410] engine: Provide the default implementation to MetricsRegister
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-03 08:23:06 +00:00
9206ce5cd2 [#1410] shard: Provide the default implementation for MetricsWriter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-03 08:23:06 +00:00
6c46044c9c [#1410] shard: Move MetricsWriter interface to a separate file
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-03 08:23:06 +00:00
01e3944b31 [#1408] metabase: Fix tests
No need to specify container ID for objects created with `testutil.GenerateObjectWithCID`.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-02 14:41:40 +03:00
434048e8d9 [#1408] metabase: Fix EC search with slow and fast filters
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-02 14:36:59 +03:00
f83f7feb8c [#1391] adm: Properly check whether transfers were made
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-02 11:41:33 +03:00
62028cd7ee [#1409] adm: Uncommonize DeltaFlag
It is used only in `force-new-epoch`, it is not _common_ between
multiple commands.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-02 11:41:27 +03:00
f45e75e3eb [#1409] adm: Do not bind DeltaFlag to viper
We bind flag that could be specified in config.
This is not a config flag, just a command option.
Also fix TestInitialize failures:
```
                Error:          Received unexpected error:
                                number of epochs cannot be less than 1
                Test:           TestInitialize/16_nodes/force-new-epoch
```
Refs #1372 (945b7c740b)

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-02 11:41:22 +03:00
57c31e9802 [#1306] node: Allow tombstone_lifetime config to be loaded on the fly
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-10-02 10:22:00 +03:00
9c5ddc4dfe [#1407] tree: Set ContainerOwner in parameter for CheckAPE
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-10-02 07:21:02 +00:00
54eb005822 [#1404] go.mod: Update api-go
Fix #1398
Fix #1399

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-02 06:19:38 +00:00
a13219808a [#1375] node: Configure of the container cache size
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-01 14:05:52 +00:00
7f8a1dcf8e [#1400] adm: Support flag alphabet-wallets for commands proxy-add/remove-account
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-30 14:15:13 +03:00
d0ed29b3c7 [#1350] node: Add ability to evacuate objects from REP 1 only
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-27 15:41:17 +03:00
5f22ba6f38 [#1397] object: Correctly set namespace before APE check
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-27 11:43:29 +00:00
a5e1aa22c9 [#1394] putSvc: Fix relay
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 17:28:03 +03:00
772b471aab [#1388] lens: Add nolint annotations
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
4fbfffd44c [#1388] putSvc: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
29e4cf7ba1 [#1388] ir: Annotate cmode as nolint
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
a2ab6d4942 [#1388] node: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
bdd57c8b6b [#1388] sessionSvc: Add nolint annotations
Used as map key.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
d1d6e3471c [#1388] signSvc: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
401c398704 [#1388] metabase: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
004ff9e9bf [#1388] blobstor: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
63a567a1de [#1388] engine: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
580cd55180 [#1388] getSvc: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
e319bf403e [#1388] apeSvc: Drop unused and make annotations
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
aedb55f913 [#1388] governance: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
b69e07da7a [#1388] metrics: Mark nolint:unused metrics
Although these fields could be deleted, I annotated them so that all the
metrics used would be defined in one place.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
2bd560e528 [#1388] cli: Drop unused flag/parameter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
95597d3437 [#1388] golangci: Make unused linter stricker
Add aditional checks. The most important false positive - structs used as
map keys.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
fd18aa363b [#1385] metabase: Optimize isTomb check
As tombstone and target must have the same containerID, do not iterate
other containers.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-24 17:09:06 +03:00
76268e3ea2 [#1385] metabase: Validate that tombstone and target have the same container ID
Target container ID is taken from tombstone: cmd/frostfs-node/object.go:507
Also object of type `TOMBSTONE` contains objectID, so tombstone and
tombstoned object must have the same containerID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-24 17:09:06 +03:00
8434f3dbfc [#1385] metabase: Use Batch for delete-related operations
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-24 17:09:06 +03:00
34e6a309c6 [#1356] engine: Evacuate object from shards concurrently
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-24 12:50:19 +03:00
bdf386366c [#1297] dev: Bump neo-go version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-24 12:05:09 +03:00
839dead226 [#1297] getSvc: Return AccessDenied instead of ObjectNotFound
Do not replace the access denied error if it was received earlier.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-24 12:05:09 +03:00
3bb65ba820 [#1392] object: Fix target initialization within put streamer
* Remove `relay` field from put streamer as it's no longer used;
* Fix initialization of `Relay` object writer parameter.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-24 09:04:13 +00:00
d4493a6d08 [#1390] getSvc: Fix Head EC1.1
If local EC chunk found, but remote node is off, then `HEAD --raw` request
returns object not found.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-23 15:12:06 +03:00
0b87be804a [#1381] engine: Fix tests
Drop not required `Eventually` calls.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-23 11:10:29 +03:00
f71418b73c [#1386] frostfs-adm: Add info to error messages
These error messages bubble up to human users - adding more context helps
to find the cause of the issue faster.

Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2024-09-23 07:47:03 +00:00
c34b8acedd [#1312] Drop handling of system attributes with NeoFS prefix
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-20 11:37:18 +00:00
c290d079fd [#1312] go.mod: Update sdk-go
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-20 11:37:18 +00:00
53a90634fc [#1301] adm/morph: Add 'delete' domains
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-09-20 11:54:18 +03:00
5a53f9c4fd [#1301] go.mod: Bump frostfs-contract
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-09-20 11:54:18 +03:00
1361db91ee [#1301] adm/morph: Add flag -v to 'Tokens'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-09-20 11:54:18 +03:00
945b7c740b [#1372] adm/morph: Add delta flag to 'force-new-epoch'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-09-20 11:42:04 +03:00
61d5e140e0 [#1383] object: Add restrictions for Patch method
* `Patch` can't be applied for non-regular type object (tombstones,
  locks etc.)
* Complex object parts can't be patched. So, if an object has EC/Split
  header, it won't be patched.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-19 13:54:49 +03:00
3441fff05d [#1382] cli: Replace deprecated methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:35:48 +03:00
ac1eee091d [#1382] node: Replace deprecated methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:35:48 +03:00
a603d14d08 [#1382] ir: Replace deprecated methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:29:31 +03:00
d4be2f20d4 [#1382] morph: Replace deprecated methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:29:27 +03:00
e5c8f7ff9f [#1382] controlSvc: Replace deprecated methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:29:24 +03:00
1e7f9909da [#1382] policer: Replace deprecated methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:29:21 +03:00
b807d8c400 [#1382] go.mod: Upgrade sdk-go and api-go versions
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:29:11 +03:00
d4bec24c9f
[#1366] node, ir: Support timestamp config option, update tests
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-17 10:50:08 +03:00
ea48e928c8
[#1366] logger: Make timestamp prepending optional
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-17 10:49:51 +03:00
96308a26c6 [#1361] linter: fix funlen
Signed-off-by: Aleksey Savaitan <a.savaitan@yadro.com>
2024-09-13 15:12:49 +00:00
74a6a1da7f [#1361] add root ca cert for telemetry configuration
Signed-off-by: Aleksey Savaitan <a.savaitan@yadro.com>
2024-09-13 15:12:49 +00:00
2be1aa781d [#1266] .forgejo: Make 'fumpt' job fail on changed files
`gofumpt` always returns an exit code of 0, even when it finds
misformatted files. To make `fumpt` action behave as expected
we need to check if `gofumpt` changed any files.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-13 17:27:58 +03:00
89d0435b1d [#1374] tree: Use NewClient to create grpc connection in cache
Created grpc connection should be established, so perform Healthcheck request
to check connection is ok.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-13 15:59:26 +03:00
54fe8383a4 [#1374] tree: Use NewClient to create grpc connection for sync
Created connection will be used to sync trees, so it is ok to defer
dial to the first RPC call.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-13 15:59:26 +03:00
944160427b [#1374] cli: Drop deprecated grpc connection method
For `frostfs-cli` it is ok to use grpc-client without blocking,
as `frostfs-cli` will perform RPC call anyway.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-13 15:59:26 +03:00
bb44867491 [#1374] go.mod: Upgrade grpc to v1.66.2
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-13 15:59:00 +03:00
546d09660f [#1283] Clear systemd-notify status on exit
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-12 14:45:06 +00:00
e3764c51df [#1347] metabase: Fix EC search
For EC chunks need to return EC parent object ID as
EC chunks don't have own attributes but inherit parent's.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 13:23:29 +00:00
b33559754d [#1367] fstree: Add size hint for Delete
This allow to not to call `os.Stat` if caller already knows data size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
f345fe9a58 [#1367] writecache: Move DB related code to upgrade.go
This is done to drop this file in the future.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
3b236160a6 [#1367] writecache: Drop DB label from metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
25d2ae8aaf [#1367] writecache: Drop BBolt related config variables
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
e39378b1c3 [#1367] writecache: Add background flushing objects limiter
To limit memory usage by background flush.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
8a6e3025a0 [#1367] writecache: Flush from FSTree concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
2dd3a6f7a8 [#1367] fstree: Add IterateInfo method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
b142b6f48e [#1367] fstree: Add size to file counter
FSTree file counter used by writecache. As writecache has now only one
storage, so it is required to use real object size to get writecache
size more accurate than `count * max_object_size`.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
5f6c7cbdb1 [#1367] writecache: Drop bbolt DB
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 14:22:29 +03:00
66e17f4b8e
[#1368] cli/container: Use dedicated method to list user attributes
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-12 10:01:27 +03:00
99be4c83a7
[#1368] *: Run gofumpt
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-12 10:00:28 +03:00
ec8da40567 [#1369] Update obsolete URLs
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2024-09-11 11:31:10 +00:00
dea6f031f9 [#1331] cli/tree: Add order flag to tree get-subtree
Added `--ordered` flag to sort output by ascending FileName.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-11 11:30:28 +00:00
5fac4058e8 [#1364] cmd/common: Add tests for CreateViper and ReloadViper
Add tests for `CreateViper` and `ReloadViper` to ensure that no extra
files, except *.yaml, *.yml, *.json, are loaded from config directory.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-11 08:03:05 +00:00
2220f6a809 [#1365] Makefile: Fix HUB_IMAGE
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-09-10 16:45:15 +03:00
a812932984 [#1362] ape: Move common APE check logic to separate package
* Tree and object service have the same log for checking APE. So,
  this check should be moved to common package.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-10 12:40:34 +00:00
92fe5d90f5 [#1359] writecache: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-09 18:45:21 +03:00
4668efc0bf [#1355] metabase: Upgrade improvements
Do not fail on same latest version to run compact on upgraded metabase.
Use NoSync on compact.
Log every batch on bucket delete stage.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-09 11:37:36 +03:00
654d970fad [#1355] adm: Run metabase upgrade concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 15:54:40 +03:00
d3b209c8e1 [#1337] shard: Disable background rebuild
Since `frostfs-cli control shards rebuild` command was added,
there is no need for background rebuild now.
For failover tests used used value 1 to rebuild only schema change.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 15:19:55 +03:00
edb1747af7 [#1337] blobovniczatree: Add rebuild by overflow
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 15:19:54 +03:00
a61201a987 [#1337] config: Move rebuild_worker_count to shard section
This makes it simple to limit performance degradation for every shard
because of rebuild.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 13:57:27 +03:00
6b6eabe41c [#1337] cli: Add control shards rebuild command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 13:57:27 +03:00
d508da8397 [#1337] blobovniczatree: Add rebuild by fill percent
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 13:57:27 +03:00
007827255e [#1337] blobovniczatree: Add .rebuild temp files
This allows to reduce open/close DBs to check incompleted rebuilds.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 13:57:27 +03:00
f652518c24 [#1357] go: Fix panic caused by using range over slice len
If slice is altered in `for` loop, we cannot use range over its
length: it may cause panic if slice gets shorter.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-06 13:09:58 +03:00
273980cfb9 [#1310] object: Remove irrelevant comments
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-05 16:40:32 +03:00
108e4e07be [#1349] node: Evacuate objects without setting mode to MAINTENANCE
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-05 16:08:27 +03:00
b3deb893ba [#1310] object: Move target initialization to separate package
* Split the logic of write target initialization to different packages;
* Refactor patch and put services: since both service initialize the target
  themselves.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-05 13:03:58 +00:00
7768a482b5 [#1223] lens/tui: Add TUI app for blobovnicza
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
371d97f61a [#1223] lens/tui: Add TUI app for write cache
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
e655336390 [#1223] lens/tui: Add app help
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
ed396448ac [#1223] lens/tui: Add TUI app to explore metabase
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
9cbd32bce8 [#1223] lens/tui: Add writecache schema
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
1ae86f35a8 [#1223] lens/tui: Add metabase schema
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
b9043433a0 [#1223] scripts: Add script to populate metabase
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
a4fb7f085b
[#1348] go.mod: Update api-go and sdk-go
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-04 10:47:26 +03:00
a685fcdc96 [#1317] go.mod: Use range over int
Since Go 1.22 a "for" statement with a "range" clause is able
to iterate through integer values from zero to an upper limit.

gopatch script:
@@
var i, e expression
@@
-for i := 0; i <= e - 1; i++ {
+for i := range e {
    ...
}

@@
var i, e expression
@@
-for i := 0; i <= e; i++ {
+for i := range e + 1 {
    ...
}

@@
var i, e expression
@@
-for i := 0; i < e; i++ {
+for i := range e {
    ...
}

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-03 13:00:54 +03:00
2b3fc50681
[#1320] shard: Fix TestGCDropsObjectInhumedFromWritecache flaky test
The `TestGCDropsObjectInhumedFromWritecache` test was flaky because a
running asynchronous rebuild operation prevented GC from deleting the
object. A test-only shard option `WithDisabledRebuild` has been added
to fix this.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-02 10:26:53 +03:00
98fe24cdb7 [#1343] go.mod: Update api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-30 08:45:24 +00:00
882c068410 [#1334] metabase: Store upgrade flag
This allows to check if metabase upgrade was not completed.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-30 09:06:20 +03:00
6c2146bbc1 [#1334] metabase: Add upgrade from v2 to v3
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-30 09:06:20 +03:00
03976c6ed5 [#1341] .golangci.yml: Replace exportloopref with copyloopvar
exportloopref is deprecated.
gopatch:
```
@@
var index, value identifier
var slice expression
@@
for index, value := range slice {
...
-value := value
...
}

@@
var index, value identifier
var slice expression
@@
for index, value := range slice {
...
-index := index
...
}

@@
var value identifier
var channel expression
@@
for value := range channel {
...
-value := value
...
}
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-28 15:44:41 +00:00
7e97df4878 [#1341] Makefile: Update golangci-lint
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-28 15:44:41 +00:00
01b6f1733c [#1341] Makefile: Build linter with -trimpath
Fix error with go1.23:
```
Error: build linters: unable to load custom analyzer "truecloudlab-linters": ../linters/bin/external_linters.so, plugin.Open("/repo/frostfs/linters/bin/external_linters"): plugin was built with a different version of package cmp
Failed executing command with error: build linters: unable to load custom analyzer "truecloudlab-linters": ../linters/bin/external_linters.so, plugin.Open("/repo/frostfs/linters/bin/external_linters"): plugin was built with a different version of package cmp
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-28 15:44:41 +00:00
7abbdca064 [#1340] getSvc: Fix access denied error handling
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-28 14:02:28 +03:00
6488ddee88 [#1338] object: Fix range provider in Patch handler
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-27 16:51:12 +03:00
d6b42972a8 [#1338] object: Fix audit patch stream
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-27 16:18:29 +03:00
5e9a97fd3e [#1336] go.mod: Update api-go and sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-27 14:48:43 +03:00
fa7f9fbce2 [#1333] go.mod: Update api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-26 15:37:43 +03:00
806ea37101 [#1328] pilorama: Do not skip items in SortedByFilename
Benchmark results:
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/pilorama
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                                   │     old     │                 new                 │
                                   │   sec/op    │   sec/op     vs base                │
ForestSortedIteration/bbolt,root-8   207.2µ ± 6%   173.6µ ± 6%  -16.23% (p=0.000 n=10)
ForestSortedIteration/bbolt,leaf-8   3.910µ ± 5%   3.928µ ± 7%        ~ (p=0.529 n=10)
geomean                              28.46µ        26.11µ        -8.27%
```

They are not representative, as the worst case is when we have multiple
items of different lengths. However, `FileName` is usually less than 100
in practice, so the asymptotics is the same.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-26 06:11:32 +00:00
80099d9a2f [#1328] pilorama: Add tricky test for SortedByFilename
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-26 06:11:32 +00:00
a059a7dcf0 [#1329] cli: Skip linking objects in complex object processing
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-23 13:51:20 +03:00
bd24beecf8 [#1329] putSvc: Reset SuccessAfter for non-EC objects in EC container broadcasting
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-23 13:51:16 +03:00
dfe825b81b [#1309] test: Clean up config after tests
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-08-22 08:49:10 +00:00
76f67ea34e [#1323] metabase: Bump version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
7d0d781db1 [#1323] metabase: Drop user attribute index
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
0f08a2efba [#1323] metabase: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
7bf20c9f1f [#1323] metabase: Add expiration epoch buckets
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
2542d4f5df [#1323] metabase: Drop payload checksum index
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
15dae8685e [#1323] metabase: Drop ownerID index
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
7bca428db0 [#1322] Use new protobuf marshaler
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-22 07:17:41 +00:00
a345c972bf [#1316] lint: Fix warnings
Renamed parameters `min/max` to avoid conflicts with
predeclared identifiers.

Replaced background context with parent context without
cancellation in closer functions in frostfs-node.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-21 18:18:25 +03:00
8c1082b31a [#1316] go.mod: Bump go version to 1.22
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-21 15:09:29 +03:00
cfda9003a7 [#1318] meta: Add test TestInhumeECObject
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-20 16:21:27 +03:00
6ff0b0996b [#1318] metrics: Fix container_size_bytes for EC
When node put chunk into EC container, `policer` may remove it as redundant.
This chunk marked as removed. When parent object removed and `gc` start iterating over chunk,
node count removing chunk twice.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-20 15:33:28 +03:00
8319b59238 [#1318] Fix gofumpt issue
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-20 14:34:54 +03:00
b7acb34fa4 [#1319] treeSvc: Do not wrap error from APE
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-19 18:38:27 +03:00
41104f2383 [#1307] cli: Make cli process object.patch
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
eeab417dcf [#1307] object: Add APE check for Patch handler
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
5ed317e24c [#1307] cli: Introduce object patch command
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
e890f1b4b1 [#1307] object: Implement Patch method
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
a4a1c3f18b [#1307] go.mod: Bump frostfs-sdk-go/frostfs-api-go/v2 versions
* Also, resolve dependencies and conflicts for object service
  by creating stub for `Patch` method.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
ec1509de4e [#1262] sdnotify: Send system monotonic time on reload
The synchronized service reload protocol added in systemd version 253
requires that the service provides a MONOTONIC_USEC field alongside the
RELOADING=1 notification message for synchronization purposes. The value
carried in this field must be the system CLOCK_MONOTONIC timestamp at
the time the notification message was generated as systemd compares it
to other CLOCK_MONOTONIC timestamps taken by pid1.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-16 12:05:31 +03:00
5da41f1fe5 Revert "[#1262] sdnotify: Get rid of go:linkname for nanotime"
This reverts commit 327d364f34.

Reverted due to the problem with reload signal sent by systemd.
`frostfs-ir` service reconfigures correctly and service's
statuses are being reported to systemd. However, since we
replaced `go:linkname` & `nanotime()` with `time.Since()`,
systemd refuses to accept reload signal response from
`frostfs-ir`. To maintain correct behaviour it was decided to
revevrt systemd-related changes until a better solution is
found.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-15 16:54:45 +03:00
d055168e2a [#1135] ir: Add healthstatus RECONFIGURING
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-15 16:54:45 +03:00
80ce7c3a00 [#1284] shard: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 13:23:35 +03:00
93d63e1632 [#1284] writecache: Allow to seal writecache async
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 13:23:35 +03:00
68029d756e [#1302] writecache: Allow to specify custom page size
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 12:12:07 +03:00
fa82854af4 [#1302] writecache: Add put->flush->put benchmark
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 12:12:07 +03:00
c985b1198f [#1302] putSvc: Override SuccessAfter for non-regular objects in EC containers
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 10:42:55 +03:00
08b1f18bca [#1296] writecache: Add count limit
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 06:30:32 +00:00
36efccd862 [#1298] writecache: Add shrink flag for Seal command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-08 16:32:29 +03:00
5c01bd5be8 [#1298] writecache: Add restore-mode flag for Seal command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-07 11:20:14 +03:00
8e51d7849a [#1295] getSvc: Assemble complex EC object headers without linking object
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-06 16:48:12 +03:00
10602b55b1 [#1295] engine: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-06 09:45:47 +03:00
eeca796d2e [#1295] engine: Log object address in case of error
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-06 09:45:43 +03:00
327d364f34 [#1262] sdnotify: Get rid of go:linkname for nanotime
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-05 12:37:16 +00:00
dc3dcabadc [#1291] morph: Reconnect to the highest priority endpoint
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-02 17:54:22 +03:00
8021bacc43 [#1288] putSvc: Respect TTL for EC put
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-02 13:01:42 +00:00
ef4cea6d19 [#1266] .forgejo: Add gofumpt action
`gofumpt` was skipped by pre-commit on CI, and now is used
in a separate action.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-02 12:34:04 +03:00
a55600893e [#1266] Makefile: Specify gofumpt version
Add target to install gofumpt, fix target to run gofumpt.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-02 12:31:59 +03:00
c49982d22a [#1282] cli: Allow to external addresses first for object nodes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-01 09:16:19 +03:00
7e04083c27 [#1278] containerSvc: Validate FrostFSID subject exitence on Put
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-29 16:01:37 +03:00
a12c39667d [#1278] ir: Do not allow to create container without FrostFSID record
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-29 16:01:37 +03:00
85a77b7c21 [#1279] adm: Interpret "root" name as empty for namespace target type
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-29 12:57:40 +00:00
8377372a40 [#1276] go.mod: Update api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-26 16:44:19 +03:00
858 changed files with 20203 additions and 12749 deletions

View file

@ -1,4 +1,4 @@
FROM golang:1.22 AS builder
FROM golang:1.23 AS builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository

View file

@ -1,4 +1,4 @@
FROM golang:1.22
FROM golang:1.23
WORKDIR /tmp

View file

@ -1,4 +1,4 @@
FROM golang:1.22 AS builder
FROM golang:1.23 AS builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository

View file

@ -1,4 +1,4 @@
FROM golang:1.22 AS builder
FROM golang:1.23 AS builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository

View file

@ -1,4 +1,4 @@
FROM golang:1.22 AS builder
FROM golang:1.23 AS builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository

View file

Before

Width:  |  Height:  |  Size: 5.5 KiB

After

Width:  |  Height:  |  Size: 5.5 KiB

View file

@ -1,6 +1,10 @@
name: Build
on: [pull_request]
on:
pull_request:
push:
branches:
- master
jobs:
build:
@ -8,7 +12,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
go_versions: [ '1.21', '1.22' ]
go_versions: [ '1.22', '1.23' ]
steps:
- uses: actions/checkout@v3

View file

@ -1,5 +1,10 @@
name: Pre-commit hooks
on: [pull_request]
on:
pull_request:
push:
branches:
- master
jobs:
precommit:
@ -16,7 +21,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: 1.22
go-version: 1.23
- name: Set up Python
run: |
apt update

View file

@ -1,5 +1,10 @@
name: Tests and linters
on: [pull_request]
on:
pull_request:
push:
branches:
- master
jobs:
lint:
@ -11,7 +16,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.22'
go-version: '1.23'
cache: true
- name: Install linters
@ -25,7 +30,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
go_versions: [ '1.21', '1.22' ]
go_versions: [ '1.22', '1.23' ]
fail-fast: false
steps:
- uses: actions/checkout@v3
@ -48,7 +53,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
go-version: '1.22'
cache: true
- name: Run tests
@ -63,7 +68,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.22'
go-version: '1.23'
cache: true
- name: Install staticcheck
@ -81,7 +86,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
go-version: '1.22'
cache: true
- name: Install gopls
@ -89,3 +94,23 @@ jobs:
- name: Run gopls
run: make gopls-run
fumpt:
name: Run gofumpt
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.23'
cache: true
- name: Install gofumpt
run: make fumpt-install
- name: Run gofumpt
run: |
make fumpt
git diff --exit-code --quiet

View file

@ -1,5 +1,10 @@
name: Vulncheck
on: [pull_request]
on:
pull_request:
push:
branches:
- master
jobs:
vulncheck:
@ -13,7 +18,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22'
go-version: '1.23'
- name: Install govulncheck
run: go install golang.org/x/vuln/cmd/govulncheck@latest

View file

@ -12,7 +12,8 @@ run:
# output configuration options
output:
# colored-line-number|line-number|json|tab|checkstyle|code-climate, default is "colored-line-number"
format: tab
formats:
- format: tab
# all available settings of specific linters
linters-settings:
@ -37,6 +38,10 @@ linters-settings:
alias:
pkg: git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object
alias: objectSDK
unused:
field-writes-are-uses: false
exported-fields-are-used: false
local-variables-are-used: false
custom:
truecloudlab-linters:
path: bin/linters/external_linters.so
@ -66,7 +71,7 @@ linters:
- bidichk
- durationcheck
- exhaustive
- exportloopref
- copyloopvar
- gofmt
- goimports
- misspell
@ -82,5 +87,7 @@ linters:
- perfsprint
- testifylint
- protogetter
- intrange
- tenv
disable-all: true
fast: false

View file

@ -1,11 +0,0 @@
pipeline:
# Kludge for non-root containers under WoodPecker
fix-ownership:
image: alpine:latest
commands: chown -R 1234:1234 .
pre-commit:
image: git.frostfs.info/truecloudlab/frostfs-ci:v0.36
commands:
- export HOME="$(getent passwd $(id -u) | cut '-d:' -f6)"
- pre-commit run --hook-stage manual

View file

@ -9,6 +9,30 @@ Changelog for FrostFS Node
### Removed
### Updated
## [v0.44.0] - 2024-25-11 - Rongbuk
### Added
- Allow to prioritize nodes during GET traversal via attributes (#1439)
- Add metrics for the frostfsid cache (#1464)
- Customize constant attributes attached to every tracing span (#1488)
- Manage additional keys in the `frostfsid` contract (#1505)
- Describe `--rule` flag in detail for `frostfs-cli ape-manager` subcommands (#1519)
### Changed
- Support richer interaction with the console in `frostfs-cli container policy-playground` (#1396)
- Print address in base58 format in `frostfs-adm morph policy set-admin` (#1515)
### Fixed
- Fix EC object search (#1408)
- Fix EC object put when one of the nodes is unavailable (#1427)
### Removed
- Drop most of the eACL-related code (#1425)
- Remove `--basic-acl` flag from `frostfs-cli container create` (#1483)
### Upgrading from v0.43.0
The metabase schema has changed completely, resync is required.
## [v0.42.0]
### Added

3
CODEOWNERS Normal file
View file

@ -0,0 +1,3 @@
.* @TrueCloudLab/storage-core-committers @TrueCloudLab/storage-core-developers
.forgejo/.* @potyarkin
Makefile @potyarkin

View file

@ -4,20 +4,19 @@ SHELL = bash
REPO ?= $(shell go list -m)
VERSION ?= $(shell git describe --tags --dirty --match "v*" --always --abbrev=8 2>/dev/null || cat VERSION 2>/dev/null || echo "develop")
HUB_IMAGE ?= truecloudlab/frostfs
HUB_IMAGE ?= git.frostfs.info/truecloudlab/frostfs
HUB_TAG ?= "$(shell echo ${VERSION} | sed 's/^v//')"
GO_VERSION ?= 1.22
LINT_VERSION ?= 1.56.1
TRUECLOUDLAB_LINT_VERSION ?= 0.0.5
LINT_VERSION ?= 1.62.0
TRUECLOUDLAB_LINT_VERSION ?= 0.0.8
PROTOC_VERSION ?= 25.0
PROTOC_GEN_GO_VERSION ?= $(shell go list -f '{{.Version}}' -m google.golang.org/protobuf)
PROTOGEN_FROSTFS_VERSION ?= $(shell go list -f '{{.Version}}' -m git.frostfs.info/TrueCloudLab/frostfs-api-go/v2)
PROTOGEN_FROSTFS_VERSION ?= $(shell go list -f '{{.Version}}' -m git.frostfs.info/TrueCloudLab/frostfs-sdk-go)
PROTOC_OS_VERSION=osx-x86_64
ifeq ($(shell uname), Linux)
PROTOC_OS_VERSION=linux-x86_64
endif
STATICCHECK_VERSION ?= 2023.1.6
STATICCHECK_VERSION ?= 2024.1.1
ARCH = amd64
BIN = bin
@ -28,24 +27,21 @@ DIRS = $(BIN) $(RELEASE)
CMDS = $(notdir $(basename $(wildcard cmd/frostfs-*)))
BINS = $(addprefix $(BIN)/, $(CMDS))
# .deb package versioning
OS_RELEASE = $(shell lsb_release -cs)
PKG_VERSION ?= $(shell echo $(VERSION) | sed "s/^v//" | \
sed -E "s/(.*)-(g[a-fA-F0-9]{6,8})(.*)/\1\3~\2/" | \
sed "s/-/~/")-${OS_RELEASE}
OUTPUT_LINT_DIR ?= $(abspath $(BIN))/linters
LINT_DIR = $(OUTPUT_LINT_DIR)/golangci-lint-$(LINT_VERSION)-v$(TRUECLOUDLAB_LINT_VERSION)
TMP_DIR := .cache
PROTOBUF_DIR ?= $(abspath $(BIN))/protobuf
PROTOC_DIR ?= $(PROTOBUF_DIR)/protoc-v$(PROTOC_VERSION)
PROTOC_GEN_GO_DIR ?= $(PROTOBUF_DIR)/protoc-gen-go-$(PROTOC_GEN_GO_VERSION)
PROTOGEN_FROSTFS_DIR ?= $(PROTOBUF_DIR)/protogen-$(PROTOGEN_FROSTFS_VERSION)
STATICCHECK_DIR ?= $(abspath $(BIN))/staticcheck
STATICCHECK_VERSION_DIR ?= $(STATICCHECK_DIR)/$(STATICCHECK_VERSION)
SOURCES = $(shell find . -type f -name "*.go" -print)
GOFUMPT_VERSION ?= v0.7.0
GOFUMPT_DIR ?= $(abspath $(BIN))/gofumpt
GOFUMPT_VERSION_DIR ?= $(GOFUMPT_DIR)/$(GOFUMPT_VERSION)
GOPLS_VERSION ?= v0.15.1
GOPLS_DIR ?= $(abspath $(BIN))/gopls
GOPLS_VERSION_DIR ?= $(GOPLS_DIR)/$(GOPLS_VERSION)
@ -56,7 +52,7 @@ LOCODE_DB_PATH=$(abspath ./.cache/locode_db)
LOCODE_DB_VERSION=v0.4.0
.PHONY: help all images dep clean fmts fumpt imports test lint docker/lint
prepare-release debpackage pre-commit unpre-commit
prepare-release pre-commit unpre-commit
# To build a specific binary, use it's name prefix with bin/ as a target
# For example `make bin/frostfs-node` will build only storage node binary
@ -103,17 +99,15 @@ export-metrics: dep
# Regenerate proto files:
protoc:
@if [ ! -d "$(PROTOC_DIR)" ] || [ ! -d "$(PROTOC_GEN_GO_DIR)" ] || [ ! -d "$(PROTOGEN_FROSTFS_DIR)" ]; then \
@if [ ! -d "$(PROTOC_DIR)" ] || [ ! -d "$(PROTOGEN_FROSTFS_DIR)" ]; then \
make protoc-install; \
fi
@for f in `find . -type f -name '*.proto' -not -path './bin/*'`; do \
echo "⇒ Processing $$f "; \
$(PROTOC_DIR)/bin/protoc \
--proto_path=.:$(PROTOC_DIR)/include:/usr/local/include \
--plugin=protoc-gen-go=$(PROTOC_GEN_GO_DIR)/protoc-gen-go \
--plugin=protoc-gen-go-frostfs=$(PROTOGEN_FROSTFS_DIR)/protogen \
--go-frostfs_out=. --go-frostfs_opt=paths=source_relative \
--go_out=. --go_opt=paths=source_relative \
--go-grpc_opt=require_unimplemented_servers=false \
--go-grpc_out=. --go-grpc_opt=paths=source_relative $$f; \
done
@ -126,10 +120,8 @@ protoc-install:
@wget -q -O $(PROTOBUF_DIR)/protoc-$(PROTOC_VERSION).zip 'https://github.com/protocolbuffers/protobuf/releases/download/v$(PROTOC_VERSION)/protoc-$(PROTOC_VERSION)-$(PROTOC_OS_VERSION).zip'
@unzip -q -o $(PROTOBUF_DIR)/protoc-$(PROTOC_VERSION).zip -d $(PROTOC_DIR)
@rm $(PROTOBUF_DIR)/protoc-$(PROTOC_VERSION).zip
@echo "⇒ Installing protoc-gen-go..."
@GOBIN=$(PROTOC_GEN_GO_DIR) go install -v google.golang.org/protobuf/...@$(PROTOC_GEN_GO_VERSION)
@echo "⇒ Instaling protogen FrostFS plugin..."
@GOBIN=$(PROTOGEN_FROSTFS_DIR) go install -mod=mod -v git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/util/protogen@$(PROTOGEN_FROSTFS_VERSION)
@GOBIN=$(PROTOGEN_FROSTFS_DIR) go install -mod=mod -v git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/util/protogen@$(PROTOGEN_FROSTFS_VERSION)
# Build FrostFS component's docker image
image-%:
@ -165,10 +157,19 @@ imports:
@echo "⇒ Processing goimports check"
@goimports -w cmd/ pkg/ misc/
# Install gofumpt
fumpt-install:
@rm -rf $(GOFUMPT_DIR)
@mkdir $(GOFUMPT_DIR)
@GOBIN=$(GOFUMPT_VERSION_DIR) go install mvdan.cc/gofumpt@$(GOFUMPT_VERSION)
# Run gofumpt
fumpt:
@if [ ! -d "$(GOFUMPT_VERSION_DIR)" ]; then \
make fumpt-install; \
fi
@echo "⇒ Processing gofumpt check"
@gofumpt -l -w cmd/ pkg/ misc/
$(GOFUMPT_VERSION_DIR)/gofumpt -l -w cmd/ pkg/ misc/
# Run Unit Test with go test
test: GOFLAGS ?= "-count=1"
@ -190,7 +191,7 @@ lint-install:
@@make -C $(TMP_DIR)/linters lib CGO_ENABLED=1 OUT_DIR=$(OUTPUT_LINT_DIR)
@rm -rf $(TMP_DIR)/linters
@rmdir $(TMP_DIR) 2>/dev/null || true
@CGO_ENABLED=1 GOBIN=$(LINT_DIR) go install github.com/golangci/golangci-lint/cmd/golangci-lint@v$(LINT_VERSION)
@CGO_ENABLED=1 GOBIN=$(LINT_DIR) go install -trimpath github.com/golangci/golangci-lint/cmd/golangci-lint@v$(LINT_VERSION)
# Run linters
lint:
@ -256,19 +257,6 @@ clean:
rm -rf $(BIN)
rm -rf $(RELEASE)
# Package for Debian
debpackage:
dch -b --package frostfs-node \
--controlmaint \
--newversion $(PKG_VERSION) \
--distribution $(OS_RELEASE) \
"Please see CHANGELOG.md for code changes for $(VERSION)"
dpkg-buildpackage --no-sign -b
# Cleanup deb package build directories
debclean:
dh clean
# Download locode database
locode-download:
mkdir -p $(TMP_DIR)
@ -294,7 +282,6 @@ env-up: all
# Shutdown dev environment
env-down:
docker compose -f dev/docker-compose.yml down
docker volume rm -f frostfs-node_neo-go
docker compose -f dev/docker-compose.yml down -v
rm -rf ./$(TMP_DIR)/state
rm -rf ./$(TMP_DIR)/storage

View file

@ -1,5 +1,5 @@
<p align="center">
<img src="./.github/logo.svg" width="500px" alt="FrostFS">
<img src="./.forgejo/logo.svg" width="500px" alt="FrostFS">
</p>
<p align="center">
@ -7,9 +7,8 @@
</p>
---
[![Report](https://goreportcard.com/badge/github.com/TrueCloudLab/frostfs-node)](https://goreportcard.com/report/github.com/TrueCloudLab/frostfs-node)
![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/TrueCloudLab/frostfs-node?sort=semver)
![License](https://img.shields.io/github/license/TrueCloudLab/frostfs-node.svg?style=popout)
[![Report](https://goreportcard.com/badge/git.frostfs.info/TrueCloudLab/frostfs-node)](https://goreportcard.com/report/git.frostfs.info/TrueCloudLab/frostfs-node)
![Release (latest)](https://git.frostfs.info/TrueCloudLab/frostfs-node/badges/release.svg)
# Overview
@ -33,8 +32,8 @@ manipulate large amounts of data without paying a prohibitive price.
FrostFS has a native [gRPC API](https://git.frostfs.info/TrueCloudLab/frostfs-api) and has
protocol gateways for popular protocols such as [AWS
S3](https://github.com/TrueCloudLab/frostfs-s3-gw),
[HTTP](https://github.com/TrueCloudLab/frostfs-http-gw),
S3](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw),
[HTTP](https://git.frostfs.info/TrueCloudLab/frostfs-http-gw),
[FUSE](https://wikipedia.org/wiki/Filesystem_in_Userspace) and
[sFTP](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol) allowing
developers to integrate applications without rewriting their code.
@ -45,11 +44,11 @@ Now, we only support GNU/Linux on amd64 CPUs with AVX/AVX2 instructions. More
platforms will be officially supported after release `1.0`.
The latest version of frostfs-node works with frostfs-contract
[v0.16.0](https://github.com/TrueCloudLab/frostfs-contract/releases/tag/v0.16.0).
[v0.19.2](https://git.frostfs.info/TrueCloudLab/frostfs-contract/releases/tag/v0.19.2).
# Building
To make all binaries you need Go 1.21+ and `make`:
To make all binaries you need Go 1.22+ and `make`:
```
make all
```
@ -71,7 +70,7 @@ make docker/bin/frostfs-<name> # build a specific binary
## Docker images
To make docker images suitable for use in [frostfs-dev-env](https://github.com/TrueCloudLab/frostfs-dev-env/) use:
To make docker images suitable for use in [frostfs-dev-env](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env/) use:
```
make images
```
@ -99,7 +98,7 @@ See `frostfs-contract`'s README.md for build instructions.
4. To create container and put object into it run (container and object IDs will be different):
```
./bin/frostfs-cli container create -r 127.0.0.1:8080 --wallet ./dev/wallet.json --policy "REP 1 IN X CBF 1 SELECT 1 FROM * AS X" --basic-acl public-read-write --await
./bin/frostfs-cli container create -r 127.0.0.1:8080 --wallet ./dev/wallet.json --policy "REP 1 IN X CBF 1 SELECT 1 FROM * AS X" --await
Enter password > <- press ENTER, the is no password for wallet
CID: CfPhEuHQ2PRvM4gfBQDC4dWZY3NccovyfcnEdiq2ixju
@ -125,7 +124,7 @@ the feature/topic you are going to implement.
# Credits
FrostFS is maintained by [True Cloud Lab](https://github.com/TrueCloudLab/) with the help and
FrostFS is maintained by [True Cloud Lab](https://git.frostfs.info/TrueCloudLab/) with the help and
contributions from community members.
Please see [CREDITS](CREDITS.md) for details.

View file

@ -1 +1 @@
v0.42.0
v0.44.0

View file

@ -9,8 +9,8 @@ related configuration details.
To follow this guide you need:
- latest released version of [neo-go](https://github.com/nspcc-dev/neo-go/releases) (v0.97.2 at the moment),
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases) utility (v0.25.1 at the moment),
- latest released version of compiled [frostfs-contract](https://github.com/TrueCloudLab/frostfs-contract/releases) (v0.11.0 at the moment).
- latest released version of [frostfs-adm](https://git.frostfs.info/TrueCloudLab/frostfs-node/releases) utility (v0.42.9 at the moment),
- latest released version of compiled [frostfs-contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/releases) (v0.19.2 at the moment).
## Step 1: Prepare network configuration

View file

@ -20,6 +20,7 @@ const (
AlphabetWalletsFlagDesc = "Path to alphabet wallets dir"
LocalDumpFlag = "local-dump"
ProtoConfigPath = "protocol"
ContractsInitFlag = "contracts"
ContractsInitFlagDesc = "Path to archive with compiled FrostFS contracts (the default is to fetch the latest release from the official repository)"
ContractsURLFlag = "contracts-url"

View file

@ -128,7 +128,7 @@ func generateConfigExample(appDir string, credSize int) (string, error) {
tmpl.AlphabetDir = filepath.Join(appDir, "alphabet-wallets")
var i innerring.GlagoliticLetter
for i = 0; i < innerring.GlagoliticLetter(credSize); i++ {
for i = range innerring.GlagoliticLetter(credSize) {
tmpl.Glagolitics = append(tmpl.Glagolitics, i.String())
}

View file

@ -0,0 +1,15 @@
package metabase
import "github.com/spf13/cobra"
// RootCmd is a root command of config section.
var RootCmd = &cobra.Command{
Use: "metabase",
Short: "Section for metabase commands",
}
func init() {
RootCmd.AddCommand(UpgradeCmd)
initUpgradeCommand()
}

View file

@ -0,0 +1,150 @@
package metabase
import (
"context"
"errors"
"fmt"
"sync"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/config"
engineconfig "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/config/engine"
shardconfig "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/config/engine/shard"
morphconfig "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/config/morph"
nodeconfig "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/config/node"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/core/container"
meta "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
morphcontainer "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/container"
"github.com/spf13/cobra"
"golang.org/x/sync/errgroup"
)
const (
noCompactFlag = "no-compact"
)
var (
errNoPathsFound = errors.New("no metabase paths found")
errNoMorphEndpointsFound = errors.New("no morph endpoints found")
)
var UpgradeCmd = &cobra.Command{
Use: "upgrade",
Short: "Upgrade metabase to latest version",
RunE: upgrade,
}
func upgrade(cmd *cobra.Command, _ []string) error {
configFile, err := cmd.Flags().GetString(commonflags.ConfigFlag)
if err != nil {
return err
}
configDir, err := cmd.Flags().GetString(commonflags.ConfigDirFlag)
if err != nil {
return err
}
appCfg := config.New(configFile, configDir, config.EnvPrefix)
paths, err := getMetabasePaths(appCfg)
if err != nil {
return err
}
if len(paths) == 0 {
return errNoPathsFound
}
cmd.Println("found", len(paths), "metabases:")
for i, path := range paths {
cmd.Println(i+1, ":", path)
}
mc, err := createMorphClient(cmd.Context(), appCfg)
if err != nil {
return err
}
defer mc.Close()
civ, err := createContainerInfoProvider(mc)
if err != nil {
return err
}
noCompact, _ := cmd.Flags().GetBool(noCompactFlag)
result := make(map[string]bool)
var resultGuard sync.Mutex
eg, ctx := errgroup.WithContext(cmd.Context())
for _, path := range paths {
eg.Go(func() error {
var success bool
cmd.Println("upgrading metabase", path, "...")
if err := meta.Upgrade(ctx, path, !noCompact, civ, func(a ...any) {
cmd.Println(append([]any{time.Now().Format(time.RFC3339), ":", path, ":"}, a...)...)
}); err != nil {
cmd.Println("error: failed to upgrade metabase", path, ":", err)
} else {
success = true
cmd.Println("metabase", path, "upgraded successfully")
}
resultGuard.Lock()
result[path] = success
resultGuard.Unlock()
return nil
})
}
if err := eg.Wait(); err != nil {
return err
}
for mb, ok := range result {
if ok {
cmd.Println(mb, ": success")
} else {
cmd.Println(mb, ": failed")
}
}
return nil
}
func getMetabasePaths(appCfg *config.Config) ([]string, error) {
var paths []string
if err := engineconfig.IterateShards(appCfg, false, func(sc *shardconfig.Config) error {
paths = append(paths, sc.Metabase().Path())
return nil
}); err != nil {
return nil, fmt.Errorf("get metabase paths: %w", err)
}
return paths, nil
}
func createMorphClient(ctx context.Context, appCfg *config.Config) (*client.Client, error) {
addresses := morphconfig.RPCEndpoint(appCfg)
if len(addresses) == 0 {
return nil, errNoMorphEndpointsFound
}
key := nodeconfig.Key(appCfg)
cli, err := client.New(ctx,
key,
client.WithDialTimeout(morphconfig.DialTimeout(appCfg)),
client.WithEndpoints(addresses...),
client.WithSwitchInterval(morphconfig.SwitchInterval(appCfg)),
)
if err != nil {
return nil, fmt.Errorf("create morph client:%w", err)
}
return cli, nil
}
func createContainerInfoProvider(cli *client.Client) (container.InfoProvider, error) {
sh, err := cli.NNSContractAddress(client.NNSContainerContractName)
if err != nil {
return nil, fmt.Errorf("resolve container contract hash: %w", err)
}
cc, err := morphcontainer.NewFromMorph(cli, sh, 0)
if err != nil {
return nil, fmt.Errorf("create morph container client: %w", err)
}
return container.NewInfoProvider(func() (container.Source, error) {
return morphcontainer.AsContainerSource(cc), nil
}), nil
}
func initUpgradeCommand() {
flags := UpgradeCmd.Flags()
flags.Bool(noCompactFlag, false, "Do not compact upgraded metabase file")
}

View file

@ -5,35 +5,19 @@ import (
"encoding/json"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
parseutil "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
apeCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common/ape"
apechain "git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
namespaceTarget = "namespace"
containerTarget = "container"
userTarget = "user"
groupTarget = "group"
jsonFlag = "json"
jsonFlagDesc = "Output rule chains in JSON format"
chainIDFlag = "chain-id"
chainIDDesc = "Rule chain ID"
ruleFlag = "rule"
ruleFlagDesc = "Rule chain in text format"
pathFlag = "path"
pathFlagDesc = "path to encoded chain in JSON or binary format"
targetNameFlag = "target-name"
targetNameDesc = "Resource name in APE resource name format"
targetTypeFlag = "target-type"
targetTypeDesc = "Resource type(container/namespace)"
addrAdminFlag = "addr"
addrAdminDesc = "The address of the admins wallet"
chainNameFlag = "chain-name"
chainNameFlagDesc = "Chain name(ingress|s3)"
jsonFlag = "json"
jsonFlagDesc = "Output rule chains in JSON format"
addrAdminFlag = "addr"
addrAdminDesc = "The address of the admins wallet"
)
var (
@ -101,17 +85,17 @@ func initAddRuleChainCmd() {
addRuleChainCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
addRuleChainCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
addRuleChainCmd.Flags().String(targetTypeFlag, "", targetTypeDesc)
_ = addRuleChainCmd.MarkFlagRequired(targetTypeFlag)
addRuleChainCmd.Flags().String(targetNameFlag, "", targetNameDesc)
_ = addRuleChainCmd.MarkFlagRequired(targetNameFlag)
addRuleChainCmd.Flags().String(apeCmd.TargetTypeFlag, "", apeCmd.TargetTypeFlagDesc)
_ = addRuleChainCmd.MarkFlagRequired(apeCmd.TargetTypeFlag)
addRuleChainCmd.Flags().String(apeCmd.TargetNameFlag, "", apeCmd.TargetTypeFlagDesc)
_ = addRuleChainCmd.MarkFlagRequired(apeCmd.TargetNameFlag)
addRuleChainCmd.Flags().String(chainIDFlag, "", chainIDDesc)
_ = addRuleChainCmd.MarkFlagRequired(chainIDFlag)
addRuleChainCmd.Flags().StringArray(ruleFlag, []string{}, ruleFlagDesc)
addRuleChainCmd.Flags().String(pathFlag, "", pathFlagDesc)
addRuleChainCmd.Flags().String(chainNameFlag, ingress, chainNameFlagDesc)
addRuleChainCmd.MarkFlagsMutuallyExclusive(ruleFlag, pathFlag)
addRuleChainCmd.Flags().String(apeCmd.ChainIDFlag, "", apeCmd.ChainIDFlagDesc)
_ = addRuleChainCmd.MarkFlagRequired(apeCmd.ChainIDFlag)
addRuleChainCmd.Flags().StringArray(apeCmd.RuleFlag, []string{}, apeCmd.RuleFlagDesc)
addRuleChainCmd.Flags().String(apeCmd.PathFlag, "", apeCmd.PathFlagDesc)
addRuleChainCmd.Flags().String(apeCmd.ChainNameFlag, apeCmd.Ingress, apeCmd.ChainNameFlagDesc)
addRuleChainCmd.MarkFlagsMutuallyExclusive(apeCmd.RuleFlag, apeCmd.PathFlag)
}
func initRemoveRuleChainCmd() {
@ -120,26 +104,25 @@ func initRemoveRuleChainCmd() {
removeRuleChainCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
removeRuleChainCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
removeRuleChainCmd.Flags().String(targetTypeFlag, "", targetTypeDesc)
_ = removeRuleChainCmd.MarkFlagRequired(targetTypeFlag)
removeRuleChainCmd.Flags().String(targetNameFlag, "", targetNameDesc)
_ = removeRuleChainCmd.MarkFlagRequired(targetNameFlag)
removeRuleChainCmd.Flags().String(chainIDFlag, "", chainIDDesc)
removeRuleChainCmd.Flags().String(chainNameFlag, ingress, chainNameFlagDesc)
removeRuleChainCmd.Flags().String(apeCmd.TargetTypeFlag, "", apeCmd.TargetTypeFlagDesc)
_ = removeRuleChainCmd.MarkFlagRequired(apeCmd.TargetTypeFlag)
removeRuleChainCmd.Flags().String(apeCmd.TargetNameFlag, "", apeCmd.TargetNameFlagDesc)
_ = removeRuleChainCmd.MarkFlagRequired(apeCmd.TargetNameFlag)
removeRuleChainCmd.Flags().String(apeCmd.ChainIDFlag, "", apeCmd.ChainIDFlagDesc)
removeRuleChainCmd.Flags().String(apeCmd.ChainNameFlag, apeCmd.Ingress, apeCmd.ChainNameFlagDesc)
removeRuleChainCmd.Flags().Bool(commonflags.AllFlag, false, "Remove all chains for target")
removeRuleChainCmd.MarkFlagsMutuallyExclusive(commonflags.AllFlag, chainIDFlag)
removeRuleChainCmd.MarkFlagsMutuallyExclusive(commonflags.AllFlag, apeCmd.ChainIDFlag)
}
func initListRuleChainsCmd() {
Cmd.AddCommand(listRuleChainsCmd)
listRuleChainsCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
listRuleChainsCmd.Flags().StringP(targetTypeFlag, "t", "", targetTypeDesc)
_ = listRuleChainsCmd.MarkFlagRequired(targetTypeFlag)
listRuleChainsCmd.Flags().String(targetNameFlag, "", targetNameDesc)
_ = listRuleChainsCmd.MarkFlagRequired(targetNameFlag)
listRuleChainsCmd.Flags().StringP(apeCmd.TargetTypeFlag, "t", "", apeCmd.TargetTypeFlagDesc)
_ = listRuleChainsCmd.MarkFlagRequired(apeCmd.TargetTypeFlag)
listRuleChainsCmd.Flags().String(apeCmd.TargetNameFlag, "", apeCmd.TargetNameFlagDesc)
listRuleChainsCmd.Flags().Bool(jsonFlag, false, jsonFlagDesc)
listRuleChainsCmd.Flags().String(chainNameFlag, ingress, chainNameFlagDesc)
listRuleChainsCmd.Flags().String(apeCmd.ChainNameFlag, apeCmd.Ingress, apeCmd.ChainNameFlagDesc)
}
func initSetAdminCmd() {
@ -161,15 +144,15 @@ func initListTargetsCmd() {
Cmd.AddCommand(listTargetsCmd)
listTargetsCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
listTargetsCmd.Flags().StringP(targetTypeFlag, "t", "", targetTypeDesc)
_ = listTargetsCmd.MarkFlagRequired(targetTypeFlag)
listTargetsCmd.Flags().StringP(apeCmd.TargetTypeFlag, "t", "", apeCmd.TargetTypeFlagDesc)
_ = listTargetsCmd.MarkFlagRequired(apeCmd.TargetTypeFlag)
}
func addRuleChain(cmd *cobra.Command, _ []string) {
chain := parseChain(cmd)
chain := apeCmd.ParseChain(cmd)
target := parseTarget(cmd)
pci, ac := newPolicyContractInterface(cmd)
h, vub, err := pci.AddMorphRuleChain(parseChainName(cmd), target, chain)
h, vub, err := pci.AddMorphRuleChain(apeCmd.ParseChainName(cmd), target, chain)
cmd.Println("Waiting for transaction to persist...")
_, err = ac.Wait(h, vub, err)
commonCmd.ExitOnErr(cmd, "add rule chain error: %w", err)
@ -181,14 +164,14 @@ func removeRuleChain(cmd *cobra.Command, _ []string) {
pci, ac := newPolicyContractInterface(cmd)
removeAll, _ := cmd.Flags().GetBool(commonflags.AllFlag)
if removeAll {
h, vub, err := pci.RemoveMorphRuleChainsByTarget(parseChainName(cmd), target)
h, vub, err := pci.RemoveMorphRuleChainsByTarget(apeCmd.ParseChainName(cmd), target)
cmd.Println("Waiting for transaction to persist...")
_, err = ac.Wait(h, vub, err)
commonCmd.ExitOnErr(cmd, "remove rule chain error: %w", err)
cmd.Println("All chains for target removed successfully")
} else {
chainID := parseChainID(cmd)
h, vub, err := pci.RemoveMorphRuleChain(parseChainName(cmd), target, chainID)
chainID := apeCmd.ParseChainID(cmd)
h, vub, err := pci.RemoveMorphRuleChain(apeCmd.ParseChainName(cmd), target, chainID)
cmd.Println("Waiting for transaction to persist...")
_, err = ac.Wait(h, vub, err)
commonCmd.ExitOnErr(cmd, "remove rule chain error: %w", err)
@ -199,7 +182,7 @@ func removeRuleChain(cmd *cobra.Command, _ []string) {
func listRuleChains(cmd *cobra.Command, _ []string) {
target := parseTarget(cmd)
pci, _ := newPolicyContractReaderInterface(cmd)
chains, err := pci.ListMorphRuleChains(parseChainName(cmd), target)
chains, err := pci.ListMorphRuleChains(apeCmd.ParseChainName(cmd), target)
commonCmd.ExitOnErr(cmd, "list rule chains error: %w", err)
if len(chains) == 0 {
return
@ -210,14 +193,14 @@ func listRuleChains(cmd *cobra.Command, _ []string) {
prettyJSONFormat(cmd, chains)
} else {
for _, c := range chains {
parseutil.PrintHumanReadableAPEChain(cmd, c)
apeCmd.PrintHumanReadableAPEChain(cmd, c)
}
}
}
func setAdmin(cmd *cobra.Command, _ []string) {
s, _ := cmd.Flags().GetString(addrAdminFlag)
addr, err := util.Uint160DecodeStringLE(s)
addr, err := address.StringToUint160(s)
commonCmd.ExitOnErr(cmd, "can't decode admin addr: %w", err)
pci, ac := newPolicyContractInterface(cmd)
h, vub, err := pci.SetAdmin(addr)
@ -231,12 +214,11 @@ func getAdmin(cmd *cobra.Command, _ []string) {
pci, _ := newPolicyContractReaderInterface(cmd)
addr, err := pci.GetAdmin()
commonCmd.ExitOnErr(cmd, "unable to get admin: %w", err)
cmd.Println(addr.StringLE())
cmd.Println(address.Uint160ToString(addr))
}
func listTargets(cmd *cobra.Command, _ []string) {
typ, err := parseTargetType(cmd)
commonCmd.ExitOnErr(cmd, "parse target type error: %w", err)
typ := apeCmd.ParseTargetType(cmd)
pci, inv := newPolicyContractReaderInterface(cmd)
sid, it, err := pci.ListTargetsIterator(typ)

View file

@ -2,13 +2,12 @@ package ape
import (
"errors"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/constants"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/helper"
parseutil "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
apechain "git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain"
apeCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common/ape"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
policyengine "git.frostfs.info/TrueCloudLab/policy-engine/pkg/engine"
morph "git.frostfs.info/TrueCloudLab/policy-engine/pkg/morph/policy"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
@ -18,84 +17,29 @@ import (
"github.com/spf13/viper"
)
const (
ingress = "ingress"
s3 = "s3"
)
var mChainName = map[string]apechain.Name{
ingress: apechain.Ingress,
s3: apechain.S3,
}
var (
errUnknownTargetType = errors.New("unknown target type")
errChainIDCannotBeEmpty = errors.New("chain id cannot be empty")
errRuleIsNotParsed = errors.New("rule is not passed")
errUnsupportedChainName = errors.New("unsupported chain name")
)
var errUnknownTargetType = errors.New("unknown target type")
func parseTarget(cmd *cobra.Command) policyengine.Target {
name, _ := cmd.Flags().GetString(targetNameFlag)
typ, err := parseTargetType(cmd)
commonCmd.ExitOnErr(cmd, "read target type error: %w", err)
return policyengine.Target{
Name: name,
Type: typ,
}
}
func parseTargetType(cmd *cobra.Command) (policyengine.TargetType, error) {
typ, _ := cmd.Flags().GetString(targetTypeFlag)
typ := apeCmd.ParseTargetType(cmd)
name, _ := cmd.Flags().GetString(apeCmd.TargetNameFlag)
switch typ {
case namespaceTarget:
return policyengine.Namespace, nil
case containerTarget:
return policyengine.Container, nil
case userTarget:
return policyengine.User, nil
case groupTarget:
return policyengine.Group, nil
case policyengine.Namespace:
if name == "root" {
name = ""
}
return policyengine.NamespaceTarget(name)
case policyengine.Container:
var cnr cid.ID
commonCmd.ExitOnErr(cmd, "can't decode container ID: %w", cnr.DecodeString(name))
return policyengine.ContainerTarget(name)
case policyengine.User:
return policyengine.UserTarget(name)
case policyengine.Group:
return policyengine.GroupTarget(name)
default:
commonCmd.ExitOnErr(cmd, "read target type error: %w", errUnknownTargetType)
}
return -1, errUnknownTargetType
}
func parseChainID(cmd *cobra.Command) apechain.ID {
chainID, _ := cmd.Flags().GetString(chainIDFlag)
if chainID == "" {
commonCmd.ExitOnErr(cmd, "read chain id error: %w",
errChainIDCannotBeEmpty)
}
return apechain.ID(chainID)
}
func parseChain(cmd *cobra.Command) *apechain.Chain {
chain := new(apechain.Chain)
if rules, _ := cmd.Flags().GetStringArray(ruleFlag); len(rules) > 0 {
commonCmd.ExitOnErr(cmd, "parser error: %w", parseutil.ParseAPEChain(chain, rules))
} else if encPath, _ := cmd.Flags().GetString(pathFlag); encPath != "" {
commonCmd.ExitOnErr(cmd, "decode binary or json error: %w", parseutil.ParseAPEChainBinaryOrJSON(chain, encPath))
} else {
commonCmd.ExitOnErr(cmd, "parser error: %w", errRuleIsNotParsed)
}
chain.ID = parseChainID(cmd)
cmd.Println("Parsed chain:")
parseutil.PrintHumanReadableAPEChain(cmd, chain)
return chain
}
func parseChainName(cmd *cobra.Command) apechain.Name {
chainName, _ := cmd.Flags().GetString(chainNameFlag)
apeChainName, ok := mChainName[strings.ToLower(chainName)]
if !ok {
commonCmd.ExitOnErr(cmd, "", errUnsupportedChainName)
}
return apeChainName
panic("unreachable")
}
// invokerAdapter adapats invoker.Invoker to ContractStorageInvoker interface.
@ -109,16 +53,15 @@ func (n *invokerAdapter) GetRPCInvoker() invoker.RPCInvoke {
}
func newPolicyContractReaderInterface(cmd *cobra.Command) (*morph.ContractStorageReader, *invoker.Invoker) {
c, err := helper.GetN3Client(viper.GetViper())
c, err := helper.NewRemoteClient(viper.GetViper())
commonCmd.ExitOnErr(cmd, "unable to create NEO rpc client: %w", err)
inv := invoker.New(c, nil)
var ch util.Uint160
r := management.NewReader(inv)
nnsCs, err := helper.GetContractByID(r, 1)
commonCmd.ExitOnErr(cmd, "can't get NNS contract state: %w", err)
ch, err = helper.NNSResolveHash(inv, nnsCs.Hash, helper.DomainOf(constants.PolicyContract))
ch, err := helper.NNSResolveHash(inv, nnsCs.Hash, helper.DomainOf(constants.PolicyContract))
commonCmd.ExitOnErr(cmd, "unable to resolve policy contract hash: %w", err)
invokerAdapter := &invokerAdapter{
@ -130,10 +73,10 @@ func newPolicyContractReaderInterface(cmd *cobra.Command) (*morph.ContractStorag
}
func newPolicyContractInterface(cmd *cobra.Command) (*morph.ContractStorage, *helper.LocalActor) {
c, err := helper.GetN3Client(viper.GetViper())
c, err := helper.NewRemoteClient(viper.GetViper())
commonCmd.ExitOnErr(cmd, "unable to create NEO rpc client: %w", err)
ac, err := helper.NewLocalActor(cmd, c)
ac, err := helper.NewLocalActor(cmd, c, constants.ConsensusAccountName)
commonCmd.ExitOnErr(cmd, "can't create actor: %w", err)
var ch util.Uint160

View file

@ -51,7 +51,7 @@ func dumpBalances(cmd *cobra.Command, _ []string) error {
nmHash util.Uint160
)
c, err := helper.GetN3Client(viper.GetViper())
c, err := helper.NewRemoteClient(viper.GetViper())
if err != nil {
return err
}

View file

@ -26,7 +26,7 @@ import (
const forceConfigSet = "force"
func dumpNetworkConfig(cmd *cobra.Command, _ []string) error {
c, err := helper.GetN3Client(viper.GetViper())
c, err := helper.NewRemoteClient(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}

View file

@ -4,7 +4,6 @@ import "time"
const (
ConsensusAccountName = "consensus"
ProtoConfigPath = "protocol"
// MaxAlphabetNodes is the maximum number of candidates allowed, which is currently limited by the size
// of the invocation script.

View file

@ -76,7 +76,7 @@ func dumpContainers(cmd *cobra.Command, _ []string) error {
return fmt.Errorf("invalid filename: %w", err)
}
c, err := helper.GetN3Client(viper.GetViper())
c, err := helper.NewRemoteClient(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
@ -139,13 +139,12 @@ func dumpContainers(cmd *cobra.Command, _ []string) error {
func dumpSingleContainer(bw *io.BufBinWriter, ch util.Uint160, inv *invoker.Invoker, id []byte) (*Container, error) {
bw.Reset()
emit.AppCall(bw.BinWriter, ch, "get", callflag.All, id)
emit.AppCall(bw.BinWriter, ch, "eACL", callflag.All, id)
res, err := inv.Run(bw.Bytes())
if err != nil {
return nil, fmt.Errorf("can't get container info: %w", err)
}
if len(res.Stack) != 2 {
return nil, fmt.Errorf("%w: expected 2 items on stack", errInvalidContainerResponse)
if len(res.Stack) != 1 {
return nil, fmt.Errorf("%w: expected 1 items on stack", errInvalidContainerResponse)
}
cnt := new(Container)
@ -154,19 +153,11 @@ func dumpSingleContainer(bw *io.BufBinWriter, ch util.Uint160, inv *invoker.Invo
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
ea := new(EACL)
err = ea.FromStackItem(res.Stack[1])
if err != nil {
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if len(ea.Value) != 0 {
cnt.EACL = ea
}
return cnt, nil
}
func listContainers(cmd *cobra.Command, _ []string) error {
c, err := helper.GetN3Client(viper.GetViper())
c, err := helper.NewRemoteClient(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
@ -258,10 +249,6 @@ func restoreOrPutContainers(containers []Container, isOK func([]byte) bool, cmd
func putContainer(bw *io.BufBinWriter, ch util.Uint160, cnt Container) {
emit.AppCall(bw.BinWriter, ch, "put", callflag.All,
cnt.Value, cnt.Signature, cnt.PublicKey, cnt.Token)
if ea := cnt.EACL; ea != nil {
emit.AppCall(bw.BinWriter, ch, "setEACL", callflag.All,
ea.Value, ea.Signature, ea.PublicKey, ea.Token)
}
}
func isContainerRestored(cmd *cobra.Command, wCtx *helper.InitializeContext, containerHash util.Uint160, bw *io.BufBinWriter, hashValue util.Uint256) (bool, error) {
@ -322,15 +309,6 @@ type Container struct {
Signature []byte `json:"signature"`
PublicKey []byte `json:"public_key"`
Token []byte `json:"token"`
EACL *EACL `json:"eacl"`
}
// EACL represents extended ACL struct in contract storage.
type EACL struct {
Value []byte `json:"value"`
Signature []byte `json:"signature"`
PublicKey []byte `json:"public_key"`
Token []byte `json:"token"`
}
// ToStackItem implements stackitem.Convertible.
@ -377,50 +355,6 @@ func (c *Container) FromStackItem(item stackitem.Item) error {
return nil
}
// ToStackItem implements stackitem.Convertible.
func (c *EACL) ToStackItem() (stackitem.Item, error) {
return stackitem.NewStruct([]stackitem.Item{
stackitem.NewByteArray(c.Value),
stackitem.NewByteArray(c.Signature),
stackitem.NewByteArray(c.PublicKey),
stackitem.NewByteArray(c.Token),
}), nil
}
// FromStackItem implements stackitem.Convertible.
func (c *EACL) FromStackItem(item stackitem.Item) error {
arr, ok := item.Value().([]stackitem.Item)
if !ok || len(arr) != 4 {
return errors.New("invalid stack item type")
}
value, err := arr[0].TryBytes()
if err != nil {
return errors.New("invalid eACL value")
}
sig, err := arr[1].TryBytes()
if err != nil {
return errors.New("invalid eACL signature")
}
pub, err := arr[2].TryBytes()
if err != nil {
return errors.New("invalid eACL public key")
}
tok, err := arr[3].TryBytes()
if err != nil {
return errors.New("invalid eACL token")
}
c.Value = value
c.Signature = sig
c.PublicKey = pub
c.Token = tok
return nil
}
// getCIDFilterFunc returns filtering function for container IDs.
// Raw byte slices are used because it works with structures returned
// from contract.

View file

@ -36,7 +36,7 @@ type contractDumpInfo struct {
}
func dumpContractHashes(cmd *cobra.Command, _ []string) error {
c, err := helper.GetN3Client(viper.GetViper())
c, err := helper.NewRemoteClient(viper.GetViper())
if err != nil {
return fmt.Errorf("can't create N3 client: %w", err)
}
@ -68,7 +68,7 @@ func dumpContractHashes(cmd *cobra.Command, _ []string) error {
if irSize != 0 {
bw.Reset()
for i := 0; i < irSize; i++ {
for i := range irSize {
emit.AppCall(bw.BinWriter, cs.Hash, "resolve", callflag.ReadOnly,
helper.GetAlphabetNNSDomain(i),
int64(nns.TXT))
@ -79,7 +79,7 @@ func dumpContractHashes(cmd *cobra.Command, _ []string) error {
return fmt.Errorf("can't fetch info from NNS: %w", err)
}
for i := 0; i < irSize; i++ {
for i := range irSize {
info := contractDumpInfo{name: fmt.Sprintf("alphabet %d", i)}
if h, err := helper.ParseNNSResolveResult(alphaRes.Stack[i]); err == nil {
info.hash = h

View file

@ -0,0 +1,83 @@
package frostfsid
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var (
frostfsidAddSubjectKeyCmd = &cobra.Command{
Use: "add-subject-key",
Short: "Add a public key to the subject in frostfsid contract",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(commonflags.AlphabetWalletsFlag, cmd.Flags().Lookup(commonflags.AlphabetWalletsFlag))
_ = viper.BindPFlag(commonflags.EndpointFlag, cmd.Flags().Lookup(commonflags.EndpointFlag))
},
Run: frostfsidAddSubjectKey,
}
frostfsidRemoveSubjectKeyCmd = &cobra.Command{
Use: "remove-subject-key",
Short: "Remove a public key from the subject in frostfsid contract",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(commonflags.AlphabetWalletsFlag, cmd.Flags().Lookup(commonflags.AlphabetWalletsFlag))
_ = viper.BindPFlag(commonflags.EndpointFlag, cmd.Flags().Lookup(commonflags.EndpointFlag))
},
Run: frostfsidRemoveSubjectKey,
}
)
func initFrostfsIDAddSubjectKeyCmd() {
Cmd.AddCommand(frostfsidAddSubjectKeyCmd)
ff := frostfsidAddSubjectKeyCmd.Flags()
ff.StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
ff.String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
ff.String(subjectAddressFlag, "", "Subject address")
_ = frostfsidAddSubjectKeyCmd.MarkFlagRequired(subjectAddressFlag)
ff.String(subjectKeyFlag, "", "Public key to add")
_ = frostfsidAddSubjectKeyCmd.MarkFlagRequired(subjectKeyFlag)
}
func initFrostfsIDRemoveSubjectKeyCmd() {
Cmd.AddCommand(frostfsidRemoveSubjectKeyCmd)
ff := frostfsidRemoveSubjectKeyCmd.Flags()
ff.StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
ff.String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
ff.String(subjectAddressFlag, "", "Subject address")
_ = frostfsidAddSubjectKeyCmd.MarkFlagRequired(subjectAddressFlag)
ff.String(subjectKeyFlag, "", "Public key to remove")
_ = frostfsidAddSubjectKeyCmd.MarkFlagRequired(subjectKeyFlag)
}
func frostfsidAddSubjectKey(cmd *cobra.Command, _ []string) {
addr := getFrostfsIDSubjectAddress(cmd)
pub := getFrostfsIDSubjectKey(cmd)
ffsid, err := newFrostfsIDClient(cmd)
commonCmd.ExitOnErr(cmd, "init contract client: %w", err)
ffsid.addCall(ffsid.roCli.AddSubjectKeyCall(addr, pub))
err = ffsid.sendWait()
commonCmd.ExitOnErr(cmd, "add subject key: %w", err)
}
func frostfsidRemoveSubjectKey(cmd *cobra.Command, _ []string) {
addr := getFrostfsIDSubjectAddress(cmd)
pub := getFrostfsIDSubjectKey(cmd)
ffsid, err := newFrostfsIDClient(cmd)
commonCmd.ExitOnErr(cmd, "init contract client: %w", err)
ffsid.addCall(ffsid.roCli.RemoveSubjectKeyCall(addr, pub))
err = ffsid.sendWait()
commonCmd.ExitOnErr(cmd, "remove subject key: %w", err)
}

View file

@ -1,7 +1,6 @@
package frostfsid
import (
"errors"
"fmt"
"math/big"
"sort"
@ -61,7 +60,6 @@ var (
Use: "list-namespaces",
Short: "List all namespaces in frostfsid",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(commonflags.AlphabetWalletsFlag, cmd.Flags().Lookup(commonflags.AlphabetWalletsFlag))
_ = viper.BindPFlag(commonflags.EndpointFlag, cmd.Flags().Lookup(commonflags.EndpointFlag))
},
Run: frostfsidListNamespaces,
@ -91,7 +89,6 @@ var (
Use: "list-subjects",
Short: "List subjects in namespace",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(commonflags.AlphabetWalletsFlag, cmd.Flags().Lookup(commonflags.AlphabetWalletsFlag))
_ = viper.BindPFlag(commonflags.EndpointFlag, cmd.Flags().Lookup(commonflags.EndpointFlag))
},
Run: frostfsidListSubjects,
@ -121,7 +118,6 @@ var (
Use: "list-groups",
Short: "List groups in namespace",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(commonflags.AlphabetWalletsFlag, cmd.Flags().Lookup(commonflags.AlphabetWalletsFlag))
_ = viper.BindPFlag(commonflags.EndpointFlag, cmd.Flags().Lookup(commonflags.EndpointFlag))
},
Run: frostfsidListGroups,
@ -151,7 +147,6 @@ var (
Use: "list-group-subjects",
Short: "List subjects in group",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(commonflags.AlphabetWalletsFlag, cmd.Flags().Lookup(commonflags.AlphabetWalletsFlag))
_ = viper.BindPFlag(commonflags.EndpointFlag, cmd.Flags().Lookup(commonflags.EndpointFlag))
},
Run: frostfsidListGroupSubjects,
@ -169,7 +164,6 @@ func initFrostfsIDCreateNamespaceCmd() {
func initFrostfsIDListNamespacesCmd() {
Cmd.AddCommand(frostfsidListNamespacesCmd)
frostfsidListNamespacesCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
frostfsidListNamespacesCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
}
func initFrostfsIDCreateSubjectCmd() {
@ -193,7 +187,6 @@ func initFrostfsIDListSubjectsCmd() {
frostfsidListSubjectsCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
frostfsidListSubjectsCmd.Flags().String(namespaceFlag, "", "Namespace to list subjects")
frostfsidListSubjectsCmd.Flags().Bool(includeNamesFlag, false, "Whether include subject name (require additional requests)")
frostfsidListSubjectsCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
}
func initFrostfsIDCreateGroupCmd() {
@ -217,7 +210,6 @@ func initFrostfsIDListGroupsCmd() {
Cmd.AddCommand(frostfsidListGroupsCmd)
frostfsidListGroupsCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
frostfsidListGroupsCmd.Flags().String(namespaceFlag, "", "Namespace to list groups")
frostfsidListGroupsCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
}
func initFrostfsIDAddSubjectToGroupCmd() {
@ -242,7 +234,6 @@ func initFrostfsIDListGroupSubjectsCmd() {
frostfsidListGroupSubjectsCmd.Flags().String(namespaceFlag, "", "Namespace name")
frostfsidListGroupSubjectsCmd.Flags().Int64(groupIDFlag, 0, "Group id")
frostfsidListGroupSubjectsCmd.Flags().Bool(includeNamesFlag, false, "Whether include subject name (require additional requests)")
frostfsidListGroupSubjectsCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
}
func frostfsidCreateNamespace(cmd *cobra.Command, _ []string) {
@ -497,10 +488,6 @@ func (f *frostfsidClient) sendWaitRes() (*state.AppExecResult, error) {
}
f.bw.Reset()
if len(f.wCtx.SentTxs) == 0 {
return nil, errors.New("no transactions to wait")
}
f.wCtx.Command.Println("Waiting for transactions to persist...")
return f.roCli.Wait(f.wCtx.SentTxs[0].Hash, f.wCtx.SentTxs[0].Vub, nil)
}
@ -522,7 +509,7 @@ func readIterator(inv *invoker.Invoker, iter *result.Iterator, batchSize int, se
}
func initInvoker(cmd *cobra.Command) (*invoker.Invoker, *state.Contract, util.Uint160) {
c, err := helper.GetN3Client(viper.GetViper())
c, err := helper.NewRemoteClient(viper.GetViper())
commonCmd.ExitOnErr(cmd, "can't create N3 client: %w", err)
inv := invoker.New(c, nil)

View file

@ -1,59 +1,12 @@
package frostfsid
import (
"encoding/hex"
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/helper"
"git.frostfs.info/TrueCloudLab/frostfs-node/internal/ape"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
)
func TestFrostfsIDConfig(t *testing.T) {
pks := make([]*keys.PrivateKey, 4)
for i := range pks {
pk, err := keys.NewPrivateKey()
require.NoError(t, err)
pks[i] = pk
}
fmts := []string{
pks[0].GetScriptHash().StringLE(),
address.Uint160ToString(pks[1].GetScriptHash()),
hex.EncodeToString(pks[2].PublicKey().UncompressedBytes()),
hex.EncodeToString(pks[3].PublicKey().Bytes()),
}
for i := range fmts {
v := viper.New()
v.Set("frostfsid.admin", fmts[i])
actual, found, err := helper.GetFrostfsIDAdmin(v)
require.NoError(t, err)
require.True(t, found)
require.Equal(t, pks[i].GetScriptHash(), actual)
}
t.Run("bad key", func(t *testing.T) {
v := viper.New()
v.Set("frostfsid.admin", "abc")
_, found, err := helper.GetFrostfsIDAdmin(v)
require.Error(t, err)
require.True(t, found)
})
t.Run("missing key", func(t *testing.T) {
v := viper.New()
_, found, err := helper.GetFrostfsIDAdmin(v)
require.NoError(t, err)
require.False(t, found)
})
}
func TestNamespaceRegexp(t *testing.T) {
for _, tc := range []struct {
name string

View file

@ -12,4 +12,6 @@ func init() {
initFrostfsIDAddSubjectToGroupCmd()
initFrostfsIDRemoveSubjectFromGroupCmd()
initFrostfsIDListGroupSubjectsCmd()
initFrostfsIDAddSubjectKeyCmd()
initFrostfsIDRemoveSubjectKeyCmd()
}

View file

@ -73,7 +73,6 @@ func initializeWallets(v *viper.Viper, walletDir string, size int) ([]string, er
return nil, fmt.Errorf("can't fetch password: %w", err)
}
i := i
errG.Go(func() error {
p := filepath.Join(walletDir, innerring.GlagoliticLetter(i).String()+".json")
f, err := os.OpenFile(p, os.O_CREATE, 0o644)
@ -107,7 +106,6 @@ func initializeWallets(v *viper.Viper, walletDir string, size int) ([]string, er
// Create consensus account with 2*N/3+1 multi-signature.
bftCount := smartcontract.GetDefaultHonestNodeCount(size)
for i := range wallets {
i := i
ps := pubs.Copy()
errG.Go(func() error {
if err := addMultisigAccount(wallets[i], majCount, constants.CommitteeAccountName, passwords[i], ps); err != nil {

View file

@ -63,7 +63,7 @@ func TestGenerateAlphabet(t *testing.T) {
buf.Reset()
v.Set(commonflags.AlphabetWalletsFlag, walletDir)
require.NoError(t, GenerateAlphabetCmd.Flags().Set(commonflags.AlphabetSizeFlag, strconv.FormatUint(size, 10)))
for i := uint64(0); i < size; i++ {
for i := range uint64(size) {
buf.WriteString(strconv.FormatUint(i, 10) + "\r")
}

View file

@ -5,7 +5,6 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/constants"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/google/uuid"
"github.com/nspcc-dev/neo-go/pkg/core/state"
@ -31,42 +30,32 @@ type LocalActor struct {
// NewLocalActor create LocalActor with accounts form provided wallets.
// In case of empty wallets provided created actor with dummy account only for read operation.
func NewLocalActor(cmd *cobra.Command, c actor.RPCActor) (*LocalActor, error) {
//
// If wallets are provided, the contract client will use accounts with accName name from these wallets.
// To determine which account name should be used in a contract client, refer to how the contract
// verifies the transaction signature.
func NewLocalActor(cmd *cobra.Command, c actor.RPCActor, accName string) (*LocalActor, error) {
walletDir := config.ResolveHomePath(viper.GetString(commonflags.AlphabetWalletsFlag))
var act *actor.Actor
var accounts []*wallet.Account
if walletDir == "" {
account, err := wallet.NewAccount()
commonCmd.ExitOnErr(cmd, "unable to create dummy account: %w", err)
act, err = actor.New(c, []actor.SignerAccount{{
Signer: transaction.Signer{
Account: account.Contract.ScriptHash(),
Scopes: transaction.Global,
},
Account: account,
}})
if err != nil {
return nil, err
}
} else {
wallets, err := GetAlphabetWallets(viper.GetViper(), walletDir)
commonCmd.ExitOnErr(cmd, "unable to get alphabet wallets: %w", err)
for _, w := range wallets {
acc, err := GetWalletAccount(w, constants.CommitteeAccountName)
commonCmd.ExitOnErr(cmd, "can't find committee account: %w", err)
accounts = append(accounts, acc)
}
act, err = actor.New(c, []actor.SignerAccount{{
Signer: transaction.Signer{
Account: accounts[0].Contract.ScriptHash(),
Scopes: transaction.Global,
},
Account: accounts[0],
}})
if err != nil {
return nil, err
}
wallets, err := GetAlphabetWallets(viper.GetViper(), walletDir)
commonCmd.ExitOnErr(cmd, "unable to get alphabet wallets: %w", err)
for _, w := range wallets {
acc, err := GetWalletAccount(w, accName)
commonCmd.ExitOnErr(cmd, fmt.Sprintf("can't find %s account: %%w", accName), err)
accounts = append(accounts, acc)
}
act, err = actor.New(c, []actor.SignerAccount{{
Signer: transaction.Signer{
Account: accounts[0].Contract.ScriptHash(),
Scopes: transaction.Global,
},
Account: accounts[0],
}})
if err != nil {
return nil, err
}
return &LocalActor{
neoActor: act,

View file

@ -82,7 +82,7 @@ func GetContractDeployData(c *InitializeContext, ctrName string, keysParam []any
h, found, err = getFrostfsIDAdminFromContract(c.ReadOnlyInvoker)
}
if method != constants.UpdateMethodName || err == nil && !found {
h, found, err = GetFrostfsIDAdmin(viper.GetViper())
h, found, err = getFrostfsIDAdmin(viper.GetViper())
}
if err != nil {
return nil, err
@ -166,5 +166,6 @@ func DeployNNS(c *InitializeContext, method string) error {
return fmt.Errorf("can't send deploy transaction: %w", err)
}
c.Command.Println("NNS hash:", invokeHash.StringLE())
return c.AwaitTx()
}

View file

@ -11,7 +11,7 @@ import (
const frostfsIDAdminConfigKey = "frostfsid.admin"
func GetFrostfsIDAdmin(v *viper.Viper) (util.Uint160, bool, error) {
func getFrostfsIDAdmin(v *viper.Viper) (util.Uint160, bool, error) {
admin := v.GetString(frostfsIDAdminConfigKey)
if admin == "" {
return util.Uint160{}, false, nil

View file

@ -0,0 +1,53 @@
package helper
import (
"encoding/hex"
"testing"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/address"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
)
func TestFrostfsIDConfig(t *testing.T) {
pks := make([]*keys.PrivateKey, 4)
for i := range pks {
pk, err := keys.NewPrivateKey()
require.NoError(t, err)
pks[i] = pk
}
fmts := []string{
pks[0].GetScriptHash().StringLE(),
address.Uint160ToString(pks[1].GetScriptHash()),
hex.EncodeToString(pks[2].PublicKey().UncompressedBytes()),
hex.EncodeToString(pks[3].PublicKey().Bytes()),
}
for i := range fmts {
v := viper.New()
v.Set("frostfsid.admin", fmts[i])
actual, found, err := getFrostfsIDAdmin(v)
require.NoError(t, err)
require.True(t, found)
require.Equal(t, pks[i].GetScriptHash(), actual)
}
t.Run("bad key", func(t *testing.T) {
v := viper.New()
v.Set("frostfsid.admin", "abc")
_, found, err := getFrostfsIDAdmin(v)
require.Error(t, err)
require.True(t, found)
})
t.Run("missing key", func(t *testing.T) {
v := viper.New()
_, found, err := getFrostfsIDAdmin(v)
require.NoError(t, err)
require.False(t, found)
})
}

View file

@ -134,12 +134,12 @@ func NewInitializeContext(cmd *cobra.Command, v *viper.Viper) (*InitializeContex
return nil, err
}
accounts, err := createWalletAccounts(wallets)
accounts, err := getSingleAccounts(wallets)
if err != nil {
return nil, err
}
cliCtx, err := DefaultClientContext(c, committeeAcc)
cliCtx, err := defaultClientContext(c, committeeAcc)
if err != nil {
return nil, fmt.Errorf("client context: %w", err)
}
@ -191,7 +191,7 @@ func createClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet)
}
c, err = NewLocalClient(cmd, v, wallets, ldf.Value.String())
} else {
c, err = GetN3Client(v)
c, err = NewRemoteClient(v)
}
if err != nil {
return nil, fmt.Errorf("can't create N3 client: %w", err)
@ -211,7 +211,7 @@ func getContractsPath(cmd *cobra.Command, needContracts bool) (string, error) {
return ctrPath, nil
}
func createWalletAccounts(wallets []*wallet.Wallet) ([]*wallet.Account, error) {
func getSingleAccounts(wallets []*wallet.Wallet) ([]*wallet.Account, error) {
accounts := make([]*wallet.Account, len(wallets))
for i, w := range wallets {
acc, err := GetWalletAccount(w, constants.SingleAccountName)

View file

@ -8,6 +8,7 @@ import (
"sort"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/constants"
"github.com/google/uuid"
"github.com/nspcc-dev/neo-go/pkg/config"
@ -47,7 +48,7 @@ type LocalClient struct {
}
func NewLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet, dumpPath string) (*LocalClient, error) {
cfg, err := config.LoadFile(v.GetString(constants.ProtoConfigPath))
cfg, err := config.LoadFile(v.GetString(commonflags.ProtoConfigPath))
if err != nil {
return nil, err
}
@ -57,17 +58,59 @@ func NewLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet
return nil, err
}
m := smartcontract.GetDefaultHonestNodeCount(int(cfg.ProtocolConfiguration.ValidatorsCount))
accounts := make([]*wallet.Account, len(wallets))
for i := range accounts {
accounts[i], err = GetWalletAccount(wallets[i], constants.ConsensusAccountName)
if err != nil {
return nil, err
go bc.Run()
accounts, err := getBlockSigningAccounts(cfg.ProtocolConfiguration, wallets)
if err != nil {
return nil, err
}
if cmd.Name() != "init" {
if err := restoreDump(bc, dumpPath); err != nil {
return nil, fmt.Errorf("restore dump: %w", err)
}
}
return &LocalClient{
bc: bc,
dumpPath: dumpPath,
accounts: accounts,
}, nil
}
func restoreDump(bc *core.Blockchain, dumpPath string) error {
f, err := os.OpenFile(dumpPath, os.O_RDONLY, 0o600)
if err != nil {
return fmt.Errorf("can't open local dump: %w", err)
}
defer f.Close()
r := io.NewBinReaderFromIO(f)
var skip uint32
if bc.BlockHeight() != 0 {
skip = bc.BlockHeight() + 1
}
count := r.ReadU32LE() - skip
if err := chaindump.Restore(bc, r, skip, count, nil); err != nil {
return err
}
return nil
}
func getBlockSigningAccounts(cfg config.ProtocolConfiguration, wallets []*wallet.Wallet) ([]*wallet.Account, error) {
accounts := make([]*wallet.Account, len(wallets))
for i := range accounts {
acc, err := GetWalletAccount(wallets[i], constants.ConsensusAccountName)
if err != nil {
return nil, err
}
accounts[i] = acc
}
indexMap := make(map[string]int)
for i, pub := range cfg.ProtocolConfiguration.StandbyCommittee {
for i, pub := range cfg.StandbyCommittee {
indexMap[pub] = i
}
@ -76,37 +119,12 @@ func NewLocalClient(cmd *cobra.Command, v *viper.Viper, wallets []*wallet.Wallet
pj := accounts[j].PrivateKey().PublicKey().Bytes()
return indexMap[string(pi)] < indexMap[string(pj)]
})
sort.Slice(accounts[:cfg.ProtocolConfiguration.ValidatorsCount], func(i, j int) bool {
sort.Slice(accounts[:cfg.ValidatorsCount], func(i, j int) bool {
return accounts[i].PublicKey().Cmp(accounts[j].PublicKey()) == -1
})
go bc.Run()
if cmd.Name() != "init" {
f, err := os.OpenFile(dumpPath, os.O_RDONLY, 0o600)
if err != nil {
return nil, fmt.Errorf("can't open local dump: %w", err)
}
defer f.Close()
r := io.NewBinReaderFromIO(f)
var skip uint32
if bc.BlockHeight() != 0 {
skip = bc.BlockHeight() + 1
}
count := r.ReadU32LE() - skip
if err := chaindump.Restore(bc, r, skip, count, nil); err != nil {
return nil, fmt.Errorf("can't restore local dump: %w", err)
}
}
return &LocalClient{
bc: bc,
dumpPath: dumpPath,
accounts: accounts[:m],
}, nil
m := smartcontract.GetDefaultHonestNodeCount(int(cfg.ValidatorsCount))
return accounts[:m], nil
}
func (l *LocalClient) GetBlockCount() (uint32, error) {
@ -127,11 +145,6 @@ func (l *LocalClient) GetApplicationLog(h util.Uint256, t *trigger.Type) (*resul
return &a, nil
}
func (l *LocalClient) GetCommittee() (keys.PublicKeys, error) {
// not used by `morph init` command
panic("unexpected call")
}
// InvokeFunction is implemented via `InvokeScript`.
func (l *LocalClient) InvokeFunction(h util.Uint160, method string, sPrm []smartcontract.Parameter, ss []transaction.Signer) (*result.Invoke, error) {
var err error
@ -224,7 +237,7 @@ func (l *LocalClient) CalculateNetworkFee(tx *transaction.Transaction) (int64, e
paramz = []manifest.Parameter{{Type: smartcontract.SignatureType}}
} else if nSigs, _, ok := vm.ParseMultiSigContract(w.VerificationScript); ok {
paramz = make([]manifest.Parameter, nSigs)
for j := 0; j < nSigs; j++ {
for j := range nSigs {
paramz[j] = manifest.Parameter{Type: smartcontract.SignatureType}
}
}
@ -295,13 +308,7 @@ func (l *LocalClient) InvokeScript(script []byte, signers []transaction.Signer)
}
func (l *LocalClient) SendRawTransaction(tx *transaction.Transaction) (util.Uint256, error) {
// We need to test that transaction was formed correctly to catch as many errors as we can.
bs := tx.Bytes()
_, err := transaction.NewTransactionFromBytes(bs)
if err != nil {
return tx.Hash(), fmt.Errorf("invalid transaction: %w", err)
}
tx = tx.Copy()
l.transactions = append(l.transactions, tx)
return tx.Hash(), nil
}

View file

@ -10,7 +10,6 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/neorpc/result"
"github.com/nspcc-dev/neo-go/pkg/rpcclient"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
@ -25,15 +24,10 @@ import (
// Client represents N3 client interface capable of test-invoking scripts
// and sending signed transactions to chain.
type Client interface {
invoker.RPCInvoke
actor.RPCActor
GetBlockCount() (uint32, error)
GetNativeContracts() ([]state.Contract, error)
GetApplicationLog(util.Uint256, *trigger.Type) (*result.ApplicationLog, error)
GetVersion() (*result.Version, error)
SendRawTransaction(*transaction.Transaction) (util.Uint256, error)
GetCommittee() (keys.PublicKeys, error)
CalculateNetworkFee(tx *transaction.Transaction) (int64, error)
}
type HashVUBPair struct {
@ -48,7 +42,7 @@ type ClientContext struct {
SentTxs []HashVUBPair
}
func GetN3Client(v *viper.Viper) (Client, error) {
func NewRemoteClient(v *viper.Viper) (Client, error) {
// number of opened connections
// by neo-go client per one host
const (
@ -88,8 +82,14 @@ func GetN3Client(v *viper.Viper) (Client, error) {
return c, nil
}
func DefaultClientContext(c Client, committeeAcc *wallet.Account) (*ClientContext, error) {
commAct, err := NewActor(c, committeeAcc)
func defaultClientContext(c Client, committeeAcc *wallet.Account) (*ClientContext, error) {
commAct, err := actor.New(c, []actor.SignerAccount{{
Signer: transaction.Signer{
Account: committeeAcc.Contract.ScriptHash(),
Scopes: transaction.Global,
},
Account: committeeAcc,
}})
if err != nil {
return nil, err
}

View file

@ -72,13 +72,17 @@ func InvalidConfigValueErr(key string) error {
return fmt.Errorf("invalid %s config value from netmap contract", key)
}
func EmitNewEpochCall(bw *io.BufBinWriter, wCtx *InitializeContext, nmHash util.Uint160) error {
func EmitNewEpochCall(bw *io.BufBinWriter, wCtx *InitializeContext, nmHash util.Uint160, countEpoch int64) error {
if countEpoch <= 0 {
return errors.New("number of epochs cannot be less than 1")
}
curr, err := unwrap.Int64(wCtx.ReadOnlyInvoker.Call(nmHash, "epoch"))
if err != nil {
return errors.New("can't fetch current epoch from the netmap contract")
}
newEpoch := curr + 1
newEpoch := curr + countEpoch
wCtx.Command.Printf("Current epoch: %d, increase to %d.\n", curr, newEpoch)
// In NeoFS this is done via Notary contract. Here, however, we can form the

View file

@ -15,10 +15,8 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/constants"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"github.com/nspcc-dev/neo-go/pkg/core/state"
"github.com/nspcc-dev/neo-go/pkg/core/transaction"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/encoding/fixedn"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/actor"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/management"
"github.com/nspcc-dev/neo-go/pkg/wallet"
"github.com/spf13/viper"
@ -44,7 +42,7 @@ func openAlphabetWallets(v *viper.Viper, walletDir string) ([]*wallet.Wallet, er
var wallets []*wallet.Wallet
var letter string
for i := 0; i < constants.MaxAlphabetNodes; i++ {
for i := range constants.MaxAlphabetNodes {
letter = innerring.GlagoliticLetter(i).String()
p := filepath.Join(walletDir, letter+".json")
var w *wallet.Wallet
@ -87,16 +85,6 @@ func openAlphabetWallets(v *viper.Viper, walletDir string) ([]*wallet.Wallet, er
return wallets, nil
}
func NewActor(c actor.RPCActor, committeeAcc *wallet.Account) (*actor.Actor, error) {
return actor.New(c, []actor.SignerAccount{{
Signer: transaction.Signer{
Account: committeeAcc.Contract.ScriptHash(),
Scopes: transaction.Global,
},
Account: committeeAcc,
}})
}
func ReadContract(ctrPath, ctrName string) (*ContractState, error) {
rawNef, err := os.ReadFile(filepath.Join(ctrPath, ctrName+"_contract.nef"))
if err != nil {

View file

@ -1,6 +1,8 @@
package initialize
import (
"fmt"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/helper"
"github.com/nspcc-dev/neo-go/pkg/core/native/noderoles"
"github.com/nspcc-dev/neo-go/pkg/io"
@ -29,10 +31,14 @@ func setNotaryAndAlphabetNodes(c *helper.InitializeContext) error {
callflag.States|callflag.AllowNotify, int64(noderoles.NeoFSAlphabet), pubs)
if err := c.SendCommitteeTx(w.Bytes(), false); err != nil {
return err
return fmt.Errorf("send committee transaction: %w", err)
}
return c.AwaitTx()
err := c.AwaitTx()
if err != nil {
err = fmt.Errorf("await committee transaction: %w", err)
}
return err
}
func setRolesFinished(c *helper.InitializeContext) (bool, error) {

View file

@ -62,7 +62,7 @@ func testInitialize(t *testing.T, committeeSize int) {
v := viper.GetViper()
require.NoError(t, generateTestData(testdataDir, committeeSize))
v.Set(constants.ProtoConfigPath, filepath.Join(testdataDir, protoFileName))
v.Set(commonflags.ProtoConfigPath, filepath.Join(testdataDir, protoFileName))
// Set to the path or remove the next statement to download from the network.
require.NoError(t, Cmd.Flags().Set(commonflags.ContractsInitFlag, contractsPath))
@ -113,7 +113,7 @@ func generateTestData(dir string, size int) error {
}
var pubs []string
for i := 0; i < size; i++ {
for i := range size {
p := filepath.Join(dir, innerring.GlagoliticLetter(i).String()+".json")
w, err := wallet.NewWalletFromFile(p)
if err != nil {
@ -148,7 +148,7 @@ func generateTestData(dir string, size int) error {
}
func setTestCredentials(v *viper.Viper, size int) {
for i := 0; i < size; i++ {
for i := range size {
v.Set("credentials."+innerring.GlagoliticLetter(i).String(), strconv.FormatUint(uint64(i), 10))
}
v.Set("credentials.contract", constants.TestContractPassword)

View file

@ -3,6 +3,7 @@ package initialize
import (
"fmt"
"math/big"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/constants"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/helper"
@ -26,12 +27,12 @@ const (
initialAlphabetGASAmount = 10_000 * native.GASFactor
// initialProxyGASAmount represents the amount of GAS given to a proxy contract.
initialProxyGASAmount = 50_000 * native.GASFactor
// alphabetGasRatio is a coefficient that defines the threshold below which
// the balance of the alphabet node is considered not replenished. The value
// of this coefficient is determined empirically.
alphabetGasRatio = 5
)
func initialCommitteeGASAmount(c *helper.InitializeContext) int64 {
return (gasInitialTotalSupply - initialAlphabetGASAmount*int64(len(c.Wallets))) / 2
}
func transferFunds(c *helper.InitializeContext) error {
ok, err := transferFundsFinished(c)
if ok || err != nil {
@ -58,7 +59,7 @@ func transferFunds(c *helper.InitializeContext) error {
transferTarget{
Token: gas.Hash,
Address: c.CommitteeAcc.Contract.ScriptHash(),
Amount: (gasInitialTotalSupply - initialAlphabetGASAmount*int64(len(c.Wallets))) / 2,
Amount: initialCommitteeGASAmount(c),
},
transferTarget{
Token: neo.Hash,
@ -79,12 +80,19 @@ func transferFunds(c *helper.InitializeContext) error {
return c.AwaitTx()
}
// transferFundsFinished checks balances of accounts we transfer GAS to.
// The stage is considered finished if the balance is greater than the half of what we need to transfer.
func transferFundsFinished(c *helper.InitializeContext) (bool, error) {
acc := c.Accounts[0]
r := nep17.NewReader(c.ReadOnlyInvoker, gas.Hash)
res, err := r.BalanceOf(acc.Contract.ScriptHash())
return res.Cmp(big.NewInt(alphabetGasRatio*native.GASFactor)) == 1, err
if err != nil || res.Cmp(big.NewInt(initialAlphabetGASAmount/2)) != 1 {
return false, err
}
res, err = r.BalanceOf(c.CommitteeAcc.ScriptHash())
return res != nil && res.Cmp(big.NewInt(initialCommitteeGASAmount(c)/2)) == 1, err
}
func transferGASToProxy(c *helper.InitializeContext) error {
@ -144,5 +152,17 @@ func createNEP17MultiTransferTx(c helper.Client, acc *wallet.Account, recipients
if err != nil {
return nil, fmt.Errorf("can't create actor: %w", err)
}
return act.MakeRun(w.Bytes())
tx, err := act.MakeRun(w.Bytes())
if err != nil {
sum := make(map[util.Uint160]int64)
for _, recipient := range recipients {
sum[recipient.Token] += recipient.Amount
}
detail := make([]string, 0, len(sum))
for _, value := range sum {
detail = append(detail, fmt.Sprintf("amount=%v", value))
}
err = fmt.Errorf("transfer failed: from=%s(%s) %s: %w", acc.Label, acc.Address, strings.Join(detail, " "), err)
}
return tx, err
}

View file

@ -2,7 +2,6 @@ package initialize
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/constants"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@ -32,7 +31,7 @@ var Cmd = &cobra.Command{
_ = viper.BindPFlag(commonflags.ContainerFeeInitFlag, cmd.Flags().Lookup(containerFeeCLIFlag))
_ = viper.BindPFlag(commonflags.ContainerAliasFeeInitFlag, cmd.Flags().Lookup(containerAliasFeeCLIFlag))
_ = viper.BindPFlag(commonflags.WithdrawFeeInitFlag, cmd.Flags().Lookup(withdrawFeeCLIFlag))
_ = viper.BindPFlag(constants.ProtoConfigPath, cmd.Flags().Lookup(constants.ProtoConfigPath))
_ = viper.BindPFlag(commonflags.ProtoConfigPath, cmd.Flags().Lookup(commonflags.ProtoConfigPath))
},
RunE: initializeSideChainCmd,
}
@ -48,7 +47,7 @@ func initInitCmd() {
// Defaults are taken from neo-preodolenie.
Cmd.Flags().Uint64(containerFeeCLIFlag, 1000, "Container registration fee")
Cmd.Flags().Uint64(containerAliasFeeCLIFlag, 500, "Container alias fee")
Cmd.Flags().String(constants.ProtoConfigPath, "", "Path to the consensus node configuration")
Cmd.Flags().String(commonflags.ProtoConfigPath, "", "Path to the consensus node configuration")
Cmd.Flags().String(commonflags.LocalDumpFlag, "", "Path to the blocks dump file")
Cmd.MarkFlagsMutuallyExclusive(commonflags.ContractsInitFlag, commonflags.ContractsURLFlag)
}

View file

@ -12,6 +12,8 @@ import (
"github.com/spf13/viper"
)
const deltaFlag = "delta"
func ForceNewEpochCmd(cmd *cobra.Command, _ []string) error {
wCtx, err := helper.NewInitializeContext(cmd, viper.GetViper())
if err != nil {
@ -30,7 +32,8 @@ func ForceNewEpochCmd(cmd *cobra.Command, _ []string) error {
}
bw := io.NewBufBinWriter()
if err := helper.EmitNewEpochCall(bw, wCtx, nmHash); err != nil {
delta, _ := cmd.Flags().GetInt64(deltaFlag)
if err := helper.EmitNewEpochCall(bw, wCtx, nmHash, delta); err != nil {
return err
}

View file

@ -13,7 +13,7 @@ import (
)
func listNetmapCandidatesNodes(cmd *cobra.Command, _ []string) {
c, err := helper.GetN3Client(viper.GetViper())
c, err := helper.NewRemoteClient(viper.GetViper())
commonCmd.ExitOnErr(cmd, "can't create N3 client: %w", err)
inv := invoker.New(c, nil)

View file

@ -12,7 +12,6 @@ var (
Short: "List netmap candidates nodes",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(commonflags.EndpointFlag, cmd.Flags().Lookup(commonflags.EndpointFlag))
_ = viper.BindPFlag(commonflags.AlphabetWalletsFlag, cmd.Flags().Lookup(commonflags.AlphabetWalletsFlag))
},
Run: listNetmapCandidatesNodes,
}
@ -35,6 +34,7 @@ func initForceNewEpochCmd() {
ForceNewEpoch.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
ForceNewEpoch.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
ForceNewEpoch.Flags().String(commonflags.LocalDumpFlag, "", "Path to the blocks dump file")
ForceNewEpoch.Flags().Int64(deltaFlag, 1, "Number of epochs to increase the current epoch")
}
func init() {

View file

@ -24,7 +24,7 @@ func initRegisterCmd() {
}
func registerDomain(cmd *cobra.Command, _ []string) {
c, actor, _ := getRPCClient(cmd)
c, actor := nnsWriter(cmd)
name, _ := cmd.Flags().GetString(nnsNameFlag)
email, _ := cmd.Flags().GetString(nnsEmailFlag)
@ -42,3 +42,23 @@ func registerDomain(cmd *cobra.Command, _ []string) {
commonCmd.ExitOnErr(cmd, "register domain error: %w", err)
cmd.Println("Domain registered successfully")
}
func initDeleteCmd() {
Cmd.AddCommand(deleteCmd)
deleteCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
deleteCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
deleteCmd.Flags().String(nnsNameFlag, "", nnsNameFlagDesc)
_ = cobra.MarkFlagRequired(deleteCmd.Flags(), nnsNameFlag)
}
func deleteDomain(cmd *cobra.Command, _ []string) {
c, actor := nnsWriter(cmd)
name, _ := cmd.Flags().GetString(nnsNameFlag)
h, vub, err := c.DeleteDomain(name)
_, err = actor.Wait(h, vub, err)
commonCmd.ExitOnErr(cmd, "delete domain error: %w", err)
cmd.Println("Domain deleted successfully")
}

View file

@ -2,24 +2,37 @@ package nns
import (
client "git.frostfs.info/TrueCloudLab/frostfs-contract/rpcclient/nns"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/constants"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/helper"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/invoker"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/management"
"github.com/nspcc-dev/neo-go/pkg/util"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
func getRPCClient(cmd *cobra.Command) (*client.Contract, *helper.LocalActor, util.Uint160) {
func nnsWriter(cmd *cobra.Command) (*client.Contract, *helper.LocalActor) {
v := viper.GetViper()
c, err := helper.GetN3Client(v)
c, err := helper.NewRemoteClient(v)
commonCmd.ExitOnErr(cmd, "unable to create NEO rpc client: %w", err)
ac, err := helper.NewLocalActor(cmd, c)
ac, err := helper.NewLocalActor(cmd, c, constants.CommitteeAccountName)
commonCmd.ExitOnErr(cmd, "can't create actor: %w", err)
r := management.NewReader(ac.Invoker)
nnsCs, err := helper.GetContractByID(r, 1)
commonCmd.ExitOnErr(cmd, "can't get NNS contract state: %w", err)
return client.New(ac, nnsCs.Hash), ac, nnsCs.Hash
return client.New(ac, nnsCs.Hash), ac
}
func nnsReader(cmd *cobra.Command) (*client.ContractReader, *invoker.Invoker) {
c, err := helper.NewRemoteClient(viper.GetViper())
commonCmd.ExitOnErr(cmd, "unable to create NEO rpc client: %w", err)
inv := invoker.New(c, nil)
r := management.NewReader(inv)
nnsCs, err := helper.GetContractByID(r, 1)
commonCmd.ExitOnErr(cmd, "can't get NNS contract state: %w", err)
return client.NewReader(inv, nnsCs.Hash), inv
}

View file

@ -8,7 +8,6 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/nspcc-dev/neo-go/pkg/rpcclient/unwrap"
"github.com/nspcc-dev/neo-go/pkg/vm/stackitem"
"github.com/spf13/cobra"
)
@ -29,7 +28,6 @@ func initAddRecordCmd() {
func initGetRecordsCmd() {
Cmd.AddCommand(getRecordsCmd)
getRecordsCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
getRecordsCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
getRecordsCmd.Flags().String(nnsNameFlag, "", nnsNameFlagDesc)
getRecordsCmd.Flags().String(nnsRecordTypeFlag, "", nnsRecordTypeFlagDesc)
@ -47,8 +45,21 @@ func initDelRecordsCmd() {
_ = cobra.MarkFlagRequired(delRecordsCmd.Flags(), nnsRecordTypeFlag)
}
func initDelRecordCmd() {
Cmd.AddCommand(delRecordCmd)
delRecordCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
delRecordCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
delRecordCmd.Flags().String(nnsNameFlag, "", nnsNameFlagDesc)
delRecordCmd.Flags().String(nnsRecordTypeFlag, "", nnsRecordTypeFlagDesc)
delRecordCmd.Flags().String(nnsRecordDataFlag, "", nnsRecordDataFlagDesc)
_ = cobra.MarkFlagRequired(delRecordCmd.Flags(), nnsNameFlag)
_ = cobra.MarkFlagRequired(delRecordCmd.Flags(), nnsRecordTypeFlag)
_ = cobra.MarkFlagRequired(delRecordCmd.Flags(), nnsRecordDataFlag)
}
func addRecord(cmd *cobra.Command, _ []string) {
c, actor, _ := getRPCClient(cmd)
c, actor := nnsWriter(cmd)
name, _ := cmd.Flags().GetString(nnsNameFlag)
data, _ := cmd.Flags().GetString(nnsRecordDataFlag)
recordType, _ := cmd.Flags().GetString(nnsRecordTypeFlag)
@ -64,16 +75,16 @@ func addRecord(cmd *cobra.Command, _ []string) {
}
func getRecords(cmd *cobra.Command, _ []string) {
c, act, hash := getRPCClient(cmd)
c, inv := nnsReader(cmd)
name, _ := cmd.Flags().GetString(nnsNameFlag)
recordType, _ := cmd.Flags().GetString(nnsRecordTypeFlag)
if recordType == "" {
sid, r, err := unwrap.SessionIterator(act.Invoker.Call(hash, "getAllRecords", name))
sid, r, err := c.GetAllRecords(name)
commonCmd.ExitOnErr(cmd, "unable to get records: %w", err)
defer func() {
_ = act.Invoker.TerminateSession(sid)
_ = inv.TerminateSession(sid)
}()
items, err := act.Invoker.TraverseIterator(sid, &r, 0)
items, err := inv.TraverseIterator(sid, &r, 0)
commonCmd.ExitOnErr(cmd, "unable to get records: %w", err)
for len(items) != 0 {
for j := range items {
@ -84,7 +95,7 @@ func getRecords(cmd *cobra.Command, _ []string) {
recordTypeToString(nns.RecordType(rs[1].Value().(*big.Int).Int64())),
string(bs))
}
items, err = act.Invoker.TraverseIterator(sid, &r, 0)
items, err = inv.TraverseIterator(sid, &r, 0)
commonCmd.ExitOnErr(cmd, "unable to get records: %w", err)
}
} else {
@ -101,7 +112,7 @@ func getRecords(cmd *cobra.Command, _ []string) {
}
func delRecords(cmd *cobra.Command, _ []string) {
c, actor, _ := getRPCClient(cmd)
c, actor := nnsWriter(cmd)
name, _ := cmd.Flags().GetString(nnsNameFlag)
recordType, _ := cmd.Flags().GetString(nnsRecordTypeFlag)
typ, err := getRecordType(recordType)
@ -115,6 +126,22 @@ func delRecords(cmd *cobra.Command, _ []string) {
cmd.Println("Records removed successfully")
}
func delRecord(cmd *cobra.Command, _ []string) {
c, actor := nnsWriter(cmd)
name, _ := cmd.Flags().GetString(nnsNameFlag)
data, _ := cmd.Flags().GetString(nnsRecordDataFlag)
recordType, _ := cmd.Flags().GetString(nnsRecordTypeFlag)
typ, err := getRecordType(recordType)
commonCmd.ExitOnErr(cmd, "unable to parse record type: %w", err)
h, vub, err := c.DeleteRecord(name, typ, data)
commonCmd.ExitOnErr(cmd, "unable to delete record: %w", err)
cmd.Println("Waiting for transaction to persist...")
_, err = actor.Wait(h, vub, err)
commonCmd.ExitOnErr(cmd, "delete records error: %w", err)
cmd.Println("Record removed successfully")
}
func getRecordType(recordType string) (*big.Int, error) {
switch strings.ToUpper(recordType) {
case "A":

View file

@ -14,7 +14,7 @@ func initRenewCmd() {
}
func renewDomain(cmd *cobra.Command, _ []string) {
c, actor, _ := getRPCClient(cmd)
c, actor := nnsWriter(cmd)
name, _ := cmd.Flags().GetString(nnsNameFlag)
h, vub, err := c.Renew(name)
commonCmd.ExitOnErr(cmd, "unable to renew domain: %w", err)

View file

@ -42,6 +42,15 @@ var (
},
Run: registerDomain,
}
deleteCmd = &cobra.Command{
Use: "delete",
Short: "Delete a domain by name",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(commonflags.EndpointFlag, cmd.Flags().Lookup(commonflags.EndpointFlag))
_ = viper.BindPFlag(commonflags.AlphabetWalletsFlag, cmd.Flags().Lookup(commonflags.AlphabetWalletsFlag))
},
Run: deleteDomain,
}
renewCmd = &cobra.Command{
Use: "renew",
Short: "Increases domain expiration date",
@ -86,14 +95,25 @@ var (
},
Run: delRecords,
}
delRecordCmd = &cobra.Command{
Use: "delete-record",
Short: "Removes domain record with the specified type and data",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(commonflags.EndpointFlag, cmd.Flags().Lookup(commonflags.EndpointFlag))
_ = viper.BindPFlag(commonflags.AlphabetWalletsFlag, cmd.Flags().Lookup(commonflags.AlphabetWalletsFlag))
},
Run: delRecord,
}
)
func init() {
initTokensCmd()
initRegisterCmd()
initDeleteCmd()
initRenewCmd()
initUpdateCmd()
initAddRecordCmd()
initGetRecordsCmd()
initDelRecordsCmd()
initDelRecordCmd()
}

View file

@ -1,24 +1,65 @@
package nns
import (
"math/big"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
client "git.frostfs.info/TrueCloudLab/frostfs-contract/rpcclient/nns"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/spf13/cobra"
)
const (
verboseDesc = "Include additional information about CNAME record."
)
func initTokensCmd() {
Cmd.AddCommand(tokensCmd)
tokensCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
tokensCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
tokensCmd.Flags().BoolP(commonflags.Verbose, commonflags.VerboseShorthand, false, verboseDesc)
}
func listTokens(cmd *cobra.Command, _ []string) {
c, _, _ := getRPCClient(cmd)
c, _ := nnsReader(cmd)
it, err := c.Tokens()
commonCmd.ExitOnErr(cmd, "unable to get tokens: %w", err)
for toks, err := it.Next(10); err == nil && len(toks) > 0; toks, err = it.Next(10) {
for _, token := range toks {
cmd.Println(string(token))
output := string(token)
if verbose, _ := cmd.Flags().GetBool(commonflags.Verbose); verbose {
cname, err := getCnameRecord(c, token)
commonCmd.ExitOnErr(cmd, "", err)
if cname != "" {
output += " (CNAME: " + cname + ")"
}
}
cmd.Println(output)
}
}
}
func getCnameRecord(c *client.ContractReader, token []byte) (string, error) {
items, err := c.GetRecords(string(token), big.NewInt(int64(nns.CNAME)))
// GetRecords returns the error "not an array" if the domain does not contain records.
if err != nil && strings.Contains(err.Error(), "not an array") {
return "", nil
}
if err != nil {
return "", err
}
if len(items) == 0 {
return "", nil
}
record, err := items[0].TryBytes()
if err != nil {
return "", err
}
return string(record), nil
}

View file

@ -30,7 +30,7 @@ func initUpdateCmd() {
}
func updateSOA(cmd *cobra.Command, _ []string) {
c, actor, _ := getRPCClient(cmd)
c, actor := nnsWriter(cmd)
name, _ := cmd.Flags().GetString(nnsNameFlag)
email, _ := cmd.Flags().GetString(nnsEmailFlag)

View file

@ -53,7 +53,7 @@ func RemoveNodesCmd(cmd *cobra.Command, args []string) error {
int64(netmapcontract.NodeStateOffline), nodeKeys[i].Bytes())
}
if err := helper.EmitNewEpochCall(bw, wCtx, nmHash); err != nil {
if err := helper.EmitNewEpochCall(bw, wCtx, nmHash, 1); err != nil {
return err
}

View file

@ -89,7 +89,7 @@ func depositNotary(cmd *cobra.Command, _ []string) error {
}
func transferGas(cmd *cobra.Command, acc *wallet.Account, accHash util.Uint160, gasAmount fixedn.Fixed8, till int64) error {
c, err := helper.GetN3Client(viper.GetViper())
c, err := helper.NewRemoteClient(viper.GetViper())
if err != nil {
return err
}

View file

@ -62,7 +62,7 @@ func SetPolicyCmd(cmd *cobra.Command, args []string) error {
}
func dumpPolicyCmd(cmd *cobra.Command, _ []string) error {
c, err := helper.GetN3Client(viper.GetViper())
c, err := helper.NewRemoteClient(viper.GetViper())
commonCmd.ExitOnErr(cmd, "can't create N3 client:", err)
inv := invoker.New(c, nil)

View file

@ -30,11 +30,13 @@ var (
func initProxyAddAccount() {
AddAccountCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
AddAccountCmd.Flags().String(accountAddressFlag, "", "Wallet address string")
AddAccountCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
}
func initProxyRemoveAccount() {
RemoveAccountCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
RemoveAccountCmd.Flags().String(accountAddressFlag, "", "Wallet address string")
RemoveAccountCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
}
func init() {

View file

@ -5,6 +5,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/metabase"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/storagecfg"
"git.frostfs.info/TrueCloudLab/frostfs-node/misc"
@ -41,6 +42,7 @@ func init() {
rootCmd.AddCommand(config.RootCmd)
rootCmd.AddCommand(morph.RootCmd)
rootCmd.AddCommand(storagecfg.RootCmd)
rootCmd.AddCommand(metabase.RootCmd)
rootCmd.AddCommand(autocomplete.Command("frostfs-adm"))
rootCmd.AddCommand(gendoc.Command(rootCmd, gendoc.Options{}))

View file

@ -72,4 +72,3 @@ All other `object` sub-commands support only static sessions (2).
List of commands supporting sessions (static only):
- `create`
- `delete`
- `set-eacl`

View file

@ -2,11 +2,13 @@ package internal
import (
"bytes"
"cmp"
"context"
"errors"
"fmt"
"io"
"sort"
"os"
"slices"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/accounting"
@ -14,7 +16,6 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
containerSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
@ -76,13 +77,31 @@ func ListContainers(ctx context.Context, prm ListContainersPrm) (res ListContain
// SortedIDList returns sorted list of identifiers of user's containers.
func (x ListContainersRes) SortedIDList() []cid.ID {
list := x.cliRes.Containers()
sort.Slice(list, func(i, j int) bool {
lhs, rhs := list[i].EncodeToString(), list[j].EncodeToString()
return strings.Compare(lhs, rhs) < 0
slices.SortFunc(list, func(lhs, rhs cid.ID) int {
return strings.Compare(lhs.EncodeToString(), rhs.EncodeToString())
})
return list
}
func ListContainersStream(ctx context.Context, prm ListContainersPrm, processCnr func(id cid.ID) bool) (err error) {
cliPrm := &client.PrmContainerListStream{
XHeaders: prm.XHeaders,
OwnerID: prm.OwnerID,
Session: prm.Session,
}
rdr, err := prm.cli.ContainerListInit(ctx, *cliPrm)
if err != nil {
return fmt.Errorf("init container list: %w", err)
}
err = rdr.Iterate(processCnr)
if err != nil {
return fmt.Errorf("read container list: %w", err)
}
return
}
// PutContainerPrm groups parameters of PutContainer operation.
type PutContainerPrm struct {
Client *client.Client
@ -189,54 +208,6 @@ func DeleteContainer(ctx context.Context, prm DeleteContainerPrm) (res DeleteCon
return
}
// EACLPrm groups parameters of EACL operation.
type EACLPrm struct {
Client *client.Client
ClientParams client.PrmContainerEACL
}
// EACLRes groups the resulting values of EACL operation.
type EACLRes struct {
cliRes *client.ResContainerEACL
}
// EACL returns requested eACL table.
func (x EACLRes) EACL() eacl.Table {
return x.cliRes.Table()
}
// EACL reads eACL table from FrostFS by container ID.
//
// Returns any error which prevented the operation from completing correctly in error return.
func EACL(ctx context.Context, prm EACLPrm) (res EACLRes, err error) {
res.cliRes, err = prm.Client.ContainerEACL(ctx, prm.ClientParams)
return
}
// SetEACLPrm groups parameters of SetEACL operation.
type SetEACLPrm struct {
Client *client.Client
ClientParams client.PrmContainerSetEACL
}
// SetEACLRes groups the resulting values of SetEACL operation.
type SetEACLRes struct{}
// SetEACL requests to save an eACL table in FrostFS.
//
// Operation is asynchronous and no guaranteed even in the absence of errors.
// The required time is also not predictable.
//
// Success can be verified by reading by container identifier.
//
// Returns any error which prevented the operation from completing correctly in error return.
func SetEACL(ctx context.Context, prm SetEACLPrm) (res SetEACLRes, err error) {
_, err = prm.Client.ContainerSetEACL(ctx, prm.ClientParams)
return
}
// NetworkInfoPrm groups parameters of NetworkInfo operation.
type NetworkInfoPrm struct {
Client *client.Client
@ -611,13 +582,6 @@ type HeadObjectPrm struct {
commonObjectPrm
objectAddressPrm
rawPrm
mainOnly bool
}
// SetMainOnlyFlag sets flag to get only main fields of an object header in terms of FrostFS API.
func (x *HeadObjectPrm) SetMainOnlyFlag(v bool) {
x.mainOnly = v
}
// HeadObjectRes groups the resulting values of HeadObject operation.
@ -712,9 +676,7 @@ func SearchObjects(ctx context.Context, prm SearchObjectsPrm) (*SearchObjectsRes
for {
n, ok = rdr.Read(buf)
for i := 0; i < n; i++ {
list = append(list, buf[i])
}
list = append(list, buf[:n]...)
if !ok {
break
}
@ -725,9 +687,8 @@ func SearchObjects(ctx context.Context, prm SearchObjectsPrm) (*SearchObjectsRes
return nil, fmt.Errorf("read object list: %w", err)
}
sort.Slice(list, func(i, j int) bool {
lhs, rhs := list[i].EncodeToString(), list[j].EncodeToString()
return strings.Compare(lhs, rhs) < 0
slices.SortFunc(list, func(a, b oid.ID) int {
return strings.Compare(a.EncodeToString(), b.EncodeToString())
})
return &SearchObjectsRes{
@ -892,3 +853,65 @@ func SyncContainerSettings(ctx context.Context, prm SyncContainerPrm) (*SyncCont
return new(SyncContainerRes), nil
}
// PatchObjectPrm groups parameters of PatchObject operation.
type PatchObjectPrm struct {
commonObjectPrm
objectAddressPrm
NewAttributes []objectSDK.Attribute
ReplaceAttribute bool
PayloadPatches []PayloadPatch
}
type PayloadPatch struct {
Range objectSDK.Range
PayloadPath string
}
type PatchRes struct {
OID oid.ID
}
func Patch(ctx context.Context, prm PatchObjectPrm) (*PatchRes, error) {
patchPrm := client.PrmObjectPatch{
XHeaders: prm.xHeaders,
BearerToken: prm.bearerToken,
Session: prm.sessionToken,
Address: prm.objAddr,
}
slices.SortFunc(prm.PayloadPatches, func(a, b PayloadPatch) int {
return cmp.Compare(a.Range.GetOffset(), b.Range.GetOffset())
})
patcher, err := prm.cli.ObjectPatchInit(ctx, patchPrm)
if err != nil {
return nil, fmt.Errorf("init payload reading: %w", err)
}
if patcher.PatchAttributes(ctx, prm.NewAttributes, prm.ReplaceAttribute) {
for _, pp := range prm.PayloadPatches {
payloadFile, err := os.OpenFile(pp.PayloadPath, os.O_RDONLY, os.ModePerm)
if err != nil {
return nil, err
}
applied := patcher.PatchPayload(ctx, &pp.Range, payloadFile)
_ = payloadFile.Close()
if !applied {
break
}
}
}
res, err := patcher.Close(ctx)
if err != nil {
return nil, err
}
return &PatchRes{
OID: res.ObjectID(),
}, nil
}

View file

@ -58,6 +58,7 @@ func GetSDKClient(ctx context.Context, cmd *cobra.Command, key *ecdsa.PrivateKey
GRPCDialOptions: []grpc.DialOption{
grpc.WithChainUnaryInterceptor(tracing.NewUnaryClientInteceptor()),
grpc.WithChainStreamInterceptor(tracing.NewStreamClientInterceptor()),
grpc.WithDefaultCallOptions(grpc.WaitForReady(true)),
},
}
if timeout := viper.GetDuration(commonflags.Timeout); timeout > 0 {

View file

@ -2,7 +2,7 @@ package common
import (
"context"
"sort"
"slices"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
@ -45,15 +45,11 @@ func StartClientCommandSpan(cmd *cobra.Command) {
})
commonCmd.ExitOnErr(cmd, "init tracing: %w", err)
var components sort.StringSlice
var components []string
for c := cmd; c != nil; c = c.Parent() {
components = append(components, c.Name())
}
for i, j := 0, len(components)-1; i < j; {
components.Swap(i, j)
i++
j--
}
slices.Reverse(components)
operation := strings.Join(components, ".")
ctx, span := tracing.StartSpanFromContext(cmd.Context(), operation)

View file

@ -24,6 +24,8 @@ var testCmd = &cobra.Command{
}
func Test_getOrGenerate(t *testing.T) {
t.Cleanup(viper.Reset)
dir := t.TempDir()
wallPath := filepath.Join(dir, "wallet.json")

View file

@ -1,45 +1,19 @@
package apemanager
import (
"encoding/hex"
"errors"
"fmt"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
apeCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common/ape"
apeSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/ape"
client_sdk "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
apechain "git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain"
"git.frostfs.info/TrueCloudLab/policy-engine/pkg/engine"
"github.com/spf13/cobra"
)
const (
chainIDFlag = "chain-id"
chainIDHexFlag = "chain-id-hex"
ruleFlag = "rule"
pathFlag = "path"
)
const (
targetNameFlag = "target-name"
targetNameDesc = "Resource name in APE resource name format"
targetTypeFlag = "target-type"
targetTypeDesc = "Resource type(container/namespace)"
)
const (
defaultNamespace = ""
namespaceTarget = "namespace"
containerTarget = "container"
userTarget = "user"
groupTarget = "group"
)
var errUnknownTargetType = errors.New("unknown target type")
var addCmd = &cobra.Command{
Use: "add",
Short: "Add rule chain for a target",
@ -50,55 +24,28 @@ var addCmd = &cobra.Command{
}
func parseTarget(cmd *cobra.Command) (ct apeSDK.ChainTarget) {
typ, _ := cmd.Flags().GetString(targetTypeFlag)
name, _ := cmd.Flags().GetString(targetNameFlag)
t := apeCmd.ParseTarget(cmd)
ct.Name = name
ct.Name = t.Name
switch typ {
case namespaceTarget:
switch t.Type {
case engine.Namespace:
ct.TargetType = apeSDK.TargetTypeNamespace
case containerTarget:
var cnr cid.ID
commonCmd.ExitOnErr(cmd, "can't decode container ID: %w", cnr.DecodeString(name))
case engine.Container:
ct.TargetType = apeSDK.TargetTypeContainer
case userTarget:
case engine.User:
ct.TargetType = apeSDK.TargetTypeUser
case groupTarget:
case engine.Group:
ct.TargetType = apeSDK.TargetTypeGroup
default:
commonCmd.ExitOnErr(cmd, "read target type error: %w", errUnknownTargetType)
commonCmd.ExitOnErr(cmd, "conversion error: %w", fmt.Errorf("unknown type '%c'", t.Type))
}
return ct
}
func parseChain(cmd *cobra.Command) apeSDK.Chain {
chainID, _ := cmd.Flags().GetString(chainIDFlag)
hexEncoded, _ := cmd.Flags().GetBool(chainIDHexFlag)
chainIDRaw := []byte(chainID)
if hexEncoded {
var err error
chainIDRaw, err = hex.DecodeString(chainID)
commonCmd.ExitOnErr(cmd, "can't decode chain ID as hex: %w", err)
}
chain := new(apechain.Chain)
chain.ID = apechain.ID(chainIDRaw)
if rules, _ := cmd.Flags().GetStringArray(ruleFlag); len(rules) > 0 {
commonCmd.ExitOnErr(cmd, "parser error: %w", util.ParseAPEChain(chain, rules))
} else if encPath, _ := cmd.Flags().GetString(pathFlag); encPath != "" {
commonCmd.ExitOnErr(cmd, "decode binary or json error: %w", util.ParseAPEChainBinaryOrJSON(chain, encPath))
} else {
commonCmd.ExitOnErr(cmd, "parser error: %w", errors.New("rule is not passed"))
}
cmd.Println("Parsed chain:")
util.PrintHumanReadableAPEChain(cmd, chain)
serialized := chain.Bytes()
c := apeCmd.ParseChain(cmd)
serialized := c.Bytes()
return apeSDK.Chain{
Raw: serialized,
}
@ -127,13 +74,13 @@ func initAddCmd() {
commonflags.Init(addCmd)
ff := addCmd.Flags()
ff.StringArray(ruleFlag, []string{}, "Rule statement")
ff.String(pathFlag, "", "Path to encoded chain in JSON or binary format")
ff.String(chainIDFlag, "", "Assign ID to the parsed chain")
ff.String(targetNameFlag, "", targetNameDesc)
ff.String(targetTypeFlag, "", targetTypeDesc)
_ = addCmd.MarkFlagRequired(targetTypeFlag)
ff.Bool(chainIDHexFlag, false, "Flag to parse chain ID as hex")
ff.StringArray(apeCmd.RuleFlag, []string{}, apeCmd.RuleFlagDesc)
ff.String(apeCmd.PathFlag, "", apeCmd.PathFlagDesc)
ff.String(apeCmd.ChainIDFlag, "", apeCmd.ChainIDFlagDesc)
ff.String(apeCmd.TargetNameFlag, "", apeCmd.TargetNameFlagDesc)
ff.String(apeCmd.TargetTypeFlag, "", apeCmd.TargetTypeFlagDesc)
_ = addCmd.MarkFlagRequired(apeCmd.TargetTypeFlag)
ff.Bool(apeCmd.ChainIDHexFlag, false, apeCmd.ChainIDHexFlagDesc)
addCmd.MarkFlagsMutuallyExclusive(pathFlag, ruleFlag)
addCmd.MarkFlagsMutuallyExclusive(apeCmd.PathFlag, apeCmd.RuleFlag)
}

View file

@ -4,8 +4,8 @@ import (
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
apeutil "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
apeCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common/ape"
client_sdk "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
apechain "git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain"
"github.com/spf13/cobra"
@ -35,7 +35,7 @@ func list(cmd *cobra.Command, _ []string) {
for _, respChain := range resp.Chains {
var chain apechain.Chain
commonCmd.ExitOnErr(cmd, "decode error: %w", chain.DecodeBytes(respChain.Raw))
apeutil.PrintHumanReadableAPEChain(cmd, &chain)
apeCmd.PrintHumanReadableAPEChain(cmd, &chain)
}
}
@ -43,7 +43,7 @@ func initListCmd() {
commonflags.Init(listCmd)
ff := listCmd.Flags()
ff.String(targetNameFlag, "", targetNameDesc)
ff.String(targetTypeFlag, "", targetTypeDesc)
_ = listCmd.MarkFlagRequired(targetTypeFlag)
ff.String(apeCmd.TargetNameFlag, "", apeCmd.TargetNameFlagDesc)
ff.String(apeCmd.TargetTypeFlag, "", apeCmd.TargetTypeFlagDesc)
_ = listCmd.MarkFlagRequired(apeCmd.TargetTypeFlag)
}

View file

@ -1,29 +1,23 @@
package apemanager
import (
"encoding/hex"
"errors"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
apeCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common/ape"
client_sdk "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
)
var (
errEmptyChainID = errors.New("chain id cannot be empty")
removeCmd = &cobra.Command{
Use: "remove",
Short: "Remove rule chain for a target",
Run: remove,
PersistentPreRun: func(cmd *cobra.Command, _ []string) {
commonflags.Bind(cmd)
},
}
)
var removeCmd = &cobra.Command{
Use: "remove",
Short: "Remove rule chain for a target",
Run: remove,
PersistentPreRun: func(cmd *cobra.Command, _ []string) {
commonflags.Bind(cmd)
},
}
func remove(cmd *cobra.Command, _ []string) {
target := parseTarget(cmd)
@ -31,19 +25,9 @@ func remove(cmd *cobra.Command, _ []string) {
key := key.Get(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, key, commonflags.RPC)
chainID, _ := cmd.Flags().GetString(chainIDFlag)
if chainID == "" {
commonCmd.ExitOnErr(cmd, "read chain id error: %w", errEmptyChainID)
}
chainID := apeCmd.ParseChainID(cmd)
chainIDRaw := []byte(chainID)
hexEncoded, _ := cmd.Flags().GetBool(chainIDHexFlag)
if hexEncoded {
var err error
chainIDRaw, err = hex.DecodeString(chainID)
commonCmd.ExitOnErr(cmd, "can't decode chain ID as hex: %w", err)
}
_, err := cli.APEManagerRemoveChain(cmd.Context(), client_sdk.PrmAPEManagerRemoveChain{
ChainTarget: target,
ChainID: chainIDRaw,
@ -58,9 +42,10 @@ func initRemoveCmd() {
commonflags.Init(removeCmd)
ff := removeCmd.Flags()
ff.String(targetNameFlag, "", targetNameDesc)
ff.String(targetTypeFlag, "", targetTypeDesc)
_ = removeCmd.MarkFlagRequired(targetTypeFlag)
ff.String(chainIDFlag, "", "Chain id")
ff.Bool(chainIDHexFlag, false, "Flag to parse chain ID as hex")
ff.String(apeCmd.TargetNameFlag, "", apeCmd.TargetNameFlagDesc)
ff.String(apeCmd.TargetTypeFlag, "", apeCmd.TargetTypeFlagDesc)
_ = removeCmd.MarkFlagRequired(apeCmd.TargetTypeFlag)
ff.String(apeCmd.ChainIDFlag, "", apeCmd.ChainIDFlagDesc)
_ = removeCmd.MarkFlagRequired(apeCmd.ChainIDFlag)
ff.Bool(apeCmd.ChainIDHexFlag, false, apeCmd.ChainIDHexFlagDesc)
}

View file

@ -1,31 +1,20 @@
package bearer
import (
"errors"
"fmt"
"os"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
parseutil "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
apeCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common/ape"
apeSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/ape"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
cidSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
apechain "git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain"
"github.com/spf13/cobra"
)
var (
errChainIDCannotBeEmpty = errors.New("chain id cannot be empty")
errRuleIsNotParsed = errors.New("rule is not passed")
)
const (
chainIDFlag = "chain-id"
chainIDHexFlag = "chain-id-hex"
ruleFlag = "rule"
pathFlag = "path"
outputFlag = "output"
outputFlag = "output"
)
var generateAPEOverrideCmd = &cobra.Command{
@ -40,7 +29,7 @@ Generated APE override can be dumped to a file in JSON format that is passed to
}
func genereateAPEOverride(cmd *cobra.Command, _ []string) {
c := parseChain(cmd)
c := apeCmd.ParseChain(cmd)
targetCID, _ := cmd.Flags().GetString(commonflags.CIDFlag)
var cid cidSDK.ID
@ -77,39 +66,11 @@ func init() {
ff.StringP(commonflags.CIDFlag, "", "", "Target container ID.")
_ = cobra.MarkFlagRequired(createCmd.Flags(), commonflags.CIDFlag)
ff.StringArray(ruleFlag, []string{}, "Rule statement")
ff.String(pathFlag, "", "Path to encoded chain in JSON or binary format")
ff.String(chainIDFlag, "", "Assign ID to the parsed chain")
ff.Bool(chainIDHexFlag, false, "Flag to parse chain ID as hex")
ff.StringArray(apeCmd.RuleFlag, []string{}, "Rule statement")
ff.String(apeCmd.PathFlag, "", "Path to encoded chain in JSON or binary format")
ff.String(apeCmd.ChainIDFlag, "", "Assign ID to the parsed chain")
ff.Bool(apeCmd.ChainIDHexFlag, false, "Flag to parse chain ID as hex")
ff.String(outputFlag, "", "Output path to dump result JSON-encoded APE override")
_ = cobra.MarkFlagFilename(createCmd.Flags(), outputFlag)
}
func parseChainID(cmd *cobra.Command) apechain.ID {
chainID, _ := cmd.Flags().GetString(chainIDFlag)
if chainID == "" {
commonCmd.ExitOnErr(cmd, "read chain id error: %w",
errChainIDCannotBeEmpty)
}
return apechain.ID(chainID)
}
func parseChain(cmd *cobra.Command) *apechain.Chain {
chain := new(apechain.Chain)
if rules, _ := cmd.Flags().GetStringArray(ruleFlag); len(rules) > 0 {
commonCmd.ExitOnErr(cmd, "parser error: %w", parseutil.ParseAPEChain(chain, rules))
} else if encPath, _ := cmd.Flags().GetString(pathFlag); encPath != "" {
commonCmd.ExitOnErr(cmd, "decode binary or json error: %w", parseutil.ParseAPEChainBinaryOrJSON(chain, encPath))
} else {
commonCmd.ExitOnErr(cmd, "parser error: %w", errRuleIsNotParsed)
}
chain.ID = parseChainID(cmd)
cmd.Println("Parsed chain:")
parseutil.PrintHumanReadableAPEChain(cmd, chain)
return chain
}

View file

@ -7,22 +7,20 @@ import (
"strings"
"time"
containerApi "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
containerApi "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/container"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/acl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra"
)
var (
containerACL string
containerPolicy string
containerAttributes []string
containerAwait bool
@ -89,9 +87,6 @@ It will be stored in sidechain when inner ring will accepts it.`,
err = parseAttributes(&cnr, containerAttributes)
commonCmd.ExitOnErr(cmd, "", err)
var basicACL acl.Basic
commonCmd.ExitOnErr(cmd, "decode basic ACL string: %w", basicACL.DecodeString(containerACL))
tok := getSession(cmd)
if tok != nil {
@ -105,7 +100,6 @@ It will be stored in sidechain when inner ring will accepts it.`,
}
cnr.SetPlacementPolicy(*placementPolicy)
cnr.SetBasicACL(basicACL)
var syncContainerPrm internalclient.SyncContainerPrm
syncContainerPrm.SetClient(cli)
@ -139,7 +133,7 @@ It will be stored in sidechain when inner ring will accepts it.`,
},
}
for i := 0; i < awaitTimeout; i++ {
for range awaitTimeout {
time.Sleep(1 * time.Second)
_, err := internalclient.GetContainer(cmd.Context(), getPrm)
@ -163,10 +157,6 @@ func initContainerCreateCmd() {
flags.DurationP(commonflags.Timeout, commonflags.TimeoutShorthand, commonflags.TimeoutDefault, commonflags.TimeoutUsage)
flags.StringP(commonflags.WalletPath, commonflags.WalletPathShorthand, commonflags.WalletPathDefault, commonflags.WalletPathUsage)
flags.StringP(commonflags.Account, commonflags.AccountShorthand, commonflags.AccountDefault, commonflags.AccountUsage)
flags.StringVar(&containerACL, "basic-acl", acl.NamePrivate, fmt.Sprintf("HEX encoded basic ACL value or keywords like '%s', '%s', '%s'",
acl.NamePublicRW, acl.NamePrivate, acl.NamePublicROExtended,
))
flags.StringVarP(&containerPolicy, "policy", "p", "", "QL-encoded or JSON-encoded placement policy or path to file with it")
flags.StringSliceVarP(&containerAttributes, "attributes", "a", nil, "Comma separated pairs of container attributes in form of Key1=Value1,Key2=Value2")
flags.BoolVar(&containerAwait, "await", false, "Block execution until container is persisted")

View file

@ -110,7 +110,7 @@ Only owner of the container has a permission to remove container.`,
},
}
for i := 0; i < awaitTimeout; i++ {
for range awaitTimeout {
time.Sleep(1 * time.Second)
_, err := internalclient.GetContainer(cmd.Context(), getPrm)

View file

@ -1,68 +0,0 @@
package container
import (
"os"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
)
var getExtendedACLCmd = &cobra.Command{
Use: "get-eacl",
Short: "Get extended ACL table of container",
Long: `Get extended ACL table of container`,
Run: func(cmd *cobra.Command, _ []string) {
id := parseContainerID(cmd)
pk := key.GetOrGenerate(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
eaclPrm := internalclient.EACLPrm{
Client: cli,
ClientParams: client.PrmContainerEACL{
ContainerID: &id,
},
}
res, err := internalclient.EACL(cmd.Context(), eaclPrm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
eaclTable := res.EACL()
if containerPathTo == "" {
cmd.Println("eACL: ")
common.PrettyPrintJSON(cmd, &eaclTable, "eACL")
return
}
var data []byte
if containerJSON {
data, err = eaclTable.MarshalJSON()
commonCmd.ExitOnErr(cmd, "can't encode to JSON: %w", err)
} else {
data, err = eaclTable.Marshal()
commonCmd.ExitOnErr(cmd, "can't encode to binary: %w", err)
}
cmd.Println("dumping data to file:", containerPathTo)
err = os.WriteFile(containerPathTo, data, 0o644)
commonCmd.ExitOnErr(cmd, "could not write eACL to file: %w", err)
},
}
func initContainerGetEACLCmd() {
commonflags.Init(getExtendedACLCmd)
flags := getExtendedACLCmd.Flags()
flags.StringVar(&containerID, commonflags.CIDFlag, "", commonflags.CIDFlagUsage)
flags.StringVar(&containerPathTo, "to", "", "Path to dump encoded container (default: binary encoded)")
flags.BoolVar(&containerJSON, commonflags.JSON, false, "Encode EACL table in json format")
}

View file

@ -1,16 +1,16 @@
package container
import (
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
containerSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
// flags of list command.
@ -54,49 +54,60 @@ var listContainersCmd = &cobra.Command{
var prm internalclient.ListContainersPrm
prm.SetClient(cli)
prm.Account = idUser
res, err := internalclient.ListContainers(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
prm.OwnerID = idUser
prmGet := internalclient.GetContainerPrm{
Client: cli,
}
var containerIDs []cid.ID
err := internalclient.ListContainersStream(cmd.Context(), prm, func(id cid.ID) bool {
printContainer(cmd, prmGet, id)
return false
})
if err == nil {
return
}
if e, ok := status.FromError(err); ok && e.Code() == codes.Unimplemented {
res, err := internalclient.ListContainers(cmd.Context(), prm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
containerIDs = res.SortedIDList()
} else {
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
}
containerIDs := res.SortedIDList()
for _, cnrID := range containerIDs {
if flagVarListName == "" && !flagVarListPrintAttr {
cmd.Println(cnrID.String())
continue
}
cnrID := cnrID
prmGet.ClientParams.ContainerID = &cnrID
res, err := internalclient.GetContainer(cmd.Context(), prmGet)
if err != nil {
cmd.Printf(" failed to read attributes: %v\n", err)
continue
}
cnr := res.Container()
if cnrName := containerSDK.Name(cnr); flagVarListName != "" && cnrName != flagVarListName {
continue
}
cmd.Println(cnrID.String())
if flagVarListPrintAttr {
cnr.IterateAttributes(func(key, val string) {
if !strings.HasPrefix(key, container.SysAttributePrefix) && !strings.HasPrefix(key, container.SysAttributePrefixNeoFS) {
// FIXME(@cthulhu-rider): https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/issues/97
// Use dedicated method to skip system attributes.
cmd.Printf(" %s: %s\n", key, val)
}
})
}
printContainer(cmd, prmGet, cnrID)
}
},
}
func printContainer(cmd *cobra.Command, prmGet internalclient.GetContainerPrm, id cid.ID) {
if flagVarListName == "" && !flagVarListPrintAttr {
cmd.Println(id.String())
return
}
prmGet.ClientParams.ContainerID = &id
res, err := internalclient.GetContainer(cmd.Context(), prmGet)
if err != nil {
cmd.Printf(" failed to read attributes: %v\n", err)
return
}
cnr := res.Container()
if cnrName := containerSDK.Name(cnr); flagVarListName != "" && cnrName != flagVarListName {
return
}
cmd.Println(id.String())
if flagVarListPrintAttr {
cnr.IterateUserAttributes(func(key, val string) {
cmd.Printf(" %s: %s\n", key, val)
})
}
}
func initContainerListContainersCmd() {
commonflags.Init(listContainersCmd)

View file

@ -1,9 +1,6 @@
package container
import (
"strings"
v2object "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
@ -67,14 +64,8 @@ var listContainerObjectsCmd = &cobra.Command{
resHead, err := internalclient.HeadObject(cmd.Context(), prmHead)
if err == nil {
attrs := resHead.Header().Attributes()
for i := range attrs {
attrKey := attrs[i].Key()
if !strings.HasPrefix(attrKey, v2object.SysAttributePrefix) && !strings.HasPrefix(attrKey, v2object.SysAttributePrefixNeoFS) {
// FIXME(@cthulhu-rider): https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/issues/97
// Use dedicated method to skip system attributes.
cmd.Printf(" %s: %s\n", attrKey, attrs[i].Value())
}
for _, attr := range resHead.Header().UserAttributes() {
cmd.Printf(" %s: %s\n", attr.Key(), attr.Value())
}
} else {
cmd.Printf(" failed to read attributes: %v\n", err)

View file

@ -1,11 +1,10 @@
package container
import (
"bufio"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"strings"
@ -14,6 +13,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/chzyer/readline"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
@ -163,6 +163,16 @@ func (repl *policyPlaygroundREPL) netMap() netmap.NetMap {
return nm
}
var policyPlaygroundCompleter = readline.NewPrefixCompleter(
readline.PcItem("list"),
readline.PcItem("ls"),
readline.PcItem("add"),
readline.PcItem("load"),
readline.PcItem("remove"),
readline.PcItem("rm"),
readline.PcItem("eval"),
)
func (repl *policyPlaygroundREPL) run() error {
if len(viper.GetString(commonflags.RPC)) > 0 {
key := key.GetOrGenerate(repl.cmd)
@ -189,22 +199,38 @@ func (repl *policyPlaygroundREPL) run() error {
"rm": repl.handleRemove,
"eval": repl.handleEval,
}
for reader := bufio.NewReader(os.Stdin); ; {
fmt.Print("> ")
line, err := reader.ReadString('\n')
rl, err := readline.NewEx(&readline.Config{
Prompt: "> ",
InterruptPrompt: "^C",
AutoComplete: policyPlaygroundCompleter,
})
if err != nil {
return fmt.Errorf("error initializing readline: %w", err)
}
defer rl.Close()
var exit bool
for {
line, err := rl.Readline()
if err != nil {
if err == io.EOF {
return nil
if errors.Is(err, readline.ErrInterrupt) {
if exit {
return nil
}
exit = true
continue
}
return fmt.Errorf("reading line: %v", err)
return fmt.Errorf("reading line: %w", err)
}
exit = false
parts := strings.Fields(line)
if len(parts) == 0 {
continue
}
cmd := parts[0]
handler, exists := cmdHandlers[cmd]
if exists {
if handler, exists := cmdHandlers[cmd]; exists {
if err := handler(parts[1:]); err != nil {
fmt.Printf("error: %v\n", err)
}

View file

@ -25,8 +25,6 @@ func init() {
deleteContainerCmd,
listContainerObjectsCmd,
getContainerInfoCmd,
getExtendedACLCmd,
setExtendedACLCmd,
containerNodesCmd,
policyPlaygroundCmd,
}
@ -38,8 +36,6 @@ func init() {
initContainerDeleteCmd()
initContainerListObjectsCmd()
initContainerInfoCmd()
initContainerGetEACLCmd()
initContainerSetEACLCmd()
initContainerNodesCmd()
initContainerPolicyPlaygroundCmd()
@ -53,7 +49,6 @@ func init() {
}{
{createContainerCmd, "PUT"},
{deleteContainerCmd, "DELETE"},
{setExtendedACLCmd, "SETEACL"},
} {
commonflags.InitSession(el.cmd, "container "+el.verb)
}

View file

@ -1,108 +0,0 @@
package container
import (
"bytes"
"errors"
"time"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
)
var flagVarsSetEACL struct {
noPreCheck bool
srcPath string
}
var setExtendedACLCmd = &cobra.Command{
Use: "set-eacl",
Short: "Set new extended ACL table for container",
Long: `Set new extended ACL table for container.
Container ID in EACL table will be substituted with ID from the CLI.`,
Run: func(cmd *cobra.Command, _ []string) {
id := parseContainerID(cmd)
eaclTable := common.ReadEACL(cmd, flagVarsSetEACL.srcPath)
tok := getSession(cmd)
eaclTable.SetCID(id)
pk := key.GetOrGenerate(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
if !flagVarsSetEACL.noPreCheck {
cmd.Println("Checking the ability to modify access rights in the container...")
extendable, err := internalclient.IsACLExtendable(cmd.Context(), cli, id)
commonCmd.ExitOnErr(cmd, "Extensibility check failure: %w", err)
if !extendable {
commonCmd.ExitOnErr(cmd, "", errors.New("container ACL is immutable"))
}
cmd.Println("ACL extension is enabled in the container, continue processing.")
}
setEACLPrm := internalclient.SetEACLPrm{
Client: cli,
ClientParams: client.PrmContainerSetEACL{
Table: eaclTable,
Session: tok,
},
}
_, err := internalclient.SetEACL(cmd.Context(), setEACLPrm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
if containerAwait {
exp, err := eaclTable.Marshal()
commonCmd.ExitOnErr(cmd, "broken EACL table: %w", err)
cmd.Println("awaiting...")
getEACLPrm := internalclient.EACLPrm{
Client: cli,
ClientParams: client.PrmContainerEACL{
ContainerID: &id,
},
}
for i := 0; i < awaitTimeout; i++ {
time.Sleep(1 * time.Second)
res, err := internalclient.EACL(cmd.Context(), getEACLPrm)
if err == nil {
// compare binary values because EACL could have been set already
table := res.EACL()
got, err := table.Marshal()
if err != nil {
continue
}
if bytes.Equal(exp, got) {
cmd.Println("EACL has been persisted on sidechain")
return
}
}
}
commonCmd.ExitOnErr(cmd, "", errSetEACLTimeout)
}
},
}
func initContainerSetEACLCmd() {
commonflags.Init(setExtendedACLCmd)
flags := setExtendedACLCmd.Flags()
flags.StringVar(&containerID, commonflags.CIDFlag, "", commonflags.CIDFlagUsage)
flags.StringVar(&flagVarsSetEACL.srcPath, "table", "", "path to file with JSON or binary encoded EACL table")
flags.BoolVar(&containerAwait, "await", false, "block execution until EACL is persisted")
flags.BoolVar(&flagVarsSetEACL.noPreCheck, "no-precheck", false, "do not pre-check the extensibility of the container ACL")
}

View file

@ -18,9 +18,8 @@ const (
)
var (
errCreateTimeout = errors.New("timeout: container has not been persisted on sidechain")
errDeleteTimeout = errors.New("timeout: container has not been removed from sidechain")
errSetEACLTimeout = errors.New("timeout: EACL has not been persisted on sidechain")
errCreateTimeout = errors.New("timeout: container has not been persisted on sidechain")
errDeleteTimeout = errors.New("timeout: container has not been removed from sidechain")
)
func parseContainerID(cmd *cobra.Command) cid.ID {

View file

@ -1,23 +1,14 @@
package control
import (
"encoding/hex"
"errors"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
apeCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common/ape"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
apechain "git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
"github.com/spf13/cobra"
)
const (
ruleFlag = "rule"
pathFlag = "path"
)
var addRuleCmd = &cobra.Command{
Use: "add-rule",
Short: "Add local override",
@ -31,41 +22,12 @@ control add-rule --endpoint ... -w ... --address ... --chain-id ChainID --cid ..
Run: addRule,
}
func parseChain(cmd *cobra.Command) *apechain.Chain {
chainID, _ := cmd.Flags().GetString(chainIDFlag)
hexEncoded, _ := cmd.Flags().GetBool(chainIDHexFlag)
chainIDRaw := []byte(chainID)
if hexEncoded {
var err error
chainIDRaw, err = hex.DecodeString(chainID)
commonCmd.ExitOnErr(cmd, "can't decode chain ID as hex: %w", err)
}
chain := new(apechain.Chain)
chain.ID = apechain.ID(chainIDRaw)
if rules, _ := cmd.Flags().GetStringArray(ruleFlag); len(rules) > 0 {
commonCmd.ExitOnErr(cmd, "parser error: %w", util.ParseAPEChain(chain, rules))
} else if encPath, _ := cmd.Flags().GetString(pathFlag); encPath != "" {
commonCmd.ExitOnErr(cmd, "decode binary or json error: %w", util.ParseAPEChainBinaryOrJSON(chain, encPath))
} else {
commonCmd.ExitOnErr(cmd, "parser error", errors.New("rule is not passed"))
}
cmd.Println("Parsed chain:")
util.PrintHumanReadableAPEChain(cmd, chain)
return chain
}
func addRule(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd)
target := parseTarget(cmd)
parsed := parseChain(cmd)
parsed := apeCmd.ParseChain(cmd)
req := &control.AddChainLocalOverrideRequest{
Body: &control.AddChainLocalOverrideRequest_Body{
@ -94,13 +56,13 @@ func initControlAddRuleCmd() {
initControlFlags(addRuleCmd)
ff := addRuleCmd.Flags()
ff.StringArray(ruleFlag, []string{}, "Rule statement")
ff.String(pathFlag, "", "Path to encoded chain in JSON or binary format")
ff.String(chainIDFlag, "", "Assign ID to the parsed chain")
ff.String(targetNameFlag, "", targetNameDesc)
ff.String(targetTypeFlag, "", targetTypeDesc)
_ = addRuleCmd.MarkFlagRequired(targetTypeFlag)
ff.Bool(chainIDHexFlag, false, "Flag to parse chain ID as hex")
ff.StringArray(apeCmd.RuleFlag, []string{}, "Rule statement")
ff.String(apeCmd.PathFlag, "", "Path to encoded chain in JSON or binary format")
ff.String(apeCmd.ChainIDFlag, "", "Assign ID to the parsed chain")
ff.String(apeCmd.TargetNameFlag, "", apeCmd.TargetNameFlagDesc)
ff.String(apeCmd.TargetTypeFlag, "", apeCmd.TargetTypeFlagDesc)
_ = addRuleCmd.MarkFlagRequired(apeCmd.TargetTypeFlag)
ff.Bool(apeCmd.ChainIDHexFlag, false, "Flag to parse chain ID as hex")
addRuleCmd.MarkFlagsMutuallyExclusive(pathFlag, ruleFlag)
addRuleCmd.MarkFlagsMutuallyExclusive(apeCmd.PathFlag, apeCmd.RuleFlag)
}

View file

@ -1,10 +1,10 @@
package control
import (
rawclient "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
"github.com/spf13/cobra"
)

View file

@ -1,10 +1,10 @@
package control
import (
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
"github.com/spf13/cobra"
)

View file

@ -1,10 +1,10 @@
package control
import (
rawclient "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
"github.com/spf13/cobra"
)

View file

@ -1,56 +0,0 @@
package control
import (
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
"github.com/spf13/cobra"
)
const ignoreErrorsFlag = "no-errors"
var evacuateShardCmd = &cobra.Command{
Use: "evacuate",
Short: "Evacuate objects from shard",
Long: "Evacuate objects from shard to other shards",
Run: evacuateShard,
Deprecated: "use frostfs-cli control shards evacuation start",
}
func evacuateShard(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd)
req := &control.EvacuateShardRequest{Body: new(control.EvacuateShardRequest_Body)}
req.Body.Shard_ID = getShardIDList(cmd)
req.Body.IgnoreErrors, _ = cmd.Flags().GetBool(ignoreErrorsFlag)
signRequest(cmd, pk, req)
cli := getClient(cmd, pk)
var resp *control.EvacuateShardResponse
var err error
err = cli.ExecRaw(func(client *client.Client) error {
resp, err = control.EvacuateShard(client, req)
return err
})
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
cmd.Printf("Objects moved: %d\n", resp.GetBody().GetCount())
verifyResponse(cmd, resp.GetSignature(), resp.GetBody())
cmd.Println("Shard has successfully been evacuated.")
}
func initControlEvacuateShardCmd() {
initControlFlags(evacuateShardCmd)
flags := evacuateShardCmd.Flags()
flags.StringSlice(shardIDFlag, nil, "List of shard IDs in base58 encoding")
flags.Bool(shardAllFlag, false, "Process all shards")
flags.Bool(ignoreErrorsFlag, false, "Skip invalid/unreadable objects")
evacuateShardCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag)
}

View file

@ -7,19 +7,24 @@ import (
"sync/atomic"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
clientSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
)
const (
awaitFlag = "await"
noProgressFlag = "no-progress"
scopeFlag = "scope"
awaitFlag = "await"
noProgressFlag = "no-progress"
scopeFlag = "scope"
repOneOnlyFlag = "rep-one-only"
ignoreErrorsFlag = "no-errors"
containerWorkerCountFlag = "container-worker-count"
objectWorkerCountFlag = "object-worker-count"
scopeAll = "all"
scopeObjects = "objects"
@ -64,12 +69,18 @@ func startEvacuateShard(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd)
ignoreErrors, _ := cmd.Flags().GetBool(ignoreErrorsFlag)
containerWorkerCount, _ := cmd.Flags().GetUint32(containerWorkerCountFlag)
objectWorkerCount, _ := cmd.Flags().GetUint32(objectWorkerCountFlag)
repOneOnly, _ := cmd.Flags().GetBool(repOneOnlyFlag)
req := &control.StartShardEvacuationRequest{
Body: &control.StartShardEvacuationRequest_Body{
Shard_ID: getShardIDList(cmd),
IgnoreErrors: ignoreErrors,
Scope: getEvacuationScope(cmd),
Shard_ID: getShardIDList(cmd),
IgnoreErrors: ignoreErrors,
Scope: getEvacuationScope(cmd),
ContainerWorkerCount: containerWorkerCount,
ObjectWorkerCount: objectWorkerCount,
RepOneOnly: repOneOnly,
},
}
@ -371,6 +382,9 @@ func initControlStartEvacuationShardCmd() {
flags.String(scopeFlag, scopeAll, fmt.Sprintf("Evacuation scope; possible values: %s, %s, %s", scopeTrees, scopeObjects, scopeAll))
flags.Bool(awaitFlag, false, "Block execution until evacuation is completed")
flags.Bool(noProgressFlag, false, fmt.Sprintf("Print progress if %s provided", awaitFlag))
flags.Uint32(containerWorkerCountFlag, 0, "Count of concurrent container evacuation workers")
flags.Uint32(objectWorkerCountFlag, 0, "Count of concurrent object evacuation workers")
flags.Bool(repOneOnlyFlag, false, "Evacuate objects only from containers with policy 'REP 1 ...'")
startEvacuationShardCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag)
}

View file

@ -1,10 +1,10 @@
package control
import (
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
"github.com/spf13/cobra"
)

View file

@ -3,11 +3,11 @@ package control
import (
"encoding/hex"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
apecmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common/ape"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
apechain "git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain"
"github.com/spf13/cobra"
)
@ -24,8 +24,8 @@ func getRule(cmd *cobra.Command, _ []string) {
target := parseTarget(cmd)
chainID, _ := cmd.Flags().GetString(chainIDFlag)
hexEncoded, _ := cmd.Flags().GetBool(chainIDHexFlag)
chainID, _ := cmd.Flags().GetString(apecmd.ChainIDFlag)
hexEncoded, _ := cmd.Flags().GetBool(apecmd.ChainIDHexFlag)
if hexEncoded {
chainIDBytes, err := hex.DecodeString(chainID)
@ -56,16 +56,16 @@ func getRule(cmd *cobra.Command, _ []string) {
var chain apechain.Chain
commonCmd.ExitOnErr(cmd, "decode error: %w", chain.DecodeBytes(resp.GetBody().GetChain()))
util.PrintHumanReadableAPEChain(cmd, &chain)
apecmd.PrintHumanReadableAPEChain(cmd, &chain)
}
func initControGetRuleCmd() {
initControlFlags(getRuleCmd)
ff := getRuleCmd.Flags()
ff.String(targetNameFlag, "", targetNameDesc)
ff.String(targetTypeFlag, "", targetTypeDesc)
_ = getRuleCmd.MarkFlagRequired(targetTypeFlag)
ff.String(chainIDFlag, "", "Chain id")
ff.Bool(chainIDHexFlag, false, "Flag to parse chain ID as hex")
ff.String(apecmd.TargetNameFlag, "", apecmd.TargetNameFlagDesc)
ff.String(apecmd.TargetTypeFlag, "", apecmd.TargetTypeFlagDesc)
_ = getRuleCmd.MarkFlagRequired(apecmd.TargetTypeFlag)
ff.String(apecmd.ChainIDFlag, "", "Chain id")
ff.Bool(apecmd.ChainIDHexFlag, false, "Flag to parse chain ID as hex")
}

View file

@ -3,11 +3,11 @@ package control
import (
"os"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
"github.com/spf13/cobra"
)

View file

@ -3,12 +3,12 @@ package control
import (
"os"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
ircontrol "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/ir"
ircontrolsrv "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/ir/server"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
"github.com/spf13/cobra"
)

View file

@ -1,13 +1,13 @@
package control
import (
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/refs"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
ircontrol "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/ir"
ircontrolsrv "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/ir/server"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/refs"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/spf13/cobra"

View file

@ -4,11 +4,11 @@ import (
"encoding/hex"
"errors"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
ircontrol "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/ir"
ircontrolsrv "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/ir/server"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
"github.com/spf13/cobra"
)

View file

@ -1,11 +1,11 @@
package control
import (
rawclient "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
ircontrol "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/ir"
ircontrolsrv "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/ir/server"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
"github.com/spf13/cobra"
)

View file

@ -1,18 +1,16 @@
package control
import (
"errors"
"fmt"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/modules/util"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
apeCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common/ape"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
apechain "git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain"
"github.com/nspcc-dev/neo-go/cli/input"
policyengine "git.frostfs.info/TrueCloudLab/policy-engine/pkg/engine"
"github.com/spf13/cobra"
)
@ -23,65 +21,25 @@ var listRulesCmd = &cobra.Command{
Run: listRules,
}
const (
defaultNamespace = "root"
namespaceTarget = "namespace"
containerTarget = "container"
userTarget = "user"
groupTarget = "group"
)
const (
targetNameFlag = "target-name"
targetNameDesc = "Resource name in APE resource name format"
targetTypeFlag = "target-type"
targetTypeDesc = "Resource type(container/namespace)"
)
var (
errSettingDefaultValueWasDeclined = errors.New("setting default value was declined")
errUnknownTargetType = errors.New("unknown target type")
)
var engineToControlSvcType = map[policyengine.TargetType]control.ChainTarget_TargetType{
policyengine.Namespace: control.ChainTarget_NAMESPACE,
policyengine.Container: control.ChainTarget_CONTAINER,
policyengine.User: control.ChainTarget_USER,
policyengine.Group: control.ChainTarget_GROUP,
}
func parseTarget(cmd *cobra.Command) *control.ChainTarget {
typ, _ := cmd.Flags().GetString(targetTypeFlag)
name, _ := cmd.Flags().GetString(targetNameFlag)
switch typ {
case namespaceTarget:
if name == "" {
ln, err := input.ReadLine(fmt.Sprintf("Target name is not set. Confirm to use %s namespace (n|Y)> ", defaultNamespace))
commonCmd.ExitOnErr(cmd, "read line error: %w", err)
ln = strings.ToLower(ln)
if len(ln) > 0 && (ln[0] == 'n') {
commonCmd.ExitOnErr(cmd, "read namespace error: %w", errSettingDefaultValueWasDeclined)
}
name = defaultNamespace
}
return &control.ChainTarget{
Name: name,
Type: control.ChainTarget_NAMESPACE,
}
case containerTarget:
var cnr cid.ID
commonCmd.ExitOnErr(cmd, "can't decode container ID: %w", cnr.DecodeString(name))
return &control.ChainTarget{
Name: name,
Type: control.ChainTarget_CONTAINER,
}
case userTarget:
return &control.ChainTarget{
Name: name,
Type: control.ChainTarget_USER,
}
case groupTarget:
return &control.ChainTarget{
Name: name,
Type: control.ChainTarget_GROUP,
}
default:
commonCmd.ExitOnErr(cmd, "read target type error: %w", errUnknownTargetType)
target := apeCmd.ParseTarget(cmd)
typ, ok := engineToControlSvcType[target.Type]
if !ok {
commonCmd.ExitOnErr(cmd, "%w", fmt.Errorf("unknown type '%c", target.Type))
}
return &control.ChainTarget{
Name: target.Name,
Type: typ,
}
return nil
}
func listRules(cmd *cobra.Command, _ []string) {
@ -117,7 +75,7 @@ func listRules(cmd *cobra.Command, _ []string) {
for _, c := range chains {
var chain apechain.Chain
commonCmd.ExitOnErr(cmd, "decode error: %w", chain.DecodeBytes(c))
util.PrintHumanReadableAPEChain(cmd, &chain)
apeCmd.PrintHumanReadableAPEChain(cmd, &chain)
}
}
@ -125,7 +83,7 @@ func initControlListRulesCmd() {
initControlFlags(listRulesCmd)
ff := listRulesCmd.Flags()
ff.String(targetNameFlag, "", targetNameDesc)
ff.String(targetTypeFlag, "", targetTypeDesc)
_ = listRulesCmd.MarkFlagRequired(targetTypeFlag)
ff.String(apeCmd.TargetNameFlag, "", apeCmd.TargetNameFlagDesc)
ff.String(apeCmd.TargetTypeFlag, "", apeCmd.TargetTypeFlagDesc)
_ = listRulesCmd.MarkFlagRequired(apeCmd.TargetTypeFlag)
}

View file

@ -2,26 +2,20 @@ package control
import (
"bytes"
"crypto/sha256"
"fmt"
"strconv"
"text/tabwriter"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
apeCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common/ape"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
"github.com/spf13/cobra"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
const (
chainNameFlag = "chain-name"
chainNameFlagUsage = "Chain name(ingress|s3)"
)
var listTargetsCmd = &cobra.Command{
Use: "list-targets",
Short: "List local targets",
@ -32,15 +26,11 @@ var listTargetsCmd = &cobra.Command{
func listTargets(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd)
var cnr cid.ID
chainName, _ := cmd.Flags().GetString(chainNameFlag)
rawCID := make([]byte, sha256.Size)
cnr.Encode(rawCID)
chainName := apeCmd.ParseChainName(cmd)
req := &control.ListTargetsLocalOverridesRequest{
Body: &control.ListTargetsLocalOverridesRequest_Body{
ChainName: chainName,
ChainName: string(chainName),
},
}
@ -82,7 +72,7 @@ func initControlListTargetsCmd() {
initControlFlags(listTargetsCmd)
ff := listTargetsCmd.Flags()
ff.String(chainNameFlag, "", chainNameFlagUsage)
ff.String(apeCmd.ChainNameFlag, "", apeCmd.ChainNameFlagDesc)
_ = cobra.MarkFlagRequired(ff, chainNameFlag)
_ = cobra.MarkFlagRequired(ff, apeCmd.ChainNameFlag)
}

View file

@ -0,0 +1,88 @@
package control
import (
"fmt"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/api/rpc/client"
"github.com/mr-tron/base58"
"github.com/spf13/cobra"
)
const (
fillPercentFlag = "fill_percent"
)
var shardsRebuildCmd = &cobra.Command{
Use: "rebuild",
Short: "Rebuild shards",
Long: "Rebuild reclaims storage occupied by dead objects and adjusts the storage structure according to the configuration (for blobovnicza only now)",
Run: shardsRebuild,
}
func shardsRebuild(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd)
req := &control.StartShardRebuildRequest{
Body: &control.StartShardRebuildRequest_Body{
Shard_ID: getShardIDList(cmd),
TargetFillPercent: getFillPercentValue(cmd),
ConcurrencyLimit: getConcurrencyValue(cmd),
},
}
signRequest(cmd, pk, req)
cli := getClient(cmd, pk)
var resp *control.StartShardRebuildResponse
var err error
err = cli.ExecRaw(func(client *rawclient.Client) error {
resp, err = control.StartShardRebuild(client, req)
return err
})
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
verifyResponse(cmd, resp.GetSignature(), resp.GetBody())
var success, failed uint
for _, res := range resp.GetBody().GetResults() {
if res.GetSuccess() {
success++
cmd.Printf("Shard %s: OK\n", base58.Encode(res.GetShard_ID()))
} else {
failed++
cmd.Printf("Shard %s: failed with error %q\n", base58.Encode(res.GetShard_ID()), res.GetError())
}
}
cmd.Printf("Total: %d success, %d failed\n", success, failed)
}
func getFillPercentValue(cmd *cobra.Command) uint32 {
v, _ := cmd.Flags().GetUint32(fillPercentFlag)
if v <= 0 || v > 100 {
commonCmd.ExitOnErr(cmd, "invalid fill_percent value", fmt.Errorf("fill_percent value must be (0, 100], current value: %d", v))
}
return v
}
func getConcurrencyValue(cmd *cobra.Command) uint32 {
v, _ := cmd.Flags().GetUint32(concurrencyFlag)
if v <= 0 || v > 10000 {
commonCmd.ExitOnErr(cmd, "invalid concurrency value", fmt.Errorf("concurrency value must be (0, 10 000], current value: %d", v))
}
return v
}
func initControlShardRebuildCmd() {
initControlFlags(shardsRebuildCmd)
flags := shardsRebuildCmd.Flags()
flags.StringSlice(shardIDFlag, nil, "List of shard IDs in base58 encoding")
flags.Bool(shardAllFlag, false, "Process all shards")
flags.Uint32(fillPercentFlag, 80, "Target fill percent to reclaim space")
flags.Uint32(concurrencyFlag, 20, "Maximum count of concurrently rebuilding files")
setShardModeCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag)
}

Some files were not shown because too many files have changed in this diff Show more