Compare commits

...

274 commits

Author SHA1 Message Date
7edec9193c [#1451] placement: Return copy of slice from container nodes cache
Nodes from cache could be changed by traverser, if no objectID specified.
So it is required to return copy of cache's slice.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
3cf6ea745d [#1451] ec: Check all parts are saved
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
9a77527f46 [#1451] ape: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
5b1ba8e23d [#1451] ape: Perform strict APE checks for EC parts
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
9902965ff4 [#1451] writer: Sign EC parts with node's private key
As EC put request may be processed only by container node, so sign requests
with current node private to not to perform APE checks.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:18:10 +00:00
33ad753302 [#1473] policer: Add tracing span
To filter HEAD requests from policer.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-06 08:15:11 +00:00
15102e6dfd [#1471] Replace sort.Slice in some places
`slices.SortFunc` doesn't use reflection and is a bit faster.
I have done some micro-benchmarks for `[]NodeInfo`:
```
$ benchstat -col "/func" out
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/pilorama
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
       │ sort.Slice  │           slices.SortFunc           │
       │   sec/op    │   sec/op     vs base                │
Sort-8   2.130µ ± 2%   1.253µ ± 2%  -41.20% (p=0.000 n=10)
```

Haven't included them, though, as they I don't see them being used a
lot.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-06 08:07:32 +00:00
17ec84151b
[#532] cli: Respect XDG base directory spec
XDG base directory specification defines where various files
should be looked by an application. Hopefully, this makes `frostfs-cli`
more predictable and pleasant to work with. Luckily for us, golang already
has everything we need in the stdlib. This commit also gets rid of
`github.com/mitchellh/go-homedir` dependency.

Close #532
Refs #1455

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-11-05 16:27:18 +03:00
6c45a17af6
[#1467] node: Break notary deposit wait after VUB
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-11-01 16:42:02 +03:00
d19ab43500
[#1462] node: Add off-cpu profiler
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-31 11:32:13 +03:00
5bcf81d1cc
[#1466] Remove woodpecker CI
We use forgejo actions now.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-31 10:07:33 +03:00
c2effcc61c [#1465] Makefile: Update golangci-lint, fix warnings
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-31 06:39:59 +00:00
2285cfc36f
[#1464] frostfsid: Cache subject not found error
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-30 18:27:33 +03:00
e74d05c03f
[#1464] frostfsid: Add cache metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-30 18:27:32 +03:00
48862e0e63 [#1459] .golanci.yml: Add tenv linter, fix issues
Refs #1309

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-30 15:18:22 +00:00
89892d9754 [#1459] cli: Simplify slice append
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-30 15:18:22 +00:00
7ac0852364 [#1459] .golangci.yml: Add intrange linter, fix issues
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-30 15:18:22 +00:00
d28a5d2d7a
[#1448] container/ape: Ignore an error when getting a role
When getting a role in the APE checker for the container services,
an error may be returned if network maps of the previous two epochs
don't have enough nodes to fulfil a container placement policy.
It's a logical error, so we should ignore it.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-30 12:51:57 +03:00
87ac3c5279 [#1458] object: Make patch not set key before target construction
* `SignRequestPrivateKey` field should be initialized either within
  `newUntrustedTarget` or within `newTrustedTarget`. Otherwise, all
  requests are signed by local node key that makes impossible to perform
  patch on non-container node.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-10-29 15:20:28 +00:00
d5ee6d3039
[#1456] morph: Use DialerSource interface instead of internal struct
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-29 17:34:14 +03:00
433aab12bb
[#1455] cli: Handle missing home directory
go-homedir library incorrectly handles some of the errors
that could occur. It is archived, so no PR, but let's fix it on our
side. The scenario in case: executing command in an empty environment.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-29 16:37:55 +03:00
81f4cdbb91 [#1439] object: Sort nodes by priority metrics to compute GET request
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-29 08:05:09 +00:00
3cd7d23f10 [#1439] node: Reduce usage of netmapAPI.NodeInfo
Remove outdated code from `netmap` service.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-29 08:05:09 +00:00
012af5cc38 [#1406] tree: Add unit-tests for ape check
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-10-29 08:04:23 +00:00
eb5336d5ff [#1406] tree: Use delete verb instead put for Remove
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-10-29 08:04:23 +00:00
bc8d79ddf9
[#1447] services/tree: Move relaying code to a separate function
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-24 10:01:03 +03:00
29708b78d7 [#1442] cli/tree: Enchance error message if rpc-endpoint isn't defined
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-23 13:05:17 +00:00
b9284604d9 [#1442] cli/tree: Allow to specify rpc-endpoint with config file
We have several ways to specify the `rpc-endpoint`: with a flag,
with a single config file or multiple files. Before, the `rpc-endpoint`
flag was marked as required. Because `cobra` checked the required flag
presence first, it prevented specifying `rpc-endpoint` with a config file.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-23 13:05:17 +00:00
65a4320c75 [#1441] services/tree: Use grpc.WaitForReady option when creating client
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-23 11:45:44 +00:00
9a260c2e64 [#1441] network/cache: Use grpc.WaitForReady option when creating client
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-23 11:45:44 +00:00
6f798b9c4b [#1441] cli: Use grpc.WaitForReady while initializing SDK client
Before, when the target RPC server was unavailable, requests made
by CLI didn't wait for a timeout specified by the `--timeout` option
if the timeout was more than 20 seconds. It's because of the gRPC
default backoff strategy. Adding this option fixes that behavior.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-23 11:45:44 +00:00
e515dd4582
[#1444] config: Fix data race on morph component init
It could be called for every shard on metabase resync concurrently and
it is possible to get state with initialized client but not initialized
contract hashes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-23 10:41:36 +03:00
8b6ec57c61 [#1440] sdnotify: Fix status for RELOADING
Before:
```
RELOADING=1
MONOTONIC_USEC=17951246687
STATUS=RELOADING=1
MONOTONIC_USEC=17951246687
```
After:
```
RELOADING=1
MONOTONIC_USEC=17951246687
STATUS=RELOADING
```

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-21 14:25:08 +03:00
ed13387c0e
[#1438] .docker: Use go1.23 for builders
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-18 15:20:37 +03:00
5afea62ec0
[#1438] debian: Remove package scripts
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-18 15:20:37 +03:00
c0a2f20eee [#1422] multinet: Add metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
2d064d0bd8 [#1422] morph: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
ef38420623 [#1422] ir: Add dialer source
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
f7caef355b [#1422] node: Use dialer source for morph
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
fbdfd503e4 [#1422] morph: Add dialer source support
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
67798bb50e [#1422] mod: Bump neoneo-go version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
5b653aa65f [#1422] morph: Drop single client as not used
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
e314f328c4 [#1422] tree: Use dialer source for tree service connections
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
6c96cc2af6 [#1422] node: Use dialer source for SDK cache
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
74db735265 [#1422] node: Add dialer source to config
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
3304afa9d1 [#1422] config: Add multinet config
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-17 13:15:08 +00:00
b42bcdc6fa
[#1433] services/object: Put object before auxiliary info
Consider the following operations ordering:
1. Inhume(with tombstone A) --> add tombstone mark for an object
2. --> new epoch arives
3. --> GCMark is added for a tombstone A, because it is unavailable
4. Put(A) --> return error, because the object already has a GCMark

It is possible, and I have successfully reproduced it with a test on the
shard level. However, the error is related to the specific
_ordering_ of operations with engine. And triggering race-conditions like
this is only possible on a shard level currently, so no tests are
written.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-17 14:43:13 +03:00
b0c5def2d9
[#1433] shard/test: Use WithDisabledGC() option where possible
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-17 14:38:07 +03:00
90f3669399 [#1342] network/cache: Add node address to error multiClient
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 14:05:33 +00:00
07ce40e119 [#1430] adm/morph: Add NNS address display in 'deploy'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 13:56:37 +00:00
41038b2ec0 [#1431] node: Fix 'empty slice declaration using a literal'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
d83879d4b8 [#1431] node: Fix comment format
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
f6582081a4 [#1431] obj_storage/metabase: Delete unused variable
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
00b1cecfb7 [#1431] obj_storage/shard: Fix visibility of 'newMetricStore'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
63466d71b2 [#1431] engine: Delete unused constants
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
d53732f663 [#1431] engine: Delete always false condition
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
3012286452 [#1431] metabase: Fix unreachable code
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-15 08:53:59 +00:00
714ff784fa [#1431] objsvc: Use specific values in message about address mismatch
This makes troubleshooting failed operations much easier

Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2024-10-15 07:34:03 +00:00
d2a59b2de8
[#1429] lens/explorer: Fix locked object records display text
Display texts for a locked object and a list of it lockers were mistakenly swapped.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-14 15:58:14 +03:00
acd6eb1815 [#1427] object: Fix Put for EC object when node unavailable
There might be situation when context canceled earlier than traverser move to another part of the nodes.
To avoid this, need to wait for the result from concurrent put at each traverser iteration.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-11 16:53:30 +03:00
42bf03e5cc
[#1411] adm/nns: Add 'delRecord'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-11 11:33:56 +03:00
5992ee901a
[#1411] go.mod: Bump frostfs-contract to v0.20.0
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-11 11:33:36 +03:00
dfb00083d0
[#1426] go.mod: Update sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-10 14:58:46 +03:00
1134760271
[#1425] services/tree: Remove eACL mentions from bearer token parsing errors
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:49 +03:00
02bb7159a5
[#1425] services/tree: Remove eACL processing
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:48 +03:00
94302235d0
[#1425] adm: Remove eACL fetching from dump-containers
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:48 +03:00
cc5360a578
[#1425] morph/event: Rename eacl_test.go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:48 +03:00
4190fba86d
[#1425] Remove SetEACL-related code
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-09 10:56:48 +03:00
936ebbb8e5 [#1423] metabase: Hide BucketName form upper levels
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-08 18:41:16 +03:00
c065d55ca3
[#1412] metabase: Drop logging inside transaction
This could lead to hang the db.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:52 +03:00
fe9f664b57
[#1412] metabase: Drop empty user attribute buckets on upgrade
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:52 +03:00
87f4b934d1
[#1412] metabase: Run bucket drop steps on upgrade concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:52 +03:00
8093e145b3
[#1412] adm: Resolve container type by metabase upgrade
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:51 +03:00
3da168f8cf
[#1412] shard: Resolve container is indexed on metabase resync
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:51 +03:00
4572fa4874
[#1412] searchSvc: Check container is indexed
For non S3 containers it is expected to use attributes index for some
attributes.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:50 +03:00
1efa64ee72
[#1412] metabase: Add search by indexed attributes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:50 +03:00
be744ae3e6
[#1412] metabase: Index attributes for indexed containers
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:49 +03:00
1b520f7973
[#1412] engine: Add IsIndexedContainer flag
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:49 +03:00
899cd55c27
[#1412] engine: PutPrm refactoring
Use fields instead of methods.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 11:41:48 +03:00
0c49bca19c [#1415] lens/explorer: Add timeout for opening database
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-10-08 07:39:31 +00:00
5fbb2657ca
[#1419] mod: Bump sdk-go version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-08 10:04:11 +03:00
a5de74a249 [#1418] go.mod: Update api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-07 12:55:28 +00:00
fc032838c0 [#1215] blobstor/test: Cover iteration behaviour
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-07 12:44:27 +00:00
2f710d8f94 [#1414] metabase: Check parameter for CountAliveObjectsInBucket
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-10-04 17:11:15 +03:00
4dc9a1b300 [#1413] engine: Remove error counting methods from Shard
All error counting and hangling logic is present on the engine level.
Currently, we pass engine metrics with shard ID metric to shard, then
export 3 methods to manipulate these metrics.
In this commits all methods are removed and error counter is tracked on
the engine level exlusively.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-04 15:10:17 +03:00
963faa615a [#1413] engine: Cleanup shard error reporting
- `reportShardErrorBackground()` no longer differs from
  `reportShardError()`, reflect this in its name;
- reuse common pieces of code to make it simpler.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-04 15:10:17 +03:00
9a87acb87a [#1410] engine: Provide the default implementation to MetricsRegister
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-03 08:23:06 +00:00
9206ce5cd2 [#1410] shard: Provide the default implementation for MetricsWriter
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-03 08:23:06 +00:00
6c46044c9c [#1410] shard: Move MetricsWriter interface to a separate file
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-03 08:23:06 +00:00
01e3944b31 [#1408] metabase: Fix tests
No need to specify container ID for objects created with `testutil.GenerateObjectWithCID`.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-02 14:41:40 +03:00
434048e8d9 [#1408] metabase: Fix EC search with slow and fast filters
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-10-02 14:36:59 +03:00
f83f7feb8c [#1391] adm: Properly check whether transfers were made
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-02 11:41:33 +03:00
62028cd7ee [#1409] adm: Uncommonize DeltaFlag
It is used only in `force-new-epoch`, it is not _common_ between
multiple commands.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-02 11:41:27 +03:00
f45e75e3eb [#1409] adm: Do not bind DeltaFlag to viper
We bind flag that could be specified in config.
This is not a config flag, just a command option.
Also fix TestInitialize failures:
```
                Error:          Received unexpected error:
                                number of epochs cannot be less than 1
                Test:           TestInitialize/16_nodes/force-new-epoch
```
Refs #1372 (945b7c740b)

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-02 11:41:22 +03:00
57c31e9802 [#1306] node: Allow tombstone_lifetime config to be loaded on the fly
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-10-02 10:22:00 +03:00
9c5ddc4dfe [#1407] tree: Set ContainerOwner in parameter for CheckAPE
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-10-02 07:21:02 +00:00
54eb005822 [#1404] go.mod: Update api-go
Fix #1398
Fix #1399

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-10-02 06:19:38 +00:00
a13219808a [#1375] node: Configure of the container cache size
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-10-01 14:05:52 +00:00
7f8a1dcf8e [#1400] adm: Support flag alphabet-wallets for commands proxy-add/remove-account
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-30 14:15:13 +03:00
d0ed29b3c7 [#1350] node: Add ability to evacuate objects from REP 1 only
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-27 15:41:17 +03:00
5f22ba6f38 [#1397] object: Correctly set namespace before APE check
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-27 11:43:29 +00:00
a5e1aa22c9 [#1394] putSvc: Fix relay
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 17:28:03 +03:00
772b471aab [#1388] lens: Add nolint annotations
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
4fbfffd44c [#1388] putSvc: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
29e4cf7ba1 [#1388] ir: Annotate cmode as nolint
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
a2ab6d4942 [#1388] node: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
bdd57c8b6b [#1388] sessionSvc: Add nolint annotations
Used as map key.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
d1d6e3471c [#1388] signSvc: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
401c398704 [#1388] metabase: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
004ff9e9bf [#1388] blobstor: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
63a567a1de [#1388] engine: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
580cd55180 [#1388] getSvc: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
e319bf403e [#1388] apeSvc: Drop unused and make annotations
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
aedb55f913 [#1388] governance: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
b69e07da7a [#1388] metrics: Mark nolint:unused metrics
Although these fields could be deleted, I annotated them so that all the
metrics used would be defined in one place.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
2bd560e528 [#1388] cli: Drop unused flag/parameter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
95597d3437 [#1388] golangci: Make unused linter stricker
Add aditional checks. The most important false positive - structs used as
map keys.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-25 08:55:38 +00:00
fd18aa363b [#1385] metabase: Optimize isTomb check
As tombstone and target must have the same containerID, do not iterate
other containers.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-24 17:09:06 +03:00
76268e3ea2 [#1385] metabase: Validate that tombstone and target have the same container ID
Target container ID is taken from tombstone: cmd/frostfs-node/object.go:507
Also object of type `TOMBSTONE` contains objectID, so tombstone and
tombstoned object must have the same containerID.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-24 17:09:06 +03:00
8434f3dbfc [#1385] metabase: Use Batch for delete-related operations
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-24 17:09:06 +03:00
34e6a309c6 [#1356] engine: Evacuate object from shards concurrently
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-24 12:50:19 +03:00
bdf386366c [#1297] dev: Bump neo-go version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-24 12:05:09 +03:00
839dead226 [#1297] getSvc: Return AccessDenied instead of ObjectNotFound
Do not replace the access denied error if it was received earlier.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-24 12:05:09 +03:00
3bb65ba820 [#1392] object: Fix target initialization within put streamer
* Remove `relay` field from put streamer as it's no longer used;
* Fix initialization of `Relay` object writer parameter.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-24 09:04:13 +00:00
d4493a6d08 [#1390] getSvc: Fix Head EC1.1
If local EC chunk found, but remote node is off, then `HEAD --raw` request
returns object not found.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-23 15:12:06 +03:00
0b87be804a [#1381] engine: Fix tests
Drop not required `Eventually` calls.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-23 11:10:29 +03:00
f71418b73c [#1386] frostfs-adm: Add info to error messages
These error messages bubble up to human users - adding more context helps
to find the cause of the issue faster.

Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2024-09-23 07:47:03 +00:00
c34b8acedd [#1312] Drop handling of system attributes with NeoFS prefix
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-20 11:37:18 +00:00
c290d079fd [#1312] go.mod: Update sdk-go
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-20 11:37:18 +00:00
53a90634fc [#1301] adm/morph: Add 'delete' domains
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-09-20 11:54:18 +03:00
5a53f9c4fd [#1301] go.mod: Bump frostfs-contract
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-09-20 11:54:18 +03:00
1361db91ee [#1301] adm/morph: Add flag -v to 'Tokens'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-09-20 11:54:18 +03:00
945b7c740b [#1372] adm/morph: Add delta flag to 'force-new-epoch'
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-09-20 11:42:04 +03:00
61d5e140e0 [#1383] object: Add restrictions for Patch method
* `Patch` can't be applied for non-regular type object (tombstones,
  locks etc.)
* Complex object parts can't be patched. So, if an object has EC/Split
  header, it won't be patched.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-19 13:54:49 +03:00
3441fff05d [#1382] cli: Replace deprecated methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:35:48 +03:00
ac1eee091d [#1382] node: Replace deprecated methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:35:48 +03:00
a603d14d08 [#1382] ir: Replace deprecated methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:29:31 +03:00
d4be2f20d4 [#1382] morph: Replace deprecated methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:29:27 +03:00
e5c8f7ff9f [#1382] controlSvc: Replace deprecated methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:29:24 +03:00
1e7f9909da [#1382] policer: Replace deprecated methods
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:29:21 +03:00
b807d8c400 [#1382] go.mod: Upgrade sdk-go and api-go versions
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-18 14:29:11 +03:00
d4bec24c9f
[#1366] node, ir: Support timestamp config option, update tests
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-17 10:50:08 +03:00
ea48e928c8
[#1366] logger: Make timestamp prepending optional
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-17 10:49:51 +03:00
96308a26c6 [#1361] linter: fix funlen
Signed-off-by: Aleksey Savaitan <a.savaitan@yadro.com>
2024-09-13 15:12:49 +00:00
74a6a1da7f [#1361] add root ca cert for telemetry configuration
Signed-off-by: Aleksey Savaitan <a.savaitan@yadro.com>
2024-09-13 15:12:49 +00:00
2be1aa781d [#1266] .forgejo: Make 'fumpt' job fail on changed files
`gofumpt` always returns an exit code of 0, even when it finds
misformatted files. To make `fumpt` action behave as expected
we need to check if `gofumpt` changed any files.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-13 17:27:58 +03:00
89d0435b1d [#1374] tree: Use NewClient to create grpc connection in cache
Created grpc connection should be established, so perform Healthcheck request
to check connection is ok.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-13 15:59:26 +03:00
54fe8383a4 [#1374] tree: Use NewClient to create grpc connection for sync
Created connection will be used to sync trees, so it is ok to defer
dial to the first RPC call.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-13 15:59:26 +03:00
944160427b [#1374] cli: Drop deprecated grpc connection method
For `frostfs-cli` it is ok to use grpc-client without blocking,
as `frostfs-cli` will perform RPC call anyway.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-13 15:59:26 +03:00
bb44867491 [#1374] go.mod: Upgrade grpc to v1.66.2
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-13 15:59:00 +03:00
546d09660f [#1283] Clear systemd-notify status on exit
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-12 14:45:06 +00:00
e3764c51df [#1347] metabase: Fix EC search
For EC chunks need to return EC parent object ID as
EC chunks don't have own attributes but inherit parent's.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 13:23:29 +00:00
b33559754d [#1367] fstree: Add size hint for Delete
This allow to not to call `os.Stat` if caller already knows data size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
f345fe9a58 [#1367] writecache: Move DB related code to upgrade.go
This is done to drop this file in the future.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
3b236160a6 [#1367] writecache: Drop DB label from metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
25d2ae8aaf [#1367] writecache: Drop BBolt related config variables
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
e39378b1c3 [#1367] writecache: Add background flushing objects limiter
To limit memory usage by background flush.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
8a6e3025a0 [#1367] writecache: Flush from FSTree concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
2dd3a6f7a8 [#1367] fstree: Add IterateInfo method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
b142b6f48e [#1367] fstree: Add size to file counter
FSTree file counter used by writecache. As writecache has now only one
storage, so it is required to use real object size to get writecache
size more accurate than `count * max_object_size`.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
5f6c7cbdb1 [#1367] writecache: Drop bbolt DB
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 14:22:29 +03:00
66e17f4b8e
[#1368] cli/container: Use dedicated method to list user attributes
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-12 10:01:27 +03:00
99be4c83a7
[#1368] *: Run gofumpt
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-12 10:00:28 +03:00
ec8da40567 [#1369] Update obsolete URLs
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2024-09-11 11:31:10 +00:00
dea6f031f9 [#1331] cli/tree: Add order flag to tree get-subtree
Added `--ordered` flag to sort output by ascending FileName.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-11 11:30:28 +00:00
5fac4058e8 [#1364] cmd/common: Add tests for CreateViper and ReloadViper
Add tests for `CreateViper` and `ReloadViper` to ensure that no extra
files, except *.yaml, *.yml, *.json, are loaded from config directory.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-11 08:03:05 +00:00
2220f6a809 [#1365] Makefile: Fix HUB_IMAGE
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-09-10 16:45:15 +03:00
a812932984 [#1362] ape: Move common APE check logic to separate package
* Tree and object service have the same log for checking APE. So,
  this check should be moved to common package.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-10 12:40:34 +00:00
92fe5d90f5 [#1359] writecache: Drop unused
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-09 18:45:21 +03:00
4668efc0bf [#1355] metabase: Upgrade improvements
Do not fail on same latest version to run compact on upgraded metabase.
Use NoSync on compact.
Log every batch on bucket delete stage.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-09 11:37:36 +03:00
654d970fad [#1355] adm: Run metabase upgrade concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 15:54:40 +03:00
d3b209c8e1 [#1337] shard: Disable background rebuild
Since `frostfs-cli control shards rebuild` command was added,
there is no need for background rebuild now.
For failover tests used used value 1 to rebuild only schema change.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 15:19:55 +03:00
edb1747af7 [#1337] blobovniczatree: Add rebuild by overflow
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 15:19:54 +03:00
a61201a987 [#1337] config: Move rebuild_worker_count to shard section
This makes it simple to limit performance degradation for every shard
because of rebuild.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 13:57:27 +03:00
6b6eabe41c [#1337] cli: Add control shards rebuild command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 13:57:27 +03:00
d508da8397 [#1337] blobovniczatree: Add rebuild by fill percent
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 13:57:27 +03:00
007827255e [#1337] blobovniczatree: Add .rebuild temp files
This allows to reduce open/close DBs to check incompleted rebuilds.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 13:57:27 +03:00
f652518c24 [#1357] go: Fix panic caused by using range over slice len
If slice is altered in `for` loop, we cannot use range over its
length: it may cause panic if slice gets shorter.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-06 13:09:58 +03:00
273980cfb9 [#1310] object: Remove irrelevant comments
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-05 16:40:32 +03:00
108e4e07be [#1349] node: Evacuate objects without setting mode to MAINTENANCE
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-05 16:08:27 +03:00
b3deb893ba [#1310] object: Move target initialization to separate package
* Split the logic of write target initialization to different packages;
* Refactor patch and put services: since both service initialize the target
  themselves.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-05 13:03:58 +00:00
7768a482b5 [#1223] lens/tui: Add TUI app for blobovnicza
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
371d97f61a [#1223] lens/tui: Add TUI app for write cache
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
e655336390 [#1223] lens/tui: Add app help
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
ed396448ac [#1223] lens/tui: Add TUI app to explore metabase
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
9cbd32bce8 [#1223] lens/tui: Add writecache schema
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
1ae86f35a8 [#1223] lens/tui: Add metabase schema
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
b9043433a0 [#1223] scripts: Add script to populate metabase
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
a4fb7f085b
[#1348] go.mod: Update api-go and sdk-go
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-04 10:47:26 +03:00
a685fcdc96 [#1317] go.mod: Use range over int
Since Go 1.22 a "for" statement with a "range" clause is able
to iterate through integer values from zero to an upper limit.

gopatch script:
@@
var i, e expression
@@
-for i := 0; i <= e - 1; i++ {
+for i := range e {
    ...
}

@@
var i, e expression
@@
-for i := 0; i <= e; i++ {
+for i := range e + 1 {
    ...
}

@@
var i, e expression
@@
-for i := 0; i < e; i++ {
+for i := range e {
    ...
}

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-03 13:00:54 +03:00
2b3fc50681
[#1320] shard: Fix TestGCDropsObjectInhumedFromWritecache flaky test
The `TestGCDropsObjectInhumedFromWritecache` test was flaky because a
running asynchronous rebuild operation prevented GC from deleting the
object. A test-only shard option `WithDisabledRebuild` has been added
to fix this.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-02 10:26:53 +03:00
98fe24cdb7 [#1343] go.mod: Update api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-30 08:45:24 +00:00
882c068410 [#1334] metabase: Store upgrade flag
This allows to check if metabase upgrade was not completed.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-30 09:06:20 +03:00
6c2146bbc1 [#1334] metabase: Add upgrade from v2 to v3
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-30 09:06:20 +03:00
03976c6ed5 [#1341] .golangci.yml: Replace exportloopref with copyloopvar
exportloopref is deprecated.
gopatch:
```
@@
var index, value identifier
var slice expression
@@
for index, value := range slice {
...
-value := value
...
}

@@
var index, value identifier
var slice expression
@@
for index, value := range slice {
...
-index := index
...
}

@@
var value identifier
var channel expression
@@
for value := range channel {
...
-value := value
...
}
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-28 15:44:41 +00:00
7e97df4878 [#1341] Makefile: Update golangci-lint
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-28 15:44:41 +00:00
01b6f1733c [#1341] Makefile: Build linter with -trimpath
Fix error with go1.23:
```
Error: build linters: unable to load custom analyzer "truecloudlab-linters": ../linters/bin/external_linters.so, plugin.Open("/repo/frostfs/linters/bin/external_linters"): plugin was built with a different version of package cmp
Failed executing command with error: build linters: unable to load custom analyzer "truecloudlab-linters": ../linters/bin/external_linters.so, plugin.Open("/repo/frostfs/linters/bin/external_linters"): plugin was built with a different version of package cmp
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-28 15:44:41 +00:00
7abbdca064 [#1340] getSvc: Fix access denied error handling
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-28 14:02:28 +03:00
6488ddee88 [#1338] object: Fix range provider in Patch handler
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-27 16:51:12 +03:00
d6b42972a8 [#1338] object: Fix audit patch stream
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-27 16:18:29 +03:00
5e9a97fd3e [#1336] go.mod: Update api-go and sdk-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-27 14:48:43 +03:00
fa7f9fbce2 [#1333] go.mod: Update api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-26 15:37:43 +03:00
806ea37101 [#1328] pilorama: Do not skip items in SortedByFilename
Benchmark results:
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/pilorama
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                                   │     old     │                 new                 │
                                   │   sec/op    │   sec/op     vs base                │
ForestSortedIteration/bbolt,root-8   207.2µ ± 6%   173.6µ ± 6%  -16.23% (p=0.000 n=10)
ForestSortedIteration/bbolt,leaf-8   3.910µ ± 5%   3.928µ ± 7%        ~ (p=0.529 n=10)
geomean                              28.46µ        26.11µ        -8.27%
```

They are not representative, as the worst case is when we have multiple
items of different lengths. However, `FileName` is usually less than 100
in practice, so the asymptotics is the same.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-26 06:11:32 +00:00
80099d9a2f [#1328] pilorama: Add tricky test for SortedByFilename
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-26 06:11:32 +00:00
a059a7dcf0 [#1329] cli: Skip linking objects in complex object processing
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-23 13:51:20 +03:00
bd24beecf8 [#1329] putSvc: Reset SuccessAfter for non-EC objects in EC container broadcasting
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-23 13:51:16 +03:00
dfe825b81b [#1309] test: Clean up config after tests
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-08-22 08:49:10 +00:00
76f67ea34e [#1323] metabase: Bump version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
7d0d781db1 [#1323] metabase: Drop user attribute index
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
0f08a2efba [#1323] metabase: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
7bf20c9f1f [#1323] metabase: Add expiration epoch buckets
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
2542d4f5df [#1323] metabase: Drop payload checksum index
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
15dae8685e [#1323] metabase: Drop ownerID index
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
7bca428db0 [#1322] Use new protobuf marshaler
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-22 07:17:41 +00:00
a345c972bf [#1316] lint: Fix warnings
Renamed parameters `min/max` to avoid conflicts with
predeclared identifiers.

Replaced background context with parent context without
cancellation in closer functions in frostfs-node.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-21 18:18:25 +03:00
8c1082b31a [#1316] go.mod: Bump go version to 1.22
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-21 15:09:29 +03:00
cfda9003a7 [#1318] meta: Add test TestInhumeECObject
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-20 16:21:27 +03:00
6ff0b0996b [#1318] metrics: Fix container_size_bytes for EC
When node put chunk into EC container, `policer` may remove it as redundant.
This chunk marked as removed. When parent object removed and `gc` start iterating over chunk,
node count removing chunk twice.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-20 15:33:28 +03:00
8319b59238 [#1318] Fix gofumpt issue
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-20 14:34:54 +03:00
b7acb34fa4 [#1319] treeSvc: Do not wrap error from APE
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-19 18:38:27 +03:00
41104f2383 [#1307] cli: Make cli process object.patch
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
eeab417dcf [#1307] object: Add APE check for Patch handler
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
5ed317e24c [#1307] cli: Introduce object patch command
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
e890f1b4b1 [#1307] object: Implement Patch method
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
a4a1c3f18b [#1307] go.mod: Bump frostfs-sdk-go/frostfs-api-go/v2 versions
* Also, resolve dependencies and conflicts for object service
  by creating stub for `Patch` method.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
ec1509de4e [#1262] sdnotify: Send system monotonic time on reload
The synchronized service reload protocol added in systemd version 253
requires that the service provides a MONOTONIC_USEC field alongside the
RELOADING=1 notification message for synchronization purposes. The value
carried in this field must be the system CLOCK_MONOTONIC timestamp at
the time the notification message was generated as systemd compares it
to other CLOCK_MONOTONIC timestamps taken by pid1.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-16 12:05:31 +03:00
5da41f1fe5 Revert "[#1262] sdnotify: Get rid of go:linkname for nanotime"
This reverts commit 327d364f34.

Reverted due to the problem with reload signal sent by systemd.
`frostfs-ir` service reconfigures correctly and service's
statuses are being reported to systemd. However, since we
replaced `go:linkname` & `nanotime()` with `time.Since()`,
systemd refuses to accept reload signal response from
`frostfs-ir`. To maintain correct behaviour it was decided to
revevrt systemd-related changes until a better solution is
found.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-15 16:54:45 +03:00
d055168e2a [#1135] ir: Add healthstatus RECONFIGURING
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-15 16:54:45 +03:00
80ce7c3a00 [#1284] shard: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 13:23:35 +03:00
93d63e1632 [#1284] writecache: Allow to seal writecache async
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 13:23:35 +03:00
68029d756e [#1302] writecache: Allow to specify custom page size
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 12:12:07 +03:00
fa82854af4 [#1302] writecache: Add put->flush->put benchmark
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 12:12:07 +03:00
c985b1198f [#1302] putSvc: Override SuccessAfter for non-regular objects in EC containers
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 10:42:55 +03:00
08b1f18bca [#1296] writecache: Add count limit
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 06:30:32 +00:00
36efccd862 [#1298] writecache: Add shrink flag for Seal command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-08 16:32:29 +03:00
5c01bd5be8 [#1298] writecache: Add restore-mode flag for Seal command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-07 11:20:14 +03:00
8e51d7849a [#1295] getSvc: Assemble complex EC object headers without linking object
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-06 16:48:12 +03:00
10602b55b1 [#1295] engine: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-06 09:45:47 +03:00
eeca796d2e [#1295] engine: Log object address in case of error
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-06 09:45:43 +03:00
327d364f34 [#1262] sdnotify: Get rid of go:linkname for nanotime
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-05 12:37:16 +00:00
dc3dcabadc [#1291] morph: Reconnect to the highest priority endpoint
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-02 17:54:22 +03:00
8021bacc43 [#1288] putSvc: Respect TTL for EC put
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-02 13:01:42 +00:00
ef4cea6d19 [#1266] .forgejo: Add gofumpt action
`gofumpt` was skipped by pre-commit on CI, and now is used
in a separate action.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-02 12:34:04 +03:00
a55600893e [#1266] Makefile: Specify gofumpt version
Add target to install gofumpt, fix target to run gofumpt.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-02 12:31:59 +03:00
c49982d22a [#1282] cli: Allow to external addresses first for object nodes
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-01 09:16:19 +03:00
7e04083c27 [#1278] containerSvc: Validate FrostFSID subject exitence on Put
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-29 16:01:37 +03:00
a12c39667d [#1278] ir: Do not allow to create container without FrostFSID record
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-29 16:01:37 +03:00
85a77b7c21 [#1279] adm: Interpret "root" name as empty for namespace target type
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-29 12:57:40 +00:00
8377372a40 [#1276] go.mod: Update api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-26 16:44:19 +03:00
dd459d399f [#1274] go.mod: Update neo-go version that fixes ws-client
* Update go.mod;
* This neo-go package version contains fix for the wsclient that
  allows to morph event listener refresh the invalidated websocket
  connection to neo-go.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-26 14:21:19 +03:00
7fd7961dfa [#1271] getSvc: Fix local EC chunk get
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-24 08:52:16 +03:00
8398a8b4b3 [#1271] getSvc: Fix head --raw assemble for EC
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-24 08:52:08 +03:00
9c2c76ca32
[#1259] Move pkg/util/locode to frostfs-locode-db
Removed pkg/util/locode package, added
git.frostfs.info/TrueCloudLab/frostfs-locode-db/pkg/locode dependency.

Signed-off-by: George Bartolomey <george@bh4.ru>
2024-07-23 14:59:14 +03:00
1032075a21
[#1259] cli: Remove locode subcommand
Removed `frostfs-cli util locode` subcommand.
Alternative command could be found in
`git.frostfs.info/TrueCloudLab/frostfs-locode-db`.

Signed-off-by: George Bartolomey <george@bh4.ru>
2024-07-23 14:59:14 +03:00
94f07b4778 [#1245] docker: Fix warnings
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-07-23 12:36:11 +03:00
7ddba70030 [#1264] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-23 08:42:29 +00:00
658e3cb92f [#1264] go.mod: Update bbolt to v1.3.10
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-23 08:42:29 +00:00
9ee1bd3669
[#1265] *: Run gofumpt
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-07-23 10:36:32 +03:00
ca4d6df1cc Release v0.42.0
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-23 09:14:36 +03:00
18182e578e [#1261] shard: Fix delete objects from FSTree
Replace nil storageID with empty like by shard.Get.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-22 14:38:09 +03:00
3119f2fbc3 [#1257] metabase: Delete EC gc marks and split info
EC parent and split gc marks should be deleted after last EC chunk delete.
Also should delete split info for EC parent and split parent.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-19 17:14:14 +03:00
e377a92d97 [#1258] audit: Fix panic in LogRequest method
* Make `LogRequest` process `req=nil` to avoid panic.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-19 11:27:33 +03:00
eadcea8df0 [#1249] object: Remove all APE pre-checks in handlers
* Methods `Head`, `Get`, `GetRangeHash` should no longer use APE pre-checks
  as that leads only to incorrect rule chain processing for requests:
  1. Immediate return with `NoRuleFound` may be unexpected as some `Allow`
     rule is actually defined but can't be matched yet as it gets no object
     attributes;
  2. Immdediate return with `Allow` may be incorrect as some `Deny` rule
     is actually defined but can't bet matched yet as it gets no object
     attirbutes;
  3. Pre-check breaks compatibility for converted EACL-tables.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-18 13:52:43 +00:00
5e5ee545b6 [#1254] policer: Fix svacer warning
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-18 11:30:39 +00:00
27caa8a72f [#1256] metabase: Put split parent object ID for EC chunks
It is required to save split parent ID too, not only split ID.
Otherwise inhume operation works incorrect: shard with last part may be skipped
and parent object will be available.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-17 18:08:07 +03:00
e83d39e33f [#1253] deleteSvc: Use copy of common parameters
getSvc may change the values of some fields, so Head will affect Delete
or Put. In this case, the change is necessary so that the session token
is stored in the tombstone object (EC assemble calls `ForgetTokens`).

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-17 14:24:27 +03:00
fc383ea6ae [#1253] getSvc: Fix EC objects get
Now EC objects assembling is performed concurrently.
Also fixed issue with an error in case of getting
EC object via non-container node.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-17 14:24:27 +03:00
00a88b9936 [#1251] *: Run gofumpt
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-17 11:25:07 +03:00
3940bc17c1 [#1251] pilorama: Allow traversing multiple branches in parallel
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-17 11:25:07 +03:00
e5767c9002 [#1250] *: Reformat proto filets with clang-format
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 15:13:08 +00:00
286df198c9 [#1248] placement: Decouple ContainerNodes() cache from the placement builder.
Also, write tests.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 12:21:30 +00:00
3a48b282b6 [#1248] placement: Use epoch to track netmap versions
Previously we used pointer, this could have worked,
because most of the time, the netmap is cached.
This didn't work, however, because `lastNm` field was always nil.

Rework the mechanism completely:
1. Use epoch to track netmap versions, as it it simpler and
   is unrelated to the TTL of an underlying cache.
2. Fix a bug where the epoch could change while mutex was unlocked.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 12:21:30 +00:00
21431f22c0 [#1248] placement: Use cid.ID as key in the cache
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-16 12:21:30 +00:00
d5dc14c639 [#1243] object: Make APE checker set x-headers to request properties
* Update go.mod, go.sum;
* Add x-headers to request properties;
* Add a unit-test.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-16 07:28:42 +00:00
589 changed files with 16524 additions and 8946 deletions

View file

@ -1,4 +1,4 @@
FROM golang:1.22 as builder FROM golang:1.23 AS builder
ARG BUILD=now ARG BUILD=now
ARG VERSION=dev ARG VERSION=dev
ARG REPO=repository ARG REPO=repository

View file

@ -1,4 +1,4 @@
FROM golang:1.22 FROM golang:1.23
WORKDIR /tmp WORKDIR /tmp

View file

@ -1,4 +1,4 @@
FROM golang:1.22 as builder FROM golang:1.23 AS builder
ARG BUILD=now ARG BUILD=now
ARG VERSION=dev ARG VERSION=dev
ARG REPO=repository ARG REPO=repository

View file

@ -1,4 +1,4 @@
FROM golang:1.22 as builder FROM golang:1.23 AS builder
ARG BUILD=now ARG BUILD=now
ARG VERSION=dev ARG VERSION=dev
ARG REPO=repository ARG REPO=repository

View file

@ -1,4 +1,4 @@
FROM golang:1.22 as builder FROM golang:1.23 AS builder
ARG BUILD=now ARG BUILD=now
ARG VERSION=dev ARG VERSION=dev
ARG REPO=repository ARG REPO=repository

View file

@ -8,7 +8,7 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
strategy: strategy:
matrix: matrix:
go_versions: [ '1.21', '1.22' ] go_versions: [ '1.22', '1.23' ]
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v3

View file

@ -16,7 +16,7 @@ jobs:
- name: Set up Go - name: Set up Go
uses: actions/setup-go@v3 uses: actions/setup-go@v3
with: with:
go-version: 1.22 go-version: 1.23
- name: Set up Python - name: Set up Python
run: | run: |
apt update apt update

View file

@ -11,7 +11,7 @@ jobs:
- name: Set up Go - name: Set up Go
uses: actions/setup-go@v3 uses: actions/setup-go@v3
with: with:
go-version: '1.22' go-version: '1.23'
cache: true cache: true
- name: Install linters - name: Install linters
@ -25,7 +25,7 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
strategy: strategy:
matrix: matrix:
go_versions: [ '1.21', '1.22' ] go_versions: [ '1.22', '1.23' ]
fail-fast: false fail-fast: false
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v3
@ -48,7 +48,7 @@ jobs:
- name: Set up Go - name: Set up Go
uses: actions/setup-go@v3 uses: actions/setup-go@v3
with: with:
go-version: '1.21' go-version: '1.22'
cache: true cache: true
- name: Run tests - name: Run tests
@ -63,7 +63,7 @@ jobs:
- name: Set up Go - name: Set up Go
uses: actions/setup-go@v3 uses: actions/setup-go@v3
with: with:
go-version: '1.22' go-version: '1.23'
cache: true cache: true
- name: Install staticcheck - name: Install staticcheck
@ -81,7 +81,7 @@ jobs:
- name: Set up Go - name: Set up Go
uses: actions/setup-go@v3 uses: actions/setup-go@v3
with: with:
go-version: '1.21' go-version: '1.22'
cache: true cache: true
- name: Install gopls - name: Install gopls
@ -89,3 +89,23 @@ jobs:
- name: Run gopls - name: Run gopls
run: make gopls-run run: make gopls-run
fumpt:
name: Run gofumpt
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.23'
cache: true
- name: Install gofumpt
run: make fumpt-install
- name: Run gofumpt
run: |
make fumpt
git diff --exit-code --quiet

View file

@ -13,7 +13,7 @@ jobs:
- name: Setup Go - name: Setup Go
uses: actions/setup-go@v3 uses: actions/setup-go@v3
with: with:
go-version: '1.22' go-version: '1.23'
- name: Install govulncheck - name: Install govulncheck
run: go install golang.org/x/vuln/cmd/govulncheck@latest run: go install golang.org/x/vuln/cmd/govulncheck@latest

View file

@ -12,7 +12,8 @@ run:
# output configuration options # output configuration options
output: output:
# colored-line-number|line-number|json|tab|checkstyle|code-climate, default is "colored-line-number" # colored-line-number|line-number|json|tab|checkstyle|code-climate, default is "colored-line-number"
format: tab formats:
- format: tab
# all available settings of specific linters # all available settings of specific linters
linters-settings: linters-settings:
@ -37,6 +38,10 @@ linters-settings:
alias: alias:
pkg: git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object pkg: git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object
alias: objectSDK alias: objectSDK
unused:
field-writes-are-uses: false
exported-fields-are-used: false
local-variables-are-used: false
custom: custom:
truecloudlab-linters: truecloudlab-linters:
path: bin/linters/external_linters.so path: bin/linters/external_linters.so
@ -66,7 +71,7 @@ linters:
- bidichk - bidichk
- durationcheck - durationcheck
- exhaustive - exhaustive
- exportloopref - copyloopvar
- gofmt - gofmt
- goimports - goimports
- misspell - misspell
@ -82,5 +87,7 @@ linters:
- perfsprint - perfsprint
- testifylint - testifylint
- protogetter - protogetter
- intrange
- tenv
disable-all: true disable-all: true
fast: false fast: false

View file

@ -1,11 +0,0 @@
pipeline:
# Kludge for non-root containers under WoodPecker
fix-ownership:
image: alpine:latest
commands: chown -R 1234:1234 .
pre-commit:
image: git.frostfs.info/truecloudlab/frostfs-ci:v0.36
commands:
- export HOME="$(getent passwd $(id -u) | cut '-d:' -f6)"
- pre-commit run --hook-stage manual

View file

@ -9,6 +9,30 @@ Changelog for FrostFS Node
### Removed ### Removed
### Updated ### Updated
## [v0.42.0]
### Added
- Add audit logs for gRPC requests (#1184)
- Add CLI command to convert eACL to APE (#1189)
- Add `--await` flag to `control set-status` (#60)
- `app_info` metric for binary version (#1154)
- `--quiet` flag for healthcheck command (#1209)
### Changed
- Deprecate Container.SetEACL RPC (#1219)
### Fixed
- Take groups into account during APE processing (#1190)
- Handle double SIGHUP correctly (#1145)
- Handle empty filenames in tree listing (#1074)
- Handle duplicate tree nodes in the split-brain scenario (#1234, #1251)
- Remove APE pre-check in Object.GET/HEAD/RANGE RPC (#1249)
- Delete EC gc marks and split info (#1257)
- Do not search for non-existent objects on deletion (#1261)
### Updated
- Make putting EC chunks more robust (#1233)
## [v0.41.0] ## [v0.41.0]
### Added ### Added

View file

@ -4,20 +4,19 @@ SHELL = bash
REPO ?= $(shell go list -m) REPO ?= $(shell go list -m)
VERSION ?= $(shell git describe --tags --dirty --match "v*" --always --abbrev=8 2>/dev/null || cat VERSION 2>/dev/null || echo "develop") VERSION ?= $(shell git describe --tags --dirty --match "v*" --always --abbrev=8 2>/dev/null || cat VERSION 2>/dev/null || echo "develop")
HUB_IMAGE ?= truecloudlab/frostfs HUB_IMAGE ?= git.frostfs.info/truecloudlab/frostfs
HUB_TAG ?= "$(shell echo ${VERSION} | sed 's/^v//')" HUB_TAG ?= "$(shell echo ${VERSION} | sed 's/^v//')"
GO_VERSION ?= 1.22 GO_VERSION ?= 1.22
LINT_VERSION ?= 1.56.1 LINT_VERSION ?= 1.61.0
TRUECLOUDLAB_LINT_VERSION ?= 0.0.5 TRUECLOUDLAB_LINT_VERSION ?= 0.0.7
PROTOC_VERSION ?= 25.0 PROTOC_VERSION ?= 25.0
PROTOC_GEN_GO_VERSION ?= $(shell go list -f '{{.Version}}' -m google.golang.org/protobuf)
PROTOGEN_FROSTFS_VERSION ?= $(shell go list -f '{{.Version}}' -m git.frostfs.info/TrueCloudLab/frostfs-api-go/v2) PROTOGEN_FROSTFS_VERSION ?= $(shell go list -f '{{.Version}}' -m git.frostfs.info/TrueCloudLab/frostfs-api-go/v2)
PROTOC_OS_VERSION=osx-x86_64 PROTOC_OS_VERSION=osx-x86_64
ifeq ($(shell uname), Linux) ifeq ($(shell uname), Linux)
PROTOC_OS_VERSION=linux-x86_64 PROTOC_OS_VERSION=linux-x86_64
endif endif
STATICCHECK_VERSION ?= 2023.1.6 STATICCHECK_VERSION ?= 2024.1.1
ARCH = amd64 ARCH = amd64
BIN = bin BIN = bin
@ -28,24 +27,21 @@ DIRS = $(BIN) $(RELEASE)
CMDS = $(notdir $(basename $(wildcard cmd/frostfs-*))) CMDS = $(notdir $(basename $(wildcard cmd/frostfs-*)))
BINS = $(addprefix $(BIN)/, $(CMDS)) BINS = $(addprefix $(BIN)/, $(CMDS))
# .deb package versioning
OS_RELEASE = $(shell lsb_release -cs)
PKG_VERSION ?= $(shell echo $(VERSION) | sed "s/^v//" | \
sed -E "s/(.*)-(g[a-fA-F0-9]{6,8})(.*)/\1\3~\2/" | \
sed "s/-/~/")-${OS_RELEASE}
OUTPUT_LINT_DIR ?= $(abspath $(BIN))/linters OUTPUT_LINT_DIR ?= $(abspath $(BIN))/linters
LINT_DIR = $(OUTPUT_LINT_DIR)/golangci-lint-$(LINT_VERSION)-v$(TRUECLOUDLAB_LINT_VERSION) LINT_DIR = $(OUTPUT_LINT_DIR)/golangci-lint-$(LINT_VERSION)-v$(TRUECLOUDLAB_LINT_VERSION)
TMP_DIR := .cache TMP_DIR := .cache
PROTOBUF_DIR ?= $(abspath $(BIN))/protobuf PROTOBUF_DIR ?= $(abspath $(BIN))/protobuf
PROTOC_DIR ?= $(PROTOBUF_DIR)/protoc-v$(PROTOC_VERSION) PROTOC_DIR ?= $(PROTOBUF_DIR)/protoc-v$(PROTOC_VERSION)
PROTOC_GEN_GO_DIR ?= $(PROTOBUF_DIR)/protoc-gen-go-$(PROTOC_GEN_GO_VERSION)
PROTOGEN_FROSTFS_DIR ?= $(PROTOBUF_DIR)/protogen-$(PROTOGEN_FROSTFS_VERSION) PROTOGEN_FROSTFS_DIR ?= $(PROTOBUF_DIR)/protogen-$(PROTOGEN_FROSTFS_VERSION)
STATICCHECK_DIR ?= $(abspath $(BIN))/staticcheck STATICCHECK_DIR ?= $(abspath $(BIN))/staticcheck
STATICCHECK_VERSION_DIR ?= $(STATICCHECK_DIR)/$(STATICCHECK_VERSION) STATICCHECK_VERSION_DIR ?= $(STATICCHECK_DIR)/$(STATICCHECK_VERSION)
SOURCES = $(shell find . -type f -name "*.go" -print) SOURCES = $(shell find . -type f -name "*.go" -print)
GOFUMPT_VERSION ?= v0.7.0
GOFUMPT_DIR ?= $(abspath $(BIN))/gofumpt
GOFUMPT_VERSION_DIR ?= $(GOFUMPT_DIR)/$(GOFUMPT_VERSION)
GOPLS_VERSION ?= v0.15.1 GOPLS_VERSION ?= v0.15.1
GOPLS_DIR ?= $(abspath $(BIN))/gopls GOPLS_DIR ?= $(abspath $(BIN))/gopls
GOPLS_VERSION_DIR ?= $(GOPLS_DIR)/$(GOPLS_VERSION) GOPLS_VERSION_DIR ?= $(GOPLS_DIR)/$(GOPLS_VERSION)
@ -56,7 +52,7 @@ LOCODE_DB_PATH=$(abspath ./.cache/locode_db)
LOCODE_DB_VERSION=v0.4.0 LOCODE_DB_VERSION=v0.4.0
.PHONY: help all images dep clean fmts fumpt imports test lint docker/lint .PHONY: help all images dep clean fmts fumpt imports test lint docker/lint
prepare-release debpackage pre-commit unpre-commit prepare-release pre-commit unpre-commit
# To build a specific binary, use it's name prefix with bin/ as a target # To build a specific binary, use it's name prefix with bin/ as a target
# For example `make bin/frostfs-node` will build only storage node binary # For example `make bin/frostfs-node` will build only storage node binary
@ -103,17 +99,15 @@ export-metrics: dep
# Regenerate proto files: # Regenerate proto files:
protoc: protoc:
@if [ ! -d "$(PROTOC_DIR)" ] || [ ! -d "$(PROTOC_GEN_GO_DIR)" ] || [ ! -d "$(PROTOGEN_FROSTFS_DIR)" ]; then \ @if [ ! -d "$(PROTOC_DIR)" ] || [ ! -d "$(PROTOGEN_FROSTFS_DIR)" ]; then \
make protoc-install; \ make protoc-install; \
fi fi
@for f in `find . -type f -name '*.proto' -not -path './bin/*'`; do \ @for f in `find . -type f -name '*.proto' -not -path './bin/*'`; do \
echo "⇒ Processing $$f "; \ echo "⇒ Processing $$f "; \
$(PROTOC_DIR)/bin/protoc \ $(PROTOC_DIR)/bin/protoc \
--proto_path=.:$(PROTOC_DIR)/include:/usr/local/include \ --proto_path=.:$(PROTOC_DIR)/include:/usr/local/include \
--plugin=protoc-gen-go=$(PROTOC_GEN_GO_DIR)/protoc-gen-go \
--plugin=protoc-gen-go-frostfs=$(PROTOGEN_FROSTFS_DIR)/protogen \ --plugin=protoc-gen-go-frostfs=$(PROTOGEN_FROSTFS_DIR)/protogen \
--go-frostfs_out=. --go-frostfs_opt=paths=source_relative \ --go-frostfs_out=. --go-frostfs_opt=paths=source_relative \
--go_out=. --go_opt=paths=source_relative \
--go-grpc_opt=require_unimplemented_servers=false \ --go-grpc_opt=require_unimplemented_servers=false \
--go-grpc_out=. --go-grpc_opt=paths=source_relative $$f; \ --go-grpc_out=. --go-grpc_opt=paths=source_relative $$f; \
done done
@ -126,8 +120,6 @@ protoc-install:
@wget -q -O $(PROTOBUF_DIR)/protoc-$(PROTOC_VERSION).zip 'https://github.com/protocolbuffers/protobuf/releases/download/v$(PROTOC_VERSION)/protoc-$(PROTOC_VERSION)-$(PROTOC_OS_VERSION).zip' @wget -q -O $(PROTOBUF_DIR)/protoc-$(PROTOC_VERSION).zip 'https://github.com/protocolbuffers/protobuf/releases/download/v$(PROTOC_VERSION)/protoc-$(PROTOC_VERSION)-$(PROTOC_OS_VERSION).zip'
@unzip -q -o $(PROTOBUF_DIR)/protoc-$(PROTOC_VERSION).zip -d $(PROTOC_DIR) @unzip -q -o $(PROTOBUF_DIR)/protoc-$(PROTOC_VERSION).zip -d $(PROTOC_DIR)
@rm $(PROTOBUF_DIR)/protoc-$(PROTOC_VERSION).zip @rm $(PROTOBUF_DIR)/protoc-$(PROTOC_VERSION).zip
@echo "⇒ Installing protoc-gen-go..."
@GOBIN=$(PROTOC_GEN_GO_DIR) go install -v google.golang.org/protobuf/...@$(PROTOC_GEN_GO_VERSION)
@echo "⇒ Instaling protogen FrostFS plugin..." @echo "⇒ Instaling protogen FrostFS plugin..."
@GOBIN=$(PROTOGEN_FROSTFS_DIR) go install -mod=mod -v git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/util/protogen@$(PROTOGEN_FROSTFS_VERSION) @GOBIN=$(PROTOGEN_FROSTFS_DIR) go install -mod=mod -v git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/util/protogen@$(PROTOGEN_FROSTFS_VERSION)
@ -165,10 +157,19 @@ imports:
@echo "⇒ Processing goimports check" @echo "⇒ Processing goimports check"
@goimports -w cmd/ pkg/ misc/ @goimports -w cmd/ pkg/ misc/
# Install gofumpt
fumpt-install:
@rm -rf $(GOFUMPT_DIR)
@mkdir $(GOFUMPT_DIR)
@GOBIN=$(GOFUMPT_VERSION_DIR) go install mvdan.cc/gofumpt@$(GOFUMPT_VERSION)
# Run gofumpt # Run gofumpt
fumpt: fumpt:
@if [ ! -d "$(GOFUMPT_VERSION_DIR)" ]; then \
make fumpt-install; \
fi
@echo "⇒ Processing gofumpt check" @echo "⇒ Processing gofumpt check"
@gofumpt -l -w cmd/ pkg/ misc/ $(GOFUMPT_VERSION_DIR)/gofumpt -l -w cmd/ pkg/ misc/
# Run Unit Test with go test # Run Unit Test with go test
test: GOFLAGS ?= "-count=1" test: GOFLAGS ?= "-count=1"
@ -190,7 +191,7 @@ lint-install:
@@make -C $(TMP_DIR)/linters lib CGO_ENABLED=1 OUT_DIR=$(OUTPUT_LINT_DIR) @@make -C $(TMP_DIR)/linters lib CGO_ENABLED=1 OUT_DIR=$(OUTPUT_LINT_DIR)
@rm -rf $(TMP_DIR)/linters @rm -rf $(TMP_DIR)/linters
@rmdir $(TMP_DIR) 2>/dev/null || true @rmdir $(TMP_DIR) 2>/dev/null || true
@CGO_ENABLED=1 GOBIN=$(LINT_DIR) go install github.com/golangci/golangci-lint/cmd/golangci-lint@v$(LINT_VERSION) @CGO_ENABLED=1 GOBIN=$(LINT_DIR) go install -trimpath github.com/golangci/golangci-lint/cmd/golangci-lint@v$(LINT_VERSION)
# Run linters # Run linters
lint: lint:
@ -256,19 +257,6 @@ clean:
rm -rf $(BIN) rm -rf $(BIN)
rm -rf $(RELEASE) rm -rf $(RELEASE)
# Package for Debian
debpackage:
dch -b --package frostfs-node \
--controlmaint \
--newversion $(PKG_VERSION) \
--distribution $(OS_RELEASE) \
"Please see CHANGELOG.md for code changes for $(VERSION)"
dpkg-buildpackage --no-sign -b
# Cleanup deb package build directories
debclean:
dh clean
# Download locode database # Download locode database
locode-download: locode-download:
mkdir -p $(TMP_DIR) mkdir -p $(TMP_DIR)

View file

@ -7,9 +7,8 @@
</p> </p>
--- ---
[![Report](https://goreportcard.com/badge/github.com/TrueCloudLab/frostfs-node)](https://goreportcard.com/report/github.com/TrueCloudLab/frostfs-node) [![Report](https://goreportcard.com/badge/git.frostfs.info/TrueCloudLab/frostfs-node)](https://goreportcard.com/report/git.frostfs.info/TrueCloudLab/frostfs-node)
![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/TrueCloudLab/frostfs-node?sort=semver) ![Release (latest)](https://git.frostfs.info/TrueCloudLab/frostfs-node/badges/release.svg)
![License](https://img.shields.io/github/license/TrueCloudLab/frostfs-node.svg?style=popout)
# Overview # Overview
@ -33,8 +32,8 @@ manipulate large amounts of data without paying a prohibitive price.
FrostFS has a native [gRPC API](https://git.frostfs.info/TrueCloudLab/frostfs-api) and has FrostFS has a native [gRPC API](https://git.frostfs.info/TrueCloudLab/frostfs-api) and has
protocol gateways for popular protocols such as [AWS protocol gateways for popular protocols such as [AWS
S3](https://github.com/TrueCloudLab/frostfs-s3-gw), S3](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw),
[HTTP](https://github.com/TrueCloudLab/frostfs-http-gw), [HTTP](https://git.frostfs.info/TrueCloudLab/frostfs-http-gw),
[FUSE](https://wikipedia.org/wiki/Filesystem_in_Userspace) and [FUSE](https://wikipedia.org/wiki/Filesystem_in_Userspace) and
[sFTP](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol) allowing [sFTP](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol) allowing
developers to integrate applications without rewriting their code. developers to integrate applications without rewriting their code.
@ -45,11 +44,11 @@ Now, we only support GNU/Linux on amd64 CPUs with AVX/AVX2 instructions. More
platforms will be officially supported after release `1.0`. platforms will be officially supported after release `1.0`.
The latest version of frostfs-node works with frostfs-contract The latest version of frostfs-node works with frostfs-contract
[v0.16.0](https://github.com/TrueCloudLab/frostfs-contract/releases/tag/v0.16.0). [v0.19.2](https://git.frostfs.info/TrueCloudLab/frostfs-contract/releases/tag/v0.19.2).
# Building # Building
To make all binaries you need Go 1.21+ and `make`: To make all binaries you need Go 1.22+ and `make`:
``` ```
make all make all
``` ```
@ -71,7 +70,7 @@ make docker/bin/frostfs-<name> # build a specific binary
## Docker images ## Docker images
To make docker images suitable for use in [frostfs-dev-env](https://github.com/TrueCloudLab/frostfs-dev-env/) use: To make docker images suitable for use in [frostfs-dev-env](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env/) use:
``` ```
make images make images
``` ```
@ -125,7 +124,7 @@ the feature/topic you are going to implement.
# Credits # Credits
FrostFS is maintained by [True Cloud Lab](https://github.com/TrueCloudLab/) with the help and FrostFS is maintained by [True Cloud Lab](https://git.frostfs.info/TrueCloudLab/) with the help and
contributions from community members. contributions from community members.
Please see [CREDITS](CREDITS.md) for details. Please see [CREDITS](CREDITS.md) for details.

View file

@ -1 +1 @@
v0.41.0 v0.42.0

View file

@ -9,8 +9,8 @@ related configuration details.
To follow this guide you need: To follow this guide you need:
- latest released version of [neo-go](https://github.com/nspcc-dev/neo-go/releases) (v0.97.2 at the moment), - latest released version of [neo-go](https://github.com/nspcc-dev/neo-go/releases) (v0.97.2 at the moment),
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases) utility (v0.25.1 at the moment), - latest released version of [frostfs-adm](https://git.frostfs.info/TrueCloudLab/frostfs-node/releases) utility (v0.42.9 at the moment),
- latest released version of compiled [frostfs-contract](https://github.com/TrueCloudLab/frostfs-contract/releases) (v0.11.0 at the moment). - latest released version of compiled [frostfs-contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/releases) (v0.19.2 at the moment).
## Step 1: Prepare network configuration ## Step 1: Prepare network configuration

View file

@ -128,7 +128,7 @@ func generateConfigExample(appDir string, credSize int) (string, error) {
tmpl.AlphabetDir = filepath.Join(appDir, "alphabet-wallets") tmpl.AlphabetDir = filepath.Join(appDir, "alphabet-wallets")
var i innerring.GlagoliticLetter var i innerring.GlagoliticLetter
for i = 0; i < innerring.GlagoliticLetter(credSize); i++ { for i = range innerring.GlagoliticLetter(credSize) {
tmpl.Glagolitics = append(tmpl.Glagolitics, i.String()) tmpl.Glagolitics = append(tmpl.Glagolitics, i.String())
} }

View file

@ -0,0 +1,15 @@
package metabase
import "github.com/spf13/cobra"
// RootCmd is a root command of config section.
var RootCmd = &cobra.Command{
Use: "metabase",
Short: "Section for metabase commands",
}
func init() {
RootCmd.AddCommand(UpgradeCmd)
initUpgradeCommand()
}

View file

@ -0,0 +1,150 @@
package metabase
import (
"context"
"errors"
"fmt"
"sync"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/config"
engineconfig "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/config/engine"
shardconfig "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/config/engine/shard"
morphconfig "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/config/morph"
nodeconfig "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/config/node"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/core/container"
meta "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client"
morphcontainer "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/morph/client/container"
"github.com/spf13/cobra"
"golang.org/x/sync/errgroup"
)
const (
noCompactFlag = "no-compact"
)
var (
errNoPathsFound = errors.New("no metabase paths found")
errNoMorphEndpointsFound = errors.New("no morph endpoints found")
)
var UpgradeCmd = &cobra.Command{
Use: "upgrade",
Short: "Upgrade metabase to latest version",
RunE: upgrade,
}
func upgrade(cmd *cobra.Command, _ []string) error {
configFile, err := cmd.Flags().GetString(commonflags.ConfigFlag)
if err != nil {
return err
}
configDir, err := cmd.Flags().GetString(commonflags.ConfigDirFlag)
if err != nil {
return err
}
appCfg := config.New(configFile, configDir, config.EnvPrefix)
paths, err := getMetabasePaths(appCfg)
if err != nil {
return err
}
if len(paths) == 0 {
return errNoPathsFound
}
cmd.Println("found", len(paths), "metabases:")
for i, path := range paths {
cmd.Println(i+1, ":", path)
}
mc, err := createMorphClient(cmd.Context(), appCfg)
if err != nil {
return err
}
defer mc.Close()
civ, err := createContainerInfoProvider(mc)
if err != nil {
return err
}
noCompact, _ := cmd.Flags().GetBool(noCompactFlag)
result := make(map[string]bool)
var resultGuard sync.Mutex
eg, ctx := errgroup.WithContext(cmd.Context())
for _, path := range paths {
eg.Go(func() error {
var success bool
cmd.Println("upgrading metabase", path, "...")
if err := meta.Upgrade(ctx, path, !noCompact, civ, func(a ...any) {
cmd.Println(append([]any{time.Now().Format(time.RFC3339), ":", path, ":"}, a...)...)
}); err != nil {
cmd.Println("error: failed to upgrade metabase", path, ":", err)
} else {
success = true
cmd.Println("metabase", path, "upgraded successfully")
}
resultGuard.Lock()
result[path] = success
resultGuard.Unlock()
return nil
})
}
if err := eg.Wait(); err != nil {
return err
}
for mb, ok := range result {
if ok {
cmd.Println(mb, ": success")
} else {
cmd.Println(mb, ": failed")
}
}
return nil
}
func getMetabasePaths(appCfg *config.Config) ([]string, error) {
var paths []string
if err := engineconfig.IterateShards(appCfg, false, func(sc *shardconfig.Config) error {
paths = append(paths, sc.Metabase().Path())
return nil
}); err != nil {
return nil, fmt.Errorf("get metabase paths: %w", err)
}
return paths, nil
}
func createMorphClient(ctx context.Context, appCfg *config.Config) (*client.Client, error) {
addresses := morphconfig.RPCEndpoint(appCfg)
if len(addresses) == 0 {
return nil, errNoMorphEndpointsFound
}
key := nodeconfig.Key(appCfg)
cli, err := client.New(ctx,
key,
client.WithDialTimeout(morphconfig.DialTimeout(appCfg)),
client.WithEndpoints(addresses...),
client.WithSwitchInterval(morphconfig.SwitchInterval(appCfg)),
)
if err != nil {
return nil, fmt.Errorf("create morph client:%w", err)
}
return cli, nil
}
func createContainerInfoProvider(cli *client.Client) (container.InfoProvider, error) {
sh, err := cli.NNSContractAddress(client.NNSContainerContractName)
if err != nil {
return nil, fmt.Errorf("resolve container contract hash: %w", err)
}
cc, err := morphcontainer.NewFromMorph(cli, sh, 0, morphcontainer.TryNotary())
if err != nil {
return nil, fmt.Errorf("create morph container client: %w", err)
}
return container.NewInfoProvider(func() (container.Source, error) {
return morphcontainer.AsContainerSource(cc), nil
}), nil
}
func initUpgradeCommand() {
flags := UpgradeCmd.Flags()
flags.Bool(noCompactFlag, false, "Do not compact upgraded metabase file")
}

View file

@ -38,6 +38,12 @@ var (
func parseTarget(cmd *cobra.Command) policyengine.Target { func parseTarget(cmd *cobra.Command) policyengine.Target {
name, _ := cmd.Flags().GetString(targetNameFlag) name, _ := cmd.Flags().GetString(targetNameFlag)
typ, err := parseTargetType(cmd) typ, err := parseTargetType(cmd)
// interpret "root" namespace as empty
if typ == policyengine.Namespace && name == "root" {
name = ""
}
commonCmd.ExitOnErr(cmd, "read target type error: %w", err) commonCmd.ExitOnErr(cmd, "read target type error: %w", err)
return policyengine.Target{ return policyengine.Target{

View file

@ -139,13 +139,12 @@ func dumpContainers(cmd *cobra.Command, _ []string) error {
func dumpSingleContainer(bw *io.BufBinWriter, ch util.Uint160, inv *invoker.Invoker, id []byte) (*Container, error) { func dumpSingleContainer(bw *io.BufBinWriter, ch util.Uint160, inv *invoker.Invoker, id []byte) (*Container, error) {
bw.Reset() bw.Reset()
emit.AppCall(bw.BinWriter, ch, "get", callflag.All, id) emit.AppCall(bw.BinWriter, ch, "get", callflag.All, id)
emit.AppCall(bw.BinWriter, ch, "eACL", callflag.All, id)
res, err := inv.Run(bw.Bytes()) res, err := inv.Run(bw.Bytes())
if err != nil { if err != nil {
return nil, fmt.Errorf("can't get container info: %w", err) return nil, fmt.Errorf("can't get container info: %w", err)
} }
if len(res.Stack) != 2 { if len(res.Stack) != 1 {
return nil, fmt.Errorf("%w: expected 2 items on stack", errInvalidContainerResponse) return nil, fmt.Errorf("%w: expected 1 items on stack", errInvalidContainerResponse)
} }
cnt := new(Container) cnt := new(Container)
@ -154,14 +153,6 @@ func dumpSingleContainer(bw *io.BufBinWriter, ch util.Uint160, inv *invoker.Invo
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err) return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
} }
ea := new(EACL)
err = ea.FromStackItem(res.Stack[1])
if err != nil {
return nil, fmt.Errorf("%w: %v", errInvalidContainerResponse, err)
}
if len(ea.Value) != 0 {
cnt.EACL = ea
}
return cnt, nil return cnt, nil
} }
@ -258,10 +249,6 @@ func restoreOrPutContainers(containers []Container, isOK func([]byte) bool, cmd
func putContainer(bw *io.BufBinWriter, ch util.Uint160, cnt Container) { func putContainer(bw *io.BufBinWriter, ch util.Uint160, cnt Container) {
emit.AppCall(bw.BinWriter, ch, "put", callflag.All, emit.AppCall(bw.BinWriter, ch, "put", callflag.All,
cnt.Value, cnt.Signature, cnt.PublicKey, cnt.Token) cnt.Value, cnt.Signature, cnt.PublicKey, cnt.Token)
if ea := cnt.EACL; ea != nil {
emit.AppCall(bw.BinWriter, ch, "setEACL", callflag.All,
ea.Value, ea.Signature, ea.PublicKey, ea.Token)
}
} }
func isContainerRestored(cmd *cobra.Command, wCtx *helper.InitializeContext, containerHash util.Uint160, bw *io.BufBinWriter, hashValue util.Uint256) (bool, error) { func isContainerRestored(cmd *cobra.Command, wCtx *helper.InitializeContext, containerHash util.Uint160, bw *io.BufBinWriter, hashValue util.Uint256) (bool, error) {
@ -322,15 +309,6 @@ type Container struct {
Signature []byte `json:"signature"` Signature []byte `json:"signature"`
PublicKey []byte `json:"public_key"` PublicKey []byte `json:"public_key"`
Token []byte `json:"token"` Token []byte `json:"token"`
EACL *EACL `json:"eacl"`
}
// EACL represents extended ACL struct in contract storage.
type EACL struct {
Value []byte `json:"value"`
Signature []byte `json:"signature"`
PublicKey []byte `json:"public_key"`
Token []byte `json:"token"`
} }
// ToStackItem implements stackitem.Convertible. // ToStackItem implements stackitem.Convertible.
@ -377,50 +355,6 @@ func (c *Container) FromStackItem(item stackitem.Item) error {
return nil return nil
} }
// ToStackItem implements stackitem.Convertible.
func (c *EACL) ToStackItem() (stackitem.Item, error) {
return stackitem.NewStruct([]stackitem.Item{
stackitem.NewByteArray(c.Value),
stackitem.NewByteArray(c.Signature),
stackitem.NewByteArray(c.PublicKey),
stackitem.NewByteArray(c.Token),
}), nil
}
// FromStackItem implements stackitem.Convertible.
func (c *EACL) FromStackItem(item stackitem.Item) error {
arr, ok := item.Value().([]stackitem.Item)
if !ok || len(arr) != 4 {
return errors.New("invalid stack item type")
}
value, err := arr[0].TryBytes()
if err != nil {
return errors.New("invalid eACL value")
}
sig, err := arr[1].TryBytes()
if err != nil {
return errors.New("invalid eACL signature")
}
pub, err := arr[2].TryBytes()
if err != nil {
return errors.New("invalid eACL public key")
}
tok, err := arr[3].TryBytes()
if err != nil {
return errors.New("invalid eACL token")
}
c.Value = value
c.Signature = sig
c.PublicKey = pub
c.Token = tok
return nil
}
// getCIDFilterFunc returns filtering function for container IDs. // getCIDFilterFunc returns filtering function for container IDs.
// Raw byte slices are used because it works with structures returned // Raw byte slices are used because it works with structures returned
// from contract. // from contract.

View file

@ -68,7 +68,7 @@ func dumpContractHashes(cmd *cobra.Command, _ []string) error {
if irSize != 0 { if irSize != 0 {
bw.Reset() bw.Reset()
for i := 0; i < irSize; i++ { for i := range irSize {
emit.AppCall(bw.BinWriter, cs.Hash, "resolve", callflag.ReadOnly, emit.AppCall(bw.BinWriter, cs.Hash, "resolve", callflag.ReadOnly,
helper.GetAlphabetNNSDomain(i), helper.GetAlphabetNNSDomain(i),
int64(nns.TXT)) int64(nns.TXT))
@ -79,7 +79,7 @@ func dumpContractHashes(cmd *cobra.Command, _ []string) error {
return fmt.Errorf("can't fetch info from NNS: %w", err) return fmt.Errorf("can't fetch info from NNS: %w", err)
} }
for i := 0; i < irSize; i++ { for i := range irSize {
info := contractDumpInfo{name: fmt.Sprintf("alphabet %d", i)} info := contractDumpInfo{name: fmt.Sprintf("alphabet %d", i)}
if h, err := helper.ParseNNSResolveResult(alphaRes.Stack[i]); err == nil { if h, err := helper.ParseNNSResolveResult(alphaRes.Stack[i]); err == nil {
info.hash = h info.hash = h

View file

@ -73,7 +73,6 @@ func initializeWallets(v *viper.Viper, walletDir string, size int) ([]string, er
return nil, fmt.Errorf("can't fetch password: %w", err) return nil, fmt.Errorf("can't fetch password: %w", err)
} }
i := i
errG.Go(func() error { errG.Go(func() error {
p := filepath.Join(walletDir, innerring.GlagoliticLetter(i).String()+".json") p := filepath.Join(walletDir, innerring.GlagoliticLetter(i).String()+".json")
f, err := os.OpenFile(p, os.O_CREATE, 0o644) f, err := os.OpenFile(p, os.O_CREATE, 0o644)
@ -107,7 +106,6 @@ func initializeWallets(v *viper.Viper, walletDir string, size int) ([]string, er
// Create consensus account with 2*N/3+1 multi-signature. // Create consensus account with 2*N/3+1 multi-signature.
bftCount := smartcontract.GetDefaultHonestNodeCount(size) bftCount := smartcontract.GetDefaultHonestNodeCount(size)
for i := range wallets { for i := range wallets {
i := i
ps := pubs.Copy() ps := pubs.Copy()
errG.Go(func() error { errG.Go(func() error {
if err := addMultisigAccount(wallets[i], majCount, constants.CommitteeAccountName, passwords[i], ps); err != nil { if err := addMultisigAccount(wallets[i], majCount, constants.CommitteeAccountName, passwords[i], ps); err != nil {

View file

@ -63,7 +63,7 @@ func TestGenerateAlphabet(t *testing.T) {
buf.Reset() buf.Reset()
v.Set(commonflags.AlphabetWalletsFlag, walletDir) v.Set(commonflags.AlphabetWalletsFlag, walletDir)
require.NoError(t, GenerateAlphabetCmd.Flags().Set(commonflags.AlphabetSizeFlag, strconv.FormatUint(size, 10))) require.NoError(t, GenerateAlphabetCmd.Flags().Set(commonflags.AlphabetSizeFlag, strconv.FormatUint(size, 10)))
for i := uint64(0); i < size; i++ { for i := range uint64(size) {
buf.WriteString(strconv.FormatUint(i, 10) + "\r") buf.WriteString(strconv.FormatUint(i, 10) + "\r")
} }

View file

@ -166,5 +166,6 @@ func DeployNNS(c *InitializeContext, method string) error {
return fmt.Errorf("can't send deploy transaction: %w", err) return fmt.Errorf("can't send deploy transaction: %w", err)
} }
c.Command.Println("NNS hash:", invokeHash.StringLE())
return c.AwaitTx() return c.AwaitTx()
} }

View file

@ -224,7 +224,7 @@ func (l *LocalClient) CalculateNetworkFee(tx *transaction.Transaction) (int64, e
paramz = []manifest.Parameter{{Type: smartcontract.SignatureType}} paramz = []manifest.Parameter{{Type: smartcontract.SignatureType}}
} else if nSigs, _, ok := vm.ParseMultiSigContract(w.VerificationScript); ok { } else if nSigs, _, ok := vm.ParseMultiSigContract(w.VerificationScript); ok {
paramz = make([]manifest.Parameter, nSigs) paramz = make([]manifest.Parameter, nSigs)
for j := 0; j < nSigs; j++ { for j := range nSigs {
paramz[j] = manifest.Parameter{Type: smartcontract.SignatureType} paramz[j] = manifest.Parameter{Type: smartcontract.SignatureType}
} }
} }

View file

@ -72,13 +72,17 @@ func InvalidConfigValueErr(key string) error {
return fmt.Errorf("invalid %s config value from netmap contract", key) return fmt.Errorf("invalid %s config value from netmap contract", key)
} }
func EmitNewEpochCall(bw *io.BufBinWriter, wCtx *InitializeContext, nmHash util.Uint160) error { func EmitNewEpochCall(bw *io.BufBinWriter, wCtx *InitializeContext, nmHash util.Uint160, countEpoch int64) error {
if countEpoch <= 0 {
return errors.New("number of epochs cannot be less than 1")
}
curr, err := unwrap.Int64(wCtx.ReadOnlyInvoker.Call(nmHash, "epoch")) curr, err := unwrap.Int64(wCtx.ReadOnlyInvoker.Call(nmHash, "epoch"))
if err != nil { if err != nil {
return errors.New("can't fetch current epoch from the netmap contract") return errors.New("can't fetch current epoch from the netmap contract")
} }
newEpoch := curr + 1 newEpoch := curr + countEpoch
wCtx.Command.Printf("Current epoch: %d, increase to %d.\n", curr, newEpoch) wCtx.Command.Printf("Current epoch: %d, increase to %d.\n", curr, newEpoch)
// In NeoFS this is done via Notary contract. Here, however, we can form the // In NeoFS this is done via Notary contract. Here, however, we can form the

View file

@ -44,7 +44,7 @@ func openAlphabetWallets(v *viper.Viper, walletDir string) ([]*wallet.Wallet, er
var wallets []*wallet.Wallet var wallets []*wallet.Wallet
var letter string var letter string
for i := 0; i < constants.MaxAlphabetNodes; i++ { for i := range constants.MaxAlphabetNodes {
letter = innerring.GlagoliticLetter(i).String() letter = innerring.GlagoliticLetter(i).String()
p := filepath.Join(walletDir, letter+".json") p := filepath.Join(walletDir, letter+".json")
var w *wallet.Wallet var w *wallet.Wallet

View file

@ -1,6 +1,8 @@
package initialize package initialize
import ( import (
"fmt"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/helper" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/helper"
"github.com/nspcc-dev/neo-go/pkg/core/native/noderoles" "github.com/nspcc-dev/neo-go/pkg/core/native/noderoles"
"github.com/nspcc-dev/neo-go/pkg/io" "github.com/nspcc-dev/neo-go/pkg/io"
@ -29,10 +31,14 @@ func setNotaryAndAlphabetNodes(c *helper.InitializeContext) error {
callflag.States|callflag.AllowNotify, int64(noderoles.NeoFSAlphabet), pubs) callflag.States|callflag.AllowNotify, int64(noderoles.NeoFSAlphabet), pubs)
if err := c.SendCommitteeTx(w.Bytes(), false); err != nil { if err := c.SendCommitteeTx(w.Bytes(), false); err != nil {
return err return fmt.Errorf("send committee transaction: %w", err)
} }
return c.AwaitTx() err := c.AwaitTx()
if err != nil {
err = fmt.Errorf("await committee transaction: %w", err)
}
return err
} }
func setRolesFinished(c *helper.InitializeContext) (bool, error) { func setRolesFinished(c *helper.InitializeContext) (bool, error) {

View file

@ -113,7 +113,7 @@ func generateTestData(dir string, size int) error {
} }
var pubs []string var pubs []string
for i := 0; i < size; i++ { for i := range size {
p := filepath.Join(dir, innerring.GlagoliticLetter(i).String()+".json") p := filepath.Join(dir, innerring.GlagoliticLetter(i).String()+".json")
w, err := wallet.NewWalletFromFile(p) w, err := wallet.NewWalletFromFile(p)
if err != nil { if err != nil {
@ -148,7 +148,7 @@ func generateTestData(dir string, size int) error {
} }
func setTestCredentials(v *viper.Viper, size int) { func setTestCredentials(v *viper.Viper, size int) {
for i := 0; i < size; i++ { for i := range size {
v.Set("credentials."+innerring.GlagoliticLetter(i).String(), strconv.FormatUint(uint64(i), 10)) v.Set("credentials."+innerring.GlagoliticLetter(i).String(), strconv.FormatUint(uint64(i), 10))
} }
v.Set("credentials.contract", constants.TestContractPassword) v.Set("credentials.contract", constants.TestContractPassword)

View file

@ -3,6 +3,7 @@ package initialize
import ( import (
"fmt" "fmt"
"math/big" "math/big"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/constants" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/constants"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/helper" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph/helper"
@ -26,12 +27,12 @@ const (
initialAlphabetGASAmount = 10_000 * native.GASFactor initialAlphabetGASAmount = 10_000 * native.GASFactor
// initialProxyGASAmount represents the amount of GAS given to a proxy contract. // initialProxyGASAmount represents the amount of GAS given to a proxy contract.
initialProxyGASAmount = 50_000 * native.GASFactor initialProxyGASAmount = 50_000 * native.GASFactor
// alphabetGasRatio is a coefficient that defines the threshold below which
// the balance of the alphabet node is considered not replenished. The value
// of this coefficient is determined empirically.
alphabetGasRatio = 5
) )
func initialCommitteeGASAmount(c *helper.InitializeContext) int64 {
return (gasInitialTotalSupply - initialAlphabetGASAmount*int64(len(c.Wallets))) / 2
}
func transferFunds(c *helper.InitializeContext) error { func transferFunds(c *helper.InitializeContext) error {
ok, err := transferFundsFinished(c) ok, err := transferFundsFinished(c)
if ok || err != nil { if ok || err != nil {
@ -58,7 +59,7 @@ func transferFunds(c *helper.InitializeContext) error {
transferTarget{ transferTarget{
Token: gas.Hash, Token: gas.Hash,
Address: c.CommitteeAcc.Contract.ScriptHash(), Address: c.CommitteeAcc.Contract.ScriptHash(),
Amount: (gasInitialTotalSupply - initialAlphabetGASAmount*int64(len(c.Wallets))) / 2, Amount: initialCommitteeGASAmount(c),
}, },
transferTarget{ transferTarget{
Token: neo.Hash, Token: neo.Hash,
@ -79,12 +80,19 @@ func transferFunds(c *helper.InitializeContext) error {
return c.AwaitTx() return c.AwaitTx()
} }
// transferFundsFinished checks balances of accounts we transfer GAS to.
// The stage is considered finished if the balance is greater than the half of what we need to transfer.
func transferFundsFinished(c *helper.InitializeContext) (bool, error) { func transferFundsFinished(c *helper.InitializeContext) (bool, error) {
acc := c.Accounts[0] acc := c.Accounts[0]
r := nep17.NewReader(c.ReadOnlyInvoker, gas.Hash) r := nep17.NewReader(c.ReadOnlyInvoker, gas.Hash)
res, err := r.BalanceOf(acc.Contract.ScriptHash()) res, err := r.BalanceOf(acc.Contract.ScriptHash())
return res.Cmp(big.NewInt(alphabetGasRatio*native.GASFactor)) == 1, err if err != nil || res.Cmp(big.NewInt(initialAlphabetGASAmount/2)) != 1 {
return false, err
}
res, err = r.BalanceOf(c.CommitteeAcc.ScriptHash())
return res != nil && res.Cmp(big.NewInt(initialCommitteeGASAmount(c)/2)) == 1, err
} }
func transferGASToProxy(c *helper.InitializeContext) error { func transferGASToProxy(c *helper.InitializeContext) error {
@ -144,5 +152,17 @@ func createNEP17MultiTransferTx(c helper.Client, acc *wallet.Account, recipients
if err != nil { if err != nil {
return nil, fmt.Errorf("can't create actor: %w", err) return nil, fmt.Errorf("can't create actor: %w", err)
} }
return act.MakeRun(w.Bytes()) tx, err := act.MakeRun(w.Bytes())
if err != nil {
sum := make(map[util.Uint160]int64)
for _, recipient := range recipients {
sum[recipient.Token] += recipient.Amount
}
detail := make([]string, 0, len(sum))
for _, value := range sum {
detail = append(detail, fmt.Sprintf("amount=%v", value))
}
err = fmt.Errorf("transfer failed: from=%s(%s) %s: %w", acc.Label, acc.Address, strings.Join(detail, " "), err)
}
return tx, err
} }

View file

@ -12,6 +12,8 @@ import (
"github.com/spf13/viper" "github.com/spf13/viper"
) )
const deltaFlag = "delta"
func ForceNewEpochCmd(cmd *cobra.Command, _ []string) error { func ForceNewEpochCmd(cmd *cobra.Command, _ []string) error {
wCtx, err := helper.NewInitializeContext(cmd, viper.GetViper()) wCtx, err := helper.NewInitializeContext(cmd, viper.GetViper())
if err != nil { if err != nil {
@ -30,7 +32,8 @@ func ForceNewEpochCmd(cmd *cobra.Command, _ []string) error {
} }
bw := io.NewBufBinWriter() bw := io.NewBufBinWriter()
if err := helper.EmitNewEpochCall(bw, wCtx, nmHash); err != nil { delta, _ := cmd.Flags().GetInt64(deltaFlag)
if err := helper.EmitNewEpochCall(bw, wCtx, nmHash, delta); err != nil {
return err return err
} }

View file

@ -35,6 +35,7 @@ func initForceNewEpochCmd() {
ForceNewEpoch.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc) ForceNewEpoch.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
ForceNewEpoch.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc) ForceNewEpoch.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
ForceNewEpoch.Flags().String(commonflags.LocalDumpFlag, "", "Path to the blocks dump file") ForceNewEpoch.Flags().String(commonflags.LocalDumpFlag, "", "Path to the blocks dump file")
ForceNewEpoch.Flags().Int64(deltaFlag, 1, "Number of epochs to increase the current epoch")
} }
func init() { func init() {

View file

@ -42,3 +42,23 @@ func registerDomain(cmd *cobra.Command, _ []string) {
commonCmd.ExitOnErr(cmd, "register domain error: %w", err) commonCmd.ExitOnErr(cmd, "register domain error: %w", err)
cmd.Println("Domain registered successfully") cmd.Println("Domain registered successfully")
} }
func initDeleteCmd() {
Cmd.AddCommand(deleteCmd)
deleteCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
deleteCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
deleteCmd.Flags().String(nnsNameFlag, "", nnsNameFlagDesc)
_ = cobra.MarkFlagRequired(deleteCmd.Flags(), nnsNameFlag)
}
func deleteDomain(cmd *cobra.Command, _ []string) {
c, actor, _ := getRPCClient(cmd)
name, _ := cmd.Flags().GetString(nnsNameFlag)
h, vub, err := c.DeleteDomain(name)
_, err = actor.Wait(h, vub, err)
commonCmd.ExitOnErr(cmd, "delete domain error: %w", err)
cmd.Println("Domain deleted successfully")
}

View file

@ -47,6 +47,19 @@ func initDelRecordsCmd() {
_ = cobra.MarkFlagRequired(delRecordsCmd.Flags(), nnsRecordTypeFlag) _ = cobra.MarkFlagRequired(delRecordsCmd.Flags(), nnsRecordTypeFlag)
} }
func initDelRecordCmd() {
Cmd.AddCommand(delRecordCmd)
delRecordCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
delRecordCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
delRecordCmd.Flags().String(nnsNameFlag, "", nnsNameFlagDesc)
delRecordCmd.Flags().String(nnsRecordTypeFlag, "", nnsRecordTypeFlagDesc)
delRecordCmd.Flags().String(nnsRecordDataFlag, "", nnsRecordDataFlagDesc)
_ = cobra.MarkFlagRequired(delRecordCmd.Flags(), nnsNameFlag)
_ = cobra.MarkFlagRequired(delRecordCmd.Flags(), nnsRecordTypeFlag)
_ = cobra.MarkFlagRequired(delRecordCmd.Flags(), nnsRecordDataFlag)
}
func addRecord(cmd *cobra.Command, _ []string) { func addRecord(cmd *cobra.Command, _ []string) {
c, actor, _ := getRPCClient(cmd) c, actor, _ := getRPCClient(cmd)
name, _ := cmd.Flags().GetString(nnsNameFlag) name, _ := cmd.Flags().GetString(nnsNameFlag)
@ -115,6 +128,22 @@ func delRecords(cmd *cobra.Command, _ []string) {
cmd.Println("Records removed successfully") cmd.Println("Records removed successfully")
} }
func delRecord(cmd *cobra.Command, _ []string) {
c, actor, _ := getRPCClient(cmd)
name, _ := cmd.Flags().GetString(nnsNameFlag)
data, _ := cmd.Flags().GetString(nnsRecordDataFlag)
recordType, _ := cmd.Flags().GetString(nnsRecordTypeFlag)
typ, err := getRecordType(recordType)
commonCmd.ExitOnErr(cmd, "unable to parse record type: %w", err)
h, vub, err := c.DeleteRecord(name, typ, data)
commonCmd.ExitOnErr(cmd, "unable to delete record: %w", err)
cmd.Println("Waiting for transaction to persist...")
_, err = actor.Wait(h, vub, err)
commonCmd.ExitOnErr(cmd, "delete records error: %w", err)
cmd.Println("Record removed successfully")
}
func getRecordType(recordType string) (*big.Int, error) { func getRecordType(recordType string) (*big.Int, error) {
switch strings.ToUpper(recordType) { switch strings.ToUpper(recordType) {
case "A": case "A":

View file

@ -42,6 +42,15 @@ var (
}, },
Run: registerDomain, Run: registerDomain,
} }
deleteCmd = &cobra.Command{
Use: "delete",
Short: "Delete a domain by name",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(commonflags.EndpointFlag, cmd.Flags().Lookup(commonflags.EndpointFlag))
_ = viper.BindPFlag(commonflags.AlphabetWalletsFlag, cmd.Flags().Lookup(commonflags.AlphabetWalletsFlag))
},
Run: deleteDomain,
}
renewCmd = &cobra.Command{ renewCmd = &cobra.Command{
Use: "renew", Use: "renew",
Short: "Increases domain expiration date", Short: "Increases domain expiration date",
@ -86,14 +95,25 @@ var (
}, },
Run: delRecords, Run: delRecords,
} }
delRecordCmd = &cobra.Command{
Use: "delete-record",
Short: "Removes domain record with the specified type and data",
PreRun: func(cmd *cobra.Command, _ []string) {
_ = viper.BindPFlag(commonflags.EndpointFlag, cmd.Flags().Lookup(commonflags.EndpointFlag))
_ = viper.BindPFlag(commonflags.AlphabetWalletsFlag, cmd.Flags().Lookup(commonflags.AlphabetWalletsFlag))
},
Run: delRecord,
}
) )
func init() { func init() {
initTokensCmd() initTokensCmd()
initRegisterCmd() initRegisterCmd()
initDeleteCmd()
initRenewCmd() initRenewCmd()
initUpdateCmd() initUpdateCmd()
initAddRecordCmd() initAddRecordCmd()
initGetRecordsCmd() initGetRecordsCmd()
initDelRecordsCmd() initDelRecordsCmd()
initDelRecordCmd()
} }

View file

@ -1,15 +1,25 @@
package nns package nns
import ( import (
"math/big"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-contract/nns"
client "git.frostfs.info/TrueCloudLab/frostfs-contract/rpcclient/nns"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common" commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
const (
verboseDesc = "Include additional information about CNAME record."
)
func initTokensCmd() { func initTokensCmd() {
Cmd.AddCommand(tokensCmd) Cmd.AddCommand(tokensCmd)
tokensCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc) tokensCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
tokensCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc) tokensCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
tokensCmd.Flags().BoolP(commonflags.Verbose, commonflags.VerboseShorthand, false, verboseDesc)
} }
func listTokens(cmd *cobra.Command, _ []string) { func listTokens(cmd *cobra.Command, _ []string) {
@ -18,7 +28,39 @@ func listTokens(cmd *cobra.Command, _ []string) {
commonCmd.ExitOnErr(cmd, "unable to get tokens: %w", err) commonCmd.ExitOnErr(cmd, "unable to get tokens: %w", err)
for toks, err := it.Next(10); err == nil && len(toks) > 0; toks, err = it.Next(10) { for toks, err := it.Next(10); err == nil && len(toks) > 0; toks, err = it.Next(10) {
for _, token := range toks { for _, token := range toks {
cmd.Println(string(token)) output := string(token)
if verbose, _ := cmd.Flags().GetBool(commonflags.Verbose); verbose {
cname, err := getCnameRecord(c, token)
commonCmd.ExitOnErr(cmd, "", err)
if cname != "" {
output += " (CNAME: " + cname + ")"
}
}
cmd.Println(output)
} }
} }
} }
func getCnameRecord(c *client.Contract, token []byte) (string, error) {
items, err := c.GetRecords(string(token), big.NewInt(int64(nns.CNAME)))
// GetRecords returns the error "not an array" if the domain does not contain records.
if err != nil && strings.Contains(err.Error(), "not an array") {
return "", nil
}
if err != nil {
return "", err
}
if len(items) == 0 {
return "", nil
}
record, err := items[0].TryBytes()
if err != nil {
return "", err
}
return string(record), nil
}

View file

@ -53,7 +53,7 @@ func RemoveNodesCmd(cmd *cobra.Command, args []string) error {
int64(netmapcontract.NodeStateOffline), nodeKeys[i].Bytes()) int64(netmapcontract.NodeStateOffline), nodeKeys[i].Bytes())
} }
if err := helper.EmitNewEpochCall(bw, wCtx, nmHash); err != nil { if err := helper.EmitNewEpochCall(bw, wCtx, nmHash, 1); err != nil {
return err return err
} }

View file

@ -30,11 +30,13 @@ var (
func initProxyAddAccount() { func initProxyAddAccount() {
AddAccountCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc) AddAccountCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
AddAccountCmd.Flags().String(accountAddressFlag, "", "Wallet address string") AddAccountCmd.Flags().String(accountAddressFlag, "", "Wallet address string")
AddAccountCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
} }
func initProxyRemoveAccount() { func initProxyRemoveAccount() {
RemoveAccountCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc) RemoveAccountCmd.Flags().StringP(commonflags.EndpointFlag, commonflags.EndpointFlagShort, "", commonflags.EndpointFlagDesc)
RemoveAccountCmd.Flags().String(accountAddressFlag, "", "Wallet address string") RemoveAccountCmd.Flags().String(accountAddressFlag, "", "Wallet address string")
RemoveAccountCmd.Flags().String(commonflags.AlphabetWalletsFlag, "", commonflags.AlphabetWalletsFlagDesc)
} }
func init() { func init() {

View file

@ -5,6 +5,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/metabase"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/storagecfg" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/storagecfg"
"git.frostfs.info/TrueCloudLab/frostfs-node/misc" "git.frostfs.info/TrueCloudLab/frostfs-node/misc"
@ -41,6 +42,7 @@ func init() {
rootCmd.AddCommand(config.RootCmd) rootCmd.AddCommand(config.RootCmd)
rootCmd.AddCommand(morph.RootCmd) rootCmd.AddCommand(morph.RootCmd)
rootCmd.AddCommand(storagecfg.RootCmd) rootCmd.AddCommand(storagecfg.RootCmd)
rootCmd.AddCommand(metabase.RootCmd)
rootCmd.AddCommand(autocomplete.Command("frostfs-adm")) rootCmd.AddCommand(autocomplete.Command("frostfs-adm"))
rootCmd.AddCommand(gendoc.Command(rootCmd, gendoc.Options{})) rootCmd.AddCommand(gendoc.Command(rootCmd, gendoc.Options{}))

View file

@ -2,10 +2,13 @@ package internal
import ( import (
"bytes" "bytes"
"cmp"
"context" "context"
"errors" "errors"
"fmt" "fmt"
"io" "io"
"os"
"slices"
"sort" "sort"
"strings" "strings"
@ -14,7 +17,6 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
containerSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container" containerSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
@ -189,54 +191,6 @@ func DeleteContainer(ctx context.Context, prm DeleteContainerPrm) (res DeleteCon
return return
} }
// EACLPrm groups parameters of EACL operation.
type EACLPrm struct {
Client *client.Client
ClientParams client.PrmContainerEACL
}
// EACLRes groups the resulting values of EACL operation.
type EACLRes struct {
cliRes *client.ResContainerEACL
}
// EACL returns requested eACL table.
func (x EACLRes) EACL() eacl.Table {
return x.cliRes.Table()
}
// EACL reads eACL table from FrostFS by container ID.
//
// Returns any error which prevented the operation from completing correctly in error return.
func EACL(ctx context.Context, prm EACLPrm) (res EACLRes, err error) {
res.cliRes, err = prm.Client.ContainerEACL(ctx, prm.ClientParams)
return
}
// SetEACLPrm groups parameters of SetEACL operation.
type SetEACLPrm struct {
Client *client.Client
ClientParams client.PrmContainerSetEACL
}
// SetEACLRes groups the resulting values of SetEACL operation.
type SetEACLRes struct{}
// SetEACL requests to save an eACL table in FrostFS.
//
// Operation is asynchronous and no guaranteed even in the absence of errors.
// The required time is also not predictable.
//
// Success can be verified by reading by container identifier.
//
// Returns any error which prevented the operation from completing correctly in error return.
func SetEACL(ctx context.Context, prm SetEACLPrm) (res SetEACLRes, err error) {
_, err = prm.Client.ContainerSetEACL(ctx, prm.ClientParams)
return
}
// NetworkInfoPrm groups parameters of NetworkInfo operation. // NetworkInfoPrm groups parameters of NetworkInfo operation.
type NetworkInfoPrm struct { type NetworkInfoPrm struct {
Client *client.Client Client *client.Client
@ -611,13 +565,6 @@ type HeadObjectPrm struct {
commonObjectPrm commonObjectPrm
objectAddressPrm objectAddressPrm
rawPrm rawPrm
mainOnly bool
}
// SetMainOnlyFlag sets flag to get only main fields of an object header in terms of FrostFS API.
func (x *HeadObjectPrm) SetMainOnlyFlag(v bool) {
x.mainOnly = v
} }
// HeadObjectRes groups the resulting values of HeadObject operation. // HeadObjectRes groups the resulting values of HeadObject operation.
@ -712,9 +659,7 @@ func SearchObjects(ctx context.Context, prm SearchObjectsPrm) (*SearchObjectsRes
for { for {
n, ok = rdr.Read(buf) n, ok = rdr.Read(buf)
for i := 0; i < n; i++ { list = append(list, buf[:n]...)
list = append(list, buf[i])
}
if !ok { if !ok {
break break
} }
@ -725,9 +670,8 @@ func SearchObjects(ctx context.Context, prm SearchObjectsPrm) (*SearchObjectsRes
return nil, fmt.Errorf("read object list: %w", err) return nil, fmt.Errorf("read object list: %w", err)
} }
sort.Slice(list, func(i, j int) bool { slices.SortFunc(list, func(a, b oid.ID) int {
lhs, rhs := list[i].EncodeToString(), list[j].EncodeToString() return strings.Compare(a.EncodeToString(), b.EncodeToString())
return strings.Compare(lhs, rhs) < 0
}) })
return &SearchObjectsRes{ return &SearchObjectsRes{
@ -892,3 +836,65 @@ func SyncContainerSettings(ctx context.Context, prm SyncContainerPrm) (*SyncCont
return new(SyncContainerRes), nil return new(SyncContainerRes), nil
} }
// PatchObjectPrm groups parameters of PatchObject operation.
type PatchObjectPrm struct {
commonObjectPrm
objectAddressPrm
NewAttributes []objectSDK.Attribute
ReplaceAttribute bool
PayloadPatches []PayloadPatch
}
type PayloadPatch struct {
Range objectSDK.Range
PayloadPath string
}
type PatchRes struct {
OID oid.ID
}
func Patch(ctx context.Context, prm PatchObjectPrm) (*PatchRes, error) {
patchPrm := client.PrmObjectPatch{
XHeaders: prm.xHeaders,
BearerToken: prm.bearerToken,
Session: prm.sessionToken,
Address: prm.objAddr,
}
slices.SortFunc(prm.PayloadPatches, func(a, b PayloadPatch) int {
return cmp.Compare(a.Range.GetOffset(), b.Range.GetOffset())
})
patcher, err := prm.cli.ObjectPatchInit(ctx, patchPrm)
if err != nil {
return nil, fmt.Errorf("init payload reading: %w", err)
}
if patcher.PatchAttributes(ctx, prm.NewAttributes, prm.ReplaceAttribute) {
for _, pp := range prm.PayloadPatches {
payloadFile, err := os.OpenFile(pp.PayloadPath, os.O_RDONLY, os.ModePerm)
if err != nil {
return nil, err
}
applied := patcher.PatchPayload(ctx, &pp.Range, payloadFile)
_ = payloadFile.Close()
if !applied {
break
}
}
}
res, err := patcher.Close(ctx)
if err != nil {
return nil, err
}
return &PatchRes{
OID: res.ObjectID(),
}, nil
}

View file

@ -58,6 +58,7 @@ func GetSDKClient(ctx context.Context, cmd *cobra.Command, key *ecdsa.PrivateKey
GRPCDialOptions: []grpc.DialOption{ GRPCDialOptions: []grpc.DialOption{
grpc.WithChainUnaryInterceptor(tracing.NewUnaryClientInteceptor()), grpc.WithChainUnaryInterceptor(tracing.NewUnaryClientInteceptor()),
grpc.WithChainStreamInterceptor(tracing.NewStreamClientInterceptor()), grpc.WithChainStreamInterceptor(tracing.NewStreamClientInterceptor()),
grpc.WithDefaultCallOptions(grpc.WaitForReady(true)),
}, },
} }
if timeout := viper.GetDuration(commonflags.Timeout); timeout > 0 { if timeout := viper.GetDuration(commonflags.Timeout); timeout > 0 {

View file

@ -2,7 +2,7 @@ package common
import ( import (
"context" "context"
"sort" "slices"
"strings" "strings"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
@ -45,15 +45,11 @@ func StartClientCommandSpan(cmd *cobra.Command) {
}) })
commonCmd.ExitOnErr(cmd, "init tracing: %w", err) commonCmd.ExitOnErr(cmd, "init tracing: %w", err)
var components sort.StringSlice var components []string
for c := cmd; c != nil; c = c.Parent() { for c := cmd; c != nil; c = c.Parent() {
components = append(components, c.Name()) components = append(components, c.Name())
} }
for i, j := 0, len(components)-1; i < j; { slices.Reverse(components)
components.Swap(i, j)
i++
j--
}
operation := strings.Join(components, ".") operation := strings.Join(components, ".")
ctx, span := tracing.StartSpanFromContext(cmd.Context(), operation) ctx, span := tracing.StartSpanFromContext(cmd.Context(), operation)

View file

@ -24,6 +24,8 @@ var testCmd = &cobra.Command{
} }
func Test_getOrGenerate(t *testing.T) { func Test_getOrGenerate(t *testing.T) {
t.Cleanup(viper.Reset)
dir := t.TempDir() dir := t.TempDir()
wallPath := filepath.Join(dir, "wallet.json") wallPath := filepath.Join(dir, "wallet.json")

View file

@ -31,7 +31,6 @@ const (
) )
const ( const (
defaultNamespace = ""
namespaceTarget = "namespace" namespaceTarget = "namespace"
containerTarget = "container" containerTarget = "container"
userTarget = "user" userTarget = "user"

View file

@ -139,7 +139,7 @@ It will be stored in sidechain when inner ring will accepts it.`,
}, },
} }
for i := 0; i < awaitTimeout; i++ { for range awaitTimeout {
time.Sleep(1 * time.Second) time.Sleep(1 * time.Second)
_, err := internalclient.GetContainer(cmd.Context(), getPrm) _, err := internalclient.GetContainer(cmd.Context(), getPrm)

View file

@ -110,7 +110,7 @@ Only owner of the container has a permission to remove container.`,
}, },
} }
for i := 0; i < awaitTimeout; i++ { for range awaitTimeout {
time.Sleep(1 * time.Second) time.Sleep(1 * time.Second)
_, err := internalclient.GetContainer(cmd.Context(), getPrm) _, err := internalclient.GetContainer(cmd.Context(), getPrm)

View file

@ -1,68 +0,0 @@
package container
import (
"os"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
)
var getExtendedACLCmd = &cobra.Command{
Use: "get-eacl",
Short: "Get extended ACL table of container",
Long: `Get extended ACL table of container`,
Run: func(cmd *cobra.Command, _ []string) {
id := parseContainerID(cmd)
pk := key.GetOrGenerate(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
eaclPrm := internalclient.EACLPrm{
Client: cli,
ClientParams: client.PrmContainerEACL{
ContainerID: &id,
},
}
res, err := internalclient.EACL(cmd.Context(), eaclPrm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
eaclTable := res.EACL()
if containerPathTo == "" {
cmd.Println("eACL: ")
common.PrettyPrintJSON(cmd, &eaclTable, "eACL")
return
}
var data []byte
if containerJSON {
data, err = eaclTable.MarshalJSON()
commonCmd.ExitOnErr(cmd, "can't encode to JSON: %w", err)
} else {
data, err = eaclTable.Marshal()
commonCmd.ExitOnErr(cmd, "can't encode to binary: %w", err)
}
cmd.Println("dumping data to file:", containerPathTo)
err = os.WriteFile(containerPathTo, data, 0o644)
commonCmd.ExitOnErr(cmd, "could not write eACL to file: %w", err)
},
}
func initContainerGetEACLCmd() {
commonflags.Init(getExtendedACLCmd)
flags := getExtendedACLCmd.Flags()
flags.StringVar(&containerID, commonflags.CIDFlag, "", commonflags.CIDFlagUsage)
flags.StringVar(&containerPathTo, "to", "", "Path to dump encoded container (default: binary encoded)")
flags.BoolVar(&containerJSON, commonflags.JSON, false, "Encode EACL table in json format")
}

View file

@ -1,9 +1,6 @@
package container package container
import ( import (
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client" internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
@ -70,7 +67,6 @@ var listContainersCmd = &cobra.Command{
continue continue
} }
cnrID := cnrID
prmGet.ClientParams.ContainerID = &cnrID prmGet.ClientParams.ContainerID = &cnrID
res, err := internalclient.GetContainer(cmd.Context(), prmGet) res, err := internalclient.GetContainer(cmd.Context(), prmGet)
if err != nil { if err != nil {
@ -85,12 +81,8 @@ var listContainersCmd = &cobra.Command{
cmd.Println(cnrID.String()) cmd.Println(cnrID.String())
if flagVarListPrintAttr { if flagVarListPrintAttr {
cnr.IterateAttributes(func(key, val string) { cnr.IterateUserAttributes(func(key, val string) {
if !strings.HasPrefix(key, container.SysAttributePrefix) && !strings.HasPrefix(key, container.SysAttributePrefixNeoFS) {
// FIXME(@cthulhu-rider): https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/issues/97
// Use dedicated method to skip system attributes.
cmd.Printf(" %s: %s\n", key, val) cmd.Printf(" %s: %s\n", key, val)
}
}) })
} }
} }

View file

@ -1,9 +1,6 @@
package container package container
import ( import (
"strings"
v2object "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client" internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
@ -67,14 +64,8 @@ var listContainerObjectsCmd = &cobra.Command{
resHead, err := internalclient.HeadObject(cmd.Context(), prmHead) resHead, err := internalclient.HeadObject(cmd.Context(), prmHead)
if err == nil { if err == nil {
attrs := resHead.Header().Attributes() for _, attr := range resHead.Header().UserAttributes() {
for i := range attrs { cmd.Printf(" %s: %s\n", attr.Key(), attr.Value())
attrKey := attrs[i].Key()
if !strings.HasPrefix(attrKey, v2object.SysAttributePrefix) && !strings.HasPrefix(attrKey, v2object.SysAttributePrefixNeoFS) {
// FIXME(@cthulhu-rider): https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/issues/97
// Use dedicated method to skip system attributes.
cmd.Printf(" %s: %s\n", attrKey, attrs[i].Value())
}
} }
} else { } else {
cmd.Printf(" failed to read attributes: %v\n", err) cmd.Printf(" failed to read attributes: %v\n", err)

View file

@ -25,8 +25,6 @@ func init() {
deleteContainerCmd, deleteContainerCmd,
listContainerObjectsCmd, listContainerObjectsCmd,
getContainerInfoCmd, getContainerInfoCmd,
getExtendedACLCmd,
setExtendedACLCmd,
containerNodesCmd, containerNodesCmd,
policyPlaygroundCmd, policyPlaygroundCmd,
} }
@ -38,8 +36,6 @@ func init() {
initContainerDeleteCmd() initContainerDeleteCmd()
initContainerListObjectsCmd() initContainerListObjectsCmd()
initContainerInfoCmd() initContainerInfoCmd()
initContainerGetEACLCmd()
initContainerSetEACLCmd()
initContainerNodesCmd() initContainerNodesCmd()
initContainerPolicyPlaygroundCmd() initContainerPolicyPlaygroundCmd()
@ -53,7 +49,6 @@ func init() {
}{ }{
{createContainerCmd, "PUT"}, {createContainerCmd, "PUT"},
{deleteContainerCmd, "DELETE"}, {deleteContainerCmd, "DELETE"},
{setExtendedACLCmd, "SETEACL"},
} { } {
commonflags.InitSession(el.cmd, "container "+el.verb) commonflags.InitSession(el.cmd, "container "+el.verb)
} }

View file

@ -1,108 +0,0 @@
package container
import (
"bytes"
"errors"
"time"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
)
var flagVarsSetEACL struct {
noPreCheck bool
srcPath string
}
var setExtendedACLCmd = &cobra.Command{
Use: "set-eacl",
Short: "Set new extended ACL table for container",
Long: `Set new extended ACL table for container.
Container ID in EACL table will be substituted with ID from the CLI.`,
Run: func(cmd *cobra.Command, _ []string) {
id := parseContainerID(cmd)
eaclTable := common.ReadEACL(cmd, flagVarsSetEACL.srcPath)
tok := getSession(cmd)
eaclTable.SetCID(id)
pk := key.GetOrGenerate(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
if !flagVarsSetEACL.noPreCheck {
cmd.Println("Checking the ability to modify access rights in the container...")
extendable, err := internalclient.IsACLExtendable(cmd.Context(), cli, id)
commonCmd.ExitOnErr(cmd, "Extensibility check failure: %w", err)
if !extendable {
commonCmd.ExitOnErr(cmd, "", errors.New("container ACL is immutable"))
}
cmd.Println("ACL extension is enabled in the container, continue processing.")
}
setEACLPrm := internalclient.SetEACLPrm{
Client: cli,
ClientParams: client.PrmContainerSetEACL{
Table: eaclTable,
Session: tok,
},
}
_, err := internalclient.SetEACL(cmd.Context(), setEACLPrm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
if containerAwait {
exp, err := eaclTable.Marshal()
commonCmd.ExitOnErr(cmd, "broken EACL table: %w", err)
cmd.Println("awaiting...")
getEACLPrm := internalclient.EACLPrm{
Client: cli,
ClientParams: client.PrmContainerEACL{
ContainerID: &id,
},
}
for i := 0; i < awaitTimeout; i++ {
time.Sleep(1 * time.Second)
res, err := internalclient.EACL(cmd.Context(), getEACLPrm)
if err == nil {
// compare binary values because EACL could have been set already
table := res.EACL()
got, err := table.Marshal()
if err != nil {
continue
}
if bytes.Equal(exp, got) {
cmd.Println("EACL has been persisted on sidechain")
return
}
}
}
commonCmd.ExitOnErr(cmd, "", errSetEACLTimeout)
}
},
}
func initContainerSetEACLCmd() {
commonflags.Init(setExtendedACLCmd)
flags := setExtendedACLCmd.Flags()
flags.StringVar(&containerID, commonflags.CIDFlag, "", commonflags.CIDFlagUsage)
flags.StringVar(&flagVarsSetEACL.srcPath, "table", "", "path to file with JSON or binary encoded EACL table")
flags.BoolVar(&containerAwait, "await", false, "block execution until EACL is persisted")
flags.BoolVar(&flagVarsSetEACL.noPreCheck, "no-precheck", false, "do not pre-check the extensibility of the container ACL")
}

View file

@ -20,7 +20,6 @@ const (
var ( var (
errCreateTimeout = errors.New("timeout: container has not been persisted on sidechain") errCreateTimeout = errors.New("timeout: container has not been persisted on sidechain")
errDeleteTimeout = errors.New("timeout: container has not been removed from sidechain") errDeleteTimeout = errors.New("timeout: container has not been removed from sidechain")
errSetEACLTimeout = errors.New("timeout: EACL has not been persisted on sidechain")
) )
func parseContainerID(cmd *cobra.Command) cid.ID { func parseContainerID(cmd *cobra.Command) cid.ID {

View file

@ -20,6 +20,10 @@ const (
awaitFlag = "await" awaitFlag = "await"
noProgressFlag = "no-progress" noProgressFlag = "no-progress"
scopeFlag = "scope" scopeFlag = "scope"
repOneOnlyFlag = "rep-one-only"
containerWorkerCountFlag = "container-worker-count"
objectWorkerCountFlag = "object-worker-count"
scopeAll = "all" scopeAll = "all"
scopeObjects = "objects" scopeObjects = "objects"
@ -64,12 +68,18 @@ func startEvacuateShard(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd) pk := key.Get(cmd)
ignoreErrors, _ := cmd.Flags().GetBool(ignoreErrorsFlag) ignoreErrors, _ := cmd.Flags().GetBool(ignoreErrorsFlag)
containerWorkerCount, _ := cmd.Flags().GetUint32(containerWorkerCountFlag)
objectWorkerCount, _ := cmd.Flags().GetUint32(objectWorkerCountFlag)
repOneOnly, _ := cmd.Flags().GetBool(repOneOnlyFlag)
req := &control.StartShardEvacuationRequest{ req := &control.StartShardEvacuationRequest{
Body: &control.StartShardEvacuationRequest_Body{ Body: &control.StartShardEvacuationRequest_Body{
Shard_ID: getShardIDList(cmd), Shard_ID: getShardIDList(cmd),
IgnoreErrors: ignoreErrors, IgnoreErrors: ignoreErrors,
Scope: getEvacuationScope(cmd), Scope: getEvacuationScope(cmd),
ContainerWorkerCount: containerWorkerCount,
ObjectWorkerCount: objectWorkerCount,
RepOneOnly: repOneOnly,
}, },
} }
@ -371,6 +381,9 @@ func initControlStartEvacuationShardCmd() {
flags.String(scopeFlag, scopeAll, fmt.Sprintf("Evacuation scope; possible values: %s, %s, %s", scopeTrees, scopeObjects, scopeAll)) flags.String(scopeFlag, scopeAll, fmt.Sprintf("Evacuation scope; possible values: %s, %s, %s", scopeTrees, scopeObjects, scopeAll))
flags.Bool(awaitFlag, false, "Block execution until evacuation is completed") flags.Bool(awaitFlag, false, "Block execution until evacuation is completed")
flags.Bool(noProgressFlag, false, fmt.Sprintf("Print progress if %s provided", awaitFlag)) flags.Bool(noProgressFlag, false, fmt.Sprintf("Print progress if %s provided", awaitFlag))
flags.Uint32(containerWorkerCountFlag, 0, "Count of concurrent container evacuation workers")
flags.Uint32(objectWorkerCountFlag, 0, "Count of concurrent object evacuation workers")
flags.Bool(repOneOnlyFlag, false, "Evacuate objects only from containers with policy 'REP 1 ...'")
startEvacuationShardCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag) startEvacuationShardCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag)
} }

View file

@ -0,0 +1,88 @@
package control
import (
"fmt"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
"github.com/mr-tron/base58"
"github.com/spf13/cobra"
)
const (
fillPercentFlag = "fill_percent"
)
var shardsRebuildCmd = &cobra.Command{
Use: "rebuild",
Short: "Rebuild shards",
Long: "Rebuild reclaims storage occupied by dead objects and adjusts the storage structure according to the configuration (for blobovnicza only now)",
Run: shardsRebuild,
}
func shardsRebuild(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd)
req := &control.StartShardRebuildRequest{
Body: &control.StartShardRebuildRequest_Body{
Shard_ID: getShardIDList(cmd),
TargetFillPercent: getFillPercentValue(cmd),
ConcurrencyLimit: getConcurrencyValue(cmd),
},
}
signRequest(cmd, pk, req)
cli := getClient(cmd, pk)
var resp *control.StartShardRebuildResponse
var err error
err = cli.ExecRaw(func(client *rawclient.Client) error {
resp, err = control.StartShardRebuild(client, req)
return err
})
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
verifyResponse(cmd, resp.GetSignature(), resp.GetBody())
var success, failed uint
for _, res := range resp.GetBody().GetResults() {
if res.GetSuccess() {
success++
cmd.Printf("Shard %s: OK\n", base58.Encode(res.GetShard_ID()))
} else {
failed++
cmd.Printf("Shard %s: failed with error %q\n", base58.Encode(res.GetShard_ID()), res.GetError())
}
}
cmd.Printf("Total: %d success, %d failed\n", success, failed)
}
func getFillPercentValue(cmd *cobra.Command) uint32 {
v, _ := cmd.Flags().GetUint32(fillPercentFlag)
if v <= 0 || v > 100 {
commonCmd.ExitOnErr(cmd, "invalid fill_percent value", fmt.Errorf("fill_percent value must be (0, 100], current value: %d", v))
}
return v
}
func getConcurrencyValue(cmd *cobra.Command) uint32 {
v, _ := cmd.Flags().GetUint32(concurrencyFlag)
if v <= 0 || v > 10000 {
commonCmd.ExitOnErr(cmd, "invalid concurrency value", fmt.Errorf("concurrency value must be (0, 10 000], current value: %d", v))
}
return v
}
func initControlShardRebuildCmd() {
initControlFlags(shardsRebuildCmd)
flags := shardsRebuildCmd.Flags()
flags.StringSlice(shardIDFlag, nil, "List of shard IDs in base58 encoding")
flags.Bool(shardAllFlag, false, "Process all shards")
flags.Uint32(fillPercentFlag, 80, "Target fill percent to reclaim space")
flags.Uint32(concurrencyFlag, 20, "Maximum count of concurrently rebuilding files")
setShardModeCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag)
}

View file

@ -84,7 +84,7 @@ func setNetmapStatus(cmd *cobra.Command, _ []string) {
body.SetStatus(control.NetmapStatus_MAINTENANCE) body.SetStatus(control.NetmapStatus_MAINTENANCE)
if force { if force {
body.SetForceMaintenance() body.SetForceMaintenance(true)
common.PrintVerbose(cmd, "Local maintenance will be forced.") common.PrintVerbose(cmd, "Local maintenance will be forced.")
} }
targetStatus = control.NetmapStatus_MAINTENANCE targetStatus = control.NetmapStatus_MAINTENANCE

View file

@ -19,6 +19,7 @@ func initControlShardsCmd() {
shardsCmd.AddCommand(doctorCmd) shardsCmd.AddCommand(doctorCmd)
shardsCmd.AddCommand(writecacheShardCmd) shardsCmd.AddCommand(writecacheShardCmd)
shardsCmd.AddCommand(shardsDetachCmd) shardsCmd.AddCommand(shardsDetachCmd)
shardsCmd.AddCommand(shardsRebuildCmd)
initControlShardsListCmd() initControlShardsListCmd()
initControlSetShardModeCmd() initControlSetShardModeCmd()
@ -28,4 +29,5 @@ func initControlShardsCmd() {
initControlDoctorCmd() initControlDoctorCmd()
initControlShardsWritecacheCmd() initControlShardsWritecacheCmd()
initControlShardsDetachCmd() initControlShardsDetachCmd()
initControlShardRebuildCmd()
} }

View file

@ -61,7 +61,7 @@ func listShards(cmd *cobra.Command, _ []string) {
} }
} }
func prettyPrintShardsJSON(cmd *cobra.Command, ii []*control.ShardInfo) { func prettyPrintShardsJSON(cmd *cobra.Command, ii []control.ShardInfo) {
out := make([]map[string]any, 0, len(ii)) out := make([]map[string]any, 0, len(ii))
for _, i := range ii { for _, i := range ii {
out = append(out, map[string]any{ out = append(out, map[string]any{
@ -72,6 +72,7 @@ func prettyPrintShardsJSON(cmd *cobra.Command, ii []*control.ShardInfo) {
"writecache": i.GetWritecachePath(), "writecache": i.GetWritecachePath(),
"pilorama": i.GetPiloramaPath(), "pilorama": i.GetPiloramaPath(),
"error_count": i.GetErrorCount(), "error_count": i.GetErrorCount(),
"evacuation_in_progress": i.GetEvacuationInProgress(),
}) })
} }
@ -83,7 +84,7 @@ func prettyPrintShardsJSON(cmd *cobra.Command, ii []*control.ShardInfo) {
cmd.Print(buf.String()) // pretty printer emits newline, so no need for Println cmd.Print(buf.String()) // pretty printer emits newline, so no need for Println
} }
func prettyPrintShards(cmd *cobra.Command, ii []*control.ShardInfo) { func prettyPrintShards(cmd *cobra.Command, ii []control.ShardInfo) {
for _, i := range ii { for _, i := range ii {
pathPrinter := func(name, path string) string { pathPrinter := func(name, path string) string {
if path == "" { if path == "" {
@ -105,7 +106,8 @@ func prettyPrintShards(cmd *cobra.Command, ii []*control.ShardInfo) {
sb.String()+ sb.String()+
pathPrinter("Write-cache", i.GetWritecachePath())+ pathPrinter("Write-cache", i.GetWritecachePath())+
pathPrinter("Pilorama", i.GetPiloramaPath())+ pathPrinter("Pilorama", i.GetPiloramaPath())+
fmt.Sprintf("Error count: %d\n", i.GetErrorCount()), fmt.Sprintf("Error count: %d\n", i.GetErrorCount())+
fmt.Sprintf("Evacuation in progress: %t\n", i.GetEvacuationInProgress()),
base58.Encode(i.GetShard_ID()), base58.Encode(i.GetShard_ID()),
shardModeToString(i.GetMode()), shardModeToString(i.GetMode()),
) )
@ -121,7 +123,7 @@ func shardModeToString(m control.ShardMode) string {
return "unknown" return "unknown"
} }
func sortShardsByID(ii []*control.ShardInfo) { func sortShardsByID(ii []control.ShardInfo) {
sort.Slice(ii, func(i, j int) bool { sort.Slice(ii, func(i, j int) bool {
return bytes.Compare(ii[i].GetShard_ID(), ii[j].GetShard_ID()) < 0 return bytes.Compare(ii[i].GetShard_ID(), ii[j].GetShard_ID()) < 0
}) })

View file

@ -117,10 +117,10 @@ func setShardMode(cmd *cobra.Command, _ []string) {
req.SetBody(body) req.SetBody(body)
body.SetMode(mode) body.SetMode(mode)
body.SetShardIDList(getShardIDList(cmd)) body.SetShard_ID(getShardIDList(cmd))
reset, _ := cmd.Flags().GetBool(shardClearErrorsFlag) reset, _ := cmd.Flags().GetBool(shardClearErrorsFlag)
body.ClearErrorCounter(reset) body.SetResetErrorCounter(reset)
signRequest(cmd, pk, req) signRequest(cmd, pk, req)

View file

@ -44,7 +44,7 @@ func verifyResponse(cmd *cobra.Command,
GetSign() []byte GetSign() []byte
}, },
body interface { body interface {
StableMarshal([]byte) []byte MarshalProtobuf([]byte) []byte
}, },
) { ) {
if sigControl == nil { if sigControl == nil {
@ -60,7 +60,7 @@ func verifyResponse(cmd *cobra.Command,
var sig frostfscrypto.Signature var sig frostfscrypto.Signature
commonCmd.ExitOnErr(cmd, "can't read signature: %w", sig.ReadFromV2(sigV2)) commonCmd.ExitOnErr(cmd, "can't read signature: %w", sig.ReadFromV2(sigV2))
if !sig.Verify(body.StableMarshal(nil)) { if !sig.Verify(body.MarshalProtobuf(nil)) {
commonCmd.ExitOnErr(cmd, "", errors.New("invalid response signature")) commonCmd.ExitOnErr(cmd, "", errors.New("invalid response signature"))
} }
} }

View file

@ -9,6 +9,12 @@ import (
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
const (
asyncFlag = "async"
restoreModeFlag = "restore-mode"
shrinkFlag = "shrink"
)
var writecacheShardCmd = &cobra.Command{ var writecacheShardCmd = &cobra.Command{
Use: "writecache", Use: "writecache",
Short: "Operations with storage node's write-cache", Short: "Operations with storage node's write-cache",
@ -26,10 +32,16 @@ func sealWritecache(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd) pk := key.Get(cmd)
ignoreErrors, _ := cmd.Flags().GetBool(ignoreErrorsFlag) ignoreErrors, _ := cmd.Flags().GetBool(ignoreErrorsFlag)
async, _ := cmd.Flags().GetBool(asyncFlag)
restoreMode, _ := cmd.Flags().GetBool(restoreModeFlag)
shrink, _ := cmd.Flags().GetBool(shrinkFlag)
req := &control.SealWriteCacheRequest{Body: &control.SealWriteCacheRequest_Body{ req := &control.SealWriteCacheRequest{Body: &control.SealWriteCacheRequest_Body{
Shard_ID: getShardIDList(cmd), Shard_ID: getShardIDList(cmd),
IgnoreErrors: ignoreErrors, IgnoreErrors: ignoreErrors,
Async: async,
RestoreMode: restoreMode,
Shrink: shrink,
}} }}
signRequest(cmd, pk, req) signRequest(cmd, pk, req)
@ -68,6 +80,9 @@ func initControlShardsWritecacheCmd() {
ff.StringSlice(shardIDFlag, nil, "List of shard IDs in base58 encoding") ff.StringSlice(shardIDFlag, nil, "List of shard IDs in base58 encoding")
ff.Bool(shardAllFlag, false, "Process all shards") ff.Bool(shardAllFlag, false, "Process all shards")
ff.Bool(ignoreErrorsFlag, true, "Skip invalid/unreadable objects") ff.Bool(ignoreErrorsFlag, true, "Skip invalid/unreadable objects")
ff.Bool(asyncFlag, false, "Run operation in background")
ff.Bool(restoreModeFlag, false, "Restore writecache's mode after sealing")
ff.Bool(shrinkFlag, false, "Shrink writecache's internal storage")
sealWritecacheShardCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag) sealWritecacheShardCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag)
} }

View file

@ -49,14 +49,14 @@ func prettyPrintNodeInfo(cmd *cobra.Command, i netmap.NodeInfo) {
cmd.Println("key:", hex.EncodeToString(i.PublicKey())) cmd.Println("key:", hex.EncodeToString(i.PublicKey()))
var stateWord string var stateWord string
switch { switch i.Status() {
default: default:
stateWord = "<undefined>" stateWord = "<undefined>"
case i.IsOnline(): case netmap.Online:
stateWord = "online" stateWord = "online"
case i.IsOffline(): case netmap.Offline:
stateWord = "offline" stateWord = "offline"
case i.IsMaintenance(): case netmap.Maintenance:
stateWord = "maintenance" stateWord = "maintenance"
} }

View file

@ -38,7 +38,6 @@ func initObjectHeadCmd() {
_ = objectHeadCmd.MarkFlagRequired(commonflags.OIDFlag) _ = objectHeadCmd.MarkFlagRequired(commonflags.OIDFlag)
flags.String(fileFlag, "", "File to write header to. Default: stdout.") flags.String(fileFlag, "", "File to write header to. Default: stdout.")
flags.Bool("main-only", false, "Return only main fields")
flags.Bool(commonflags.JSON, false, "Marshal output in JSON") flags.Bool(commonflags.JSON, false, "Marshal output in JSON")
flags.Bool("proto", false, "Marshal output in Protobuf") flags.Bool("proto", false, "Marshal output in Protobuf")
flags.Bool(rawFlag, false, rawFlagDesc) flags.Bool(rawFlag, false, rawFlagDesc)
@ -49,7 +48,6 @@ func getObjectHeader(cmd *cobra.Command, _ []string) {
var obj oid.ID var obj oid.ID
objAddr := readObjectAddress(cmd, &cnr, &obj) objAddr := readObjectAddress(cmd, &cnr, &obj)
mainOnly, _ := cmd.Flags().GetBool("main-only")
pk := key.GetOrGenerate(cmd) pk := key.GetOrGenerate(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC) cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
@ -62,7 +60,6 @@ func getObjectHeader(cmd *cobra.Command, _ []string) {
raw, _ := cmd.Flags().GetBool(rawFlag) raw, _ := cmd.Flags().GetBool(rawFlag)
prm.SetRawFlag(raw) prm.SetRawFlag(raw)
prm.SetAddress(objAddr) prm.SetAddress(objAddr)
prm.SetMainOnlyFlag(mainOnly)
res, err := internalclient.HeadObject(cmd.Context(), prm) res, err := internalclient.HeadObject(cmd.Context(), prm)
if err != nil { if err != nil {

View file

@ -31,6 +31,7 @@ import (
const ( const (
verifyPresenceAllFlag = "verify-presence-all" verifyPresenceAllFlag = "verify-presence-all"
preferInternalAddressesFlag = "prefer-internal-addresses"
) )
var ( var (
@ -97,6 +98,7 @@ func initObjectNodesCmd() {
flags.Bool(verifyPresenceAllFlag, false, "Verify the actual presence of the object on all netmap nodes.") flags.Bool(verifyPresenceAllFlag, false, "Verify the actual presence of the object on all netmap nodes.")
flags.Bool(commonflags.JSON, false, "Print information about the object placement as json.") flags.Bool(commonflags.JSON, false, "Print information about the object placement as json.")
flags.Bool(preferInternalAddressesFlag, false, "Use internal addresses first to get object info.")
} }
func objectNodes(cmd *cobra.Command, _ []string) { func objectNodes(cmd *cobra.Command, _ []string) {
@ -170,7 +172,7 @@ func getComplexObjectParts(cmd *cobra.Command, cnrID cid.ID, objID oid.ID, cli *
func getCompexObjectMembers(cmd *cobra.Command, cnrID cid.ID, objID oid.ID, cli *client.Client, prmHead internalclient.HeadObjectPrm, errSplitInfo *objectSDK.SplitInfoError) []oid.ID { func getCompexObjectMembers(cmd *cobra.Command, cnrID cid.ID, objID oid.ID, cli *client.Client, prmHead internalclient.HeadObjectPrm, errSplitInfo *objectSDK.SplitInfoError) []oid.ID {
splitInfo := errSplitInfo.SplitInfo() splitInfo := errSplitInfo.SplitInfo()
if members, ok := tryGetSplitMembersByLinkingObject(cmd, splitInfo, prmHead, cnrID, false); ok { if members, ok := tryGetSplitMembersByLinkingObject(cmd, splitInfo, prmHead, cnrID); ok {
return members return members
} }
@ -183,6 +185,7 @@ func getCompexObjectMembers(cmd *cobra.Command, cnrID cid.ID, objID oid.ID, cli
func flattenComplexMembersIfECContainer(cmd *cobra.Command, cnrID cid.ID, members []oid.ID, prmHead internalclient.HeadObjectPrm) []phyObject { func flattenComplexMembersIfECContainer(cmd *cobra.Command, cnrID cid.ID, members []oid.ID, prmHead internalclient.HeadObjectPrm) []phyObject {
result := make([]phyObject, 0, len(members)) result := make([]phyObject, 0, len(members))
var hasNonEC, hasEC bool
var resultGuard sync.Mutex var resultGuard sync.Mutex
if len(members) == 0 { if len(members) == 0 {
@ -191,31 +194,8 @@ func flattenComplexMembersIfECContainer(cmd *cobra.Command, cnrID cid.ID, member
prmHead.SetRawFlag(true) // to get an error instead of whole object prmHead.SetRawFlag(true) // to get an error instead of whole object
first := members[0]
var addrObj oid.Address
addrObj.SetContainer(cnrID)
addrObj.SetObject(first)
prmHead.SetAddress(addrObj)
_, err := internalclient.HeadObject(cmd.Context(), prmHead)
var ecInfoError *objectSDK.ECInfoError
if errors.As(err, &ecInfoError) {
chunks := getECObjectChunks(cmd, cnrID, first, ecInfoError)
result = append(result, chunks...)
} else if err == nil { // not EC object, so all members must be phy objects
for _, member := range members {
result = append(result, phyObject{
containerID: cnrID,
objectID: member,
})
}
return result
} else {
commonCmd.ExitOnErr(cmd, "failed to flatten parts of complex object: %w", err)
}
eg, egCtx := errgroup.WithContext(cmd.Context()) eg, egCtx := errgroup.WithContext(cmd.Context())
for idx := 1; idx < len(members); idx++ { for idx := range members {
partObjID := members[idx] partObjID := members[idx]
eg.Go(func() error { eg.Go(func() error {
@ -225,24 +205,44 @@ func flattenComplexMembersIfECContainer(cmd *cobra.Command, cnrID cid.ID, member
partAddr.SetObject(partObjID) partAddr.SetObject(partObjID)
partHeadPrm.SetAddress(partAddr) partHeadPrm.SetAddress(partAddr)
_, err := internalclient.HeadObject(egCtx, partHeadPrm) obj, err := internalclient.HeadObject(egCtx, partHeadPrm)
if err != nil {
var ecInfoError *objectSDK.ECInfoError var ecInfoError *objectSDK.ECInfoError
if errors.As(err, &ecInfoError) { if errors.As(err, &ecInfoError) {
chunks := getECObjectChunks(cmd, cnrID, partObjID, ecInfoError) resultGuard.Lock()
defer resultGuard.Unlock()
result = append(result, getECObjectChunks(cmd, cnrID, partObjID, ecInfoError)...)
hasEC = true
return nil
}
return err
}
if obj.Header().Type() != objectSDK.TypeRegular {
commonCmd.ExitOnErr(cmd, "failed to flatten parts of complex object: %w", fmt.Errorf("object '%s' with type '%s' is not supported as part of complex object", partAddr, obj.Header().Type()))
}
if len(obj.Header().Children()) > 0 {
// linking object is not data object, so skip it
return nil
}
resultGuard.Lock() resultGuard.Lock()
defer resultGuard.Unlock() defer resultGuard.Unlock()
result = append(result, chunks...) result = append(result, phyObject{
containerID: cnrID,
objectID: partObjID,
})
hasNonEC = true
return nil return nil
} else if err == nil {
return errMalformedComplexObject
}
return err
}) })
} }
commonCmd.ExitOnErr(cmd, "failed to flatten parts of complex object: %w", eg.Wait()) commonCmd.ExitOnErr(cmd, "failed to flatten parts of complex object: %w", eg.Wait())
if hasEC && hasNonEC {
commonCmd.ExitOnErr(cmd, "failed to flatten parts of complex object: %w", errMalformedComplexObject)
}
return result return result
} }
@ -393,8 +393,6 @@ func getActualPlacement(cmd *cobra.Command, netmap *netmapSDK.NetMap, pk *ecdsa.
eg, egCtx := errgroup.WithContext(cmd.Context()) eg, egCtx := errgroup.WithContext(cmd.Context())
for _, cand := range candidates { for _, cand := range candidates {
cand := cand
eg.Go(func() error { eg.Go(func() error {
cli, err := createClient(egCtx, cmd, cand, pk) cli, err := createClient(egCtx, cmd, cand, pk)
if err != nil { if err != nil {
@ -405,7 +403,6 @@ func getActualPlacement(cmd *cobra.Command, netmap *netmapSDK.NetMap, pk *ecdsa.
} }
for _, object := range objects { for _, object := range objects {
object := object
eg.Go(func() error { eg.Go(func() error {
stored, err := isObjectStoredOnNode(egCtx, cmd, object.containerID, object.objectID, cli, pk) stored, err := isObjectStoredOnNode(egCtx, cmd, object.containerID, object.objectID, cli, pk)
resultMtx.Lock() resultMtx.Lock()
@ -449,11 +446,20 @@ func getNodesToCheckObjectExistance(cmd *cobra.Command, netmap *netmapSDK.NetMap
func createClient(ctx context.Context, cmd *cobra.Command, candidate netmapSDK.NodeInfo, pk *ecdsa.PrivateKey) (*client.Client, error) { func createClient(ctx context.Context, cmd *cobra.Command, candidate netmapSDK.NodeInfo, pk *ecdsa.PrivateKey) (*client.Client, error) {
var cli *client.Client var cli *client.Client
var addresses []string var addresses []string
if preferInternal, _ := cmd.Flags().GetBool(preferInternalAddressesFlag); preferInternal {
candidate.IterateNetworkEndpoints(func(s string) bool { candidate.IterateNetworkEndpoints(func(s string) bool {
addresses = append(addresses, s) addresses = append(addresses, s)
return false return false
}) })
addresses = append(addresses, candidate.ExternalAddresses()...) addresses = append(addresses, candidate.ExternalAddresses()...)
} else {
addresses = append(addresses, candidate.ExternalAddresses()...)
candidate.IterateNetworkEndpoints(func(s string) bool {
addresses = append(addresses, s)
return false
})
}
var lastErr error var lastErr error
for _, address := range addresses { for _, address := range addresses {
var networkAddr network.Address var networkAddr network.Address

View file

@ -0,0 +1,151 @@
package object
import (
"fmt"
"strconv"
"strings"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/spf13/cobra"
)
const (
newAttrsFlagName = "new-attrs"
replaceAttrsFlagName = "replace-attrs"
rangeFlagName = "range"
payloadFlagName = "payload"
)
var objectPatchCmd = &cobra.Command{
Use: "patch",
Run: patch,
Short: "Patch FrostFS object",
Long: "Patch FrostFS object. Each range passed to the command requires to pass a corresponding patch payload.",
Example: `
frostfs-cli -c config.yml -r 127.0.0.1:8080 object patch --cid <CID> --oid <OID> --new-attrs 'key1=val1,key2=val2' --replace-attrs
frostfs-cli -c config.yml -r 127.0.0.1:8080 object patch --cid <CID> --oid <OID> --range offX:lnX --payload /path/to/payloadX --range offY:lnY --payload /path/to/payloadY
frostfs-cli -c config.yml -r 127.0.0.1:8080 object patch --cid <CID> --oid <OID> --new-attrs 'key1=val1,key2=val2' --replace-attrs --range offX:lnX --payload /path/to/payload
`,
}
func initObjectPatchCmd() {
commonflags.Init(objectPatchCmd)
initFlagSession(objectPatchCmd, "PATCH")
flags := objectPatchCmd.Flags()
flags.String(commonflags.CIDFlag, "", commonflags.CIDFlagUsage)
_ = objectRangeCmd.MarkFlagRequired(commonflags.CIDFlag)
flags.String(commonflags.OIDFlag, "", commonflags.OIDFlagUsage)
_ = objectRangeCmd.MarkFlagRequired(commonflags.OIDFlag)
flags.String(newAttrsFlagName, "", "New object attributes in form of Key1=Value1,Key2=Value2")
flags.Bool(replaceAttrsFlagName, false, "Replace object attributes by new ones.")
flags.StringSlice(rangeFlagName, []string{}, "Range to which patch payload is applied. Format: offset:length")
flags.StringSlice(payloadFlagName, []string{}, "Path to file with patch payload.")
}
func patch(cmd *cobra.Command, _ []string) {
var cnr cid.ID
var obj oid.ID
objAddr := readObjectAddress(cmd, &cnr, &obj)
ranges, err := getRangeSlice(cmd)
commonCmd.ExitOnErr(cmd, "", err)
payloads := patchPayloadPaths(cmd)
if len(ranges) != len(payloads) {
commonCmd.ExitOnErr(cmd, "", fmt.Errorf("the number of ranges and payloads are not equal: ranges = %d, payloads = %d", len(ranges), len(payloads)))
}
newAttrs, err := parseNewObjectAttrs(cmd)
commonCmd.ExitOnErr(cmd, "can't parse new object attributes: %w", err)
replaceAttrs, _ := cmd.Flags().GetBool(replaceAttrsFlagName)
pk := key.GetOrGenerate(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
var prm internalclient.PatchObjectPrm
prm.SetClient(cli)
Prepare(cmd, &prm)
ReadOrOpenSession(cmd, &prm, pk, cnr, nil)
prm.SetAddress(objAddr)
prm.NewAttributes = newAttrs
prm.ReplaceAttribute = replaceAttrs
for i := range ranges {
prm.PayloadPatches = append(prm.PayloadPatches, internalclient.PayloadPatch{
Range: ranges[i],
PayloadPath: payloads[i],
})
}
res, err := internalclient.Patch(cmd.Context(), prm)
if err != nil {
commonCmd.ExitOnErr(cmd, "can't patch the object: %w", err)
}
cmd.Println("Patched object ID: ", res.OID.EncodeToString())
}
func parseNewObjectAttrs(cmd *cobra.Command) ([]objectSDK.Attribute, error) {
var rawAttrs []string
raw := cmd.Flag(newAttrsFlagName).Value.String()
if len(raw) != 0 {
rawAttrs = strings.Split(raw, ",")
}
attrs := make([]objectSDK.Attribute, len(rawAttrs), len(rawAttrs)+2) // name + timestamp attributes
for i := range rawAttrs {
k, v, found := strings.Cut(rawAttrs[i], "=")
if !found {
return nil, fmt.Errorf("invalid attribute format: %s", rawAttrs[i])
}
attrs[i].SetKey(k)
attrs[i].SetValue(v)
}
return attrs, nil
}
func getRangeSlice(cmd *cobra.Command) ([]objectSDK.Range, error) {
v, _ := cmd.Flags().GetStringSlice(rangeFlagName)
if len(v) == 0 {
return []objectSDK.Range{}, nil
}
rs := make([]objectSDK.Range, len(v))
for i := range v {
before, after, found := strings.Cut(v[i], rangeSep)
if !found {
return nil, fmt.Errorf("invalid range specifier: %s", v[i])
}
offset, err := strconv.ParseUint(before, 10, 64)
if err != nil {
return nil, fmt.Errorf("invalid '%s' range offset specifier: %w", v[i], err)
}
length, err := strconv.ParseUint(after, 10, 64)
if err != nil {
return nil, fmt.Errorf("invalid '%s' range length specifier: %w", v[i], err)
}
rs[i].SetOffset(offset)
rs[i].SetLength(length)
}
return rs, nil
}
func patchPayloadPaths(cmd *cobra.Command) []string {
v, _ := cmd.Flags().GetStringSlice(payloadFlagName)
return v
}

View file

@ -29,6 +29,7 @@ func init() {
objectRangeCmd, objectRangeCmd,
objectLockCmd, objectLockCmd,
objectNodesCmd, objectNodesCmd,
objectPatchCmd,
} }
Cmd.AddCommand(objectChildCommands...) Cmd.AddCommand(objectChildCommands...)
@ -39,6 +40,7 @@ func init() {
} }
initObjectPutCmd() initObjectPutCmd()
initObjectPatchCmd()
initObjectDeleteCmd() initObjectDeleteCmd()
initObjectGetCmd() initObjectGetCmd()
initObjectSearchCmd() initObjectSearchCmd()

View file

@ -306,6 +306,8 @@ func finalizeSession(cmd *cobra.Command, dst SessionPrm, tok *session.Object, ke
case *internal.PutObjectPrm: case *internal.PutObjectPrm:
common.PrintVerbose(cmd, "Binding session to object PUT...") common.PrintVerbose(cmd, "Binding session to object PUT...")
tok.ForVerb(session.VerbObjectPut) tok.ForVerb(session.VerbObjectPut)
case *internal.PatchObjectPrm:
tok.ForVerb(session.VerbObjectPatch)
case *internal.DeleteObjectPrm: case *internal.DeleteObjectPrm:
common.PrintVerbose(cmd, "Binding session to object DELETE...") common.PrintVerbose(cmd, "Binding session to object DELETE...")
tok.ForVerb(session.VerbObjectDelete) tok.ForVerb(session.VerbObjectDelete)
@ -372,7 +374,7 @@ func collectObjectRelatives(cmd *cobra.Command, cli *client.Client, cnr cid.ID,
common.PrintVerbose(cmd, "Split information received - object is virtual.") common.PrintVerbose(cmd, "Split information received - object is virtual.")
splitInfo := errSplit.SplitInfo() splitInfo := errSplit.SplitInfo()
if members, ok := tryGetSplitMembersByLinkingObject(cmd, splitInfo, prmHead, cnr, true); ok { if members, ok := tryGetSplitMembersByLinkingObject(cmd, splitInfo, prmHead, cnr); ok {
return members return members
} }
@ -388,7 +390,7 @@ func collectObjectRelatives(cmd *cobra.Command, cli *client.Client, cnr cid.ID,
return nil return nil
} }
func tryGetSplitMembersByLinkingObject(cmd *cobra.Command, splitInfo *objectSDK.SplitInfo, prmHead internal.HeadObjectPrm, cnr cid.ID, withLinking bool) ([]oid.ID, bool) { func tryGetSplitMembersByLinkingObject(cmd *cobra.Command, splitInfo *objectSDK.SplitInfo, prmHead internal.HeadObjectPrm, cnr cid.ID) ([]oid.ID, bool) {
// collect split chain by the descending ease of operations (ease is evaluated heuristically). // collect split chain by the descending ease of operations (ease is evaluated heuristically).
// If any approach fails, we don't try the next since we assume that it will fail too. // If any approach fails, we don't try the next since we assume that it will fail too.
@ -409,11 +411,8 @@ func tryGetSplitMembersByLinkingObject(cmd *cobra.Command, splitInfo *objectSDK.
common.PrintVerbose(cmd, "Received split members from the linking object: %v", children) common.PrintVerbose(cmd, "Received split members from the linking object: %v", children)
if withLinking {
return append(children, idLinking), true return append(children, idLinking), true
} }
return children, true
}
// linking object is not required for // linking object is not required for
// object collecting // object collecting

View file

@ -21,7 +21,6 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-node/misc" "git.frostfs.info/TrueCloudLab/frostfs-node/misc"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/config" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/gendoc" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/gendoc"
"github.com/mitchellh/go-homedir"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
) )
@ -112,15 +111,17 @@ func initConfig() {
// Use config file from the flag. // Use config file from the flag.
viper.SetConfigFile(cfgFile) viper.SetConfigFile(cfgFile)
} else { } else {
// Find home directory. // Find config directory.
home, err := homedir.Dir() configDir, err := os.UserConfigDir()
commonCmd.ExitOnErr(rootCmd, "", err) if err != nil {
common.PrintVerbose(rootCmd, "Get config dir: %s", err)
// Search config in `$HOME/.config/frostfs-cli/` with name "config.yaml" } else {
viper.AddConfigPath(filepath.Join(home, ".config", "frostfs-cli")) // Search config in `$XDG_CONFIG_HOME/frostfs-cli/` with name "config.yaml"
viper.AddConfigPath(filepath.Join(configDir, "frostfs-cli"))
viper.SetConfigName("config") viper.SetConfigName("config")
viper.SetConfigType("yaml") viper.SetConfigType("yaml")
} }
}
viper.SetEnvPrefix(envPrefix) viper.SetEnvPrefix(envPrefix)
viper.AutomaticEnv() // read in environment variables that match viper.AutomaticEnv() // read in environment variables that match

View file

@ -30,8 +30,6 @@ func initAddCmd() {
ff := addCmd.Flags() ff := addCmd.Flags()
ff.StringSlice(metaFlagKey, nil, "Meta pairs in the form of Key1=[0x]Value1,Key2=[0x]Value2") ff.StringSlice(metaFlagKey, nil, "Meta pairs in the form of Key1=[0x]Value1,Key2=[0x]Value2")
ff.Uint64(parentIDFlagKey, 0, "Parent node ID") ff.Uint64(parentIDFlagKey, 0, "Parent node ID")
_ = cobra.MarkFlagRequired(ff, commonflags.RPC)
} }
func add(cmd *cobra.Command, _ []string) { func add(cmd *cobra.Command, _ []string) {
@ -47,9 +45,10 @@ func add(cmd *cobra.Command, _ []string) {
meta, err := parseMeta(cmd) meta, err := parseMeta(cmd)
commonCmd.ExitOnErr(cmd, "meta data parsing: %w", err) commonCmd.ExitOnErr(cmd, "meta data parsing: %w", err)
ctx := cmd.Context() ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx) cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err) commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size) rawCID := make([]byte, sha256.Size)
@ -77,13 +76,13 @@ func add(cmd *cobra.Command, _ []string) {
cmd.Println("Node ID: ", resp.GetBody().GetNodeId()) cmd.Println("Node ID: ", resp.GetBody().GetNodeId())
} }
func parseMeta(cmd *cobra.Command) ([]*tree.KeyValue, error) { func parseMeta(cmd *cobra.Command) ([]tree.KeyValue, error) {
raws, _ := cmd.Flags().GetStringSlice(metaFlagKey) raws, _ := cmd.Flags().GetStringSlice(metaFlagKey)
if len(raws) == 0 { if len(raws) == 0 {
return nil, nil return nil, nil
} }
pairs := make([]*tree.KeyValue, 0, len(raws)) pairs := make([]tree.KeyValue, 0, len(raws))
for i := range raws { for i := range raws {
k, v, found := strings.Cut(raws[i], "=") k, v, found := strings.Cut(raws[i], "=")
if !found { if !found {
@ -94,7 +93,7 @@ func parseMeta(cmd *cobra.Command) ([]*tree.KeyValue, error) {
pair.Key = k pair.Key = k
pair.Value = []byte(v) pair.Value = []byte(v)
pairs = append(pairs, &pair) pairs = append(pairs, pair)
} }
return pairs, nil return pairs, nil

View file

@ -36,7 +36,6 @@ func initAddByPathCmd() {
ff.String(pathFlagKey, "", "Path to a node") ff.String(pathFlagKey, "", "Path to a node")
ff.StringSlice(metaFlagKey, nil, "Meta pairs in the form of Key1=[0x]Value1,Key2=[0x]Value2") ff.StringSlice(metaFlagKey, nil, "Meta pairs in the form of Key1=[0x]Value1,Key2=[0x]Value2")
_ = cobra.MarkFlagRequired(ff, commonflags.RPC)
_ = cobra.MarkFlagRequired(ff, pathFlagKey) _ = cobra.MarkFlagRequired(ff, pathFlagKey)
} }
@ -50,9 +49,10 @@ func addByPath(cmd *cobra.Command, _ []string) {
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err) commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
tid, _ := cmd.Flags().GetString(treeIDFlagKey) tid, _ := cmd.Flags().GetString(treeIDFlagKey)
ctx := cmd.Context() ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx) cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err) commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size) rawCID := make([]byte, sha256.Size)

View file

@ -2,14 +2,16 @@ package tree
import ( import (
"context" "context"
"fmt"
"strings" "strings"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags" "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree"
metrics "git.frostfs.info/TrueCloudLab/frostfs-observability/metrics/grpc" metrics "git.frostfs.info/TrueCloudLab/frostfs-observability/metrics/grpc"
tracing "git.frostfs.info/TrueCloudLab/frostfs-observability/tracing/grpc" tracing "git.frostfs.info/TrueCloudLab/frostfs-observability/tracing/grpc"
"github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
"google.golang.org/grpc" "google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure" "google.golang.org/grpc/credentials/insecure"
@ -17,15 +19,20 @@ import (
// _client returns grpc Tree service client. Should be removed // _client returns grpc Tree service client. Should be removed
// after making Tree API public. // after making Tree API public.
func _client(ctx context.Context) (tree.TreeServiceClient, error) { func _client() (tree.TreeServiceClient, error) {
var netAddr network.Address var netAddr network.Address
err := netAddr.FromString(viper.GetString(commonflags.RPC))
rpcEndpoint := viper.GetString(commonflags.RPC)
if rpcEndpoint == "" {
return nil, fmt.Errorf("%s is not defined", commonflags.RPC)
}
err := netAddr.FromString(rpcEndpoint)
if err != nil { if err != nil {
return nil, err return nil, err
} }
opts := []grpc.DialOption{ opts := []grpc.DialOption{
grpc.WithBlock(),
grpc.WithChainUnaryInterceptor( grpc.WithChainUnaryInterceptor(
metrics.NewUnaryClientInterceptor(), metrics.NewUnaryClientInterceptor(),
tracing.NewUnaryClientInteceptor(), tracing.NewUnaryClientInteceptor(),
@ -34,18 +41,21 @@ func _client(ctx context.Context) (tree.TreeServiceClient, error) {
metrics.NewStreamClientInterceptor(), metrics.NewStreamClientInterceptor(),
tracing.NewStreamClientInterceptor(), tracing.NewStreamClientInterceptor(),
), ),
grpc.WithDefaultCallOptions(grpc.WaitForReady(true)),
} }
if !strings.HasPrefix(netAddr.URIAddr(), "grpcs:") { if !strings.HasPrefix(netAddr.URIAddr(), "grpcs:") {
opts = append(opts, grpc.WithTransportCredentials(insecure.NewCredentials())) opts = append(opts, grpc.WithTransportCredentials(insecure.NewCredentials()))
} }
// a default connection establishing timeout cc, err := grpc.NewClient(netAddr.URIAddr(), opts...)
const defaultClientConnectTimeout = time.Second * 2
ctx, cancel := context.WithTimeout(ctx, defaultClientConnectTimeout)
cc, err := grpc.DialContext(ctx, netAddr.URIAddr(), opts...)
cancel()
return tree.NewTreeServiceClient(cc), err return tree.NewTreeServiceClient(cc), err
} }
func contextWithTimeout(cmd *cobra.Command) (context.Context, context.CancelFunc) {
if timeout := viper.GetDuration(commonflags.Timeout); timeout > 0 {
common.PrintVerbose(cmd, "Set request timeout to %s.", timeout)
return context.WithTimeout(cmd.Context(), timeout)
}
return context.WithTimeout(cmd.Context(), commonflags.TimeoutDefault)
}

View file

@ -36,8 +36,6 @@ func initGetByPathCmd() {
ff.String(pathFlagKey, "", "Path to a node") ff.String(pathFlagKey, "", "Path to a node")
ff.Bool(latestOnlyFlagKey, false, "Look only for the latest version of a node") ff.Bool(latestOnlyFlagKey, false, "Look only for the latest version of a node")
_ = cobra.MarkFlagRequired(ff, commonflags.RPC)
} }
func getByPath(cmd *cobra.Command, _ []string) { func getByPath(cmd *cobra.Command, _ []string) {
@ -50,9 +48,10 @@ func getByPath(cmd *cobra.Command, _ []string) {
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err) commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
tid, _ := cmd.Flags().GetString(treeIDFlagKey) tid, _ := cmd.Flags().GetString(treeIDFlagKey)
ctx := cmd.Context() ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx) cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err) commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size) rawCID := make([]byte, sha256.Size)

View file

@ -30,8 +30,6 @@ func initGetOpLogCmd() {
ff := getOpLogCmd.Flags() ff := getOpLogCmd.Flags()
ff.Uint64(heightFlagKey, 0, "Height to start with") ff.Uint64(heightFlagKey, 0, "Height to start with")
ff.Uint64(countFlagKey, 10, "Logged operations count") ff.Uint64(countFlagKey, 10, "Logged operations count")
_ = cobra.MarkFlagRequired(ff, commonflags.RPC)
} }
func getOpLog(cmd *cobra.Command, _ []string) { func getOpLog(cmd *cobra.Command, _ []string) {
@ -44,9 +42,10 @@ func getOpLog(cmd *cobra.Command, _ []string) {
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err) commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
tid, _ := cmd.Flags().GetString(treeIDFlagKey) tid, _ := cmd.Flags().GetString(treeIDFlagKey)
ctx := cmd.Context() ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx) cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err) commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size) rawCID := make([]byte, sha256.Size)

View file

@ -20,15 +20,14 @@ var healthcheckCmd = &cobra.Command{
func initHealthcheckCmd() { func initHealthcheckCmd() {
commonflags.Init(healthcheckCmd) commonflags.Init(healthcheckCmd)
ff := healthcheckCmd.Flags()
_ = cobra.MarkFlagRequired(ff, commonflags.RPC)
} }
func healthcheck(cmd *cobra.Command, _ []string) { func healthcheck(cmd *cobra.Command, _ []string) {
pk := key.GetOrGenerate(cmd) pk := key.GetOrGenerate(cmd)
ctx := cmd.Context() ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx) cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err) commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
req := &tree.HealthcheckRequest{ req := &tree.HealthcheckRequest{

View file

@ -26,8 +26,6 @@ func initListCmd() {
ff := listCmd.Flags() ff := listCmd.Flags()
ff.String(commonflags.CIDFlag, "", commonflags.CIDFlagUsage) ff.String(commonflags.CIDFlag, "", commonflags.CIDFlagUsage)
_ = listCmd.MarkFlagRequired(commonflags.CIDFlag) _ = listCmd.MarkFlagRequired(commonflags.CIDFlag)
_ = cobra.MarkFlagRequired(ff, commonflags.RPC)
} }
func list(cmd *cobra.Command, _ []string) { func list(cmd *cobra.Command, _ []string) {
@ -38,9 +36,10 @@ func list(cmd *cobra.Command, _ []string) {
err := cnr.DecodeString(cidString) err := cnr.DecodeString(cidString)
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err) commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
ctx := cmd.Context() ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx) cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err) commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size) rawCID := make([]byte, sha256.Size)

View file

@ -33,8 +33,6 @@ func initMoveCmd() {
_ = getSubtreeCmd.MarkFlagRequired(nodeIDFlagKey) _ = getSubtreeCmd.MarkFlagRequired(nodeIDFlagKey)
_ = getSubtreeCmd.MarkFlagRequired(parentIDFlagKey) _ = getSubtreeCmd.MarkFlagRequired(parentIDFlagKey)
_ = cobra.MarkFlagRequired(ff, commonflags.RPC)
} }
func move(cmd *cobra.Command, _ []string) { func move(cmd *cobra.Command, _ []string) {
@ -45,9 +43,10 @@ func move(cmd *cobra.Command, _ []string) {
err := cnr.DecodeString(cidString) err := cnr.DecodeString(cidString)
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err) commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
ctx := cmd.Context() ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx) cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err) commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size) rawCID := make([]byte, sha256.Size)
@ -66,7 +65,7 @@ func move(cmd *cobra.Command, _ []string) {
Body: &tree.GetSubTreeRequest_Body{ Body: &tree.GetSubTreeRequest_Body{
ContainerId: rawCID, ContainerId: rawCID,
TreeId: tid, TreeId: tid,
RootId: nid, RootId: []uint64{nid},
Depth: 1, Depth: 1,
BearerToken: bt, BearerToken: bt,
}, },
@ -75,7 +74,7 @@ func move(cmd *cobra.Command, _ []string) {
resp, err := cli.GetSubTree(ctx, subTreeReq) resp, err := cli.GetSubTree(ctx, subTreeReq)
commonCmd.ExitOnErr(cmd, "rpc call: %w", err) commonCmd.ExitOnErr(cmd, "rpc call: %w", err)
var meta []*tree.KeyValue var meta []tree.KeyValue
subtreeResp, err := resp.Recv() subtreeResp, err := resp.Recv()
for ; err == nil; subtreeResp, err = resp.Recv() { for ; err == nil; subtreeResp, err = resp.Recv() {
meta = subtreeResp.GetBody().GetMeta() meta = subtreeResp.GetBody().GetMeta()

View file

@ -29,8 +29,6 @@ func initRemoveCmd() {
ff.Uint64(nodeIDFlagKey, 0, "Node ID.") ff.Uint64(nodeIDFlagKey, 0, "Node ID.")
_ = getSubtreeCmd.MarkFlagRequired(nodeIDFlagKey) _ = getSubtreeCmd.MarkFlagRequired(nodeIDFlagKey)
_ = cobra.MarkFlagRequired(ff, commonflags.RPC)
} }
func remove(cmd *cobra.Command, _ []string) { func remove(cmd *cobra.Command, _ []string) {
@ -41,9 +39,10 @@ func remove(cmd *cobra.Command, _ []string) {
err := cnr.DecodeString(cidString) err := cnr.DecodeString(cidString)
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err) commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
ctx := cmd.Context() ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx) cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err) commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size) rawCID := make([]byte, sha256.Size)

View file

@ -49,6 +49,7 @@ const (
heightFlagKey = "height" heightFlagKey = "height"
countFlagKey = "count" countFlagKey = "count"
depthFlagKey = "depth" depthFlagKey = "depth"
orderFlagKey = "ordered"
) )
func initCTID(cmd *cobra.Command) { func initCTID(cmd *cobra.Command) {

View file

@ -30,11 +30,10 @@ func initGetSubtreeCmd() {
ff := getSubtreeCmd.Flags() ff := getSubtreeCmd.Flags()
ff.Uint64(rootIDFlagKey, 0, "Root ID to traverse from.") ff.Uint64(rootIDFlagKey, 0, "Root ID to traverse from.")
ff.Uint32(depthFlagKey, 10, "Traversal depth.") ff.Uint32(depthFlagKey, 10, "Traversal depth.")
ff.Bool(orderFlagKey, false, "Sort output by ascending FileName.")
_ = getSubtreeCmd.MarkFlagRequired(commonflags.CIDFlag) _ = getSubtreeCmd.MarkFlagRequired(commonflags.CIDFlag)
_ = getSubtreeCmd.MarkFlagRequired(treeIDFlagKey) _ = getSubtreeCmd.MarkFlagRequired(treeIDFlagKey)
_ = cobra.MarkFlagRequired(ff, commonflags.RPC)
} }
func getSubTree(cmd *cobra.Command, _ []string) { func getSubTree(cmd *cobra.Command, _ []string) {
@ -45,9 +44,10 @@ func getSubTree(cmd *cobra.Command, _ []string) {
err := cnr.DecodeString(cidString) err := cnr.DecodeString(cidString)
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err) commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
ctx := cmd.Context() ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx) cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err) commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size) rawCID := make([]byte, sha256.Size)
@ -59,6 +59,13 @@ func getSubTree(cmd *cobra.Command, _ []string) {
depth, _ := cmd.Flags().GetUint32(depthFlagKey) depth, _ := cmd.Flags().GetUint32(depthFlagKey)
order, _ := cmd.Flags().GetBool(orderFlagKey)
bodyOrder := tree.GetSubTreeRequest_Body_Order_None
if order {
bodyOrder = tree.GetSubTreeRequest_Body_Order_Asc
}
var bt []byte var bt []byte
if t := common.ReadBearerToken(cmd, bearerFlagKey); t != nil { if t := common.ReadBearerToken(cmd, bearerFlagKey); t != nil {
bt = t.Marshal() bt = t.Marshal()
@ -68,9 +75,12 @@ func getSubTree(cmd *cobra.Command, _ []string) {
Body: &tree.GetSubTreeRequest_Body{ Body: &tree.GetSubTreeRequest_Body{
ContainerId: rawCID, ContainerId: rawCID,
TreeId: tid, TreeId: tid,
RootId: rid, RootId: []uint64{rid},
Depth: depth, Depth: depth,
BearerToken: bt, BearerToken: bt,
OrderBy: &tree.GetSubTreeRequest_Body_Order{
Direction: bodyOrder,
},
}, },
} }
@ -83,10 +93,15 @@ func getSubTree(cmd *cobra.Command, _ []string) {
for ; err == nil; subtreeResp, err = resp.Recv() { for ; err == nil; subtreeResp, err = resp.Recv() {
b := subtreeResp.GetBody() b := subtreeResp.GetBody()
if len(b.GetNodeId()) == 1 {
cmd.Printf("Node ID: %d\n", b.GetNodeId()) cmd.Printf("Node ID: %d\n", b.GetNodeId())
cmd.Println("\tParent ID: ", b.GetParentId()) cmd.Println("\tParent ID: ", b.GetParentId())
cmd.Println("\tTimestamp: ", b.GetTimestamp()) cmd.Println("\tTimestamp: ", b.GetTimestamp())
} else {
cmd.Printf("Node IDs: %v\n", b.GetNodeId())
cmd.Println("\tParent IDs: ", b.GetParentId())
cmd.Println("\tTimestamps: ", b.GetTimestamp())
}
if meta := b.GetMeta(); len(meta) > 0 { if meta := b.GetMeta(); len(meta) > 0 {
cmd.Println("\tMeta pairs: ") cmd.Println("\tMeta pairs: ")

View file

@ -33,7 +33,7 @@ func PrettyPrintTableBACL(cmd *cobra.Command, bacl *acl.Basic) {
fmt.Fprintln(w, strings.Join(bits, "\t")) fmt.Fprintln(w, strings.Join(bits, "\t"))
// Footer // Footer
footer := []string{"X F"} footer := []string{"X F"}
for i := 0; i < 7; i++ { for range 7 {
footer = append(footer, "U S O B") footer = append(footer, "U S O B")
} }
fmt.Fprintln(w, strings.Join(footer, "\t")) fmt.Fprintln(w, strings.Join(footer, "\t"))

View file

@ -239,6 +239,8 @@ func parseAction(lexeme string) ([]string, bool, error) {
return []string{nativeschema.MethodRangeObject}, true, nil return []string{nativeschema.MethodRangeObject}, true, nil
case "object.hash": case "object.hash":
return []string{nativeschema.MethodHashObject}, true, nil return []string{nativeschema.MethodHashObject}, true, nil
case "object.patch":
return []string{nativeschema.MethodPatchObject}, true, nil
case "object.*": case "object.*":
return []string{ return []string{
nativeschema.MethodPutObject, nativeschema.MethodPutObject,
@ -248,6 +250,7 @@ func parseAction(lexeme string) ([]string, bool, error) {
nativeschema.MethodSearchObject, nativeschema.MethodSearchObject,
nativeschema.MethodRangeObject, nativeschema.MethodRangeObject,
nativeschema.MethodHashObject, nativeschema.MethodHashObject,
nativeschema.MethodPatchObject,
}, true, nil }, true, nil
case "container.put": case "container.put":
return []string{nativeschema.MethodPutContainer}, false, nil return []string{nativeschema.MethodPutContainer}, false, nil
@ -255,10 +258,6 @@ func parseAction(lexeme string) ([]string, bool, error) {
return []string{nativeschema.MethodDeleteContainer}, false, nil return []string{nativeschema.MethodDeleteContainer}, false, nil
case "container.get": case "container.get":
return []string{nativeschema.MethodGetContainer}, false, nil return []string{nativeschema.MethodGetContainer}, false, nil
case "container.setcontainereacl":
return []string{nativeschema.MethodSetContainerEACL}, false, nil
case "container.getcontainereacl":
return []string{nativeschema.MethodGetContainerEACL}, false, nil
case "container.list": case "container.list":
return []string{nativeschema.MethodListContainers}, false, nil return []string{nativeschema.MethodListContainers}, false, nil
case "container.*": case "container.*":
@ -266,8 +265,6 @@ func parseAction(lexeme string) ([]string, bool, error) {
nativeschema.MethodPutContainer, nativeschema.MethodPutContainer,
nativeschema.MethodDeleteContainer, nativeschema.MethodDeleteContainer,
nativeschema.MethodGetContainer, nativeschema.MethodGetContainer,
nativeschema.MethodSetContainerEACL,
nativeschema.MethodGetContainerEACL,
nativeschema.MethodListContainers, nativeschema.MethodListContainers,
}, false, nil }, false, nil
default: default:

View file

@ -1,18 +0,0 @@
package util
import (
"github.com/spf13/cobra"
)
// locode section.
var locodeCmd = &cobra.Command{
Use: "locode",
Short: "Working with FrostFS UN/LOCODE database",
}
func initLocodeCmd() {
locodeCmd.AddCommand(locodeGenerateCmd, locodeInfoCmd)
initUtilLocodeInfoCmd()
initUtilLocodeGenerateCmd()
}

View file

@ -1,96 +0,0 @@
package util
import (
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
locodedb "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/db"
airportsdb "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/db/airports"
locodebolt "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/db/boltdb"
continentsdb "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/db/continents/geojson"
csvlocode "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/table/csv"
"github.com/spf13/cobra"
)
type namesDB struct {
*airportsdb.DB
*csvlocode.Table
}
const (
locodeGenerateInputFlag = "in"
locodeGenerateSubDivFlag = "subdiv"
locodeGenerateAirportsFlag = "airports"
locodeGenerateCountriesFlag = "countries"
locodeGenerateContinentsFlag = "continents"
locodeGenerateOutputFlag = "out"
)
var (
locodeGenerateInPaths []string
locodeGenerateSubDivPath string
locodeGenerateAirportsPath string
locodeGenerateCountriesPath string
locodeGenerateContinentsPath string
locodeGenerateOutPath string
locodeGenerateCmd = &cobra.Command{
Use: "generate",
Short: "Generate UN/LOCODE database for FrostFS",
Run: func(cmd *cobra.Command, _ []string) {
locodeDB := csvlocode.New(
csvlocode.Prm{
Path: locodeGenerateInPaths[0],
SubDivPath: locodeGenerateSubDivPath,
},
csvlocode.WithExtraPaths(locodeGenerateInPaths[1:]...),
)
airportDB := airportsdb.New(airportsdb.Prm{
AirportsPath: locodeGenerateAirportsPath,
CountriesPath: locodeGenerateCountriesPath,
})
continentsDB := continentsdb.New(continentsdb.Prm{
Path: locodeGenerateContinentsPath,
})
targetDB := locodebolt.New(locodebolt.Prm{
Path: locodeGenerateOutPath,
})
err := targetDB.Open()
commonCmd.ExitOnErr(cmd, "", err)
defer targetDB.Close()
names := &namesDB{
DB: airportDB,
Table: locodeDB,
}
err = locodedb.FillDatabase(locodeDB, airportDB, continentsDB, names, targetDB)
commonCmd.ExitOnErr(cmd, "", err)
},
}
)
func initUtilLocodeGenerateCmd() {
flags := locodeGenerateCmd.Flags()
flags.StringSliceVar(&locodeGenerateInPaths, locodeGenerateInputFlag, nil, "List of paths to UN/LOCODE tables (csv)")
_ = locodeGenerateCmd.MarkFlagRequired(locodeGenerateInputFlag)
flags.StringVar(&locodeGenerateSubDivPath, locodeGenerateSubDivFlag, "", "Path to UN/LOCODE subdivision database (csv)")
_ = locodeGenerateCmd.MarkFlagRequired(locodeGenerateSubDivFlag)
flags.StringVar(&locodeGenerateAirportsPath, locodeGenerateAirportsFlag, "", "Path to OpenFlights airport database (csv)")
_ = locodeGenerateCmd.MarkFlagRequired(locodeGenerateAirportsFlag)
flags.StringVar(&locodeGenerateCountriesPath, locodeGenerateCountriesFlag, "", "Path to OpenFlights country database (csv)")
_ = locodeGenerateCmd.MarkFlagRequired(locodeGenerateCountriesFlag)
flags.StringVar(&locodeGenerateContinentsPath, locodeGenerateContinentsFlag, "", "Path to continent polygons (GeoJSON)")
_ = locodeGenerateCmd.MarkFlagRequired(locodeGenerateContinentsFlag)
flags.StringVar(&locodeGenerateOutPath, locodeGenerateOutputFlag, "", "Target path for generated database")
_ = locodeGenerateCmd.MarkFlagRequired(locodeGenerateOutputFlag)
}

View file

@ -1,56 +0,0 @@
package util
import (
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
locodedb "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/db"
locodebolt "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/db/boltdb"
"github.com/spf13/cobra"
)
const (
locodeInfoDBFlag = "db"
locodeInfoCodeFlag = "locode"
)
var (
locodeInfoDBPath string
locodeInfoCode string
locodeInfoCmd = &cobra.Command{
Use: "info",
Short: "Print information about UN/LOCODE from FrostFS database",
Run: func(cmd *cobra.Command, _ []string) {
targetDB := locodebolt.New(locodebolt.Prm{
Path: locodeInfoDBPath,
}, locodebolt.ReadOnly())
err := targetDB.Open()
commonCmd.ExitOnErr(cmd, "", err)
defer targetDB.Close()
record, err := locodedb.LocodeRecord(targetDB, locodeInfoCode)
commonCmd.ExitOnErr(cmd, "", err)
cmd.Printf("Country: %s\n", record.CountryName())
cmd.Printf("Location: %s\n", record.LocationName())
cmd.Printf("Continent: %s\n", record.Continent())
if subDivCode := record.SubDivCode(); subDivCode != "" {
cmd.Printf("Subdivision: [%s] %s\n", subDivCode, record.SubDivName())
}
geoPoint := record.GeoPoint()
cmd.Printf("Coordinates: %0.2f, %0.2f\n", geoPoint.Latitude(), geoPoint.Longitude())
},
}
)
func initUtilLocodeInfoCmd() {
flags := locodeInfoCmd.Flags()
flags.StringVar(&locodeInfoDBPath, locodeInfoDBFlag, "", "Path to FrostFS UN/LOCODE database")
_ = locodeInfoCmd.MarkFlagRequired(locodeInfoDBFlag)
flags.StringVar(&locodeInfoCode, locodeInfoCodeFlag, "", "UN/LOCODE")
_ = locodeInfoCmd.MarkFlagRequired(locodeInfoCodeFlag)
}

View file

@ -23,11 +23,9 @@ func init() {
signCmd, signCmd,
convertCmd, convertCmd,
keyerCmd, keyerCmd,
locodeCmd,
) )
initSignCmd() initSignCmd()
initConvertCmd() initConvertCmd()
initKeyerCmd() initKeyerCmd()
initLocodeCmd()
} }

View file

@ -7,6 +7,7 @@ import (
configViper "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common/config" configViper "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-node/internal/logs"
control "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/ir"
"github.com/spf13/viper" "github.com/spf13/viper"
"go.uber.org/zap" "go.uber.org/zap"
) )
@ -40,6 +41,8 @@ func reloadConfig() error {
if err != nil { if err != nil {
return err return err
} }
logPrm.PrependTimestamp = cfg.GetBool("logger.timestamp")
return logPrm.Reload() return logPrm.Reload()
} }
@ -81,6 +84,10 @@ func watchForSignal(cancel func()) {
return return
case <-sighupCh: case <-sighupCh:
log.Info(logs.FrostFSNodeSIGHUPHasBeenReceivedRereadingConfiguration) log.Info(logs.FrostFSNodeSIGHUPHasBeenReceivedRereadingConfiguration)
if !innerRing.CompareAndSwapHealthStatus(control.HealthStatus_READY, control.HealthStatus_RECONFIGURING) {
log.Info(logs.FrostFSNodeSIGHUPSkip)
break
}
err := reloadConfig() err := reloadConfig()
if err != nil { if err != nil {
log.Error(logs.FrostFSNodeConfigurationReading, zap.Error(err)) log.Error(logs.FrostFSNodeConfigurationReading, zap.Error(err))
@ -92,6 +99,7 @@ func watchForSignal(cancel func()) {
if err != nil { if err != nil {
log.Error(logs.FrostFSNodeConfigurationReading, zap.Error(err)) log.Error(logs.FrostFSNodeConfigurationReading, zap.Error(err))
} }
innerRing.CompareAndSwapHealthStatus(control.HealthStatus_RECONFIGURING, control.HealthStatus_READY)
log.Info(logs.FrostFSNodeConfigurationHasBeenReloadedSuccessfully) log.Info(logs.FrostFSNodeConfigurationHasBeenReloadedSuccessfully)
} }
} }

View file

@ -9,6 +9,7 @@ import (
func defaultConfiguration(cfg *viper.Viper) { func defaultConfiguration(cfg *viper.Viper) {
cfg.SetDefault("logger.level", "info") cfg.SetDefault("logger.level", "info")
cfg.SetDefault("logger.destination", "stdout") cfg.SetDefault("logger.destination", "stdout")
cfg.SetDefault("logger.timestamp", false)
setPprofDefaults(cfg) setPprofDefaults(cfg)
@ -47,6 +48,8 @@ func defaultConfiguration(cfg *viper.Viper) {
cfg.SetDefault("node.kludge_compatibility_mode", false) cfg.SetDefault("node.kludge_compatibility_mode", false)
cfg.SetDefault("audit.enabled", false) cfg.SetDefault("audit.enabled", false)
setMultinetDefaults(cfg)
} }
func setControlDefaults(cfg *viper.Viper) { func setControlDefaults(cfg *viper.Viper) {
@ -130,3 +133,11 @@ func setMorphDefaults(cfg *viper.Viper) {
cfg.SetDefault("morph.validators", []string{}) cfg.SetDefault("morph.validators", []string{})
cfg.SetDefault("morph.switch_interval", 2*time.Minute) cfg.SetDefault("morph.switch_interval", 2*time.Minute)
} }
func setMultinetDefaults(cfg *viper.Viper) {
cfg.SetDefault("multinet.enabled", false)
cfg.SetDefault("multinet.balancer", "")
cfg.SetDefault("multinet.restrict", false)
cfg.SetDefault("multinet.fallback_delay", "0s")
cfg.SetDefault("multinet.subnets", "")
}

View file

@ -13,6 +13,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-node/misc" "git.frostfs.info/TrueCloudLab/frostfs-node/misc"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger" "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/sdnotify"
"github.com/spf13/viper" "github.com/spf13/viper"
"go.uber.org/zap" "go.uber.org/zap"
) )
@ -78,6 +79,8 @@ func main() {
) )
exitErr(err) exitErr(err)
logPrm.SamplingHook = metrics.LogMetrics().GetSamplingHook() logPrm.SamplingHook = metrics.LogMetrics().GetSamplingHook()
logPrm.PrependTimestamp = cfg.GetBool("logger.timestamp")
log, err = logger.NewLogger(logPrm) log, err = logger.NewLogger(logPrm)
exitErr(err) exitErr(err)
@ -124,4 +127,8 @@ func shutdown() {
zap.String("error", err.Error()), zap.String("error", err.Error()),
) )
} }
if err := sdnotify.ClearStatus(); err != nil {
log.Error(logs.FailedToReportStatusToSystemd, zap.Error(err))
}
} }

View file

@ -19,7 +19,7 @@ var Root = &cobra.Command{
} }
func init() { func init() {
Root.AddCommand(listCMD, inspectCMD) Root.AddCommand(listCMD, inspectCMD, tuiCMD)
} }
func openBlobovnicza(cmd *cobra.Command) *blobovnicza.Blobovnicza { func openBlobovnicza(cmd *cobra.Command) *blobovnicza.Blobovnicza {

View file

@ -0,0 +1,68 @@
package blobovnicza
import (
"context"
"fmt"
common "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal"
schema "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/blobovnicza"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/tui"
"github.com/rivo/tview"
"github.com/spf13/cobra"
)
var tuiCMD = &cobra.Command{
Use: "explore",
Short: "Blobovnicza exploration with a terminal UI",
Long: `Launch a terminal UI to explore blobovnicza and search for data.
Available search filters:
- cid CID
- oid OID
- addr CID/OID
`,
Run: tuiFunc,
}
var initialPrompt string
func init() {
common.AddComponentPathFlag(tuiCMD, &vPath)
tuiCMD.Flags().StringVar(
&initialPrompt,
"filter",
"",
"Filter prompt to start with, format 'tag:value [+ tag:value]...'",
)
}
func tuiFunc(cmd *cobra.Command, _ []string) {
common.ExitOnErr(cmd, runTUI(cmd))
}
func runTUI(cmd *cobra.Command) error {
db, err := tui.OpenDB(vPath, false)
if err != nil {
return fmt.Errorf("couldn't open database: %w", err)
}
defer db.Close()
ctx, cancel := context.WithCancel(cmd.Context())
defer cancel()
app := tview.NewApplication()
ui := tui.NewUI(ctx, app, db, schema.BlobovniczaParser, nil)
_ = ui.AddFilter("cid", tui.CIDParser, "CID")
_ = ui.AddFilter("oid", tui.OIDParser, "OID")
_ = ui.AddCompositeFilter("addr", tui.AddressParser, "CID/OID")
err = ui.WithPrompt(initialPrompt)
if err != nil {
return fmt.Errorf("invalid filter prompt: %w", err)
}
app.SetRoot(ui, true).SetFocus(ui)
return app.Run()
}

View file

@ -32,6 +32,7 @@ func init() {
inspectCMD, inspectCMD,
listGraveyardCMD, listGraveyardCMD,
listGarbageCMD, listGarbageCMD,
tuiCMD,
) )
} }

View file

@ -0,0 +1,71 @@
package meta
import (
"context"
"fmt"
common "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal"
schema "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/metabase"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/tui"
"github.com/rivo/tview"
"github.com/spf13/cobra"
)
var tuiCMD = &cobra.Command{
Use: "explore",
Short: "Metabase exploration with a terminal UI",
Long: `Launch a terminal UI to explore metabase and search for data.
Available search filters:
- cid CID
- oid OID
- addr CID/OID
- attr key[/value]
`,
Run: tuiFunc,
}
var initialPrompt string
func init() {
common.AddComponentPathFlag(tuiCMD, &vPath)
tuiCMD.Flags().StringVar(
&initialPrompt,
"filter",
"",
"Filter prompt to start with, format 'tag:value [+ tag:value]...'",
)
}
func tuiFunc(cmd *cobra.Command, _ []string) {
common.ExitOnErr(cmd, runTUI(cmd))
}
func runTUI(cmd *cobra.Command) error {
db, err := tui.OpenDB(vPath, false)
if err != nil {
return fmt.Errorf("couldn't open database: %w", err)
}
defer db.Close()
// Need if app was stopped with Ctrl-C.
ctx, cancel := context.WithCancel(cmd.Context())
defer cancel()
app := tview.NewApplication()
ui := tui.NewUI(ctx, app, db, schema.MetabaseParser, nil)
_ = ui.AddFilter("cid", tui.CIDParser, "CID")
_ = ui.AddFilter("oid", tui.OIDParser, "OID")
_ = ui.AddCompositeFilter("addr", tui.AddressParser, "CID/OID")
_ = ui.AddCompositeFilter("attr", tui.AttributeParser, "key[/value]")
err = ui.WithPrompt(initialPrompt)
if err != nil {
return fmt.Errorf("invalid filter prompt: %w", err)
}
app.SetRoot(ui, true).SetFocus(ui)
return app.Run()
}

View file

@ -0,0 +1,96 @@
package blobovnicza
import (
"encoding/binary"
"errors"
"fmt"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/mr-tron/base58"
)
var BlobovniczaParser = common.WithFallback(
common.Any(
MetaBucketParser,
BucketParser,
),
common.RawParser.ToFallbackParser(),
)
func MetaBucketParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if value != nil {
return nil, nil, errors.New("not a bucket")
}
if string(key) != "META" {
return nil, nil, errors.New("invalid bucket name")
}
return &MetaBucket{}, MetaRecordParser, nil
}
func MetaRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
var r MetaRecord
if len(key) == 0 {
return nil, nil, errors.New("invalid key")
}
r.label = string(key)
r.count = binary.LittleEndian.Uint64(value)
return &r, nil, nil
}
func BucketParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if value != nil {
return nil, nil, errors.New("not a bucket")
}
size, n := binary.Varint(key)
if n <= 0 {
return nil, nil, errors.New("invalid size")
}
return &Bucket{size: size}, RecordParser, nil
}
func RecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
parts := strings.Split(string(key), "/")
if len(parts) != 2 {
return nil, nil, errors.New("invalid key, expected address string <CID>/<OID>")
}
cnrRaw, err := base58.Decode(parts[0])
if err != nil {
return nil, nil, errors.New("can't decode CID string")
}
objRaw, err := base58.Decode(parts[1])
if err != nil {
return nil, nil, errors.New("can't decode OID string")
}
cnr := cid.ID{}
if err := cnr.Decode(cnrRaw); err != nil {
return nil, nil, fmt.Errorf("can't decode CID: %w", err)
}
obj := oid.ID{}
if err := obj.Decode(objRaw); err != nil {
return nil, nil, fmt.Errorf("can't decode OID: %w", err)
}
var r Record
r.addr.SetContainer(cnr)
r.addr.SetObject(obj)
if err := r.object.Unmarshal(value); err != nil {
return nil, nil, errors.New("can't unmarshal object")
}
return &r, nil, nil
}

View file

@ -0,0 +1,101 @@
package blobovnicza
import (
"fmt"
"strconv"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/davecgh/go-spew/spew"
"github.com/gdamore/tcell/v2"
"github.com/rivo/tview"
)
type (
MetaBucket struct{}
MetaRecord struct {
label string
count uint64
}
Bucket struct {
size int64
}
Record struct {
addr oid.Address
object objectSDK.Object
}
)
func (b *MetaBucket) String() string {
return common.FormatSimple("META", tcell.ColorLime)
}
func (b *MetaBucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *MetaBucket) Filter(string, any) common.FilterResult {
return common.No
}
func (r *MetaRecord) String() string {
return fmt.Sprintf("%-11s %c %d", r.label, tview.Borders.Vertical, r.count)
}
func (r *MetaRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *MetaRecord) Filter(string, any) common.FilterResult {
return common.No
}
func (b *Bucket) String() string {
return common.FormatSimple(strconv.FormatInt(b.size, 10), tcell.ColorLime)
}
func (b *Bucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *Bucket) Filter(typ string, _ any) common.FilterResult {
switch typ {
case "cid":
return common.Maybe
case "oid":
return common.Maybe
default:
return common.No
}
}
func (r *Record) String() string {
return fmt.Sprintf(
"CID %s OID %s %c Object {...}",
common.FormatSimple(fmt.Sprintf("%-44s", r.addr.Container()), tcell.ColorAqua),
common.FormatSimple(fmt.Sprintf("%-44s", r.addr.Object()), tcell.ColorAqua),
tview.Borders.Vertical,
)
}
func (r *Record) DetailedString() string {
return spew.Sdump(*r)
}
func (r *Record) Filter(typ string, val any) common.FilterResult {
switch typ {
case "cid":
id := val.(cid.ID)
return common.IfThenElse(r.addr.Container().Equals(id), common.Yes, common.No)
case "oid":
id := val.(oid.ID)
return common.IfThenElse(r.addr.Object().Equals(id), common.Yes, common.No)
default:
return common.No
}
}

View file

@ -0,0 +1,43 @@
package common
import (
"fmt"
"strconv"
"github.com/gdamore/tcell/v2"
)
type FormatOptions struct {
Color tcell.Color
Bold,
Italic,
Underline,
StrikeThrough bool
}
func Format(s string, opts FormatOptions) string {
var boldTag, italicTag, underlineTag, strikeThroughTag string
switch {
case opts.Bold:
boldTag = "b"
case opts.Italic:
italicTag = "i"
case opts.Underline:
underlineTag = "u"
case opts.StrikeThrough:
strikeThroughTag = "s"
}
attrs := fmt.Sprintf(
"%s%s%s%s", boldTag, italicTag, underlineTag, strikeThroughTag,
)
color := strconv.FormatInt(int64(opts.Color.Hex()), 16)
return fmt.Sprintf("[#%06s::%s]%s[-::-]", color, attrs, s)
}
func FormatSimple(s string, c tcell.Color) string {
return Format(s, FormatOptions{Color: c})
}

View file

@ -0,0 +1,31 @@
package common
import (
"github.com/davecgh/go-spew/spew"
"github.com/gdamore/tcell/v2"
"github.com/mr-tron/base58"
)
type RawEntry struct {
// key and value used for record dump.
// nolint:unused
key, value []byte
}
var RawParser Parser = rawParser
func rawParser(key, value []byte) (SchemaEntry, Parser, error) {
return &RawEntry{key: key, value: value}, rawParser, nil
}
func (r *RawEntry) String() string {
return FormatSimple(base58.Encode(r.key), tcell.ColorRed)
}
func (r *RawEntry) DetailedString() string {
return spew.Sdump(r)
}
func (r *RawEntry) Filter(string, any) FilterResult {
return No
}

View file

@ -0,0 +1,81 @@
package common
import (
"errors"
"fmt"
)
type FilterResult byte
const (
No FilterResult = iota
Maybe
Yes
)
func IfThenElse(condition bool, onSuccess, onFailure FilterResult) FilterResult {
var res FilterResult
if condition {
res = onSuccess
} else {
res = onFailure
}
return res
}
type SchemaEntry interface {
String() string
DetailedString() string
Filter(typ string, val any) FilterResult
}
type (
Parser func(key, value []byte) (SchemaEntry, Parser, error)
FallbackParser func(key, value []byte) (SchemaEntry, Parser)
)
func Any(parsers ...Parser) Parser {
return func(key, value []byte) (SchemaEntry, Parser, error) {
var errs error
for _, parser := range parsers {
ret, next, err := parser(key, value)
if err == nil {
return ret, next, nil
}
errs = errors.Join(errs, err)
}
return nil, nil, fmt.Errorf("no parser succeeded: %w", errs)
}
}
func WithFallback(parser Parser, fallback FallbackParser) Parser {
if parser == nil {
return fallback.ToParser()
}
return func(key, value []byte) (SchemaEntry, Parser, error) {
entry, next, err := parser(key, value)
if err == nil {
return entry, WithFallback(next, fallback), nil
}
return fallback.ToParser()(key, value)
}
}
func (fp FallbackParser) ToParser() Parser {
return func(key, value []byte) (SchemaEntry, Parser, error) {
entry, next := fp(key, value)
return entry, next, nil
}
}
func (p Parser) ToFallbackParser() FallbackParser {
return func(key, value []byte) (SchemaEntry, Parser) {
entry, next, err := p(key, value)
if err != nil {
panic(fmt.Errorf(
"couldn't use that parser as a fallback parser, it returned an error: %w", err,
))
}
return entry, next
}
}

View file

@ -0,0 +1,29 @@
package buckets
import (
"github.com/davecgh/go-spew/spew"
)
func (b *PrefixBucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *PrefixContainerBucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *UserBucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *ContainerBucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *UserAttributeKeyBucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *UserAttributeValueBucket) DetailedString() string {
return spew.Sdump(*b)
}

View file

@ -0,0 +1,81 @@
package buckets
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
)
func (b *PrefixBucket) Filter(typ string, _ any) common.FilterResult {
switch typ {
case "cid":
return b.resolvers.cidResolver(false)
case "oid":
return b.resolvers.oidResolver(false)
default:
return common.No
}
}
func (b *PrefixContainerBucket) Filter(typ string, val any) common.FilterResult {
switch typ {
case "cid":
id := val.(cid.ID)
return b.resolvers.cidResolver(b.id.Equals(id))
case "oid":
return b.resolvers.oidResolver(false)
default:
return common.No
}
}
func (b *UserBucket) Filter(typ string, _ any) common.FilterResult {
switch typ {
case "cid":
return b.resolvers.cidResolver(false)
case "oid":
return b.resolvers.oidResolver(false)
default:
return common.No
}
}
func (b *ContainerBucket) Filter(typ string, val any) common.FilterResult {
switch typ {
case "cid":
id := val.(cid.ID)
return b.resolvers.cidResolver(b.id.Equals(id))
case "oid":
return b.resolvers.oidResolver(false)
default:
return common.No
}
}
func (b *UserAttributeKeyBucket) Filter(typ string, val any) common.FilterResult {
switch typ {
case "cid":
id := val.(cid.ID)
return common.IfThenElse(b.id.Equals(id), common.Yes, common.No)
case "oid":
return common.Maybe
case "key":
key := val.(string)
return common.IfThenElse(b.key == key, common.Yes, common.No)
case "value":
return common.Maybe
default:
return common.No
}
}
func (b *UserAttributeValueBucket) Filter(typ string, val any) common.FilterResult {
switch typ {
case "oid":
return common.Maybe
case "value":
value := val.(string)
return common.IfThenElse(b.value == value, common.Yes, common.No)
default:
return common.No
}
}

View file

@ -0,0 +1,111 @@
package buckets
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/metabase/records"
)
var (
GraveyardParser = NewPrefixBucketParser(Graveyard, records.GraveyardRecordParser, Resolvers{
cidResolver: LenientResolver,
oidResolver: LenientResolver,
})
GarbageParser = NewPrefixBucketParser(Garbage, records.GarbageRecordParser, Resolvers{
cidResolver: LenientResolver,
oidResolver: LenientResolver,
})
ContainerVolumeParser = NewPrefixBucketParser(ContainerVolume, records.ContainerVolumeRecordParser, Resolvers{
cidResolver: LenientResolver,
oidResolver: StrictResolver,
})
LockedParser = NewPrefixBucketParser(
Locked,
NewContainerBucketParser(
records.LockedRecordParser,
Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
},
),
Resolvers{
cidResolver: LenientResolver,
oidResolver: LenientResolver,
},
)
ShardInfoParser = NewPrefixBucketParser(ShardInfo, records.ShardInfoRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: StrictResolver,
})
PrimaryParser = NewPrefixContainerBucketParser(Primary, records.ObjectRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
LockersParser = NewPrefixContainerBucketParser(Lockers, records.ObjectRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
TombstoneParser = NewPrefixContainerBucketParser(Tombstone, records.ObjectRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
SmallParser = NewPrefixContainerBucketParser(Small, records.SmallRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
RootParser = NewPrefixContainerBucketParser(Root, records.RootRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
OwnerParser = NewPrefixContainerBucketParser(
Owner,
NewUserBucketParser(
records.OwnerRecordParser,
Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
},
),
Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
},
)
UserAttributeParser = NewUserAttributeKeyBucketParser(
NewUserAttributeValueBucketParser(records.UserAttributeRecordParser),
)
PayloadHashParser = NewPrefixContainerBucketParser(PayloadHash, records.PayloadHashRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: StrictResolver,
})
ParentParser = NewPrefixContainerBucketParser(Parent, records.ParentRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
SplitParser = NewPrefixContainerBucketParser(Split, records.SplitRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: StrictResolver,
})
ContainerCountersParser = NewPrefixBucketParser(ContainerCounters, records.ContainerCountersRecordParser, Resolvers{
cidResolver: LenientResolver,
oidResolver: StrictResolver,
})
ECInfoParser = NewPrefixContainerBucketParser(ECInfo, records.ECInfoRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
)

Some files were not shown because too many files have changed in this diff Show more