Compare commits

...

117 commits

Author SHA1 Message Date
d4bec24c9f
[#1366] node, ir: Support timestamp config option, update tests
All checks were successful
DCO action / DCO (pull_request) Successful in 1m2s
Tests and linters / Run gofumpt (pull_request) Successful in 1m9s
Vulncheck / Vulncheck (pull_request) Successful in 1m31s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m6s
Build / Build Components (pull_request) Successful in 2m14s
Tests and linters / gopls check (pull_request) Successful in 2m35s
Tests and linters / Staticcheck (pull_request) Successful in 2m42s
Tests and linters / Lint (pull_request) Successful in 3m32s
Tests and linters / Tests (pull_request) Successful in 4m10s
Tests and linters / Tests with -race (pull_request) Successful in 5m32s
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-17 10:50:08 +03:00
ea48e928c8
[#1366] logger: Make timestamp prepending optional
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-17 10:49:51 +03:00
96308a26c6 [#1361] linter: fix funlen
Signed-off-by: Aleksey Savaitan <a.savaitan@yadro.com>
2024-09-13 15:12:49 +00:00
74a6a1da7f [#1361] add root ca cert for telemetry configuration
Signed-off-by: Aleksey Savaitan <a.savaitan@yadro.com>
2024-09-13 15:12:49 +00:00
2be1aa781d [#1266] .forgejo: Make 'fumpt' job fail on changed files
All checks were successful
DCO action / DCO (pull_request) Successful in 1m4s
Tests and linters / Run gofumpt (pull_request) Successful in 59s
Vulncheck / Vulncheck (pull_request) Successful in 2m24s
Tests and linters / Staticcheck (pull_request) Successful in 2m27s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m43s
Build / Build Components (pull_request) Successful in 2m50s
Tests and linters / gopls check (pull_request) Successful in 3m2s
Tests and linters / Lint (pull_request) Successful in 3m10s
Tests and linters / Tests (pull_request) Successful in 4m36s
Tests and linters / Tests with -race (pull_request) Successful in 6m24s
`gofumpt` always returns an exit code of 0, even when it finds
misformatted files. To make `fumpt` action behave as expected
we need to check if `gofumpt` changed any files.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-13 17:27:58 +03:00
89d0435b1d [#1374] tree: Use NewClient to create grpc connection in cache
All checks were successful
DCO action / DCO (pull_request) Successful in 1m2s
Pre-commit hooks / Pre-commit (pull_request) Successful in 1m53s
Vulncheck / Vulncheck (pull_request) Successful in 2m4s
Build / Build Components (pull_request) Successful in 2m39s
Tests and linters / Run gofumpt (pull_request) Successful in 2m30s
Tests and linters / Staticcheck (pull_request) Successful in 2m38s
Tests and linters / gopls check (pull_request) Successful in 3m2s
Tests and linters / Lint (pull_request) Successful in 3m14s
Tests and linters / Tests (pull_request) Successful in 6m4s
Tests and linters / Tests with -race (pull_request) Successful in 6m7s
Created grpc connection should be established, so perform Healthcheck request
to check connection is ok.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-13 15:59:26 +03:00
54fe8383a4 [#1374] tree: Use NewClient to create grpc connection for sync
Created connection will be used to sync trees, so it is ok to defer
dial to the first RPC call.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-13 15:59:26 +03:00
944160427b [#1374] cli: Drop deprecated grpc connection method
For `frostfs-cli` it is ok to use grpc-client without blocking,
as `frostfs-cli` will perform RPC call anyway.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-13 15:59:26 +03:00
bb44867491 [#1374] go.mod: Upgrade grpc to v1.66.2
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-13 15:59:00 +03:00
546d09660f [#1283] Clear systemd-notify status on exit
All checks were successful
DCO action / DCO (pull_request) Successful in 3m9s
Vulncheck / Vulncheck (pull_request) Successful in 3m13s
Tests and linters / Run gofumpt (pull_request) Successful in 3m26s
Tests and linters / Staticcheck (pull_request) Successful in 4m9s
Tests and linters / Tests with -race (pull_request) Successful in 4m23s
Tests and linters / Lint (pull_request) Successful in 4m49s
Pre-commit hooks / Pre-commit (pull_request) Successful in 5m33s
Tests and linters / gopls check (pull_request) Successful in 5m59s
Build / Build Components (pull_request) Successful in 6m26s
Tests and linters / Tests (pull_request) Successful in 10m23s
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-12 14:45:06 +00:00
e3764c51df [#1347] metabase: Fix EC search
For EC chunks need to return EC parent object ID as
EC chunks don't have own attributes but inherit parent's.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 13:23:29 +00:00
b33559754d [#1367] fstree: Add size hint for Delete
All checks were successful
DCO action / DCO (pull_request) Successful in 1m2s
Vulncheck / Vulncheck (pull_request) Successful in 1m20s
Tests and linters / Run gofumpt (pull_request) Successful in 1m44s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m6s
Build / Build Components (pull_request) Successful in 2m19s
Tests and linters / gopls check (pull_request) Successful in 2m34s
Tests and linters / Staticcheck (pull_request) Successful in 2m36s
Tests and linters / Lint (pull_request) Successful in 3m29s
Tests and linters / Tests (pull_request) Successful in 4m0s
Tests and linters / Tests with -race (pull_request) Successful in 5m48s
This allow to not to call `os.Stat` if caller already knows data size.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
f345fe9a58 [#1367] writecache: Move DB related code to upgrade.go
This is done to drop this file in the future.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
3b236160a6 [#1367] writecache: Drop DB label from metrics
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
25d2ae8aaf [#1367] writecache: Drop BBolt related config variables
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
e39378b1c3 [#1367] writecache: Add background flushing objects limiter
To limit memory usage by background flush.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
8a6e3025a0 [#1367] writecache: Flush from FSTree concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
2dd3a6f7a8 [#1367] fstree: Add IterateInfo method
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
b142b6f48e [#1367] fstree: Add size to file counter
FSTree file counter used by writecache. As writecache has now only one
storage, so it is required to use real object size to get writecache
size more accurate than `count * max_object_size`.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 15:06:33 +03:00
5f6c7cbdb1 [#1367] writecache: Drop bbolt DB
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-12 14:22:29 +03:00
66e17f4b8e
[#1368] cli/container: Use dedicated method to list user attributes
All checks were successful
DCO action / DCO (pull_request) Successful in 1m45s
Vulncheck / Vulncheck (pull_request) Successful in 2m11s
Tests and linters / Run gofumpt (pull_request) Successful in 2m37s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m50s
Build / Build Components (pull_request) Successful in 3m6s
Tests and linters / Staticcheck (pull_request) Successful in 3m1s
Tests and linters / Lint (pull_request) Successful in 3m57s
Tests and linters / gopls check (pull_request) Successful in 4m44s
Tests and linters / Tests (pull_request) Successful in 5m56s
Tests and linters / Tests with -race (pull_request) Successful in 6m0s
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-12 10:01:27 +03:00
99be4c83a7
[#1368] *: Run gofumpt
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-12 10:00:28 +03:00
ec8da40567 [#1369] Update obsolete URLs
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2024-09-11 11:31:10 +00:00
dea6f031f9 [#1331] cli/tree: Add order flag to tree get-subtree
Added `--ordered` flag to sort output by ascending FileName.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-11 11:30:28 +00:00
5fac4058e8 [#1364] cmd/common: Add tests for CreateViper and ReloadViper
All checks were successful
DCO action / DCO (pull_request) Successful in 51s
Tests and linters / Run gofumpt (pull_request) Successful in 1m41s
Vulncheck / Vulncheck (pull_request) Successful in 1m54s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m13s
Build / Build Components (pull_request) Successful in 2m21s
Tests and linters / gopls check (pull_request) Successful in 2m31s
Tests and linters / Staticcheck (pull_request) Successful in 2m34s
Tests and linters / Lint (pull_request) Successful in 3m30s
Tests and linters / Tests (pull_request) Successful in 3m35s
Tests and linters / Tests with -race (pull_request) Successful in 3m37s
Add tests for `CreateViper` and `ReloadViper` to ensure that no extra
files, except *.yaml, *.yml, *.json, are loaded from config directory.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-11 08:03:05 +00:00
2220f6a809 [#1365] Makefile: Fix HUB_IMAGE
All checks were successful
DCO action / DCO (pull_request) Successful in 59s
Vulncheck / Vulncheck (pull_request) Successful in 1m33s
Tests and linters / Run gofumpt (pull_request) Successful in 1m40s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m10s
Build / Build Components (pull_request) Successful in 2m16s
Tests and linters / gopls check (pull_request) Successful in 2m36s
Tests and linters / Staticcheck (pull_request) Successful in 2m36s
Tests and linters / Tests (pull_request) Successful in 3m32s
Tests and linters / Lint (pull_request) Successful in 3m33s
Tests and linters / Tests with -race (pull_request) Successful in 3m34s
Signed-off-by: Alexander Chuprov <a.chuprov@yadro.com>
2024-09-10 16:45:15 +03:00
a812932984 [#1362] ape: Move common APE check logic to separate package
* Tree and object service have the same log for checking APE. So,
  this check should be moved to common package.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-10 12:40:34 +00:00
92fe5d90f5 [#1359] writecache: Drop unused
All checks were successful
DCO action / DCO (pull_request) Successful in 2m45s
Vulncheck / Vulncheck (pull_request) Successful in 3m13s
Tests and linters / Run gofumpt (pull_request) Successful in 3m48s
Tests and linters / Staticcheck (pull_request) Successful in 3m57s
Tests and linters / Tests with -race (pull_request) Successful in 4m15s
Tests and linters / Lint (pull_request) Successful in 4m38s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m40s
Tests and linters / gopls check (pull_request) Successful in 4m41s
Build / Build Components (pull_request) Successful in 5m10s
Tests and linters / Tests (pull_request) Successful in 6m20s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-09 18:45:21 +03:00
4668efc0bf [#1355] metabase: Upgrade improvements
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m16s
DCO action / DCO (pull_request) Successful in 1m31s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m5s
Tests and linters / Run gofumpt (pull_request) Successful in 1m59s
Build / Build Components (pull_request) Successful in 2m13s
Tests and linters / gopls check (pull_request) Successful in 2m31s
Tests and linters / Staticcheck (pull_request) Successful in 2m38s
Tests and linters / Lint (pull_request) Successful in 3m15s
Tests and linters / Tests (pull_request) Successful in 3m32s
Tests and linters / Tests with -race (pull_request) Successful in 3m39s
Do not fail on same latest version to run compact on upgraded metabase.
Use NoSync on compact.
Log every batch on bucket delete stage.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-09 11:37:36 +03:00
654d970fad [#1355] adm: Run metabase upgrade concurrently
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 15:54:40 +03:00
d3b209c8e1 [#1337] shard: Disable background rebuild
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m3s
DCO action / DCO (pull_request) Successful in 1m28s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m2s
Build / Build Components (pull_request) Successful in 2m15s
Tests and linters / Run gofumpt (pull_request) Successful in 2m10s
Tests and linters / gopls check (pull_request) Successful in 2m21s
Tests and linters / Staticcheck (pull_request) Successful in 2m55s
Tests and linters / Lint (pull_request) Successful in 3m29s
Tests and linters / Tests (pull_request) Successful in 3m47s
Tests and linters / Tests with -race (pull_request) Successful in 3m58s
Since `frostfs-cli control shards rebuild` command was added,
there is no need for background rebuild now.
For failover tests used used value 1 to rebuild only schema change.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 15:19:55 +03:00
edb1747af7 [#1337] blobovniczatree: Add rebuild by overflow
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 15:19:54 +03:00
a61201a987 [#1337] config: Move rebuild_worker_count to shard section
This makes it simple to limit performance degradation for every shard
because of rebuild.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 13:57:27 +03:00
6b6eabe41c [#1337] cli: Add control shards rebuild command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 13:57:27 +03:00
d508da8397 [#1337] blobovniczatree: Add rebuild by fill percent
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 13:57:27 +03:00
007827255e [#1337] blobovniczatree: Add .rebuild temp files
This allows to reduce open/close DBs to check incompleted rebuilds.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-09-06 13:57:27 +03:00
f652518c24 [#1357] go: Fix panic caused by using range over slice len
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 1m12s
DCO action / DCO (pull_request) Successful in 1m28s
Pre-commit hooks / Pre-commit (pull_request) Successful in 1m53s
Tests and linters / Run gofumpt (pull_request) Successful in 1m55s
Build / Build Components (pull_request) Successful in 2m14s
Tests and linters / gopls check (pull_request) Successful in 2m22s
Tests and linters / Staticcheck (pull_request) Successful in 2m30s
Tests and linters / Lint (pull_request) Successful in 3m15s
Tests and linters / Tests (pull_request) Successful in 3m25s
Tests and linters / Tests with -race (pull_request) Successful in 3m31s
If slice is altered in `for` loop, we cannot use range over its
length: it may cause panic if slice gets shorter.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-06 13:09:58 +03:00
273980cfb9 [#1310] object: Remove irrelevant comments
Some checks failed
DCO action / DCO (pull_request) Successful in 51s
Vulncheck / Vulncheck (pull_request) Successful in 1m49s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m9s
Build / Build Components (pull_request) Successful in 2m23s
Tests and linters / Run gofumpt (pull_request) Successful in 2m15s
Tests and linters / gopls check (pull_request) Successful in 2m37s
Tests and linters / Staticcheck (pull_request) Successful in 3m6s
Tests and linters / Lint (pull_request) Successful in 3m45s
Tests and linters / Tests with -race (pull_request) Failing after 4m37s
Tests and linters / Tests (pull_request) Failing after 2m3s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-05 16:40:32 +03:00
108e4e07be [#1349] node: Evacuate objects without setting mode to MAINTENANCE
All checks were successful
DCO action / DCO (pull_request) Successful in 1m31s
Vulncheck / Vulncheck (pull_request) Successful in 1m32s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m12s
Tests and linters / Run gofumpt (pull_request) Successful in 2m21s
Build / Build Components (pull_request) Successful in 2m32s
Tests and linters / gopls check (pull_request) Successful in 2m36s
Tests and linters / Staticcheck (pull_request) Successful in 2m55s
Tests and linters / Tests with -race (pull_request) Successful in 3m26s
Tests and linters / Lint (pull_request) Successful in 3m31s
Tests and linters / Tests (pull_request) Successful in 3m40s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-09-05 16:08:27 +03:00
b3deb893ba [#1310] object: Move target initialization to separate package
* Split the logic of write target initialization to different packages;
* Refactor patch and put services: since both service initialize the target
  themselves.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-09-05 13:03:58 +00:00
7768a482b5 [#1223] lens/tui: Add TUI app for blobovnicza
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
371d97f61a [#1223] lens/tui: Add TUI app for write cache
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
e655336390 [#1223] lens/tui: Add app help
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
ed396448ac [#1223] lens/tui: Add TUI app to explore metabase
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
9cbd32bce8 [#1223] lens/tui: Add writecache schema
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
1ae86f35a8 [#1223] lens/tui: Add metabase schema
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
b9043433a0 [#1223] scripts: Add script to populate metabase
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-05 08:03:52 +00:00
a4fb7f085b
[#1348] go.mod: Update api-go and sdk-go
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 2m36s
Pre-commit hooks / Pre-commit (pull_request) Successful in 3m3s
Vulncheck / Vulncheck (pull_request) Successful in 2m50s
Tests and linters / Tests (1.22) (pull_request) Successful in 3m3s
DCO action / DCO (pull_request) Successful in 2m47s
Tests and linters / Lint (pull_request) Successful in 3m39s
Tests and linters / Staticcheck (pull_request) Successful in 3m37s
Tests and linters / Tests (1.23) (pull_request) Successful in 3m43s
Build / Build Components (1.22) (pull_request) Successful in 3m32s
Build / Build Components (1.23) (pull_request) Successful in 3m32s
Tests and linters / gopls check (pull_request) Successful in 4m10s
Tests and linters / Tests with -race (pull_request) Successful in 4m16s
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-04 10:47:26 +03:00
a685fcdc96 [#1317] go.mod: Use range over int
All checks were successful
DCO action / DCO (pull_request) Successful in 2m41s
Tests and linters / Run gofumpt (pull_request) Successful in 2m32s
Vulncheck / Vulncheck (pull_request) Successful in 2m38s
Build / Build Components (1.23) (pull_request) Successful in 3m0s
Build / Build Components (1.22) (pull_request) Successful in 3m3s
Pre-commit hooks / Pre-commit (pull_request) Successful in 3m33s
Tests and linters / Tests (1.22) (pull_request) Successful in 3m34s
Tests and linters / Tests (1.23) (pull_request) Successful in 3m36s
Tests and linters / Staticcheck (pull_request) Successful in 3m35s
Tests and linters / Lint (pull_request) Successful in 4m18s
Tests and linters / Tests with -race (pull_request) Successful in 4m20s
Tests and linters / gopls check (pull_request) Successful in 4m25s
Since Go 1.22 a "for" statement with a "range" clause is able
to iterate through integer values from zero to an upper limit.

gopatch script:
@@
var i, e expression
@@
-for i := 0; i <= e - 1; i++ {
+for i := range e {
    ...
}

@@
var i, e expression
@@
-for i := 0; i <= e; i++ {
+for i := range e + 1 {
    ...
}

@@
var i, e expression
@@
-for i := 0; i < e; i++ {
+for i := range e {
    ...
}

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-09-03 13:00:54 +03:00
2b3fc50681
[#1320] shard: Fix TestGCDropsObjectInhumedFromWritecache flaky test
All checks were successful
DCO action / DCO (pull_request) Successful in 1m27s
Tests and linters / Run gofumpt (pull_request) Successful in 2m25s
Build / Build Components (1.23) (pull_request) Successful in 2m46s
Vulncheck / Vulncheck (pull_request) Successful in 2m37s
Build / Build Components (1.22) (pull_request) Successful in 2m55s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m50s
Tests and linters / Staticcheck (pull_request) Successful in 3m33s
Tests and linters / Lint (pull_request) Successful in 3m45s
Tests and linters / Tests (1.22) (pull_request) Successful in 3m45s
Tests and linters / Tests (1.23) (pull_request) Successful in 3m42s
Tests and linters / Tests with -race (pull_request) Successful in 4m22s
Tests and linters / gopls check (pull_request) Successful in 4m25s
The `TestGCDropsObjectInhumedFromWritecache` test was flaky because a
running asynchronous rebuild operation prevented GC from deleting the
object. A test-only shard option `WithDisabledRebuild` has been added
to fix this.

Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-09-02 10:26:53 +03:00
98fe24cdb7 [#1343] go.mod: Update api-go
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-30 08:45:24 +00:00
882c068410 [#1334] metabase: Store upgrade flag
All checks were successful
DCO action / DCO (pull_request) Successful in 1m35s
Tests and linters / Run gofumpt (pull_request) Successful in 1m52s
Build / Build Components (1.22) (pull_request) Successful in 2m40s
Build / Build Components (1.23) (pull_request) Successful in 2m39s
Vulncheck / Vulncheck (pull_request) Successful in 2m20s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m53s
Tests and linters / Tests (1.22) (pull_request) Successful in 3m11s
Tests and linters / Tests (1.23) (pull_request) Successful in 3m13s
Tests and linters / Staticcheck (pull_request) Successful in 3m46s
Tests and linters / Tests with -race (pull_request) Successful in 4m16s
Tests and linters / Lint (pull_request) Successful in 4m23s
Tests and linters / gopls check (pull_request) Successful in 4m14s
This allows to check if metabase upgrade was not completed.

Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-30 09:06:20 +03:00
6c2146bbc1 [#1334] metabase: Add upgrade from v2 to v3
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-30 09:06:20 +03:00
03976c6ed5 [#1341] .golangci.yml: Replace exportloopref with copyloopvar
exportloopref is deprecated.
gopatch:
```
@@
var index, value identifier
var slice expression
@@
for index, value := range slice {
...
-value := value
...
}

@@
var index, value identifier
var slice expression
@@
for index, value := range slice {
...
-index := index
...
}

@@
var value identifier
var channel expression
@@
for value := range channel {
...
-value := value
...
}
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-28 15:44:41 +00:00
7e97df4878 [#1341] Makefile: Update golangci-lint
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-28 15:44:41 +00:00
01b6f1733c [#1341] Makefile: Build linter with -trimpath
Fix error with go1.23:
```
Error: build linters: unable to load custom analyzer "truecloudlab-linters": ../linters/bin/external_linters.so, plugin.Open("/repo/frostfs/linters/bin/external_linters"): plugin was built with a different version of package cmp
Failed executing command with error: build linters: unable to load custom analyzer "truecloudlab-linters": ../linters/bin/external_linters.so, plugin.Open("/repo/frostfs/linters/bin/external_linters"): plugin was built with a different version of package cmp
```

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-28 15:44:41 +00:00
7abbdca064 [#1340] getSvc: Fix access denied error handling
All checks were successful
DCO action / DCO (pull_request) Successful in 4m23s
Tests and linters / Run gofumpt (pull_request) Successful in 4m14s
Vulncheck / Vulncheck (pull_request) Successful in 4m33s
Build / Build Components (1.22) (pull_request) Successful in 4m56s
Build / Build Components (1.23) (pull_request) Successful in 4m59s
Tests and linters / Tests (1.22) (pull_request) Successful in 5m2s
Pre-commit hooks / Pre-commit (pull_request) Successful in 5m6s
Tests and linters / Tests (1.23) (pull_request) Successful in 5m8s
Tests and linters / Lint (pull_request) Successful in 5m51s
Tests and linters / gopls check (pull_request) Successful in 6m13s
Tests and linters / Tests with -race (pull_request) Successful in 6m19s
Tests and linters / Staticcheck (pull_request) Successful in 1m48s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-28 14:02:28 +03:00
6488ddee88 [#1338] object: Fix range provider in Patch handler
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 1m50s
DCO action / DCO (pull_request) Successful in 1m54s
Vulncheck / Vulncheck (pull_request) Successful in 2m29s
Tests and linters / Tests (1.23) (pull_request) Successful in 2m58s
Tests and linters / Tests (1.22) (pull_request) Successful in 2m59s
Tests and linters / Lint (pull_request) Successful in 3m17s
Pre-commit hooks / Pre-commit (pull_request) Successful in 3m3s
Build / Build Components (1.23) (pull_request) Successful in 3m12s
Build / Build Components (1.22) (pull_request) Successful in 3m15s
Tests and linters / Staticcheck (pull_request) Successful in 3m35s
Tests and linters / gopls check (pull_request) Successful in 3m52s
Tests and linters / Tests with -race (pull_request) Successful in 4m11s
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-27 16:51:12 +03:00
d6b42972a8 [#1338] object: Fix audit patch stream
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-27 16:18:29 +03:00
5e9a97fd3e [#1336] go.mod: Update api-go and sdk-go
All checks were successful
DCO action / DCO (pull_request) Successful in 5m13s
Build / Build Components (1.22) (pull_request) Successful in 6m30s
Build / Build Components (1.23) (pull_request) Successful in 6m31s
Pre-commit hooks / Pre-commit (pull_request) Successful in 6m59s
Tests and linters / Run gofumpt (pull_request) Successful in 57s
Tests and linters / Tests (1.22) (pull_request) Successful in 2m21s
Tests and linters / Tests (1.23) (pull_request) Successful in 2m19s
Tests and linters / Staticcheck (pull_request) Successful in 2m26s
Tests and linters / Lint (pull_request) Successful in 2m49s
Tests and linters / gopls check (pull_request) Successful in 3m6s
Tests and linters / Tests with -race (pull_request) Successful in 3m16s
Vulncheck / Vulncheck (pull_request) Successful in 45s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-27 14:48:43 +03:00
fa7f9fbce2 [#1333] go.mod: Update api-go
All checks were successful
DCO action / DCO (pull_request) Successful in 3m21s
Tests and linters / Run gofumpt (pull_request) Successful in 3m23s
Build / Build Components (1.22) (pull_request) Successful in 4m1s
Build / Build Components (1.23) (pull_request) Successful in 3m59s
Vulncheck / Vulncheck (pull_request) Successful in 4m24s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m49s
Tests and linters / Staticcheck (pull_request) Successful in 5m26s
Tests and linters / Tests (1.23) (pull_request) Successful in 5m30s
Tests and linters / Tests (1.22) (pull_request) Successful in 5m34s
Tests and linters / gopls check (pull_request) Successful in 5m45s
Tests and linters / Lint (pull_request) Successful in 5m58s
Tests and linters / Tests with -race (pull_request) Successful in 5m59s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-26 15:37:43 +03:00
806ea37101 [#1328] pilorama: Do not skip items in SortedByFilename
Benchmark results:
```
goos: linux
goarch: amd64
pkg: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/pilorama
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
                                   │     old     │                 new                 │
                                   │   sec/op    │   sec/op     vs base                │
ForestSortedIteration/bbolt,root-8   207.2µ ± 6%   173.6µ ± 6%  -16.23% (p=0.000 n=10)
ForestSortedIteration/bbolt,leaf-8   3.910µ ± 5%   3.928µ ± 7%        ~ (p=0.529 n=10)
geomean                              28.46µ        26.11µ        -8.27%
```

They are not representative, as the worst case is when we have multiple
items of different lengths. However, `FileName` is usually less than 100
in practice, so the asymptotics is the same.

Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-26 06:11:32 +00:00
80099d9a2f [#1328] pilorama: Add tricky test for SortedByFilename
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-26 06:11:32 +00:00
a059a7dcf0 [#1329] cli: Skip linking objects in complex object processing
All checks were successful
DCO action / DCO (pull_request) Successful in 2m41s
Pre-commit hooks / Pre-commit (pull_request) Successful in 3m42s
Tests and linters / Run gofumpt (pull_request) Successful in 3m50s
Vulncheck / Vulncheck (pull_request) Successful in 3m55s
Tests and linters / Tests (1.23) (pull_request) Successful in 4m8s
Tests and linters / Tests (1.22) (pull_request) Successful in 4m11s
Build / Build Components (1.22) (pull_request) Successful in 4m32s
Tests and linters / Tests with -race (pull_request) Successful in 4m20s
Build / Build Components (1.23) (pull_request) Successful in 4m33s
Tests and linters / Staticcheck (pull_request) Successful in 5m9s
Tests and linters / Lint (pull_request) Successful in 5m34s
Tests and linters / gopls check (pull_request) Successful in 5m24s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-23 13:51:20 +03:00
bd24beecf8 [#1329] putSvc: Reset SuccessAfter for non-EC objects in EC container broadcasting
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-23 13:51:16 +03:00
dfe825b81b [#1309] test: Clean up config after tests
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-08-22 08:49:10 +00:00
76f67ea34e [#1323] metabase: Bump version
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
7d0d781db1 [#1323] metabase: Drop user attribute index
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
0f08a2efba [#1323] metabase: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
7bf20c9f1f [#1323] metabase: Add expiration epoch buckets
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
2542d4f5df [#1323] metabase: Drop payload checksum index
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
15dae8685e [#1323] metabase: Drop ownerID index
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-22 08:21:40 +00:00
7bca428db0 [#1322] Use new protobuf marshaler
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-08-22 07:17:41 +00:00
a345c972bf [#1316] lint: Fix warnings
All checks were successful
DCO action / DCO (pull_request) Successful in 1m43s
Build / Build Components (1.22) (pull_request) Successful in 2m24s
Vulncheck / Vulncheck (pull_request) Successful in 2m12s
Build / Build Components (1.23) (pull_request) Successful in 2m36s
Tests and linters / Run gofumpt (pull_request) Successful in 2m21s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m48s
Tests and linters / Tests (1.23) (pull_request) Successful in 2m56s
Tests and linters / Staticcheck (pull_request) Successful in 3m8s
Tests and linters / Lint (pull_request) Successful in 3m26s
Tests and linters / Tests with -race (pull_request) Successful in 3m38s
Tests and linters / gopls check (pull_request) Successful in 3m54s
Tests and linters / Tests (1.22) (pull_request) Successful in 1m28s
Renamed parameters `min/max` to avoid conflicts with
predeclared identifiers.

Replaced background context with parent context without
cancellation in closer functions in frostfs-node.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-21 18:18:25 +03:00
8c1082b31a [#1316] go.mod: Bump go version to 1.22
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-21 15:09:29 +03:00
cfda9003a7 [#1318] meta: Add test TestInhumeECObject
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 2m37s
DCO action / DCO (pull_request) Successful in 2m34s
Vulncheck / Vulncheck (pull_request) Successful in 2m57s
Build / Build Components (1.22) (pull_request) Successful in 3m27s
Pre-commit hooks / Pre-commit (pull_request) Successful in 3m48s
Build / Build Components (1.21) (pull_request) Successful in 3m38s
Tests and linters / Tests (1.22) (pull_request) Successful in 4m0s
Tests and linters / Tests (1.21) (pull_request) Successful in 4m5s
Tests and linters / Staticcheck (pull_request) Successful in 4m7s
Tests and linters / Lint (pull_request) Successful in 4m20s
Tests and linters / Tests with -race (pull_request) Successful in 4m37s
Tests and linters / gopls check (pull_request) Successful in 4m52s
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-20 16:21:27 +03:00
6ff0b0996b [#1318] metrics: Fix container_size_bytes for EC
All checks were successful
DCO action / DCO (pull_request) Successful in 1m41s
Vulncheck / Vulncheck (pull_request) Successful in 1m47s
Build / Build Components (1.22) (pull_request) Successful in 2m15s
Build / Build Components (1.21) (pull_request) Successful in 2m18s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m24s
Tests and linters / Run gofumpt (pull_request) Successful in 1m4s
Tests and linters / Tests (1.21) (pull_request) Successful in 2m9s
Tests and linters / Staticcheck (pull_request) Successful in 3m23s
Tests and linters / Lint (pull_request) Successful in 3m52s
Tests and linters / gopls check (pull_request) Successful in 3m43s
Tests and linters / Tests with -race (pull_request) Successful in 3m47s
Tests and linters / Tests (1.22) (pull_request) Successful in 1m33s
When node put chunk into EC container, `policer` may remove it as redundant.
This chunk marked as removed. When parent object removed and `gc` start iterating over chunk,
node count removing chunk twice.

Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-20 15:33:28 +03:00
8319b59238 [#1318] Fix gofumpt issue
Signed-off-by: Anton Nikiforov <an.nikiforov@yadro.com>
2024-08-20 14:34:54 +03:00
b7acb34fa4 [#1319] treeSvc: Do not wrap error from APE
All checks were successful
Tests and linters / Run gofumpt (pull_request) Successful in 2m43s
DCO action / DCO (pull_request) Successful in 2m56s
Tests and linters / Tests (1.21) (pull_request) Successful in 4m24s
Vulncheck / Vulncheck (pull_request) Successful in 4m12s
Tests and linters / Tests (1.22) (pull_request) Successful in 4m23s
Tests and linters / Staticcheck (pull_request) Successful in 4m18s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m32s
Tests and linters / Tests with -race (pull_request) Successful in 4m22s
Build / Build Components (1.21) (pull_request) Successful in 4m52s
Build / Build Components (1.22) (pull_request) Successful in 4m50s
Tests and linters / Lint (pull_request) Successful in 4m55s
Tests and linters / gopls check (pull_request) Successful in 4m59s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-19 18:38:27 +03:00
41104f2383 [#1307] cli: Make cli process object.patch
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
eeab417dcf [#1307] object: Add APE check for Patch handler
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
5ed317e24c [#1307] cli: Introduce object patch command
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
e890f1b4b1 [#1307] object: Implement Patch method
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
a4a1c3f18b [#1307] go.mod: Bump frostfs-sdk-go/frostfs-api-go/v2 versions
* Also, resolve dependencies and conflicts for object service
  by creating stub for `Patch` method.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-08-16 14:13:09 +00:00
ec1509de4e [#1262] sdnotify: Send system monotonic time on reload
All checks were successful
DCO action / DCO (pull_request) Successful in 1m40s
Tests and linters / Run gofumpt (pull_request) Successful in 1m37s
Vulncheck / Vulncheck (pull_request) Successful in 2m12s
Build / Build Components (1.22) (pull_request) Successful in 2m33s
Build / Build Components (1.21) (pull_request) Successful in 2m35s
Tests and linters / Tests (1.21) (pull_request) Successful in 2m34s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m45s
Tests and linters / Tests (1.22) (pull_request) Successful in 3m2s
Tests and linters / Staticcheck (pull_request) Successful in 3m3s
Tests and linters / Lint (pull_request) Successful in 3m27s
Tests and linters / Tests with -race (pull_request) Successful in 3m42s
Tests and linters / gopls check (pull_request) Successful in 3m51s
The synchronized service reload protocol added in systemd version 253
requires that the service provides a MONOTONIC_USEC field alongside the
RELOADING=1 notification message for synchronization purposes. The value
carried in this field must be the system CLOCK_MONOTONIC timestamp at
the time the notification message was generated as systemd compares it
to other CLOCK_MONOTONIC timestamps taken by pid1.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-16 12:05:31 +03:00
5da41f1fe5 Revert "[#1262] sdnotify: Get rid of go:linkname for nanotime"
All checks were successful
DCO action / DCO (pull_request) Successful in 1m22s
Tests and linters / Run gofumpt (pull_request) Successful in 1m28s
Build / Build Components (1.21) (pull_request) Successful in 2m25s
Build / Build Components (1.22) (pull_request) Successful in 2m23s
Vulncheck / Vulncheck (pull_request) Successful in 2m9s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m45s
Tests and linters / Staticcheck (pull_request) Successful in 3m1s
Tests and linters / Tests (1.22) (pull_request) Successful in 3m5s
Tests and linters / Tests (1.21) (pull_request) Successful in 3m10s
Tests and linters / Lint (pull_request) Successful in 3m24s
Tests and linters / Tests with -race (pull_request) Successful in 3m29s
Tests and linters / gopls check (pull_request) Successful in 3m39s
This reverts commit 327d364f34.

Reverted due to the problem with reload signal sent by systemd.
`frostfs-ir` service reconfigures correctly and service's
statuses are being reported to systemd. However, since we
replaced `go:linkname` & `nanotime()` with `time.Since()`,
systemd refuses to accept reload signal response from
`frostfs-ir`. To maintain correct behaviour it was decided to
revevrt systemd-related changes until a better solution is
found.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-15 16:54:45 +03:00
d055168e2a [#1135] ir: Add healthstatus RECONFIGURING
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-15 16:54:45 +03:00
80ce7c3a00 [#1284] shard: Resolve funlen linter
All checks were successful
DCO action / DCO (pull_request) Successful in 1m34s
Vulncheck / Vulncheck (pull_request) Successful in 2m0s
Tests and linters / Run gofumpt (pull_request) Successful in 2m14s
Build / Build Components (1.22) (pull_request) Successful in 2m41s
Build / Build Components (1.21) (pull_request) Successful in 2m43s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m50s
Tests and linters / Tests (1.21) (pull_request) Successful in 2m51s
Tests and linters / Tests (1.22) (pull_request) Successful in 2m51s
Tests and linters / Tests with -race (pull_request) Successful in 2m56s
Tests and linters / Staticcheck (pull_request) Successful in 2m57s
Tests and linters / Lint (pull_request) Successful in 3m27s
Tests and linters / gopls check (pull_request) Successful in 3m56s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 13:23:35 +03:00
93d63e1632 [#1284] writecache: Allow to seal writecache async
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 13:23:35 +03:00
68029d756e [#1302] writecache: Allow to specify custom page size
All checks were successful
DCO action / DCO (pull_request) Successful in 1m57s
Vulncheck / Vulncheck (pull_request) Successful in 2m6s
Tests and linters / Run gofumpt (pull_request) Successful in 2m9s
Build / Build Components (1.22) (pull_request) Successful in 2m37s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m49s
Tests and linters / Tests (1.21) (pull_request) Successful in 3m1s
Tests and linters / Tests (1.22) (pull_request) Successful in 3m4s
Tests and linters / Staticcheck (pull_request) Successful in 3m0s
Tests and linters / Tests with -race (pull_request) Successful in 3m9s
Tests and linters / Lint (pull_request) Successful in 3m30s
Tests and linters / gopls check (pull_request) Successful in 3m40s
Build / Build Components (1.21) (pull_request) Successful in 1m12s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 12:12:07 +03:00
fa82854af4 [#1302] writecache: Add put->flush->put benchmark
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 12:12:07 +03:00
c985b1198f [#1302] putSvc: Override SuccessAfter for non-regular objects in EC containers
All checks were successful
DCO action / DCO (pull_request) Successful in 3m16s
Vulncheck / Vulncheck (pull_request) Successful in 3m21s
Tests and linters / Run gofumpt (pull_request) Successful in 3m35s
Build / Build Components (1.21) (pull_request) Successful in 4m6s
Build / Build Components (1.22) (pull_request) Successful in 4m4s
Pre-commit hooks / Pre-commit (pull_request) Successful in 4m5s
Tests and linters / Tests (1.22) (pull_request) Successful in 4m1s
Tests and linters / Tests (1.21) (pull_request) Successful in 4m6s
Tests and linters / Staticcheck (pull_request) Successful in 4m15s
Tests and linters / Tests with -race (pull_request) Successful in 4m37s
Tests and linters / Lint (pull_request) Successful in 4m48s
Tests and linters / gopls check (pull_request) Successful in 4m55s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 10:42:55 +03:00
08b1f18bca [#1296] writecache: Add count limit
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-09 06:30:32 +00:00
36efccd862 [#1298] writecache: Add shrink flag for Seal command
All checks were successful
DCO action / DCO (pull_request) Successful in 1m23s
Vulncheck / Vulncheck (pull_request) Successful in 1m48s
Tests and linters / Run gofumpt (pull_request) Successful in 1m54s
Build / Build Components (1.22) (pull_request) Successful in 2m23s
Build / Build Components (1.21) (pull_request) Successful in 2m28s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m53s
Tests and linters / Tests (1.21) (pull_request) Successful in 2m53s
Tests and linters / Staticcheck (pull_request) Successful in 2m48s
Tests and linters / Tests (1.22) (pull_request) Successful in 2m54s
Tests and linters / Tests with -race (pull_request) Successful in 2m58s
Tests and linters / Lint (pull_request) Successful in 3m24s
Tests and linters / gopls check (pull_request) Successful in 3m36s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-08 16:32:29 +03:00
5c01bd5be8 [#1298] writecache: Add restore-mode flag for Seal command
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-07 11:20:14 +03:00
8e51d7849a [#1295] getSvc: Assemble complex EC object headers without linking object
All checks were successful
DCO action / DCO (pull_request) Successful in 1m36s
Build / Build Components (1.21) (pull_request) Successful in 2m23s
Vulncheck / Vulncheck (pull_request) Successful in 2m7s
Tests and linters / Run gofumpt (pull_request) Successful in 2m11s
Build / Build Components (1.22) (pull_request) Successful in 2m40s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m49s
Tests and linters / Staticcheck (pull_request) Successful in 2m56s
Tests and linters / Tests (1.21) (pull_request) Successful in 3m5s
Tests and linters / Tests (1.22) (pull_request) Successful in 3m5s
Tests and linters / Tests with -race (pull_request) Successful in 3m8s
Tests and linters / Lint (pull_request) Successful in 3m35s
Tests and linters / gopls check (pull_request) Successful in 4m4s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-06 16:48:12 +03:00
10602b55b1 [#1295] engine: Resolve funlen linter
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-06 09:45:47 +03:00
eeca796d2e [#1295] engine: Log object address in case of error
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-06 09:45:43 +03:00
327d364f34 [#1262] sdnotify: Get rid of go:linkname for nanotime
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-05 12:37:16 +00:00
dc3dcabadc [#1291] morph: Reconnect to the highest priority endpoint
All checks were successful
DCO action / DCO (pull_request) Successful in 3m13s
Vulncheck / Vulncheck (pull_request) Successful in 5m3s
Tests and linters / Run gofumpt (pull_request) Successful in 4m49s
Tests and linters / Staticcheck (pull_request) Successful in 5m20s
Tests and linters / Tests (1.22) (pull_request) Successful in 5m25s
Pre-commit hooks / Pre-commit (pull_request) Successful in 5m40s
Tests and linters / Tests (1.21) (pull_request) Successful in 5m39s
Tests and linters / Tests with -race (pull_request) Successful in 5m35s
Tests and linters / Lint (pull_request) Successful in 5m46s
Build / Build Components (1.22) (pull_request) Successful in 6m8s
Build / Build Components (1.21) (pull_request) Successful in 6m11s
Tests and linters / gopls check (pull_request) Successful in 6m21s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-02 17:54:22 +03:00
8021bacc43 [#1288] putSvc: Respect TTL for EC put
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-02 13:01:42 +00:00
ef4cea6d19 [#1266] .forgejo: Add gofumpt action
All checks were successful
DCO action / DCO (pull_request) Successful in 1m51s
Vulncheck / Vulncheck (pull_request) Successful in 2m14s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m44s
Tests and linters / Run gofumpt (pull_request) Successful in 2m36s
Tests and linters / Tests (1.22) (pull_request) Successful in 2m47s
Tests and linters / Staticcheck (pull_request) Successful in 2m46s
Build / Build Components (1.22) (pull_request) Successful in 2m45s
Build / Build Components (1.21) (pull_request) Successful in 2m48s
Tests and linters / Tests with -race (pull_request) Successful in 3m0s
Tests and linters / Tests (1.21) (pull_request) Successful in 3m15s
Tests and linters / Lint (pull_request) Successful in 3m25s
Tests and linters / gopls check (pull_request) Successful in 3m59s
`gofumpt` was skipped by pre-commit on CI, and now is used
in a separate action.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-02 12:34:04 +03:00
a55600893e [#1266] Makefile: Specify gofumpt version
Add target to install gofumpt, fix target to run gofumpt.

Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-08-02 12:31:59 +03:00
c49982d22a [#1282] cli: Allow to external addresses first for object nodes
Some checks failed
DCO action / DCO (pull_request) Successful in 1m18s
Vulncheck / Vulncheck (pull_request) Successful in 2m15s
Pre-commit hooks / Pre-commit (pull_request) Successful in 2m39s
Build / Build Components (1.22) (pull_request) Successful in 3m20s
Tests and linters / Staticcheck (pull_request) Successful in 3m34s
Tests and linters / Lint (pull_request) Successful in 3m42s
Build / Build Components (1.21) (pull_request) Successful in 3m32s
Tests and linters / Tests with -race (pull_request) Successful in 3m54s
Tests and linters / gopls check (pull_request) Successful in 3m57s
Tests and linters / Tests (1.22) (pull_request) Successful in 1m26s
Tests and linters / Tests (1.21) (pull_request) Failing after 50s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-08-01 09:16:19 +03:00
7e04083c27 [#1278] containerSvc: Validate FrostFSID subject exitence on Put
All checks were successful
DCO action / DCO (pull_request) Successful in 1m52s
Vulncheck / Vulncheck (pull_request) Successful in 2m18s
Build / Build Components (1.21) (pull_request) Successful in 2m52s
Tests and linters / Lint (pull_request) Successful in 2m56s
Build / Build Components (1.22) (pull_request) Successful in 2m32s
Tests and linters / Staticcheck (pull_request) Successful in 3m2s
Tests and linters / gopls check (pull_request) Successful in 3m25s
Tests and linters / Tests (1.22) (pull_request) Successful in 5m16s
Pre-commit hooks / Pre-commit (pull_request) Successful in 5m3s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m5s
Tests and linters / Tests with -race (pull_request) Successful in 8m53s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-29 16:01:37 +03:00
a12c39667d [#1278] ir: Do not allow to create container without FrostFSID record
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-29 16:01:37 +03:00
85a77b7c21 [#1279] adm: Interpret "root" name as empty for namespace target type
Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-29 12:57:40 +00:00
8377372a40 [#1276] go.mod: Update api-go
All checks were successful
Tests and linters / Staticcheck (pull_request) Successful in 2m54s
Build / Build Components (1.22) (pull_request) Successful in 3m28s
Build / Build Components (1.21) (pull_request) Successful in 3m16s
Tests and linters / Lint (pull_request) Successful in 4m14s
Tests and linters / Tests with -race (pull_request) Successful in 8m2s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m41s
Tests and linters / Tests (1.22) (pull_request) Successful in 8m54s
Tests and linters / gopls check (pull_request) Successful in 2m53s
DCO action / DCO (pull_request) Successful in 46s
Vulncheck / Vulncheck (pull_request) Successful in 1m4s
Pre-commit hooks / Pre-commit (pull_request) Successful in 1m47s
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-26 16:44:19 +03:00
dd459d399f [#1274] go.mod: Update neo-go version that fixes ws-client
All checks were successful
DCO action / DCO (pull_request) Successful in 1m1s
Vulncheck / Vulncheck (pull_request) Successful in 1m27s
Pre-commit hooks / Pre-commit (pull_request) Successful in 3m8s
Tests and linters / gopls check (pull_request) Successful in 3m12s
Build / Build Components (1.21) (pull_request) Successful in 3m27s
Build / Build Components (1.22) (pull_request) Successful in 3m43s
Tests and linters / Tests (1.22) (pull_request) Successful in 3m39s
Tests and linters / Staticcheck (pull_request) Successful in 3m38s
Tests and linters / Lint (pull_request) Successful in 4m29s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m2s
Tests and linters / Tests with -race (pull_request) Successful in 9m10s
* Update go.mod;
* This neo-go package version contains fix for the wsclient that
  allows to morph event listener refresh the invalidated websocket
  connection to neo-go.

Signed-off-by: Airat Arifullin <a.arifullin@yadro.com>
2024-07-26 14:21:19 +03:00
7fd7961dfa [#1271] getSvc: Fix local EC chunk get
All checks were successful
Build / Build Components (1.21) (pull_request) Successful in 4m22s
Tests and linters / Staticcheck (pull_request) Successful in 4m56s
Tests and linters / gopls check (pull_request) Successful in 5m59s
Tests and linters / Tests (1.21) (pull_request) Successful in 8m20s
DCO action / DCO (pull_request) Successful in 35s
Build / Build Components (1.22) (pull_request) Successful in 1m15s
Vulncheck / Vulncheck (pull_request) Successful in 59s
Tests and linters / Lint (pull_request) Successful in 2m24s
Pre-commit hooks / Pre-commit (pull_request) Successful in 3m1s
Tests and linters / Tests (1.22) (pull_request) Successful in 6m59s
Tests and linters / Tests with -race (pull_request) Successful in 7m32s
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-24 08:52:16 +03:00
8398a8b4b3 [#1271] getSvc: Fix head --raw assemble for EC
Signed-off-by: Dmitrii Stepanov <d.stepanov@yadro.com>
2024-07-24 08:52:08 +03:00
9c2c76ca32
[#1259] Move pkg/util/locode to frostfs-locode-db
All checks were successful
Vulncheck / Vulncheck (pull_request) Successful in 4m47s
DCO action / DCO (pull_request) Successful in 4m40s
Build / Build Components (1.21) (pull_request) Successful in 7m27s
Tests and linters / Staticcheck (pull_request) Successful in 8m22s
Tests and linters / gopls check (pull_request) Successful in 10m56s
Build / Build Components (1.22) (pull_request) Successful in 11m48s
Tests and linters / Tests (1.22) (pull_request) Successful in 16m11s
Tests and linters / Lint (pull_request) Successful in 16m27s
Pre-commit hooks / Pre-commit (pull_request) Successful in 19m12s
Tests and linters / Tests (1.21) (pull_request) Successful in 20m31s
Tests and linters / Tests with -race (pull_request) Successful in 8m56s
Removed pkg/util/locode package, added
git.frostfs.info/TrueCloudLab/frostfs-locode-db/pkg/locode dependency.

Signed-off-by: George Bartolomey <george@bh4.ru>
2024-07-23 14:59:14 +03:00
1032075a21
[#1259] cli: Remove locode subcommand
Removed `frostfs-cli util locode` subcommand.
Alternative command could be found in
`git.frostfs.info/TrueCloudLab/frostfs-locode-db`.

Signed-off-by: George Bartolomey <george@bh4.ru>
2024-07-23 14:59:14 +03:00
94f07b4778 [#1245] docker: Fix warnings
All checks were successful
Build / Build Components (1.21) (pull_request) Successful in 2m14s
DCO action / DCO (pull_request) Successful in 2m42s
Build / Build Components (1.22) (pull_request) Successful in 3m49s
Vulncheck / Vulncheck (pull_request) Successful in 3m53s
Tests and linters / Staticcheck (pull_request) Successful in 5m2s
Tests and linters / gopls check (pull_request) Successful in 5m9s
Tests and linters / Lint (pull_request) Successful in 6m9s
Pre-commit hooks / Pre-commit (pull_request) Successful in 8m56s
Tests and linters / Tests (1.21) (pull_request) Successful in 9m35s
Tests and linters / Tests (1.22) (pull_request) Successful in 9m47s
Tests and linters / Tests with -race (pull_request) Successful in 9m58s
Signed-off-by: Ekaterina Lebedeva <ekaterina.lebedeva@yadro.com>
2024-07-23 12:36:11 +03:00
7ddba70030 [#1264] go.mod: Update dependencies
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-23 08:42:29 +00:00
658e3cb92f [#1264] go.mod: Update bbolt to v1.3.10
Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
2024-07-23 08:42:29 +00:00
9ee1bd3669
[#1265] *: Run gofumpt
All checks were successful
DCO action / DCO (pull_request) Successful in 7m50s
Vulncheck / Vulncheck (pull_request) Successful in 11m12s
Build / Build Components (1.22) (pull_request) Successful in 15m55s
Build / Build Components (1.21) (pull_request) Successful in 16m2s
Pre-commit hooks / Pre-commit (pull_request) Successful in 20m47s
Tests and linters / Staticcheck (pull_request) Successful in 4m8s
Tests and linters / gopls check (pull_request) Successful in 4m16s
Tests and linters / Lint (pull_request) Successful in 5m35s
Tests and linters / Tests with -race (pull_request) Successful in 7m58s
Tests and linters / Tests (1.22) (pull_request) Successful in 8m12s
Tests and linters / Tests (1.21) (pull_request) Successful in 2m59s
Signed-off-by: Aleksey Savchuk <a.savchuk@yadro.com>
2024-07-23 10:36:32 +03:00
405 changed files with 10408 additions and 6330 deletions

View file

@ -1,4 +1,4 @@
FROM golang:1.22 as builder
FROM golang:1.22 AS builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository

View file

@ -1,4 +1,4 @@
FROM golang:1.22 as builder
FROM golang:1.22 AS builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository

View file

@ -1,4 +1,4 @@
FROM golang:1.22 as builder
FROM golang:1.22 AS builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository

View file

@ -1,4 +1,4 @@
FROM golang:1.22 as builder
FROM golang:1.22 AS builder
ARG BUILD=now
ARG VERSION=dev
ARG REPO=repository

View file

@ -8,7 +8,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
go_versions: [ '1.21', '1.22' ]
go_versions: [ '1.22', '1.23' ]
steps:
- uses: actions/checkout@v3

View file

@ -16,7 +16,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: 1.22
go-version: 1.23
- name: Set up Python
run: |
apt update

View file

@ -11,7 +11,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.22'
go-version: '1.23'
cache: true
- name: Install linters
@ -25,7 +25,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
go_versions: [ '1.21', '1.22' ]
go_versions: [ '1.22', '1.23' ]
fail-fast: false
steps:
- uses: actions/checkout@v3
@ -48,7 +48,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
go-version: '1.22'
cache: true
- name: Run tests
@ -63,7 +63,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.22'
go-version: '1.23'
cache: true
- name: Install staticcheck
@ -81,7 +81,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.21'
go-version: '1.22'
cache: true
- name: Install gopls
@ -89,3 +89,23 @@ jobs:
- name: Run gopls
run: make gopls-run
fumpt:
name: Run gofumpt
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '1.23'
cache: true
- name: Install gofumpt
run: make fumpt-install
- name: Run gofumpt
run: |
make fumpt
git diff --exit-code --quiet

View file

@ -13,7 +13,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22'
go-version: '1.23'
- name: Install govulncheck
run: go install golang.org/x/vuln/cmd/govulncheck@latest

View file

@ -12,7 +12,8 @@ run:
# output configuration options
output:
# colored-line-number|line-number|json|tab|checkstyle|code-climate, default is "colored-line-number"
format: tab
formats:
- format: tab
# all available settings of specific linters
linters-settings:
@ -66,7 +67,7 @@ linters:
- bidichk
- durationcheck
- exhaustive
- exportloopref
- copyloopvar
- gofmt
- goimports
- misspell

View file

@ -4,20 +4,19 @@ SHELL = bash
REPO ?= $(shell go list -m)
VERSION ?= $(shell git describe --tags --dirty --match "v*" --always --abbrev=8 2>/dev/null || cat VERSION 2>/dev/null || echo "develop")
HUB_IMAGE ?= truecloudlab/frostfs
HUB_IMAGE ?= git.frostfs.info/truecloudlab/frostfs
HUB_TAG ?= "$(shell echo ${VERSION} | sed 's/^v//')"
GO_VERSION ?= 1.22
LINT_VERSION ?= 1.56.1
TRUECLOUDLAB_LINT_VERSION ?= 0.0.5
LINT_VERSION ?= 1.60.3
TRUECLOUDLAB_LINT_VERSION ?= 0.0.7
PROTOC_VERSION ?= 25.0
PROTOC_GEN_GO_VERSION ?= $(shell go list -f '{{.Version}}' -m google.golang.org/protobuf)
PROTOGEN_FROSTFS_VERSION ?= $(shell go list -f '{{.Version}}' -m git.frostfs.info/TrueCloudLab/frostfs-api-go/v2)
PROTOC_OS_VERSION=osx-x86_64
ifeq ($(shell uname), Linux)
PROTOC_OS_VERSION=linux-x86_64
endif
STATICCHECK_VERSION ?= 2023.1.6
STATICCHECK_VERSION ?= 2024.1.1
ARCH = amd64
BIN = bin
@ -39,13 +38,16 @@ LINT_DIR = $(OUTPUT_LINT_DIR)/golangci-lint-$(LINT_VERSION)-v$(TRUECLOUDLAB_LINT
TMP_DIR := .cache
PROTOBUF_DIR ?= $(abspath $(BIN))/protobuf
PROTOC_DIR ?= $(PROTOBUF_DIR)/protoc-v$(PROTOC_VERSION)
PROTOC_GEN_GO_DIR ?= $(PROTOBUF_DIR)/protoc-gen-go-$(PROTOC_GEN_GO_VERSION)
PROTOGEN_FROSTFS_DIR ?= $(PROTOBUF_DIR)/protogen-$(PROTOGEN_FROSTFS_VERSION)
STATICCHECK_DIR ?= $(abspath $(BIN))/staticcheck
STATICCHECK_VERSION_DIR ?= $(STATICCHECK_DIR)/$(STATICCHECK_VERSION)
SOURCES = $(shell find . -type f -name "*.go" -print)
GOFUMPT_VERSION ?= v0.7.0
GOFUMPT_DIR ?= $(abspath $(BIN))/gofumpt
GOFUMPT_VERSION_DIR ?= $(GOFUMPT_DIR)/$(GOFUMPT_VERSION)
GOPLS_VERSION ?= v0.15.1
GOPLS_DIR ?= $(abspath $(BIN))/gopls
GOPLS_VERSION_DIR ?= $(GOPLS_DIR)/$(GOPLS_VERSION)
@ -103,17 +105,15 @@ export-metrics: dep
# Regenerate proto files:
protoc:
@if [ ! -d "$(PROTOC_DIR)" ] || [ ! -d "$(PROTOC_GEN_GO_DIR)" ] || [ ! -d "$(PROTOGEN_FROSTFS_DIR)" ]; then \
@if [ ! -d "$(PROTOC_DIR)" ] || [ ! -d "$(PROTOGEN_FROSTFS_DIR)" ]; then \
make protoc-install; \
fi
@for f in `find . -type f -name '*.proto' -not -path './bin/*'`; do \
echo "⇒ Processing $$f "; \
$(PROTOC_DIR)/bin/protoc \
--proto_path=.:$(PROTOC_DIR)/include:/usr/local/include \
--plugin=protoc-gen-go=$(PROTOC_GEN_GO_DIR)/protoc-gen-go \
--plugin=protoc-gen-go-frostfs=$(PROTOGEN_FROSTFS_DIR)/protogen \
--go-frostfs_out=. --go-frostfs_opt=paths=source_relative \
--go_out=. --go_opt=paths=source_relative \
--go-grpc_opt=require_unimplemented_servers=false \
--go-grpc_out=. --go-grpc_opt=paths=source_relative $$f; \
done
@ -126,8 +126,6 @@ protoc-install:
@wget -q -O $(PROTOBUF_DIR)/protoc-$(PROTOC_VERSION).zip 'https://github.com/protocolbuffers/protobuf/releases/download/v$(PROTOC_VERSION)/protoc-$(PROTOC_VERSION)-$(PROTOC_OS_VERSION).zip'
@unzip -q -o $(PROTOBUF_DIR)/protoc-$(PROTOC_VERSION).zip -d $(PROTOC_DIR)
@rm $(PROTOBUF_DIR)/protoc-$(PROTOC_VERSION).zip
@echo "⇒ Installing protoc-gen-go..."
@GOBIN=$(PROTOC_GEN_GO_DIR) go install -v google.golang.org/protobuf/...@$(PROTOC_GEN_GO_VERSION)
@echo "⇒ Instaling protogen FrostFS plugin..."
@GOBIN=$(PROTOGEN_FROSTFS_DIR) go install -mod=mod -v git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/util/protogen@$(PROTOGEN_FROSTFS_VERSION)
@ -165,10 +163,19 @@ imports:
@echo "⇒ Processing goimports check"
@goimports -w cmd/ pkg/ misc/
# Install gofumpt
fumpt-install:
@rm -rf $(GOFUMPT_DIR)
@mkdir $(GOFUMPT_DIR)
@GOBIN=$(GOFUMPT_VERSION_DIR) go install mvdan.cc/gofumpt@$(GOFUMPT_VERSION)
# Run gofumpt
fumpt:
@if [ ! -d "$(GOFUMPT_VERSION_DIR)" ]; then \
make fumpt-install; \
fi
@echo "⇒ Processing gofumpt check"
@gofumpt -l -w cmd/ pkg/ misc/
$(GOFUMPT_VERSION_DIR)/gofumpt -l -w cmd/ pkg/ misc/
# Run Unit Test with go test
test: GOFLAGS ?= "-count=1"
@ -190,7 +197,7 @@ lint-install:
@@make -C $(TMP_DIR)/linters lib CGO_ENABLED=1 OUT_DIR=$(OUTPUT_LINT_DIR)
@rm -rf $(TMP_DIR)/linters
@rmdir $(TMP_DIR) 2>/dev/null || true
@CGO_ENABLED=1 GOBIN=$(LINT_DIR) go install github.com/golangci/golangci-lint/cmd/golangci-lint@v$(LINT_VERSION)
@CGO_ENABLED=1 GOBIN=$(LINT_DIR) go install -trimpath github.com/golangci/golangci-lint/cmd/golangci-lint@v$(LINT_VERSION)
# Run linters
lint:

View file

@ -7,9 +7,8 @@
</p>
---
[![Report](https://goreportcard.com/badge/github.com/TrueCloudLab/frostfs-node)](https://goreportcard.com/report/github.com/TrueCloudLab/frostfs-node)
![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/TrueCloudLab/frostfs-node?sort=semver)
![License](https://img.shields.io/github/license/TrueCloudLab/frostfs-node.svg?style=popout)
[![Report](https://goreportcard.com/badge/git.frostfs.info/TrueCloudLab/frostfs-node)](https://goreportcard.com/report/git.frostfs.info/TrueCloudLab/frostfs-node)
![Release (latest)](https://git.frostfs.info/TrueCloudLab/frostfs-node/badges/release.svg)
# Overview
@ -33,8 +32,8 @@ manipulate large amounts of data without paying a prohibitive price.
FrostFS has a native [gRPC API](https://git.frostfs.info/TrueCloudLab/frostfs-api) and has
protocol gateways for popular protocols such as [AWS
S3](https://github.com/TrueCloudLab/frostfs-s3-gw),
[HTTP](https://github.com/TrueCloudLab/frostfs-http-gw),
S3](https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw),
[HTTP](https://git.frostfs.info/TrueCloudLab/frostfs-http-gw),
[FUSE](https://wikipedia.org/wiki/Filesystem_in_Userspace) and
[sFTP](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol) allowing
developers to integrate applications without rewriting their code.
@ -45,11 +44,11 @@ Now, we only support GNU/Linux on amd64 CPUs with AVX/AVX2 instructions. More
platforms will be officially supported after release `1.0`.
The latest version of frostfs-node works with frostfs-contract
[v0.16.0](https://github.com/TrueCloudLab/frostfs-contract/releases/tag/v0.16.0).
[v0.19.2](https://git.frostfs.info/TrueCloudLab/frostfs-contract/releases/tag/v0.19.2).
# Building
To make all binaries you need Go 1.21+ and `make`:
To make all binaries you need Go 1.22+ and `make`:
```
make all
```
@ -71,7 +70,7 @@ make docker/bin/frostfs-<name> # build a specific binary
## Docker images
To make docker images suitable for use in [frostfs-dev-env](https://github.com/TrueCloudLab/frostfs-dev-env/) use:
To make docker images suitable for use in [frostfs-dev-env](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env/) use:
```
make images
```
@ -125,7 +124,7 @@ the feature/topic you are going to implement.
# Credits
FrostFS is maintained by [True Cloud Lab](https://github.com/TrueCloudLab/) with the help and
FrostFS is maintained by [True Cloud Lab](https://git.frostfs.info/TrueCloudLab/) with the help and
contributions from community members.
Please see [CREDITS](CREDITS.md) for details.

View file

@ -9,8 +9,8 @@ related configuration details.
To follow this guide you need:
- latest released version of [neo-go](https://github.com/nspcc-dev/neo-go/releases) (v0.97.2 at the moment),
- latest released version of [frostfs-adm](https://github.com/TrueCloudLab/frostfs-node/releases) utility (v0.25.1 at the moment),
- latest released version of compiled [frostfs-contract](https://github.com/TrueCloudLab/frostfs-contract/releases) (v0.11.0 at the moment).
- latest released version of [frostfs-adm](https://git.frostfs.info/TrueCloudLab/frostfs-node/releases) utility (v0.42.9 at the moment),
- latest released version of compiled [frostfs-contract](https://git.frostfs.info/TrueCloudLab/frostfs-contract/releases) (v0.19.2 at the moment).
## Step 1: Prepare network configuration

View file

@ -0,0 +1,15 @@
package metabase
import "github.com/spf13/cobra"
// RootCmd is a root command of config section.
var RootCmd = &cobra.Command{
Use: "metabase",
Short: "Section for metabase commands",
}
func init() {
RootCmd.AddCommand(UpgradeCmd)
initUpgradeCommand()
}

View file

@ -0,0 +1,99 @@
package metabase
import (
"errors"
"fmt"
"sync"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/config"
engineconfig "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/config/engine"
shardconfig "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-node/config/engine/shard"
meta "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/metabase"
"github.com/spf13/cobra"
"golang.org/x/sync/errgroup"
)
const (
pathFlag = "path"
noCompactFlag = "no-compact"
)
var errNoPathsFound = errors.New("no metabase paths found")
var path string
var UpgradeCmd = &cobra.Command{
Use: "upgrade",
Short: "Upgrade metabase to latest version",
RunE: upgrade,
}
func upgrade(cmd *cobra.Command, _ []string) error {
configFile, err := cmd.Flags().GetString(commonflags.ConfigFlag)
if err != nil {
return err
}
configDir, err := cmd.Flags().GetString(commonflags.ConfigDirFlag)
if err != nil {
return err
}
noCompact, _ := cmd.Flags().GetBool(noCompactFlag)
var paths []string
if path != "" {
paths = append(paths, path)
}
appCfg := config.New(configFile, configDir, config.EnvPrefix)
if err := engineconfig.IterateShards(appCfg, false, func(sc *shardconfig.Config) error {
paths = append(paths, sc.Metabase().Path())
return nil
}); err != nil {
return fmt.Errorf("failed to get metabase paths: %w", err)
}
if len(paths) == 0 {
return errNoPathsFound
}
cmd.Println("found", len(paths), "metabases:")
for i, path := range paths {
cmd.Println(i+1, ":", path)
}
result := make(map[string]bool)
var resultGuard sync.Mutex
eg, ctx := errgroup.WithContext(cmd.Context())
for _, path := range paths {
eg.Go(func() error {
var success bool
cmd.Println("upgrading metabase", path, "...")
if err := meta.Upgrade(ctx, path, !noCompact, func(a ...any) {
cmd.Println(append([]any{time.Now().Format(time.RFC3339), ":", path, ":"}, a...)...)
}); err != nil {
cmd.Println("error: failed to upgrade metabase", path, ":", err)
} else {
success = true
cmd.Println("metabase", path, "upgraded successfully")
}
resultGuard.Lock()
result[path] = success
resultGuard.Unlock()
return nil
})
}
if err := eg.Wait(); err != nil {
return err
}
for mb, ok := range result {
if ok {
cmd.Println(mb, ": success")
} else {
cmd.Println(mb, ": failed")
}
}
return nil
}
func initUpgradeCommand() {
flags := UpgradeCmd.Flags()
flags.StringVar(&path, pathFlag, "", "Path to metabase file")
flags.Bool(noCompactFlag, false, "Do not compact upgraded metabase file")
}

View file

@ -38,6 +38,12 @@ var (
func parseTarget(cmd *cobra.Command) policyengine.Target {
name, _ := cmd.Flags().GetString(targetNameFlag)
typ, err := parseTargetType(cmd)
// interpret "root" namespace as empty
if typ == policyengine.Namespace && name == "root" {
name = ""
}
commonCmd.ExitOnErr(cmd, "read target type error: %w", err)
return policyengine.Target{

View file

@ -68,7 +68,7 @@ func dumpContractHashes(cmd *cobra.Command, _ []string) error {
if irSize != 0 {
bw.Reset()
for i := 0; i < irSize; i++ {
for i := range irSize {
emit.AppCall(bw.BinWriter, cs.Hash, "resolve", callflag.ReadOnly,
helper.GetAlphabetNNSDomain(i),
int64(nns.TXT))
@ -79,7 +79,7 @@ func dumpContractHashes(cmd *cobra.Command, _ []string) error {
return fmt.Errorf("can't fetch info from NNS: %w", err)
}
for i := 0; i < irSize; i++ {
for i := range irSize {
info := contractDumpInfo{name: fmt.Sprintf("alphabet %d", i)}
if h, err := helper.ParseNNSResolveResult(alphaRes.Stack[i]); err == nil {
info.hash = h

View file

@ -73,7 +73,6 @@ func initializeWallets(v *viper.Viper, walletDir string, size int) ([]string, er
return nil, fmt.Errorf("can't fetch password: %w", err)
}
i := i
errG.Go(func() error {
p := filepath.Join(walletDir, innerring.GlagoliticLetter(i).String()+".json")
f, err := os.OpenFile(p, os.O_CREATE, 0o644)
@ -107,7 +106,6 @@ func initializeWallets(v *viper.Viper, walletDir string, size int) ([]string, er
// Create consensus account with 2*N/3+1 multi-signature.
bftCount := smartcontract.GetDefaultHonestNodeCount(size)
for i := range wallets {
i := i
ps := pubs.Copy()
errG.Go(func() error {
if err := addMultisigAccount(wallets[i], majCount, constants.CommitteeAccountName, passwords[i], ps); err != nil {

View file

@ -224,7 +224,7 @@ func (l *LocalClient) CalculateNetworkFee(tx *transaction.Transaction) (int64, e
paramz = []manifest.Parameter{{Type: smartcontract.SignatureType}}
} else if nSigs, _, ok := vm.ParseMultiSigContract(w.VerificationScript); ok {
paramz = make([]manifest.Parameter, nSigs)
for j := 0; j < nSigs; j++ {
for j := range nSigs {
paramz[j] = manifest.Parameter{Type: smartcontract.SignatureType}
}
}

View file

@ -44,7 +44,7 @@ func openAlphabetWallets(v *viper.Viper, walletDir string) ([]*wallet.Wallet, er
var wallets []*wallet.Wallet
var letter string
for i := 0; i < constants.MaxAlphabetNodes; i++ {
for i := range constants.MaxAlphabetNodes {
letter = innerring.GlagoliticLetter(i).String()
p := filepath.Join(walletDir, letter+".json")
var w *wallet.Wallet

View file

@ -113,7 +113,7 @@ func generateTestData(dir string, size int) error {
}
var pubs []string
for i := 0; i < size; i++ {
for i := range size {
p := filepath.Join(dir, innerring.GlagoliticLetter(i).String()+".json")
w, err := wallet.NewWalletFromFile(p)
if err != nil {
@ -148,7 +148,7 @@ func generateTestData(dir string, size int) error {
}
func setTestCredentials(v *viper.Viper, size int) {
for i := 0; i < size; i++ {
for i := range size {
v.Set("credentials."+innerring.GlagoliticLetter(i).String(), strconv.FormatUint(uint64(i), 10))
}
v.Set("credentials.contract", constants.TestContractPassword)

View file

@ -5,6 +5,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/metabase"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/morph"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-adm/internal/modules/storagecfg"
"git.frostfs.info/TrueCloudLab/frostfs-node/misc"
@ -41,6 +42,7 @@ func init() {
rootCmd.AddCommand(config.RootCmd)
rootCmd.AddCommand(morph.RootCmd)
rootCmd.AddCommand(storagecfg.RootCmd)
rootCmd.AddCommand(metabase.RootCmd)
rootCmd.AddCommand(autocomplete.Command("frostfs-adm"))
rootCmd.AddCommand(gendoc.Command(rootCmd, gendoc.Options{}))

View file

@ -2,10 +2,13 @@ package internal
import (
"bytes"
"cmp"
"context"
"errors"
"fmt"
"io"
"os"
"slices"
"sort"
"strings"
@ -14,7 +17,6 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
containerSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
@ -189,54 +191,6 @@ func DeleteContainer(ctx context.Context, prm DeleteContainerPrm) (res DeleteCon
return
}
// EACLPrm groups parameters of EACL operation.
type EACLPrm struct {
Client *client.Client
ClientParams client.PrmContainerEACL
}
// EACLRes groups the resulting values of EACL operation.
type EACLRes struct {
cliRes *client.ResContainerEACL
}
// EACL returns requested eACL table.
func (x EACLRes) EACL() eacl.Table {
return x.cliRes.Table()
}
// EACL reads eACL table from FrostFS by container ID.
//
// Returns any error which prevented the operation from completing correctly in error return.
func EACL(ctx context.Context, prm EACLPrm) (res EACLRes, err error) {
res.cliRes, err = prm.Client.ContainerEACL(ctx, prm.ClientParams)
return
}
// SetEACLPrm groups parameters of SetEACL operation.
type SetEACLPrm struct {
Client *client.Client
ClientParams client.PrmContainerSetEACL
}
// SetEACLRes groups the resulting values of SetEACL operation.
type SetEACLRes struct{}
// SetEACL requests to save an eACL table in FrostFS.
//
// Operation is asynchronous and no guaranteed even in the absence of errors.
// The required time is also not predictable.
//
// Success can be verified by reading by container identifier.
//
// Returns any error which prevented the operation from completing correctly in error return.
func SetEACL(ctx context.Context, prm SetEACLPrm) (res SetEACLRes, err error) {
_, err = prm.Client.ContainerSetEACL(ctx, prm.ClientParams)
return
}
// NetworkInfoPrm groups parameters of NetworkInfo operation.
type NetworkInfoPrm struct {
Client *client.Client
@ -712,7 +666,7 @@ func SearchObjects(ctx context.Context, prm SearchObjectsPrm) (*SearchObjectsRes
for {
n, ok = rdr.Read(buf)
for i := 0; i < n; i++ {
for i := range n {
list = append(list, buf[i])
}
if !ok {
@ -892,3 +846,65 @@ func SyncContainerSettings(ctx context.Context, prm SyncContainerPrm) (*SyncCont
return new(SyncContainerRes), nil
}
// PatchObjectPrm groups parameters of PatchObject operation.
type PatchObjectPrm struct {
commonObjectPrm
objectAddressPrm
NewAttributes []objectSDK.Attribute
ReplaceAttribute bool
PayloadPatches []PayloadPatch
}
type PayloadPatch struct {
Range objectSDK.Range
PayloadPath string
}
type PatchRes struct {
OID oid.ID
}
func Patch(ctx context.Context, prm PatchObjectPrm) (*PatchRes, error) {
patchPrm := client.PrmObjectPatch{
XHeaders: prm.xHeaders,
BearerToken: prm.bearerToken,
Session: prm.sessionToken,
Address: prm.objAddr,
}
slices.SortFunc(prm.PayloadPatches, func(a, b PayloadPatch) int {
return cmp.Compare(a.Range.GetOffset(), b.Range.GetOffset())
})
patcher, err := prm.cli.ObjectPatchInit(ctx, patchPrm)
if err != nil {
return nil, fmt.Errorf("init payload reading: %w", err)
}
if patcher.PatchAttributes(ctx, prm.NewAttributes, prm.ReplaceAttribute) {
for _, pp := range prm.PayloadPatches {
payloadFile, err := os.OpenFile(pp.PayloadPath, os.O_RDONLY, os.ModePerm)
if err != nil {
return nil, err
}
applied := patcher.PatchPayload(ctx, &pp.Range, payloadFile)
_ = payloadFile.Close()
if !applied {
break
}
}
}
res, err := patcher.Close(ctx)
if err != nil {
return nil, err
}
return &PatchRes{
OID: res.ObjectID(),
}, nil
}

View file

@ -24,6 +24,8 @@ var testCmd = &cobra.Command{
}
func Test_getOrGenerate(t *testing.T) {
t.Cleanup(viper.Reset)
dir := t.TempDir()
wallPath := filepath.Join(dir, "wallet.json")

View file

@ -139,7 +139,7 @@ It will be stored in sidechain when inner ring will accepts it.`,
},
}
for i := 0; i < awaitTimeout; i++ {
for range awaitTimeout {
time.Sleep(1 * time.Second)
_, err := internalclient.GetContainer(cmd.Context(), getPrm)

View file

@ -110,7 +110,7 @@ Only owner of the container has a permission to remove container.`,
},
}
for i := 0; i < awaitTimeout; i++ {
for range awaitTimeout {
time.Sleep(1 * time.Second)
_, err := internalclient.GetContainer(cmd.Context(), getPrm)

View file

@ -1,68 +0,0 @@
package container
import (
"os"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
)
var getExtendedACLCmd = &cobra.Command{
Use: "get-eacl",
Short: "Get extended ACL table of container",
Long: `Get extended ACL table of container`,
Run: func(cmd *cobra.Command, _ []string) {
id := parseContainerID(cmd)
pk := key.GetOrGenerate(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
eaclPrm := internalclient.EACLPrm{
Client: cli,
ClientParams: client.PrmContainerEACL{
ContainerID: &id,
},
}
res, err := internalclient.EACL(cmd.Context(), eaclPrm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
eaclTable := res.EACL()
if containerPathTo == "" {
cmd.Println("eACL: ")
common.PrettyPrintJSON(cmd, &eaclTable, "eACL")
return
}
var data []byte
if containerJSON {
data, err = eaclTable.MarshalJSON()
commonCmd.ExitOnErr(cmd, "can't encode to JSON: %w", err)
} else {
data, err = eaclTable.Marshal()
commonCmd.ExitOnErr(cmd, "can't encode to binary: %w", err)
}
cmd.Println("dumping data to file:", containerPathTo)
err = os.WriteFile(containerPathTo, data, 0o644)
commonCmd.ExitOnErr(cmd, "could not write eACL to file: %w", err)
},
}
func initContainerGetEACLCmd() {
commonflags.Init(getExtendedACLCmd)
flags := getExtendedACLCmd.Flags()
flags.StringVar(&containerID, commonflags.CIDFlag, "", commonflags.CIDFlagUsage)
flags.StringVar(&containerPathTo, "to", "", "Path to dump encoded container (default: binary encoded)")
flags.BoolVar(&containerJSON, commonflags.JSON, false, "Encode EACL table in json format")
}

View file

@ -1,9 +1,6 @@
package container
import (
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
@ -70,7 +67,6 @@ var listContainersCmd = &cobra.Command{
continue
}
cnrID := cnrID
prmGet.ClientParams.ContainerID = &cnrID
res, err := internalclient.GetContainer(cmd.Context(), prmGet)
if err != nil {
@ -85,12 +81,8 @@ var listContainersCmd = &cobra.Command{
cmd.Println(cnrID.String())
if flagVarListPrintAttr {
cnr.IterateAttributes(func(key, val string) {
if !strings.HasPrefix(key, container.SysAttributePrefix) && !strings.HasPrefix(key, container.SysAttributePrefixNeoFS) {
// FIXME(@cthulhu-rider): https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/issues/97
// Use dedicated method to skip system attributes.
cmd.Printf(" %s: %s\n", key, val)
}
cnr.IterateUserAttributes(func(key, val string) {
cmd.Printf(" %s: %s\n", key, val)
})
}
}

View file

@ -25,8 +25,6 @@ func init() {
deleteContainerCmd,
listContainerObjectsCmd,
getContainerInfoCmd,
getExtendedACLCmd,
setExtendedACLCmd,
containerNodesCmd,
policyPlaygroundCmd,
}
@ -38,8 +36,6 @@ func init() {
initContainerDeleteCmd()
initContainerListObjectsCmd()
initContainerInfoCmd()
initContainerGetEACLCmd()
initContainerSetEACLCmd()
initContainerNodesCmd()
initContainerPolicyPlaygroundCmd()
@ -53,7 +49,6 @@ func init() {
}{
{createContainerCmd, "PUT"},
{deleteContainerCmd, "DELETE"},
{setExtendedACLCmd, "SETEACL"},
} {
commonflags.InitSession(el.cmd, "container "+el.verb)
}

View file

@ -1,108 +0,0 @@
package container
import (
"bytes"
"errors"
"time"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
"github.com/spf13/cobra"
)
var flagVarsSetEACL struct {
noPreCheck bool
srcPath string
}
var setExtendedACLCmd = &cobra.Command{
Use: "set-eacl",
Short: "Set new extended ACL table for container",
Long: `Set new extended ACL table for container.
Container ID in EACL table will be substituted with ID from the CLI.`,
Run: func(cmd *cobra.Command, _ []string) {
id := parseContainerID(cmd)
eaclTable := common.ReadEACL(cmd, flagVarsSetEACL.srcPath)
tok := getSession(cmd)
eaclTable.SetCID(id)
pk := key.GetOrGenerate(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
if !flagVarsSetEACL.noPreCheck {
cmd.Println("Checking the ability to modify access rights in the container...")
extendable, err := internalclient.IsACLExtendable(cmd.Context(), cli, id)
commonCmd.ExitOnErr(cmd, "Extensibility check failure: %w", err)
if !extendable {
commonCmd.ExitOnErr(cmd, "", errors.New("container ACL is immutable"))
}
cmd.Println("ACL extension is enabled in the container, continue processing.")
}
setEACLPrm := internalclient.SetEACLPrm{
Client: cli,
ClientParams: client.PrmContainerSetEACL{
Table: eaclTable,
Session: tok,
},
}
_, err := internalclient.SetEACL(cmd.Context(), setEACLPrm)
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
if containerAwait {
exp, err := eaclTable.Marshal()
commonCmd.ExitOnErr(cmd, "broken EACL table: %w", err)
cmd.Println("awaiting...")
getEACLPrm := internalclient.EACLPrm{
Client: cli,
ClientParams: client.PrmContainerEACL{
ContainerID: &id,
},
}
for i := 0; i < awaitTimeout; i++ {
time.Sleep(1 * time.Second)
res, err := internalclient.EACL(cmd.Context(), getEACLPrm)
if err == nil {
// compare binary values because EACL could have been set already
table := res.EACL()
got, err := table.Marshal()
if err != nil {
continue
}
if bytes.Equal(exp, got) {
cmd.Println("EACL has been persisted on sidechain")
return
}
}
}
commonCmd.ExitOnErr(cmd, "", errSetEACLTimeout)
}
},
}
func initContainerSetEACLCmd() {
commonflags.Init(setExtendedACLCmd)
flags := setExtendedACLCmd.Flags()
flags.StringVar(&containerID, commonflags.CIDFlag, "", commonflags.CIDFlagUsage)
flags.StringVar(&flagVarsSetEACL.srcPath, "table", "", "path to file with JSON or binary encoded EACL table")
flags.BoolVar(&containerAwait, "await", false, "block execution until EACL is persisted")
flags.BoolVar(&flagVarsSetEACL.noPreCheck, "no-precheck", false, "do not pre-check the extensibility of the container ACL")
}

View file

@ -18,9 +18,8 @@ const (
)
var (
errCreateTimeout = errors.New("timeout: container has not been persisted on sidechain")
errDeleteTimeout = errors.New("timeout: container has not been removed from sidechain")
errSetEACLTimeout = errors.New("timeout: EACL has not been persisted on sidechain")
errCreateTimeout = errors.New("timeout: container has not been persisted on sidechain")
errDeleteTimeout = errors.New("timeout: container has not been removed from sidechain")
)
func parseContainerID(cmd *cobra.Command) cid.ID {

View file

@ -0,0 +1,88 @@
package control
import (
"fmt"
rawclient "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/rpc/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control"
"github.com/mr-tron/base58"
"github.com/spf13/cobra"
)
const (
fillPercentFlag = "fill_percent"
)
var shardsRebuildCmd = &cobra.Command{
Use: "rebuild",
Short: "Rebuild shards",
Long: "Rebuild reclaims storage occupied by dead objects and adjusts the storage structure according to the configuration (for blobovnicza only now)",
Run: shardsRebuild,
}
func shardsRebuild(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd)
req := &control.StartShardRebuildRequest{
Body: &control.StartShardRebuildRequest_Body{
Shard_ID: getShardIDList(cmd),
TargetFillPercent: getFillPercentValue(cmd),
ConcurrencyLimit: getConcurrencyValue(cmd),
},
}
signRequest(cmd, pk, req)
cli := getClient(cmd, pk)
var resp *control.StartShardRebuildResponse
var err error
err = cli.ExecRaw(func(client *rawclient.Client) error {
resp, err = control.StartShardRebuild(client, req)
return err
})
commonCmd.ExitOnErr(cmd, "rpc error: %w", err)
verifyResponse(cmd, resp.GetSignature(), resp.GetBody())
var success, failed uint
for _, res := range resp.GetBody().GetResults() {
if res.GetSuccess() {
success++
cmd.Printf("Shard %s: OK\n", base58.Encode(res.GetShard_ID()))
} else {
failed++
cmd.Printf("Shard %s: failed with error %q\n", base58.Encode(res.GetShard_ID()), res.GetError())
}
}
cmd.Printf("Total: %d success, %d failed\n", success, failed)
}
func getFillPercentValue(cmd *cobra.Command) uint32 {
v, _ := cmd.Flags().GetUint32(fillPercentFlag)
if v <= 0 || v > 100 {
commonCmd.ExitOnErr(cmd, "invalid fill_percent value", fmt.Errorf("fill_percent value must be (0, 100], current value: %d", v))
}
return v
}
func getConcurrencyValue(cmd *cobra.Command) uint32 {
v, _ := cmd.Flags().GetUint32(concurrencyFlag)
if v <= 0 || v > 10000 {
commonCmd.ExitOnErr(cmd, "invalid concurrency value", fmt.Errorf("concurrency value must be (0, 10 000], current value: %d", v))
}
return v
}
func initControlShardRebuildCmd() {
initControlFlags(shardsRebuildCmd)
flags := shardsRebuildCmd.Flags()
flags.StringSlice(shardIDFlag, nil, "List of shard IDs in base58 encoding")
flags.Bool(shardAllFlag, false, "Process all shards")
flags.Uint32(fillPercentFlag, 80, "Target fill percent to reclaim space")
flags.Uint32(concurrencyFlag, 20, "Maximum count of concurrently rebuilding files")
setShardModeCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag)
}

View file

@ -84,7 +84,7 @@ func setNetmapStatus(cmd *cobra.Command, _ []string) {
body.SetStatus(control.NetmapStatus_MAINTENANCE)
if force {
body.SetForceMaintenance()
body.SetForceMaintenance(true)
common.PrintVerbose(cmd, "Local maintenance will be forced.")
}
targetStatus = control.NetmapStatus_MAINTENANCE

View file

@ -19,6 +19,7 @@ func initControlShardsCmd() {
shardsCmd.AddCommand(doctorCmd)
shardsCmd.AddCommand(writecacheShardCmd)
shardsCmd.AddCommand(shardsDetachCmd)
shardsCmd.AddCommand(shardsRebuildCmd)
initControlShardsListCmd()
initControlSetShardModeCmd()
@ -28,4 +29,5 @@ func initControlShardsCmd() {
initControlDoctorCmd()
initControlShardsWritecacheCmd()
initControlShardsDetachCmd()
initControlShardRebuildCmd()
}

View file

@ -61,17 +61,18 @@ func listShards(cmd *cobra.Command, _ []string) {
}
}
func prettyPrintShardsJSON(cmd *cobra.Command, ii []*control.ShardInfo) {
func prettyPrintShardsJSON(cmd *cobra.Command, ii []control.ShardInfo) {
out := make([]map[string]any, 0, len(ii))
for _, i := range ii {
out = append(out, map[string]any{
"shard_id": base58.Encode(i.GetShard_ID()),
"mode": shardModeToString(i.GetMode()),
"metabase": i.GetMetabasePath(),
"blobstor": i.GetBlobstor(),
"writecache": i.GetWritecachePath(),
"pilorama": i.GetPiloramaPath(),
"error_count": i.GetErrorCount(),
"shard_id": base58.Encode(i.GetShard_ID()),
"mode": shardModeToString(i.GetMode()),
"metabase": i.GetMetabasePath(),
"blobstor": i.GetBlobstor(),
"writecache": i.GetWritecachePath(),
"pilorama": i.GetPiloramaPath(),
"error_count": i.GetErrorCount(),
"evacuation_in_progress": i.GetEvacuationInProgress(),
})
}
@ -83,7 +84,7 @@ func prettyPrintShardsJSON(cmd *cobra.Command, ii []*control.ShardInfo) {
cmd.Print(buf.String()) // pretty printer emits newline, so no need for Println
}
func prettyPrintShards(cmd *cobra.Command, ii []*control.ShardInfo) {
func prettyPrintShards(cmd *cobra.Command, ii []control.ShardInfo) {
for _, i := range ii {
pathPrinter := func(name, path string) string {
if path == "" {
@ -105,7 +106,8 @@ func prettyPrintShards(cmd *cobra.Command, ii []*control.ShardInfo) {
sb.String()+
pathPrinter("Write-cache", i.GetWritecachePath())+
pathPrinter("Pilorama", i.GetPiloramaPath())+
fmt.Sprintf("Error count: %d\n", i.GetErrorCount()),
fmt.Sprintf("Error count: %d\n", i.GetErrorCount())+
fmt.Sprintf("Evacuation in progress: %t\n", i.GetEvacuationInProgress()),
base58.Encode(i.GetShard_ID()),
shardModeToString(i.GetMode()),
)
@ -121,7 +123,7 @@ func shardModeToString(m control.ShardMode) string {
return "unknown"
}
func sortShardsByID(ii []*control.ShardInfo) {
func sortShardsByID(ii []control.ShardInfo) {
sort.Slice(ii, func(i, j int) bool {
return bytes.Compare(ii[i].GetShard_ID(), ii[j].GetShard_ID()) < 0
})

View file

@ -117,10 +117,10 @@ func setShardMode(cmd *cobra.Command, _ []string) {
req.SetBody(body)
body.SetMode(mode)
body.SetShardIDList(getShardIDList(cmd))
body.SetShard_ID(getShardIDList(cmd))
reset, _ := cmd.Flags().GetBool(shardClearErrorsFlag)
body.ClearErrorCounter(reset)
body.SetResetErrorCounter(reset)
signRequest(cmd, pk, req)

View file

@ -44,7 +44,7 @@ func verifyResponse(cmd *cobra.Command,
GetSign() []byte
},
body interface {
StableMarshal([]byte) []byte
MarshalProtobuf([]byte) []byte
},
) {
if sigControl == nil {
@ -60,7 +60,7 @@ func verifyResponse(cmd *cobra.Command,
var sig frostfscrypto.Signature
commonCmd.ExitOnErr(cmd, "can't read signature: %w", sig.ReadFromV2(sigV2))
if !sig.Verify(body.StableMarshal(nil)) {
if !sig.Verify(body.MarshalProtobuf(nil)) {
commonCmd.ExitOnErr(cmd, "", errors.New("invalid response signature"))
}
}

View file

@ -9,6 +9,12 @@ import (
"github.com/spf13/cobra"
)
const (
asyncFlag = "async"
restoreModeFlag = "restore-mode"
shrinkFlag = "shrink"
)
var writecacheShardCmd = &cobra.Command{
Use: "writecache",
Short: "Operations with storage node's write-cache",
@ -26,10 +32,16 @@ func sealWritecache(cmd *cobra.Command, _ []string) {
pk := key.Get(cmd)
ignoreErrors, _ := cmd.Flags().GetBool(ignoreErrorsFlag)
async, _ := cmd.Flags().GetBool(asyncFlag)
restoreMode, _ := cmd.Flags().GetBool(restoreModeFlag)
shrink, _ := cmd.Flags().GetBool(shrinkFlag)
req := &control.SealWriteCacheRequest{Body: &control.SealWriteCacheRequest_Body{
Shard_ID: getShardIDList(cmd),
IgnoreErrors: ignoreErrors,
Async: async,
RestoreMode: restoreMode,
Shrink: shrink,
}}
signRequest(cmd, pk, req)
@ -68,6 +80,9 @@ func initControlShardsWritecacheCmd() {
ff.StringSlice(shardIDFlag, nil, "List of shard IDs in base58 encoding")
ff.Bool(shardAllFlag, false, "Process all shards")
ff.Bool(ignoreErrorsFlag, true, "Skip invalid/unreadable objects")
ff.Bool(asyncFlag, false, "Run operation in background")
ff.Bool(restoreModeFlag, false, "Restore writecache's mode after sealing")
ff.Bool(shrinkFlag, false, "Shrink writecache's internal storage")
sealWritecacheShardCmd.MarkFlagsMutuallyExclusive(shardIDFlag, shardAllFlag)
}

View file

@ -30,7 +30,8 @@ import (
)
const (
verifyPresenceAllFlag = "verify-presence-all"
verifyPresenceAllFlag = "verify-presence-all"
preferInternalAddressesFlag = "prefer-internal-addresses"
)
var (
@ -97,6 +98,7 @@ func initObjectNodesCmd() {
flags.Bool(verifyPresenceAllFlag, false, "Verify the actual presence of the object on all netmap nodes.")
flags.Bool(commonflags.JSON, false, "Print information about the object placement as json.")
flags.Bool(preferInternalAddressesFlag, false, "Use internal addresses first to get object info.")
}
func objectNodes(cmd *cobra.Command, _ []string) {
@ -170,7 +172,7 @@ func getComplexObjectParts(cmd *cobra.Command, cnrID cid.ID, objID oid.ID, cli *
func getCompexObjectMembers(cmd *cobra.Command, cnrID cid.ID, objID oid.ID, cli *client.Client, prmHead internalclient.HeadObjectPrm, errSplitInfo *objectSDK.SplitInfoError) []oid.ID {
splitInfo := errSplitInfo.SplitInfo()
if members, ok := tryGetSplitMembersByLinkingObject(cmd, splitInfo, prmHead, cnrID, false); ok {
if members, ok := tryGetSplitMembersByLinkingObject(cmd, splitInfo, prmHead, cnrID); ok {
return members
}
@ -183,6 +185,7 @@ func getCompexObjectMembers(cmd *cobra.Command, cnrID cid.ID, objID oid.ID, cli
func flattenComplexMembersIfECContainer(cmd *cobra.Command, cnrID cid.ID, members []oid.ID, prmHead internalclient.HeadObjectPrm) []phyObject {
result := make([]phyObject, 0, len(members))
var hasNonEC, hasEC bool
var resultGuard sync.Mutex
if len(members) == 0 {
@ -191,31 +194,8 @@ func flattenComplexMembersIfECContainer(cmd *cobra.Command, cnrID cid.ID, member
prmHead.SetRawFlag(true) // to get an error instead of whole object
first := members[0]
var addrObj oid.Address
addrObj.SetContainer(cnrID)
addrObj.SetObject(first)
prmHead.SetAddress(addrObj)
_, err := internalclient.HeadObject(cmd.Context(), prmHead)
var ecInfoError *objectSDK.ECInfoError
if errors.As(err, &ecInfoError) {
chunks := getECObjectChunks(cmd, cnrID, first, ecInfoError)
result = append(result, chunks...)
} else if err == nil { // not EC object, so all members must be phy objects
for _, member := range members {
result = append(result, phyObject{
containerID: cnrID,
objectID: member,
})
}
return result
} else {
commonCmd.ExitOnErr(cmd, "failed to flatten parts of complex object: %w", err)
}
eg, egCtx := errgroup.WithContext(cmd.Context())
for idx := 1; idx < len(members); idx++ {
for idx := range len(members) {
partObjID := members[idx]
eg.Go(func() error {
@ -225,24 +205,44 @@ func flattenComplexMembersIfECContainer(cmd *cobra.Command, cnrID cid.ID, member
partAddr.SetObject(partObjID)
partHeadPrm.SetAddress(partAddr)
_, err := internalclient.HeadObject(egCtx, partHeadPrm)
var ecInfoError *objectSDK.ECInfoError
if errors.As(err, &ecInfoError) {
chunks := getECObjectChunks(cmd, cnrID, partObjID, ecInfoError)
resultGuard.Lock()
defer resultGuard.Unlock()
result = append(result, chunks...)
return nil
} else if err == nil {
return errMalformedComplexObject
obj, err := internalclient.HeadObject(egCtx, partHeadPrm)
if err != nil {
var ecInfoError *objectSDK.ECInfoError
if errors.As(err, &ecInfoError) {
resultGuard.Lock()
defer resultGuard.Unlock()
result = append(result, getECObjectChunks(cmd, cnrID, partObjID, ecInfoError)...)
hasEC = true
return nil
}
return err
}
return err
if obj.Header().Type() != objectSDK.TypeRegular {
commonCmd.ExitOnErr(cmd, "failed to flatten parts of complex object: %w", fmt.Errorf("object '%s' with type '%s' is not supported as part of complex object", partAddr, obj.Header().Type()))
}
if len(obj.Header().Children()) > 0 {
// linking object is not data object, so skip it
return nil
}
resultGuard.Lock()
defer resultGuard.Unlock()
result = append(result, phyObject{
containerID: cnrID,
objectID: partObjID,
})
hasNonEC = true
return nil
})
}
commonCmd.ExitOnErr(cmd, "failed to flatten parts of complex object: %w", eg.Wait())
if hasEC && hasNonEC {
commonCmd.ExitOnErr(cmd, "failed to flatten parts of complex object: %w", errMalformedComplexObject)
}
return result
}
@ -393,8 +393,6 @@ func getActualPlacement(cmd *cobra.Command, netmap *netmapSDK.NetMap, pk *ecdsa.
eg, egCtx := errgroup.WithContext(cmd.Context())
for _, cand := range candidates {
cand := cand
eg.Go(func() error {
cli, err := createClient(egCtx, cmd, cand, pk)
if err != nil {
@ -405,7 +403,6 @@ func getActualPlacement(cmd *cobra.Command, netmap *netmapSDK.NetMap, pk *ecdsa.
}
for _, object := range objects {
object := object
eg.Go(func() error {
stored, err := isObjectStoredOnNode(egCtx, cmd, object.containerID, object.objectID, cli, pk)
resultMtx.Lock()
@ -449,11 +446,20 @@ func getNodesToCheckObjectExistance(cmd *cobra.Command, netmap *netmapSDK.NetMap
func createClient(ctx context.Context, cmd *cobra.Command, candidate netmapSDK.NodeInfo, pk *ecdsa.PrivateKey) (*client.Client, error) {
var cli *client.Client
var addresses []string
candidate.IterateNetworkEndpoints(func(s string) bool {
addresses = append(addresses, s)
return false
})
addresses = append(addresses, candidate.ExternalAddresses()...)
if preferInternal, _ := cmd.Flags().GetBool(preferInternalAddressesFlag); preferInternal {
candidate.IterateNetworkEndpoints(func(s string) bool {
addresses = append(addresses, s)
return false
})
addresses = append(addresses, candidate.ExternalAddresses()...)
} else {
addresses = append(addresses, candidate.ExternalAddresses()...)
candidate.IterateNetworkEndpoints(func(s string) bool {
addresses = append(addresses, s)
return false
})
}
var lastErr error
for _, address := range addresses {
var networkAddr network.Address

View file

@ -0,0 +1,151 @@
package object
import (
"fmt"
"strconv"
"strings"
internalclient "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/client"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/key"
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/spf13/cobra"
)
const (
newAttrsFlagName = "new-attrs"
replaceAttrsFlagName = "replace-attrs"
rangeFlagName = "range"
payloadFlagName = "payload"
)
var objectPatchCmd = &cobra.Command{
Use: "patch",
Run: patch,
Short: "Patch FrostFS object",
Long: "Patch FrostFS object. Each range passed to the command requires to pass a corresponding patch payload.",
Example: `
frostfs-cli -c config.yml -r 127.0.0.1:8080 object patch --cid <CID> --oid <OID> --new-attrs 'key1=val1,key2=val2' --replace-attrs
frostfs-cli -c config.yml -r 127.0.0.1:8080 object patch --cid <CID> --oid <OID> --range offX:lnX --payload /path/to/payloadX --range offY:lnY --payload /path/to/payloadY
frostfs-cli -c config.yml -r 127.0.0.1:8080 object patch --cid <CID> --oid <OID> --new-attrs 'key1=val1,key2=val2' --replace-attrs --range offX:lnX --payload /path/to/payload
`,
}
func initObjectPatchCmd() {
commonflags.Init(objectPatchCmd)
initFlagSession(objectPatchCmd, "PATCH")
flags := objectPatchCmd.Flags()
flags.String(commonflags.CIDFlag, "", commonflags.CIDFlagUsage)
_ = objectRangeCmd.MarkFlagRequired(commonflags.CIDFlag)
flags.String(commonflags.OIDFlag, "", commonflags.OIDFlagUsage)
_ = objectRangeCmd.MarkFlagRequired(commonflags.OIDFlag)
flags.String(newAttrsFlagName, "", "New object attributes in form of Key1=Value1,Key2=Value2")
flags.Bool(replaceAttrsFlagName, false, "Replace object attributes by new ones.")
flags.StringSlice(rangeFlagName, []string{}, "Range to which patch payload is applied. Format: offset:length")
flags.StringSlice(payloadFlagName, []string{}, "Path to file with patch payload.")
}
func patch(cmd *cobra.Command, _ []string) {
var cnr cid.ID
var obj oid.ID
objAddr := readObjectAddress(cmd, &cnr, &obj)
ranges, err := getRangeSlice(cmd)
commonCmd.ExitOnErr(cmd, "", err)
payloads := patchPayloadPaths(cmd)
if len(ranges) != len(payloads) {
commonCmd.ExitOnErr(cmd, "", fmt.Errorf("the number of ranges and payloads are not equal: ranges = %d, payloads = %d", len(ranges), len(payloads)))
}
newAttrs, err := parseNewObjectAttrs(cmd)
commonCmd.ExitOnErr(cmd, "can't parse new object attributes: %w", err)
replaceAttrs, _ := cmd.Flags().GetBool(replaceAttrsFlagName)
pk := key.GetOrGenerate(cmd)
cli := internalclient.GetSDKClientByFlag(cmd, pk, commonflags.RPC)
var prm internalclient.PatchObjectPrm
prm.SetClient(cli)
Prepare(cmd, &prm)
ReadOrOpenSession(cmd, &prm, pk, cnr, nil)
prm.SetAddress(objAddr)
prm.NewAttributes = newAttrs
prm.ReplaceAttribute = replaceAttrs
for i := range ranges {
prm.PayloadPatches = append(prm.PayloadPatches, internalclient.PayloadPatch{
Range: ranges[i],
PayloadPath: payloads[i],
})
}
res, err := internalclient.Patch(cmd.Context(), prm)
if err != nil {
commonCmd.ExitOnErr(cmd, "can't patch the object: %w", err)
}
cmd.Println("Patched object ID: ", res.OID.EncodeToString())
}
func parseNewObjectAttrs(cmd *cobra.Command) ([]objectSDK.Attribute, error) {
var rawAttrs []string
raw := cmd.Flag(newAttrsFlagName).Value.String()
if len(raw) != 0 {
rawAttrs = strings.Split(raw, ",")
}
attrs := make([]objectSDK.Attribute, len(rawAttrs), len(rawAttrs)+2) // name + timestamp attributes
for i := range rawAttrs {
k, v, found := strings.Cut(rawAttrs[i], "=")
if !found {
return nil, fmt.Errorf("invalid attribute format: %s", rawAttrs[i])
}
attrs[i].SetKey(k)
attrs[i].SetValue(v)
}
return attrs, nil
}
func getRangeSlice(cmd *cobra.Command) ([]objectSDK.Range, error) {
v, _ := cmd.Flags().GetStringSlice(rangeFlagName)
if len(v) == 0 {
return []objectSDK.Range{}, nil
}
rs := make([]objectSDK.Range, len(v))
for i := range v {
before, after, found := strings.Cut(v[i], rangeSep)
if !found {
return nil, fmt.Errorf("invalid range specifier: %s", v[i])
}
offset, err := strconv.ParseUint(before, 10, 64)
if err != nil {
return nil, fmt.Errorf("invalid '%s' range offset specifier: %w", v[i], err)
}
length, err := strconv.ParseUint(after, 10, 64)
if err != nil {
return nil, fmt.Errorf("invalid '%s' range length specifier: %w", v[i], err)
}
rs[i].SetOffset(offset)
rs[i].SetLength(length)
}
return rs, nil
}
func patchPayloadPaths(cmd *cobra.Command) []string {
v, _ := cmd.Flags().GetStringSlice(payloadFlagName)
return v
}

View file

@ -29,6 +29,7 @@ func init() {
objectRangeCmd,
objectLockCmd,
objectNodesCmd,
objectPatchCmd,
}
Cmd.AddCommand(objectChildCommands...)
@ -39,6 +40,7 @@ func init() {
}
initObjectPutCmd()
initObjectPatchCmd()
initObjectDeleteCmd()
initObjectGetCmd()
initObjectSearchCmd()

View file

@ -306,6 +306,8 @@ func finalizeSession(cmd *cobra.Command, dst SessionPrm, tok *session.Object, ke
case *internal.PutObjectPrm:
common.PrintVerbose(cmd, "Binding session to object PUT...")
tok.ForVerb(session.VerbObjectPut)
case *internal.PatchObjectPrm:
tok.ForVerb(session.VerbObjectPatch)
case *internal.DeleteObjectPrm:
common.PrintVerbose(cmd, "Binding session to object DELETE...")
tok.ForVerb(session.VerbObjectDelete)
@ -372,7 +374,7 @@ func collectObjectRelatives(cmd *cobra.Command, cli *client.Client, cnr cid.ID,
common.PrintVerbose(cmd, "Split information received - object is virtual.")
splitInfo := errSplit.SplitInfo()
if members, ok := tryGetSplitMembersByLinkingObject(cmd, splitInfo, prmHead, cnr, true); ok {
if members, ok := tryGetSplitMembersByLinkingObject(cmd, splitInfo, prmHead, cnr); ok {
return members
}
@ -388,7 +390,7 @@ func collectObjectRelatives(cmd *cobra.Command, cli *client.Client, cnr cid.ID,
return nil
}
func tryGetSplitMembersByLinkingObject(cmd *cobra.Command, splitInfo *objectSDK.SplitInfo, prmHead internal.HeadObjectPrm, cnr cid.ID, withLinking bool) ([]oid.ID, bool) {
func tryGetSplitMembersByLinkingObject(cmd *cobra.Command, splitInfo *objectSDK.SplitInfo, prmHead internal.HeadObjectPrm, cnr cid.ID) ([]oid.ID, bool) {
// collect split chain by the descending ease of operations (ease is evaluated heuristically).
// If any approach fails, we don't try the next since we assume that it will fail too.
@ -409,10 +411,7 @@ func tryGetSplitMembersByLinkingObject(cmd *cobra.Command, splitInfo *objectSDK.
common.PrintVerbose(cmd, "Received split members from the linking object: %v", children)
if withLinking {
return append(children, idLinking), true
}
return children, true
return append(children, idLinking), true
}
// linking object is not required for

View file

@ -47,9 +47,10 @@ func add(cmd *cobra.Command, _ []string) {
meta, err := parseMeta(cmd)
commonCmd.ExitOnErr(cmd, "meta data parsing: %w", err)
ctx := cmd.Context()
ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx)
cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size)
@ -77,13 +78,13 @@ func add(cmd *cobra.Command, _ []string) {
cmd.Println("Node ID: ", resp.GetBody().GetNodeId())
}
func parseMeta(cmd *cobra.Command) ([]*tree.KeyValue, error) {
func parseMeta(cmd *cobra.Command) ([]tree.KeyValue, error) {
raws, _ := cmd.Flags().GetStringSlice(metaFlagKey)
if len(raws) == 0 {
return nil, nil
}
pairs := make([]*tree.KeyValue, 0, len(raws))
pairs := make([]tree.KeyValue, 0, len(raws))
for i := range raws {
k, v, found := strings.Cut(raws[i], "=")
if !found {
@ -94,7 +95,7 @@ func parseMeta(cmd *cobra.Command) ([]*tree.KeyValue, error) {
pair.Key = k
pair.Value = []byte(v)
pairs = append(pairs, &pair)
pairs = append(pairs, pair)
}
return pairs, nil

View file

@ -50,9 +50,10 @@ func addByPath(cmd *cobra.Command, _ []string) {
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
tid, _ := cmd.Flags().GetString(treeIDFlagKey)
ctx := cmd.Context()
ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx)
cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size)

View file

@ -3,13 +3,14 @@ package tree
import (
"context"
"strings"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-cli/internal/commonflags"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree"
metrics "git.frostfs.info/TrueCloudLab/frostfs-observability/metrics/grpc"
tracing "git.frostfs.info/TrueCloudLab/frostfs-observability/tracing/grpc"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
@ -17,7 +18,7 @@ import (
// _client returns grpc Tree service client. Should be removed
// after making Tree API public.
func _client(ctx context.Context) (tree.TreeServiceClient, error) {
func _client() (tree.TreeServiceClient, error) {
var netAddr network.Address
err := netAddr.FromString(viper.GetString(commonflags.RPC))
if err != nil {
@ -25,7 +26,6 @@ func _client(ctx context.Context) (tree.TreeServiceClient, error) {
}
opts := []grpc.DialOption{
grpc.WithBlock(),
grpc.WithChainUnaryInterceptor(
metrics.NewUnaryClientInterceptor(),
tracing.NewUnaryClientInteceptor(),
@ -40,12 +40,14 @@ func _client(ctx context.Context) (tree.TreeServiceClient, error) {
opts = append(opts, grpc.WithTransportCredentials(insecure.NewCredentials()))
}
// a default connection establishing timeout
const defaultClientConnectTimeout = time.Second * 2
ctx, cancel := context.WithTimeout(ctx, defaultClientConnectTimeout)
cc, err := grpc.DialContext(ctx, netAddr.URIAddr(), opts...)
cancel()
cc, err := grpc.NewClient(netAddr.URIAddr(), opts...)
return tree.NewTreeServiceClient(cc), err
}
func contextWithTimeout(cmd *cobra.Command) (context.Context, context.CancelFunc) {
if timeout := viper.GetDuration(commonflags.Timeout); timeout > 0 {
common.PrintVerbose(cmd, "Set request timeout to %s.", timeout)
return context.WithTimeout(cmd.Context(), timeout)
}
return context.WithTimeout(cmd.Context(), commonflags.TimeoutDefault)
}

View file

@ -50,9 +50,10 @@ func getByPath(cmd *cobra.Command, _ []string) {
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
tid, _ := cmd.Flags().GetString(treeIDFlagKey)
ctx := cmd.Context()
ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx)
cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size)

View file

@ -44,9 +44,10 @@ func getOpLog(cmd *cobra.Command, _ []string) {
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
tid, _ := cmd.Flags().GetString(treeIDFlagKey)
ctx := cmd.Context()
ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx)
cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size)

View file

@ -26,9 +26,10 @@ func initHealthcheckCmd() {
func healthcheck(cmd *cobra.Command, _ []string) {
pk := key.GetOrGenerate(cmd)
ctx := cmd.Context()
ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx)
cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
req := &tree.HealthcheckRequest{

View file

@ -38,9 +38,10 @@ func list(cmd *cobra.Command, _ []string) {
err := cnr.DecodeString(cidString)
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
ctx := cmd.Context()
ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx)
cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size)

View file

@ -45,9 +45,10 @@ func move(cmd *cobra.Command, _ []string) {
err := cnr.DecodeString(cidString)
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
ctx := cmd.Context()
ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx)
cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size)
@ -75,7 +76,7 @@ func move(cmd *cobra.Command, _ []string) {
resp, err := cli.GetSubTree(ctx, subTreeReq)
commonCmd.ExitOnErr(cmd, "rpc call: %w", err)
var meta []*tree.KeyValue
var meta []tree.KeyValue
subtreeResp, err := resp.Recv()
for ; err == nil; subtreeResp, err = resp.Recv() {
meta = subtreeResp.GetBody().GetMeta()

View file

@ -41,9 +41,10 @@ func remove(cmd *cobra.Command, _ []string) {
err := cnr.DecodeString(cidString)
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
ctx := cmd.Context()
ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx)
cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size)

View file

@ -49,6 +49,7 @@ const (
heightFlagKey = "height"
countFlagKey = "count"
depthFlagKey = "depth"
orderFlagKey = "ordered"
)
func initCTID(cmd *cobra.Command) {

View file

@ -30,6 +30,7 @@ func initGetSubtreeCmd() {
ff := getSubtreeCmd.Flags()
ff.Uint64(rootIDFlagKey, 0, "Root ID to traverse from.")
ff.Uint32(depthFlagKey, 10, "Traversal depth.")
ff.Bool(orderFlagKey, false, "Sort output by ascending FileName.")
_ = getSubtreeCmd.MarkFlagRequired(commonflags.CIDFlag)
_ = getSubtreeCmd.MarkFlagRequired(treeIDFlagKey)
@ -45,9 +46,10 @@ func getSubTree(cmd *cobra.Command, _ []string) {
err := cnr.DecodeString(cidString)
commonCmd.ExitOnErr(cmd, "decode container ID string: %w", err)
ctx := cmd.Context()
ctx, cancel := contextWithTimeout(cmd)
defer cancel()
cli, err := _client(ctx)
cli, err := _client()
commonCmd.ExitOnErr(cmd, "failed to create client: %w", err)
rawCID := make([]byte, sha256.Size)
@ -59,6 +61,13 @@ func getSubTree(cmd *cobra.Command, _ []string) {
depth, _ := cmd.Flags().GetUint32(depthFlagKey)
order, _ := cmd.Flags().GetBool(orderFlagKey)
bodyOrder := tree.GetSubTreeRequest_Body_Order_None
if order {
bodyOrder = tree.GetSubTreeRequest_Body_Order_Asc
}
var bt []byte
if t := common.ReadBearerToken(cmd, bearerFlagKey); t != nil {
bt = t.Marshal()
@ -71,6 +80,9 @@ func getSubTree(cmd *cobra.Command, _ []string) {
RootId: []uint64{rid},
Depth: depth,
BearerToken: bt,
OrderBy: &tree.GetSubTreeRequest_Body_Order{
Direction: bodyOrder,
},
},
}

View file

@ -33,7 +33,7 @@ func PrettyPrintTableBACL(cmd *cobra.Command, bacl *acl.Basic) {
fmt.Fprintln(w, strings.Join(bits, "\t"))
// Footer
footer := []string{"X F"}
for i := 0; i < 7; i++ {
for range 7 {
footer = append(footer, "U S O B")
}
fmt.Fprintln(w, strings.Join(footer, "\t"))

View file

@ -239,6 +239,8 @@ func parseAction(lexeme string) ([]string, bool, error) {
return []string{nativeschema.MethodRangeObject}, true, nil
case "object.hash":
return []string{nativeschema.MethodHashObject}, true, nil
case "object.patch":
return []string{nativeschema.MethodPatchObject}, true, nil
case "object.*":
return []string{
nativeschema.MethodPutObject,
@ -248,6 +250,7 @@ func parseAction(lexeme string) ([]string, bool, error) {
nativeschema.MethodSearchObject,
nativeschema.MethodRangeObject,
nativeschema.MethodHashObject,
nativeschema.MethodPatchObject,
}, true, nil
case "container.put":
return []string{nativeschema.MethodPutContainer}, false, nil
@ -255,10 +258,6 @@ func parseAction(lexeme string) ([]string, bool, error) {
return []string{nativeschema.MethodDeleteContainer}, false, nil
case "container.get":
return []string{nativeschema.MethodGetContainer}, false, nil
case "container.setcontainereacl":
return []string{nativeschema.MethodSetContainerEACL}, false, nil
case "container.getcontainereacl":
return []string{nativeschema.MethodGetContainerEACL}, false, nil
case "container.list":
return []string{nativeschema.MethodListContainers}, false, nil
case "container.*":
@ -266,8 +265,6 @@ func parseAction(lexeme string) ([]string, bool, error) {
nativeschema.MethodPutContainer,
nativeschema.MethodDeleteContainer,
nativeschema.MethodGetContainer,
nativeschema.MethodSetContainerEACL,
nativeschema.MethodGetContainerEACL,
nativeschema.MethodListContainers,
}, false, nil
default:

View file

@ -1,18 +0,0 @@
package util
import (
"github.com/spf13/cobra"
)
// locode section.
var locodeCmd = &cobra.Command{
Use: "locode",
Short: "Working with FrostFS UN/LOCODE database",
}
func initLocodeCmd() {
locodeCmd.AddCommand(locodeGenerateCmd, locodeInfoCmd)
initUtilLocodeInfoCmd()
initUtilLocodeGenerateCmd()
}

View file

@ -1,96 +0,0 @@
package util
import (
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
locodedb "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/db"
airportsdb "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/db/airports"
locodebolt "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/db/boltdb"
continentsdb "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/db/continents/geojson"
csvlocode "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/table/csv"
"github.com/spf13/cobra"
)
type namesDB struct {
*airportsdb.DB
*csvlocode.Table
}
const (
locodeGenerateInputFlag = "in"
locodeGenerateSubDivFlag = "subdiv"
locodeGenerateAirportsFlag = "airports"
locodeGenerateCountriesFlag = "countries"
locodeGenerateContinentsFlag = "continents"
locodeGenerateOutputFlag = "out"
)
var (
locodeGenerateInPaths []string
locodeGenerateSubDivPath string
locodeGenerateAirportsPath string
locodeGenerateCountriesPath string
locodeGenerateContinentsPath string
locodeGenerateOutPath string
locodeGenerateCmd = &cobra.Command{
Use: "generate",
Short: "Generate UN/LOCODE database for FrostFS",
Run: func(cmd *cobra.Command, _ []string) {
locodeDB := csvlocode.New(
csvlocode.Prm{
Path: locodeGenerateInPaths[0],
SubDivPath: locodeGenerateSubDivPath,
},
csvlocode.WithExtraPaths(locodeGenerateInPaths[1:]...),
)
airportDB := airportsdb.New(airportsdb.Prm{
AirportsPath: locodeGenerateAirportsPath,
CountriesPath: locodeGenerateCountriesPath,
})
continentsDB := continentsdb.New(continentsdb.Prm{
Path: locodeGenerateContinentsPath,
})
targetDB := locodebolt.New(locodebolt.Prm{
Path: locodeGenerateOutPath,
})
err := targetDB.Open()
commonCmd.ExitOnErr(cmd, "", err)
defer targetDB.Close()
names := &namesDB{
DB: airportDB,
Table: locodeDB,
}
err = locodedb.FillDatabase(locodeDB, airportDB, continentsDB, names, targetDB)
commonCmd.ExitOnErr(cmd, "", err)
},
}
)
func initUtilLocodeGenerateCmd() {
flags := locodeGenerateCmd.Flags()
flags.StringSliceVar(&locodeGenerateInPaths, locodeGenerateInputFlag, nil, "List of paths to UN/LOCODE tables (csv)")
_ = locodeGenerateCmd.MarkFlagRequired(locodeGenerateInputFlag)
flags.StringVar(&locodeGenerateSubDivPath, locodeGenerateSubDivFlag, "", "Path to UN/LOCODE subdivision database (csv)")
_ = locodeGenerateCmd.MarkFlagRequired(locodeGenerateSubDivFlag)
flags.StringVar(&locodeGenerateAirportsPath, locodeGenerateAirportsFlag, "", "Path to OpenFlights airport database (csv)")
_ = locodeGenerateCmd.MarkFlagRequired(locodeGenerateAirportsFlag)
flags.StringVar(&locodeGenerateCountriesPath, locodeGenerateCountriesFlag, "", "Path to OpenFlights country database (csv)")
_ = locodeGenerateCmd.MarkFlagRequired(locodeGenerateCountriesFlag)
flags.StringVar(&locodeGenerateContinentsPath, locodeGenerateContinentsFlag, "", "Path to continent polygons (GeoJSON)")
_ = locodeGenerateCmd.MarkFlagRequired(locodeGenerateContinentsFlag)
flags.StringVar(&locodeGenerateOutPath, locodeGenerateOutputFlag, "", "Target path for generated database")
_ = locodeGenerateCmd.MarkFlagRequired(locodeGenerateOutputFlag)
}

View file

@ -1,56 +0,0 @@
package util
import (
commonCmd "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common"
locodedb "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/db"
locodebolt "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/locode/db/boltdb"
"github.com/spf13/cobra"
)
const (
locodeInfoDBFlag = "db"
locodeInfoCodeFlag = "locode"
)
var (
locodeInfoDBPath string
locodeInfoCode string
locodeInfoCmd = &cobra.Command{
Use: "info",
Short: "Print information about UN/LOCODE from FrostFS database",
Run: func(cmd *cobra.Command, _ []string) {
targetDB := locodebolt.New(locodebolt.Prm{
Path: locodeInfoDBPath,
}, locodebolt.ReadOnly())
err := targetDB.Open()
commonCmd.ExitOnErr(cmd, "", err)
defer targetDB.Close()
record, err := locodedb.LocodeRecord(targetDB, locodeInfoCode)
commonCmd.ExitOnErr(cmd, "", err)
cmd.Printf("Country: %s\n", record.CountryName())
cmd.Printf("Location: %s\n", record.LocationName())
cmd.Printf("Continent: %s\n", record.Continent())
if subDivCode := record.SubDivCode(); subDivCode != "" {
cmd.Printf("Subdivision: [%s] %s\n", subDivCode, record.SubDivName())
}
geoPoint := record.GeoPoint()
cmd.Printf("Coordinates: %0.2f, %0.2f\n", geoPoint.Latitude(), geoPoint.Longitude())
},
}
)
func initUtilLocodeInfoCmd() {
flags := locodeInfoCmd.Flags()
flags.StringVar(&locodeInfoDBPath, locodeInfoDBFlag, "", "Path to FrostFS UN/LOCODE database")
_ = locodeInfoCmd.MarkFlagRequired(locodeInfoDBFlag)
flags.StringVar(&locodeInfoCode, locodeInfoCodeFlag, "", "UN/LOCODE")
_ = locodeInfoCmd.MarkFlagRequired(locodeInfoCodeFlag)
}

View file

@ -23,11 +23,9 @@ func init() {
signCmd,
convertCmd,
keyerCmd,
locodeCmd,
)
initSignCmd()
initConvertCmd()
initKeyerCmd()
initLocodeCmd()
}

View file

@ -7,6 +7,7 @@ import (
configViper "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/internal/common/config"
"git.frostfs.info/TrueCloudLab/frostfs-node/internal/logs"
control "git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/control/ir"
"github.com/spf13/viper"
"go.uber.org/zap"
)
@ -40,6 +41,8 @@ func reloadConfig() error {
if err != nil {
return err
}
logPrm.PrependTimestamp = cfg.GetBool("logger.timestamp")
return logPrm.Reload()
}
@ -81,6 +84,10 @@ func watchForSignal(cancel func()) {
return
case <-sighupCh:
log.Info(logs.FrostFSNodeSIGHUPHasBeenReceivedRereadingConfiguration)
if !innerRing.CompareAndSwapHealthStatus(control.HealthStatus_READY, control.HealthStatus_RECONFIGURING) {
log.Info(logs.FrostFSNodeSIGHUPSkip)
break
}
err := reloadConfig()
if err != nil {
log.Error(logs.FrostFSNodeConfigurationReading, zap.Error(err))
@ -92,6 +99,7 @@ func watchForSignal(cancel func()) {
if err != nil {
log.Error(logs.FrostFSNodeConfigurationReading, zap.Error(err))
}
innerRing.CompareAndSwapHealthStatus(control.HealthStatus_RECONFIGURING, control.HealthStatus_READY)
log.Info(logs.FrostFSNodeConfigurationHasBeenReloadedSuccessfully)
}
}

View file

@ -9,6 +9,7 @@ import (
func defaultConfiguration(cfg *viper.Viper) {
cfg.SetDefault("logger.level", "info")
cfg.SetDefault("logger.destination", "stdout")
cfg.SetDefault("logger.timestamp", false)
setPprofDefaults(cfg)

View file

@ -13,6 +13,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-node/misc"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/innerring"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/logger"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/util/sdnotify"
"github.com/spf13/viper"
"go.uber.org/zap"
)
@ -78,6 +79,8 @@ func main() {
)
exitErr(err)
logPrm.SamplingHook = metrics.LogMetrics().GetSamplingHook()
logPrm.PrependTimestamp = cfg.GetBool("logger.timestamp")
log, err = logger.NewLogger(logPrm)
exitErr(err)
@ -124,4 +127,8 @@ func shutdown() {
zap.String("error", err.Error()),
)
}
if err := sdnotify.ClearStatus(); err != nil {
log.Error(logs.FailedToReportStatusToSystemd, zap.Error(err))
}
}

View file

@ -19,7 +19,7 @@ var Root = &cobra.Command{
}
func init() {
Root.AddCommand(listCMD, inspectCMD)
Root.AddCommand(listCMD, inspectCMD, tuiCMD)
}
func openBlobovnicza(cmd *cobra.Command) *blobovnicza.Blobovnicza {

View file

@ -0,0 +1,79 @@
package blobovnicza
import (
"context"
"fmt"
common "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal"
schema "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/blobovnicza"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/tui"
"github.com/rivo/tview"
"github.com/spf13/cobra"
"go.etcd.io/bbolt"
)
var tuiCMD = &cobra.Command{
Use: "explore",
Short: "Blobovnicza exploration with a terminal UI",
Long: `Launch a terminal UI to explore blobovnicza and search for data.
Available search filters:
- cid CID
- oid OID
- addr CID/OID
`,
Run: tuiFunc,
}
var initialPrompt string
func init() {
common.AddComponentPathFlag(tuiCMD, &vPath)
tuiCMD.Flags().StringVar(
&initialPrompt,
"filter",
"",
"Filter prompt to start with, format 'tag:value [+ tag:value]...'",
)
}
func tuiFunc(cmd *cobra.Command, _ []string) {
common.ExitOnErr(cmd, runTUI(cmd))
}
func runTUI(cmd *cobra.Command) error {
db, err := openDB(false)
if err != nil {
return fmt.Errorf("couldn't open database: %w", err)
}
defer db.Close()
ctx, cancel := context.WithCancel(cmd.Context())
defer cancel()
app := tview.NewApplication()
ui := tui.NewUI(ctx, app, db, schema.BlobovniczaParser, nil)
_ = ui.AddFilter("cid", tui.CIDParser, "CID")
_ = ui.AddFilter("oid", tui.OIDParser, "OID")
_ = ui.AddCompositeFilter("addr", tui.AddressParser, "CID/OID")
err = ui.WithPrompt(initialPrompt)
if err != nil {
return fmt.Errorf("invalid filter prompt: %w", err)
}
app.SetRoot(ui, true).SetFocus(ui)
return app.Run()
}
func openDB(writable bool) (*bbolt.DB, error) {
db, err := bbolt.Open(vPath, 0o600, &bbolt.Options{
ReadOnly: !writable,
})
if err != nil {
return nil, err
}
return db, nil
}

View file

@ -32,6 +32,7 @@ func init() {
inspectCMD,
listGraveyardCMD,
listGarbageCMD,
tuiCMD,
)
}

View file

@ -0,0 +1,82 @@
package meta
import (
"context"
"fmt"
common "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal"
schema "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/metabase"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/tui"
"github.com/rivo/tview"
"github.com/spf13/cobra"
"go.etcd.io/bbolt"
)
var tuiCMD = &cobra.Command{
Use: "explore",
Short: "Metabase exploration with a terminal UI",
Long: `Launch a terminal UI to explore metabase and search for data.
Available search filters:
- cid CID
- oid OID
- addr CID/OID
- attr key[/value]
`,
Run: tuiFunc,
}
var initialPrompt string
func init() {
common.AddComponentPathFlag(tuiCMD, &vPath)
tuiCMD.Flags().StringVar(
&initialPrompt,
"filter",
"",
"Filter prompt to start with, format 'tag:value [+ tag:value]...'",
)
}
func tuiFunc(cmd *cobra.Command, _ []string) {
common.ExitOnErr(cmd, runTUI(cmd))
}
func runTUI(cmd *cobra.Command) error {
db, err := openDB(false)
if err != nil {
return fmt.Errorf("couldn't open database: %w", err)
}
defer db.Close()
// Need if app was stopped with Ctrl-C.
ctx, cancel := context.WithCancel(cmd.Context())
defer cancel()
app := tview.NewApplication()
ui := tui.NewUI(ctx, app, db, schema.MetabaseParser, nil)
_ = ui.AddFilter("cid", tui.CIDParser, "CID")
_ = ui.AddFilter("oid", tui.OIDParser, "OID")
_ = ui.AddCompositeFilter("addr", tui.AddressParser, "CID/OID")
_ = ui.AddCompositeFilter("attr", tui.AttributeParser, "key[/value]")
err = ui.WithPrompt(initialPrompt)
if err != nil {
return fmt.Errorf("invalid filter prompt: %w", err)
}
app.SetRoot(ui, true).SetFocus(ui)
return app.Run()
}
func openDB(writable bool) (*bbolt.DB, error) {
db, err := bbolt.Open(vPath, 0o600, &bbolt.Options{
ReadOnly: !writable,
})
if err != nil {
return nil, err
}
return db, nil
}

View file

@ -0,0 +1,96 @@
package blobovnicza
import (
"encoding/binary"
"errors"
"fmt"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/mr-tron/base58"
)
var BlobovniczaParser = common.WithFallback(
common.Any(
MetaBucketParser,
BucketParser,
),
common.RawParser.ToFallbackParser(),
)
func MetaBucketParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if value != nil {
return nil, nil, errors.New("not a bucket")
}
if string(key) != "META" {
return nil, nil, errors.New("invalid bucket name")
}
return &MetaBucket{}, MetaRecordParser, nil
}
func MetaRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
var r MetaRecord
if len(key) == 0 {
return nil, nil, errors.New("invalid key")
}
r.label = string(key)
r.count = binary.LittleEndian.Uint64(value)
return &r, nil, nil
}
func BucketParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if value != nil {
return nil, nil, errors.New("not a bucket")
}
size, n := binary.Varint(key)
if n <= 0 {
return nil, nil, errors.New("invalid size")
}
return &Bucket{size: size}, RecordParser, nil
}
func RecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
parts := strings.Split(string(key), "/")
if len(parts) != 2 {
return nil, nil, errors.New("invalid key, expected address string <CID>/<OID>")
}
cnrRaw, err := base58.Decode(parts[0])
if err != nil {
return nil, nil, errors.New("can't decode CID string")
}
objRaw, err := base58.Decode(parts[1])
if err != nil {
return nil, nil, errors.New("can't decode OID string")
}
cnr := cid.ID{}
if err := cnr.Decode(cnrRaw); err != nil {
return nil, nil, fmt.Errorf("can't decode CID: %w", err)
}
obj := oid.ID{}
if err := obj.Decode(objRaw); err != nil {
return nil, nil, fmt.Errorf("can't decode OID: %w", err)
}
var r Record
r.addr.SetContainer(cnr)
r.addr.SetObject(obj)
if err := r.object.Unmarshal(value); err != nil {
return nil, nil, errors.New("can't unmarshal object")
}
return &r, nil, nil
}

View file

@ -0,0 +1,101 @@
package blobovnicza
import (
"fmt"
"strconv"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/davecgh/go-spew/spew"
"github.com/gdamore/tcell/v2"
"github.com/rivo/tview"
)
type (
MetaBucket struct{}
MetaRecord struct {
label string
count uint64
}
Bucket struct {
size int64
}
Record struct {
addr oid.Address
object objectSDK.Object
}
)
func (b *MetaBucket) String() string {
return common.FormatSimple("META", tcell.ColorLime)
}
func (b *MetaBucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *MetaBucket) Filter(string, any) common.FilterResult {
return common.No
}
func (r *MetaRecord) String() string {
return fmt.Sprintf("%-11s %c %d", r.label, tview.Borders.Vertical, r.count)
}
func (r *MetaRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *MetaRecord) Filter(string, any) common.FilterResult {
return common.No
}
func (b *Bucket) String() string {
return common.FormatSimple(strconv.FormatInt(b.size, 10), tcell.ColorLime)
}
func (b *Bucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *Bucket) Filter(typ string, _ any) common.FilterResult {
switch typ {
case "cid":
return common.Maybe
case "oid":
return common.Maybe
default:
return common.No
}
}
func (r *Record) String() string {
return fmt.Sprintf(
"CID %s OID %s %c Object {...}",
common.FormatSimple(fmt.Sprintf("%-44s", r.addr.Container()), tcell.ColorAqua),
common.FormatSimple(fmt.Sprintf("%-44s", r.addr.Object()), tcell.ColorAqua),
tview.Borders.Vertical,
)
}
func (r *Record) DetailedString() string {
return spew.Sdump(*r)
}
func (r *Record) Filter(typ string, val any) common.FilterResult {
switch typ {
case "cid":
id := val.(cid.ID)
return common.IfThenElse(r.addr.Container().Equals(id), common.Yes, common.No)
case "oid":
id := val.(oid.ID)
return common.IfThenElse(r.addr.Object().Equals(id), common.Yes, common.No)
default:
return common.No
}
}

View file

@ -0,0 +1,43 @@
package common
import (
"fmt"
"strconv"
"github.com/gdamore/tcell/v2"
)
type FormatOptions struct {
Color tcell.Color
Bold,
Italic,
Underline,
StrikeThrough bool
}
func Format(s string, opts FormatOptions) string {
var boldTag, italicTag, underlineTag, strikeThroughTag string
switch {
case opts.Bold:
boldTag = "b"
case opts.Italic:
italicTag = "i"
case opts.Underline:
underlineTag = "u"
case opts.StrikeThrough:
strikeThroughTag = "s"
}
attrs := fmt.Sprintf(
"%s%s%s%s", boldTag, italicTag, underlineTag, strikeThroughTag,
)
color := strconv.FormatInt(int64(opts.Color.Hex()), 16)
return fmt.Sprintf("[#%06s::%s]%s[-::-]", color, attrs, s)
}
func FormatSimple(s string, c tcell.Color) string {
return Format(s, FormatOptions{Color: c})
}

View file

@ -0,0 +1,29 @@
package common
import (
"github.com/davecgh/go-spew/spew"
"github.com/gdamore/tcell/v2"
"github.com/mr-tron/base58"
)
type RawEntry struct {
key, value []byte
}
var RawParser Parser = rawParser
func rawParser(key, value []byte) (SchemaEntry, Parser, error) {
return &RawEntry{key: key, value: value}, rawParser, nil
}
func (r *RawEntry) String() string {
return FormatSimple(base58.Encode(r.key), tcell.ColorRed)
}
func (r *RawEntry) DetailedString() string {
return spew.Sdump(r)
}
func (r *RawEntry) Filter(string, any) FilterResult {
return No
}

View file

@ -0,0 +1,81 @@
package common
import (
"errors"
"fmt"
)
type FilterResult byte
const (
No FilterResult = iota
Maybe
Yes
)
func IfThenElse(condition bool, onSuccess, onFailure FilterResult) FilterResult {
var res FilterResult
if condition {
res = onSuccess
} else {
res = onFailure
}
return res
}
type SchemaEntry interface {
String() string
DetailedString() string
Filter(typ string, val any) FilterResult
}
type (
Parser func(key, value []byte) (SchemaEntry, Parser, error)
FallbackParser func(key, value []byte) (SchemaEntry, Parser)
)
func Any(parsers ...Parser) Parser {
return func(key, value []byte) (SchemaEntry, Parser, error) {
var errs error
for _, parser := range parsers {
ret, next, err := parser(key, value)
if err == nil {
return ret, next, nil
}
errs = errors.Join(errs, err)
}
return nil, nil, fmt.Errorf("no parser succeeded: %w", errs)
}
}
func WithFallback(parser Parser, fallback FallbackParser) Parser {
if parser == nil {
return fallback.ToParser()
}
return func(key, value []byte) (SchemaEntry, Parser, error) {
entry, next, err := parser(key, value)
if err == nil {
return entry, WithFallback(next, fallback), nil
}
return fallback.ToParser()(key, value)
}
}
func (fp FallbackParser) ToParser() Parser {
return func(key, value []byte) (SchemaEntry, Parser, error) {
entry, next := fp(key, value)
return entry, next, nil
}
}
func (p Parser) ToFallbackParser() FallbackParser {
return func(key, value []byte) (SchemaEntry, Parser) {
entry, next, err := p(key, value)
if err != nil {
panic(fmt.Errorf(
"couldn't use that parser as a fallback parser, it returned an error: %w", err,
))
}
return entry, next
}
}

View file

@ -0,0 +1,29 @@
package buckets
import (
"github.com/davecgh/go-spew/spew"
)
func (b *PrefixBucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *PrefixContainerBucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *UserBucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *ContainerBucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *UserAttributeKeyBucket) DetailedString() string {
return spew.Sdump(*b)
}
func (b *UserAttributeValueBucket) DetailedString() string {
return spew.Sdump(*b)
}

View file

@ -0,0 +1,81 @@
package buckets
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
)
func (b *PrefixBucket) Filter(typ string, _ any) common.FilterResult {
switch typ {
case "cid":
return b.resolvers.cidResolver(false)
case "oid":
return b.resolvers.oidResolver(false)
default:
return common.No
}
}
func (b *PrefixContainerBucket) Filter(typ string, val any) common.FilterResult {
switch typ {
case "cid":
id := val.(cid.ID)
return b.resolvers.cidResolver(b.id.Equals(id))
case "oid":
return b.resolvers.oidResolver(false)
default:
return common.No
}
}
func (b *UserBucket) Filter(typ string, _ any) common.FilterResult {
switch typ {
case "cid":
return b.resolvers.cidResolver(false)
case "oid":
return b.resolvers.oidResolver(false)
default:
return common.No
}
}
func (b *ContainerBucket) Filter(typ string, val any) common.FilterResult {
switch typ {
case "cid":
id := val.(cid.ID)
return b.resolvers.cidResolver(b.id.Equals(id))
case "oid":
return b.resolvers.oidResolver(false)
default:
return common.No
}
}
func (b *UserAttributeKeyBucket) Filter(typ string, val any) common.FilterResult {
switch typ {
case "cid":
id := val.(cid.ID)
return common.IfThenElse(b.id.Equals(id), common.Yes, common.No)
case "oid":
return common.Maybe
case "key":
key := val.(string)
return common.IfThenElse(b.key == key, common.Yes, common.No)
case "value":
return common.Maybe
default:
return common.No
}
}
func (b *UserAttributeValueBucket) Filter(typ string, val any) common.FilterResult {
switch typ {
case "oid":
return common.Maybe
case "value":
value := val.(string)
return common.IfThenElse(b.value == value, common.Yes, common.No)
default:
return common.No
}
}

View file

@ -0,0 +1,111 @@
package buckets
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/metabase/records"
)
var (
GraveyardParser = NewPrefixBucketParser(Graveyard, records.GraveyardRecordParser, Resolvers{
cidResolver: LenientResolver,
oidResolver: LenientResolver,
})
GarbageParser = NewPrefixBucketParser(Garbage, records.GarbageRecordParser, Resolvers{
cidResolver: LenientResolver,
oidResolver: LenientResolver,
})
ContainerVolumeParser = NewPrefixBucketParser(ContainerVolume, records.ContainerVolumeRecordParser, Resolvers{
cidResolver: LenientResolver,
oidResolver: StrictResolver,
})
LockedParser = NewPrefixBucketParser(
Locked,
NewContainerBucketParser(
records.LockedRecordParser,
Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
},
),
Resolvers{
cidResolver: LenientResolver,
oidResolver: LenientResolver,
},
)
ShardInfoParser = NewPrefixBucketParser(ShardInfo, records.ShardInfoRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: StrictResolver,
})
PrimaryParser = NewPrefixContainerBucketParser(Primary, records.ObjectRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
LockersParser = NewPrefixContainerBucketParser(Lockers, records.ObjectRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
TombstoneParser = NewPrefixContainerBucketParser(Tombstone, records.ObjectRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
SmallParser = NewPrefixContainerBucketParser(Small, records.SmallRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
RootParser = NewPrefixContainerBucketParser(Root, records.RootRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
OwnerParser = NewPrefixContainerBucketParser(
Owner,
NewUserBucketParser(
records.OwnerRecordParser,
Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
},
),
Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
},
)
UserAttributeParser = NewUserAttributeKeyBucketParser(
NewUserAttributeValueBucketParser(records.UserAttributeRecordParser),
)
PayloadHashParser = NewPrefixContainerBucketParser(PayloadHash, records.PayloadHashRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: StrictResolver,
})
ParentParser = NewPrefixContainerBucketParser(Parent, records.ParentRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
SplitParser = NewPrefixContainerBucketParser(Split, records.SplitRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: StrictResolver,
})
ContainerCountersParser = NewPrefixBucketParser(ContainerCounters, records.ContainerCountersRecordParser, Resolvers{
cidResolver: LenientResolver,
oidResolver: StrictResolver,
})
ECInfoParser = NewPrefixContainerBucketParser(ECInfo, records.ECInfoRecordParser, Resolvers{
cidResolver: StrictResolver,
oidResolver: LenientResolver,
})
)

View file

@ -0,0 +1,53 @@
package buckets
type Prefix byte
const (
Graveyard Prefix = iota
Garbage
ToMoveIt
ContainerVolume
Locked
ShardInfo
Primary
Lockers
_
Tombstone
Small
Root
Owner
UserAttribute
PayloadHash
Parent
Split
ContainerCounters
ECInfo
)
var x = map[Prefix]string{
Graveyard: "Graveyard",
Garbage: "Garbage",
ToMoveIt: "To Move It",
ContainerVolume: "Container Volume",
Locked: "Locked",
ShardInfo: "Shard Info",
Primary: "Primary",
Lockers: "Lockers",
Tombstone: "Tombstone",
Small: "Small",
Root: "Root",
Owner: "Owner",
UserAttribute: "User Attribute",
PayloadHash: "Payload Hash",
Parent: "Parent",
Split: "Split",
ContainerCounters: "Container Counters",
ECInfo: "EC Info",
}
func (p Prefix) String() string {
if s, ok := x[p]; ok {
return s
}
return "Unknown Prefix"
}

View file

@ -0,0 +1,48 @@
package buckets
import (
"fmt"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
"github.com/gdamore/tcell/v2"
)
func (b *PrefixBucket) String() string {
return common.FormatSimple(
fmt.Sprintf("(%2d %-18s)", b.prefix, b.prefix), tcell.ColorLime,
)
}
func (b *PrefixContainerBucket) String() string {
return fmt.Sprintf(
"%s CID %s",
common.FormatSimple(
fmt.Sprintf("(%2d %-18s)", b.prefix, b.prefix), tcell.ColorLime,
),
common.FormatSimple(b.id.String(), tcell.ColorAqua),
)
}
func (b *UserBucket) String() string {
return "UID " + common.FormatSimple(b.id.String(), tcell.ColorAqua)
}
func (b *ContainerBucket) String() string {
return "CID " + common.FormatSimple(b.id.String(), tcell.ColorAqua)
}
func (b *UserAttributeKeyBucket) String() string {
return fmt.Sprintf("%s CID %s ATTR-KEY %s",
common.FormatSimple(
fmt.Sprintf("(%2d %-18s)", b.prefix, b.prefix), tcell.ColorLime,
),
common.FormatSimple(
fmt.Sprintf("%-44s", b.id), tcell.ColorAqua,
),
common.FormatSimple(b.key, tcell.ColorAqua),
)
}
func (b *UserAttributeValueBucket) String() string {
return "ATTR-VALUE " + common.FormatSimple(b.value, tcell.ColorAqua)
}

View file

@ -0,0 +1,166 @@
package buckets
import (
"errors"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/mr-tron/base58"
)
type (
PrefixBucket struct {
prefix Prefix
resolvers Resolvers
}
PrefixContainerBucket struct {
prefix Prefix
id cid.ID
resolvers Resolvers
}
ContainerBucket struct {
id cid.ID
resolvers Resolvers
}
UserBucket struct {
id user.ID
resolvers Resolvers
}
UserAttributeKeyBucket struct {
prefix Prefix
id cid.ID
key string
}
UserAttributeValueBucket struct {
value string
}
)
type (
FilterResolver = func(result bool) common.FilterResult
Resolvers struct {
cidResolver FilterResolver
oidResolver FilterResolver
}
)
var (
StrictResolver = func(x bool) common.FilterResult { return common.IfThenElse(x, common.Yes, common.No) }
LenientResolver = func(x bool) common.FilterResult { return common.IfThenElse(x, common.Yes, common.Maybe) }
)
var (
ErrNotBucket = errors.New("not a bucket")
ErrInvalidKeyLength = errors.New("invalid key length")
ErrInvalidValueLength = errors.New("invalid value length")
ErrInvalidPrefix = errors.New("invalid prefix")
)
func NewPrefixBucketParser(prefix Prefix, next common.Parser, resolvers Resolvers) common.Parser {
return func(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if value != nil {
return nil, nil, ErrNotBucket
}
if len(key) != 1 {
return nil, nil, ErrInvalidKeyLength
}
var b PrefixBucket
if b.prefix = Prefix(key[0]); b.prefix != prefix {
return nil, nil, ErrInvalidPrefix
}
b.resolvers = resolvers
return &b, next, nil
}
}
func NewPrefixContainerBucketParser(prefix Prefix, next common.Parser, resolvers Resolvers) common.Parser {
return func(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if value != nil {
return nil, nil, ErrNotBucket
}
if len(key) != 33 {
return nil, nil, ErrInvalidKeyLength
}
var b PrefixContainerBucket
if b.prefix = Prefix(key[0]); b.prefix != prefix {
return nil, nil, ErrInvalidPrefix
}
if err := b.id.Decode(key[1:]); err != nil {
return nil, nil, err
}
b.resolvers = resolvers
return &b, next, nil
}
}
func NewUserBucketParser(next common.Parser, resolvers Resolvers) common.Parser {
return func(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if value != nil {
return nil, nil, ErrNotBucket
}
var b UserBucket
if err := b.id.DecodeString(base58.Encode(key)); err != nil {
return nil, nil, err
}
b.resolvers = resolvers
return &b, next, nil
}
}
func NewContainerBucketParser(next common.Parser, resolvers Resolvers) common.Parser {
return func(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if value != nil {
return nil, nil, ErrNotBucket
}
if len(key) != 32 {
return nil, nil, ErrInvalidKeyLength
}
var b ContainerBucket
if err := b.id.Decode(key); err != nil {
return nil, nil, err
}
b.resolvers = resolvers
return &b, next, nil
}
}
func NewUserAttributeKeyBucketParser(next common.Parser) common.Parser {
return func(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if value != nil {
return nil, nil, ErrNotBucket
}
if len(key) < 34 {
return nil, nil, ErrInvalidKeyLength
}
var b UserAttributeKeyBucket
if b.prefix = Prefix(key[0]); b.prefix != UserAttribute {
return nil, nil, ErrInvalidPrefix
}
if err := b.id.Decode(key[1:33]); err != nil {
return nil, nil, err
}
b.key = string(key[33:])
return &b, next, nil
}
}
func NewUserAttributeValueBucketParser(next common.Parser) common.Parser {
return func(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if value != nil {
return nil, nil, ErrNotBucket
}
if len(key) == 0 {
return nil, nil, ErrInvalidKeyLength
}
var b UserAttributeValueBucket
b.value = string(key)
return &b, next, nil
}
}

View file

@ -0,0 +1,29 @@
package metabase
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/metabase/buckets"
)
var MetabaseParser = common.WithFallback(
common.Any(
buckets.GraveyardParser,
buckets.GarbageParser,
buckets.ContainerVolumeParser,
buckets.LockedParser,
buckets.ShardInfoParser,
buckets.PrimaryParser,
buckets.LockersParser,
buckets.TombstoneParser,
buckets.SmallParser,
buckets.RootParser,
buckets.OwnerParser,
buckets.UserAttributeParser,
buckets.PayloadHashParser,
buckets.ParentParser,
buckets.SplitParser,
buckets.ContainerCountersParser,
buckets.ECInfoParser,
),
common.RawParser.ToFallbackParser(),
)

View file

@ -0,0 +1,65 @@
package records
import (
"github.com/davecgh/go-spew/spew"
)
func (r *GraveyardRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *GarbageRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *ContainerVolumeRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *LockedRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *ShardInfoRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *ObjectRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *SmallRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *RootRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *OwnerRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *UserAttributeRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *PayloadHashRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *ParentRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *SplitRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *ContainerCountersRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (r *ECInfoRecord) DetailedString() string {
return spew.Sdump(*r)
}

View file

@ -0,0 +1,145 @@
package records
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
)
func (r *GraveyardRecord) Filter(typ string, val any) common.FilterResult {
switch typ {
case "cid":
id := val.(cid.ID)
return common.IfThenElse(r.object.Container().Equals(id), common.Yes, common.No)
case "oid":
id := val.(oid.ID)
return common.IfThenElse(r.object.Object().Equals(id), common.Yes, common.No)
default:
return common.No
}
}
func (r *GarbageRecord) Filter(typ string, val any) common.FilterResult {
switch typ {
case "cid":
id := val.(cid.ID)
return common.IfThenElse(r.addr.Container().Equals(id), common.Yes, common.No)
case "oid":
id := val.(oid.ID)
return common.IfThenElse(r.addr.Object().Equals(id), common.Yes, common.No)
default:
return common.No
}
}
func (r *ContainerVolumeRecord) Filter(typ string, val any) common.FilterResult {
switch typ {
case "cid":
id := val.(cid.ID)
return common.IfThenElse(r.id.Equals(id), common.Yes, common.No)
default:
return common.No
}
}
func (r *ShardInfoRecord) Filter(string, any) common.FilterResult {
return common.No
}
func (r *LockedRecord) Filter(typ string, val any) common.FilterResult {
switch typ {
case "oid":
id := val.(oid.ID)
return common.IfThenElse(r.id.Equals(id), common.Yes, common.No)
default:
return common.No
}
}
func (r *ObjectRecord) Filter(typ string, val any) common.FilterResult {
switch typ {
case "oid":
id := val.(oid.ID)
return common.IfThenElse(r.id.Equals(id), common.Yes, common.No)
default:
return common.No
}
}
func (r *SmallRecord) Filter(typ string, val any) common.FilterResult {
switch typ {
case "oid":
id := val.(oid.ID)
return common.IfThenElse(r.id.Equals(id), common.Yes, common.No)
default:
return common.No
}
}
func (r *RootRecord) Filter(typ string, val any) common.FilterResult {
switch typ {
case "oid":
id := val.(oid.ID)
return common.IfThenElse(r.id.Equals(id), common.Yes, common.No)
default:
return common.No
}
}
func (r *OwnerRecord) Filter(typ string, val any) common.FilterResult {
switch typ {
case "oid":
id := val.(oid.ID)
return common.IfThenElse(r.id.Equals(id), common.Yes, common.No)
default:
return common.No
}
}
func (r *UserAttributeRecord) Filter(typ string, val any) common.FilterResult {
switch typ {
case "oid":
id := val.(oid.ID)
return common.IfThenElse(r.id.Equals(id), common.Yes, common.No)
default:
return common.No
}
}
func (r *PayloadHashRecord) Filter(string, any) common.FilterResult {
return common.No
}
func (r *ParentRecord) Filter(typ string, val any) common.FilterResult {
switch typ {
case "oid":
id := val.(oid.ID)
return common.IfThenElse(r.parent.Equals(id), common.Yes, common.No)
default:
return common.No
}
}
func (r *SplitRecord) Filter(string, any) common.FilterResult {
return common.No
}
func (r *ContainerCountersRecord) Filter(typ string, val any) common.FilterResult {
switch typ {
case "cid":
id := val.(cid.ID)
return common.IfThenElse(r.id.Equals(id), common.Yes, common.No)
default:
return common.No
}
}
func (r *ECInfoRecord) Filter(typ string, val any) common.FilterResult {
switch typ {
case "oid":
id := val.(oid.ID)
return common.IfThenElse(r.id.Equals(id), common.Yes, common.No)
default:
return common.No
}
}

View file

@ -0,0 +1,251 @@
package records
import (
"encoding/binary"
"errors"
"strconv"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
"git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
)
var (
ErrInvalidKeyLength = errors.New("invalid key length")
ErrInvalidValueLength = errors.New("invalid value length")
ErrInvalidPrefix = errors.New("invalid prefix")
)
func GraveyardRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if len(key) != 64 {
return nil, nil, ErrInvalidKeyLength
}
if len(value) != 64 {
return nil, nil, ErrInvalidValueLength
}
var (
cnr cid.ID
obj oid.ID
r GraveyardRecord
)
_ = cnr.Decode(key[:32])
_ = obj.Decode(key[32:])
r.object.SetContainer(cnr)
r.object.SetObject(obj)
_ = cnr.Decode(value[:32])
_ = obj.Decode(value[32:])
r.tombstone.SetContainer(cnr)
r.tombstone.SetObject(obj)
return &r, nil, nil
}
func GarbageRecordParser(key, _ []byte) (common.SchemaEntry, common.Parser, error) {
if len(key) != 64 {
return nil, nil, ErrInvalidKeyLength
}
var (
cnr cid.ID
obj oid.ID
r GarbageRecord
)
_ = cnr.Decode(key[:32])
_ = obj.Decode(key[32:])
r.addr.SetContainer(cnr)
r.addr.SetObject(obj)
return &r, nil, nil
}
func ContainerVolumeRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if len(key) != 32 {
return nil, nil, ErrInvalidKeyLength
}
if len(value) != 8 {
return nil, nil, ErrInvalidValueLength
}
var r ContainerVolumeRecord
_ = r.id.Decode(key)
r.volume = binary.LittleEndian.Uint64(value)
return &r, nil, nil
}
func LockedRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
var (
r LockedRecord
err error
)
if err := r.id.Decode(key); err != nil {
return nil, nil, err
}
if r.ids, err = DecodeOIDs(value); err != nil {
return nil, nil, err
}
return &r, nil, nil
}
func ShardInfoRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if len(key) == 0 {
return nil, nil, ErrInvalidKeyLength
}
var r ShardInfoRecord
if string(key) == "id" {
r.label = string(key)
r.value = shard.ID(value).String()
return &r, nil, nil
}
if len(value) != 8 {
return nil, nil, ErrInvalidValueLength
}
r.label = string(key)
r.value = strconv.FormatUint(binary.LittleEndian.Uint64(value), 10)
return &r, nil, nil
}
func ObjectRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if len(key) != 32 {
return nil, nil, ErrInvalidKeyLength
}
var r ObjectRecord
_ = r.id.Decode(key)
if err := r.object.Unmarshal(value); err != nil {
return nil, nil, err
}
return &r, nil, nil
}
func SmallRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
var r SmallRecord
if err := r.id.Decode(key); err != nil {
return nil, nil, err
}
if len(value) != 0 {
x := string(value)
r.storageID = &x
}
return &r, nil, nil
}
func RootRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
var r RootRecord
if err := r.id.Decode(key); err != nil {
return nil, nil, err
}
if len(value) == 0 {
return &r, nil, nil
}
r.info = &objectSDK.SplitInfo{}
if err := r.info.Unmarshal(value); err != nil {
return nil, nil, err
}
return &r, nil, nil
}
func OwnerRecordParser(key, _ []byte) (common.SchemaEntry, common.Parser, error) {
var r OwnerRecord
if err := r.id.Decode(key); err != nil {
return nil, nil, err
}
return &r, nil, nil
}
func UserAttributeRecordParser(key, _ []byte) (common.SchemaEntry, common.Parser, error) {
var r UserAttributeRecord
if err := r.id.Decode(key); err != nil {
return nil, nil, err
}
return &r, nil, nil
}
func PayloadHashRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if len(key) != 32 {
return nil, nil, ErrInvalidKeyLength
}
var (
err error
r PayloadHashRecord
)
r.checksum.SetSHA256([32]byte(key))
if r.ids, err = DecodeOIDs(value); err != nil {
return nil, nil, err
}
return &r, nil, nil
}
func ParentRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
var (
r ParentRecord
err error
)
if err = r.parent.Decode(key); err != nil {
return nil, nil, err
}
if r.ids, err = DecodeOIDs(value); err != nil {
return nil, nil, err
}
return &r, nil, nil
}
func SplitRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
var (
err error
r SplitRecord
)
if err = r.id.UnmarshalBinary(key); err != nil {
return nil, nil, err
}
if r.ids, err = DecodeOIDs(value); err != nil {
return nil, nil, err
}
return &r, nil, nil
}
func ContainerCountersRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if len(value) != 24 {
return nil, nil, ErrInvalidValueLength
}
var r ContainerCountersRecord
if err := r.id.Decode(key); err != nil {
return nil, nil, err
}
r.logical = binary.LittleEndian.Uint64(value[:8])
r.physical = binary.LittleEndian.Uint64(value[8:16])
r.user = binary.LittleEndian.Uint64(value[16:24])
return &r, nil, nil
}
func ECInfoRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
var (
r ECInfoRecord
err error
)
if err := r.id.Decode(key); err != nil {
return nil, nil, err
}
if r.ids, err = DecodeOIDs(value); err != nil {
return nil, nil, err
}
return &r, nil, nil
}

View file

@ -0,0 +1,135 @@
package records
import (
"fmt"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
"github.com/gdamore/tcell/v2"
"github.com/rivo/tview"
)
func (r *GraveyardRecord) String() string {
return fmt.Sprintf(
"Object CID %s OID %s %c Tombstone CID %s OID %s",
common.FormatSimple(fmt.Sprintf("%-44s", r.object.Container()), tcell.ColorAqua),
common.FormatSimple(fmt.Sprintf("%-44s", r.object.Object()), tcell.ColorAqua),
tview.Borders.Vertical,
common.FormatSimple(fmt.Sprintf("%-44s", r.tombstone.Container()), tcell.ColorAqua),
common.FormatSimple(fmt.Sprintf("%-44s", r.tombstone.Object()), tcell.ColorAqua),
)
}
func (r *GarbageRecord) String() string {
return fmt.Sprintf(
"CID %-44s OID %-44s",
common.FormatSimple(fmt.Sprintf("%-44s", r.addr.Container()), tcell.ColorAqua),
common.FormatSimple(fmt.Sprintf("%-44s", r.addr.Object()), tcell.ColorAqua),
)
}
func (r *ContainerVolumeRecord) String() string {
return fmt.Sprintf(
"CID %-44s %c %d",
common.FormatSimple(fmt.Sprintf("%-44s", r.id), tcell.ColorAqua),
tview.Borders.Vertical,
r.volume,
)
}
func (r *LockedRecord) String() string {
return fmt.Sprintf(
"Locker OID %s %c Locked [%d]OID {...}",
common.FormatSimple(fmt.Sprintf("%-44s", r.id), tcell.ColorAqua),
tview.Borders.Vertical,
len(r.ids),
)
}
func (r *ShardInfoRecord) String() string {
return fmt.Sprintf("%-13s %c %s", r.label, tview.Borders.Vertical, r.value)
}
func (r *ObjectRecord) String() string {
return fmt.Sprintf(
"OID %s %c Object {...}",
common.FormatSimple(fmt.Sprintf("%-44s", r.id), tcell.ColorAqua),
tview.Borders.Vertical,
)
}
func (r *SmallRecord) String() string {
s := fmt.Sprintf(
"OID %s %c",
common.FormatSimple(fmt.Sprintf("%-44s", r.id), tcell.ColorAqua),
tview.Borders.Vertical,
)
if r.storageID != nil {
s = fmt.Sprintf("%s %s", s, *r.storageID)
}
return s
}
func (r *RootRecord) String() string {
s := fmt.Sprintf(
"Root OID %s %c",
common.FormatSimple(fmt.Sprintf("%-44s", r.id), tcell.ColorAqua),
tview.Borders.Vertical,
)
if r.info != nil {
s += " Split info {...}"
}
return s
}
func (r *OwnerRecord) String() string {
return "OID " + common.FormatSimple(r.id.String(), tcell.ColorAqua)
}
func (r *UserAttributeRecord) String() string {
return "OID " + common.FormatSimple(r.id.String(), tcell.ColorAqua)
}
func (r *PayloadHashRecord) String() string {
return fmt.Sprintf(
"Checksum %s %c [%d]OID {...}",
common.FormatSimple(r.checksum.String(), tcell.ColorAqua),
tview.Borders.Vertical,
len(r.ids),
)
}
func (r *ParentRecord) String() string {
return fmt.Sprintf(
"Parent OID %s %c [%d]OID {...}",
common.FormatSimple(fmt.Sprintf("%-44s", r.parent), tcell.ColorAqua),
tview.Borders.Vertical,
len(r.ids),
)
}
func (r *SplitRecord) String() string {
return fmt.Sprintf(
"Split ID %s %c [%d]OID {...}",
common.FormatSimple(r.id.String(), tcell.ColorAqua),
tview.Borders.Vertical,
len(r.ids),
)
}
func (r *ContainerCountersRecord) String() string {
return fmt.Sprintf(
"CID %s %c logical %d, physical %d, user %d",
common.FormatSimple(fmt.Sprintf("%-44s", r.id), tcell.ColorAqua),
tview.Borders.Vertical,
r.logical, r.physical, r.user,
)
}
func (r *ECInfoRecord) String() string {
return fmt.Sprintf(
"OID %s %c [%d]OID {...}",
common.FormatSimple(fmt.Sprintf("%-44s", r.id), tcell.ColorAqua),
tview.Borders.Vertical,
len(r.ids),
)
}

View file

@ -0,0 +1,82 @@
package records
import (
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/checksum"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/google/uuid"
)
type (
GraveyardRecord struct {
object, tombstone oid.Address
}
GarbageRecord struct {
addr oid.Address
}
ContainerVolumeRecord struct {
id cid.ID
volume uint64
}
LockedRecord struct {
id oid.ID
ids []oid.ID
}
ShardInfoRecord struct {
label string
value string
}
ObjectRecord struct {
id oid.ID
object objectSDK.Object
}
SmallRecord struct {
id oid.ID
storageID *string // optional
}
RootRecord struct {
id oid.ID
info *objectSDK.SplitInfo // optional
}
OwnerRecord struct {
id oid.ID
}
UserAttributeRecord struct {
id oid.ID
}
PayloadHashRecord struct {
checksum checksum.Checksum
ids []oid.ID
}
ParentRecord struct {
parent oid.ID
ids []oid.ID
}
SplitRecord struct {
id uuid.UUID
ids []oid.ID
}
ContainerCountersRecord struct {
id cid.ID
logical, physical, user uint64
}
ECInfoRecord struct {
id oid.ID
ids []oid.ID
}
)

View file

@ -0,0 +1,20 @@
package records
import (
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/nspcc-dev/neo-go/pkg/io"
)
func DecodeOIDs(data []byte) ([]oid.ID, error) {
r := io.NewBinReaderFromBuf(data)
size := r.ReadVarUint()
oids := make([]oid.ID, size)
for i := uint64(0); i < size; i++ {
if err := oids[i].Decode(r.ReadVarBytes()); err != nil {
return nil, err
}
}
return oids, nil
}

View file

@ -0,0 +1,63 @@
package writecache
import (
"bytes"
"errors"
"fmt"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/mr-tron/base58"
)
var WritecacheParser = common.WithFallback(
DefaultBucketParser,
common.RawParser.ToFallbackParser(),
)
func DefaultBucketParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
if value != nil {
return nil, nil, errors.New("not a bucket")
}
if !bytes.Equal(key, []byte{0}) {
return nil, nil, errors.New("invalid key")
}
return &DefaultBucket{}, DefaultRecordParser, nil
}
func DefaultRecordParser(key, value []byte) (common.SchemaEntry, common.Parser, error) {
parts := strings.Split(string(key), "/")
if len(parts) != 2 {
return nil, nil, errors.New("invalid key, expected address string <CID>/<OID>")
}
cnrRaw, err := base58.Decode(parts[0])
if err != nil {
return nil, nil, errors.New("can't decode CID string")
}
objRaw, err := base58.Decode(parts[1])
if err != nil {
return nil, nil, errors.New("can't decode OID string")
}
cnr := cid.ID{}
if err := cnr.Decode(cnrRaw); err != nil {
return nil, nil, fmt.Errorf("can't decode CID: %w", err)
}
obj := oid.ID{}
if err := obj.Decode(objRaw); err != nil {
return nil, nil, fmt.Errorf("can't decode OID: %w", err)
}
var r DefaultRecord
r.addr.SetContainer(cnr)
r.addr.SetObject(obj)
r.data = value[:]
return &r, nil, nil
}

View file

@ -0,0 +1,66 @@
package writecache
import (
"fmt"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/davecgh/go-spew/spew"
"github.com/gdamore/tcell/v2"
"github.com/rivo/tview"
)
type (
DefaultBucket struct{}
DefaultRecord struct {
addr oid.Address
data []byte
}
)
func (b *DefaultBucket) String() string {
return common.FormatSimple("0 Default", tcell.ColorLime)
}
func (r *DefaultRecord) String() string {
return fmt.Sprintf(
"CID %s OID %s %c Data {...}",
common.FormatSimple(fmt.Sprintf("%-44s", r.addr.Container()), tcell.ColorAqua),
common.FormatSimple(fmt.Sprintf("%-44s", r.addr.Object()), tcell.ColorAqua),
tview.Borders.Vertical,
)
}
func (b *DefaultBucket) DetailedString() string {
return spew.Sdump(*b)
}
func (r *DefaultRecord) DetailedString() string {
return spew.Sdump(*r)
}
func (b *DefaultBucket) Filter(typ string, _ any) common.FilterResult {
switch typ {
case "cid":
return common.Maybe
case "oid":
return common.Maybe
default:
return common.No
}
}
func (r *DefaultRecord) Filter(typ string, val any) common.FilterResult {
switch typ {
case "cid":
id := val.(cid.ID)
return common.IfThenElse(r.addr.Container().Equals(id), common.Yes, common.No)
case "oid":
id := val.(oid.ID)
return common.IfThenElse(r.addr.Object().Equals(id), common.Yes, common.No)
default:
return common.No
}
}

View file

@ -0,0 +1,257 @@
package tui
import (
"context"
"sync"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
"github.com/gdamore/tcell/v2"
"github.com/rivo/tview"
)
type BucketsView struct {
*tview.Box
mu sync.Mutex
view *tview.TreeView
nodeToUpdate *tview.TreeNode
ui *UI
filter *Filter
}
type bucketNode struct {
bucket *Bucket
filter *Filter
}
func NewBucketsView(ui *UI, filter *Filter) *BucketsView {
return &BucketsView{
Box: tview.NewBox(),
view: tview.NewTreeView(),
ui: ui,
filter: filter,
}
}
func (v *BucketsView) Mount(_ context.Context) error {
root := tview.NewTreeNode(".")
root.SetExpanded(false)
root.SetSelectable(false)
root.SetReference(&bucketNode{
bucket: &Bucket{NextParser: v.ui.rootParser},
filter: v.filter,
})
v.nodeToUpdate = root
v.view.SetRoot(root)
v.view.SetCurrentNode(root)
return nil
}
func (v *BucketsView) Update(ctx context.Context) error {
if v.nodeToUpdate == nil {
return nil
}
defer func() { v.nodeToUpdate = nil }()
ctx, cancel := context.WithCancel(ctx)
defer cancel()
ready := make(chan struct{})
errCh := make(chan error)
tmp := tview.NewTreeNode(v.nodeToUpdate.GetText())
tmp.SetReference(v.nodeToUpdate.GetReference())
node := v.nodeToUpdate.GetReference().(*bucketNode)
go func() {
defer close(ready)
hasBuckets, err := HasBuckets(ctx, v.ui.db, node.bucket.Path)
if err != nil {
errCh <- err
}
// Show the selected bucket's records instead.
if !hasBuckets && node.bucket.NextParser != nil {
v.ui.moveNextPage(NewRecordsView(v.ui, node.bucket, node.filter))
}
if v.nodeToUpdate.IsExpanded() {
return
}
err = v.loadNodeChildren(ctx, tmp, node.filter)
if err != nil {
errCh <- err
}
}()
select {
case <-ctx.Done():
case <-ready:
v.mu.Lock()
v.nodeToUpdate.SetChildren(tmp.GetChildren())
v.nodeToUpdate.SetExpanded(!v.nodeToUpdate.IsExpanded())
v.mu.Unlock()
case err := <-errCh:
return err
}
return nil
}
func (v *BucketsView) Unmount() {
}
func (v *BucketsView) Draw(screen tcell.Screen) {
x, y, width, height := v.GetInnerRect()
v.view.SetRect(x, y, width, height)
v.view.Draw(screen)
}
func (v *BucketsView) loadNodeChildren(
ctx context.Context, node *tview.TreeNode, filter *Filter,
) error {
parentBucket := node.GetReference().(*bucketNode).bucket
path := parentBucket.Path
parser := parentBucket.NextParser
buffer, err := LoadBuckets(ctx, v.ui.db, path, v.ui.loadBufferSize)
if err != nil {
return err
}
for item := range buffer {
if item.err != nil {
return item.err
}
bucket := item.val
bucket.Entry, bucket.NextParser, err = parser(bucket.Name, nil)
if err != nil {
return err
}
satisfies, err := v.bucketSatisfiesFilter(ctx, bucket, filter)
if err != nil {
return err
}
if !satisfies {
continue
}
child := tview.NewTreeNode(bucket.Entry.String()).
SetSelectable(true).
SetExpanded(false).
SetReference(&bucketNode{
bucket: bucket,
filter: filter.Apply(bucket.Entry),
})
node.AddChild(child)
}
return nil
}
func (v *BucketsView) bucketSatisfiesFilter(
ctx context.Context, bucket *Bucket, filter *Filter,
) (bool, error) {
// Does the current bucket satisfies the filter?
filter = filter.Apply(bucket.Entry)
if filter.Result() == common.Yes {
return true, nil
}
if filter.Result() == common.No {
return false, nil
}
ctx, cancel := context.WithCancel(ctx)
defer cancel()
// Check the current bucket's nested buckets if exist
bucketsBuffer, err := LoadBuckets(ctx, v.ui.db, bucket.Path, v.ui.loadBufferSize)
if err != nil {
return false, err
}
for item := range bucketsBuffer {
if item.err != nil {
return false, item.err
}
b := item.val
b.Entry, b.NextParser, err = bucket.NextParser(b.Name, nil)
if err != nil {
return false, err
}
satisfies, err := v.bucketSatisfiesFilter(ctx, b, filter)
if err != nil {
return false, err
}
if satisfies {
return true, nil
}
}
// Check the current bucket's nested records if exist
recordsBuffer, err := LoadRecords(ctx, v.ui.db, bucket.Path, v.ui.loadBufferSize)
if err != nil {
return false, err
}
for item := range recordsBuffer {
if item.err != nil {
return false, item.err
}
r := item.val
r.Entry, _, err = bucket.NextParser(r.Key, r.Value)
if err != nil {
return false, err
}
if filter.Apply(r.Entry).Result() == common.Yes {
return true, nil
}
}
return false, nil
}
func (v *BucketsView) InputHandler() func(event *tcell.EventKey, setFocus func(p tview.Primitive)) {
return v.WrapInputHandler(func(event *tcell.EventKey, _ func(tview.Primitive)) {
currentNode := v.view.GetCurrentNode()
if currentNode == nil {
return
}
switch event.Key() {
case tcell.KeyEnter:
// Expand or collapse the selected bucket's nested buckets,
// otherwise, navigate to that bucket's records.
v.nodeToUpdate = currentNode
case tcell.KeyCtrlR:
// Navigate to the selected bucket's records.
bucketNode := currentNode.GetReference().(*bucketNode)
v.ui.moveNextPage(NewRecordsView(v.ui, bucketNode.bucket, bucketNode.filter))
case tcell.KeyCtrlD:
// Navigate to the selected bucket's detailed view.
bucketNode := currentNode.GetReference().(*bucketNode)
v.ui.moveNextPage(NewDetailedView(bucketNode.bucket.Entry.DetailedString()))
default:
v.view.InputHandler()(event, func(tview.Primitive) {})
}
})
}

View file

@ -0,0 +1,160 @@
package tui
import (
"context"
"errors"
"fmt"
"go.etcd.io/bbolt"
)
type Item[T any] struct {
val T
err error
}
func resolvePath(tx *bbolt.Tx, path [][]byte) (*bbolt.Bucket, error) {
if len(path) == 0 {
return nil, errors.New("can't find bucket without path")
}
name := path[0]
bucket := tx.Bucket(name)
if bucket == nil {
return nil, fmt.Errorf("no bucket with name %s", name)
}
for _, name := range path[1:] {
bucket = bucket.Bucket(name)
if bucket == nil {
return nil, fmt.Errorf("no bucket with name %s", name)
}
}
return bucket, nil
}
func load[T any](
ctx context.Context, db *bbolt.DB, path [][]byte, bufferSize int,
filter func(key, value []byte) bool, transform func(key, value []byte) T,
) (<-chan Item[T], error) {
buffer := make(chan Item[T], bufferSize)
go func() {
defer close(buffer)
err := db.View(func(tx *bbolt.Tx) error {
var cursor *bbolt.Cursor
if len(path) == 0 {
cursor = tx.Cursor()
} else {
bucket, err := resolvePath(tx, path)
if err != nil {
buffer <- Item[T]{err: fmt.Errorf("can't find bucket: %w", err)}
return nil
}
cursor = bucket.Cursor()
}
key, value := cursor.First()
for {
if key == nil {
return nil
}
if filter != nil && !filter(key, value) {
key, value = cursor.Next()
continue
}
select {
case <-ctx.Done():
return nil
case buffer <- Item[T]{val: transform(key, value)}:
key, value = cursor.Next()
}
}
})
if err != nil {
buffer <- Item[T]{err: err}
}
}()
return buffer, nil
}
func LoadBuckets(
ctx context.Context, db *bbolt.DB, path [][]byte, bufferSize int,
) (<-chan Item[*Bucket], error) {
buffer, err := load(
ctx, db, path, bufferSize,
func(_, value []byte) bool {
return value == nil
},
func(key, _ []byte) *Bucket {
base := make([][]byte, 0, len(path))
base = append(base, path...)
return &Bucket{
Name: key,
Path: append(base, key),
}
},
)
if err != nil {
return nil, fmt.Errorf("can't start iterating bucket: %w", err)
}
return buffer, nil
}
func LoadRecords(
ctx context.Context, db *bbolt.DB, path [][]byte, bufferSize int,
) (<-chan Item[*Record], error) {
buffer, err := load(
ctx, db, path, bufferSize,
func(_, value []byte) bool {
return value != nil
},
func(key, value []byte) *Record {
base := make([][]byte, 0, len(path))
base = append(base, path...)
return &Record{
Key: key,
Value: value,
Path: append(base, key),
}
},
)
if err != nil {
return nil, fmt.Errorf("can't start iterating bucket: %w", err)
}
return buffer, nil
}
// HasBuckets checks if a bucket has nested buckets. It relies on assumption
// that a bucket can have either nested buckets or records but not both.
func HasBuckets(ctx context.Context, db *bbolt.DB, path [][]byte) (bool, error) {
ctx, cancel := context.WithCancel(ctx)
defer cancel()
buffer, err := load(
ctx, db, path, 1,
nil,
func(_, value []byte) []byte { return value },
)
if err != nil {
return false, err
}
x, ok := <-buffer
if !ok {
return false, nil
}
if x.err != nil {
return false, err
}
if x.val != nil {
return false, err
}
return true, nil
}

View file

@ -0,0 +1,24 @@
package tui
import (
"context"
"github.com/rivo/tview"
)
type DetailedView struct {
*tview.TextView
}
func NewDetailedView(detailed string) *DetailedView {
v := &DetailedView{
TextView: tview.NewTextView(),
}
v.SetDynamicColors(true)
v.SetText(detailed)
return v
}
func (v *DetailedView) Mount(_ context.Context) error { return nil }
func (v *DetailedView) Update(_ context.Context) error { return nil }
func (v *DetailedView) Unmount() {}

View file

@ -0,0 +1,44 @@
package tui
import (
"maps"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
)
type Filter struct {
values map[string]any
results map[string]common.FilterResult
}
func NewFilter(values map[string]any) *Filter {
f := &Filter{
values: maps.Clone(values),
results: make(map[string]common.FilterResult),
}
for tag := range values {
f.results[tag] = common.No
}
return f
}
func (f *Filter) Apply(e common.SchemaEntry) *Filter {
filter := &Filter{
values: f.values,
results: maps.Clone(f.results),
}
for tag, value := range filter.values {
filter.results[tag] = max(filter.results[tag], e.Filter(tag, value))
}
return filter
}
func (f *Filter) Result() common.FilterResult {
current := common.Yes
for _, r := range f.results {
current = min(r, current)
}
return current
}

View file

@ -0,0 +1,38 @@
[green::b]HOTKEYS[-::-]
[green::b]Navigation[-::-]
[yellow::b]Down Arrow[-::-] / [yellow::b]j[-::-]
Scroll down.
[yellow::b]Up Arrow[-::-] / [yellow::b]k[-::-]
Scroll up.
[yellow::b]Page Down[-::-] / [yellow::b]Ctrl-f[-::-]
Scroll down by a full page.
[yellow::b]Page Up[-::-] / [yellow::b]Ctrl-b[-::-]
Scroll up by a full page.
[green::b]Actions[-::-]
[yellow::b]Enter[-::-]
Perform actions based on the current context:
- In Buckets View:
- Expand/collapse the selected bucket to show/hide its nested buckets.
- If no nested buckets exist, navigate to the selected bucket's records.
- In Records View: Open the detailed view of the selected record.
[yellow::b]Escape[-::-]
Return to the previous page, opposite of [yellow::b]Enter[-::-].
Refer to the [green::b]SEARCHING[-::-] section for more specific actions.
[green::b]Alternative Action Hotkeys[-::-]
[yellow::b]Ctrl-r[-::-]
Directly navigate to the selected bucket's records.
[yellow::b]Ctrl-d[-::-]
Access the detailed view of the selected bucket.

View file

@ -0,0 +1,26 @@
[green::b]SEARCHING[-::-]
[green::b]Hotkeys[-::-]
[yellow::b]/[-::-]
Initiate the search prompt.
- The prompt follows this syntax: [yellow::b]tag:value [+ tag:value]...[-::-]
- Multiple filter can be combined with [yellow::b]+[-::-], the result is an intersection of those filters' result sets.
- Any leading and trailing whitespace will be ignored.
- An empty prompt will return all results with no filters applied.
- Refer to the [green::b]Available Search Filters[-::-] section below for a list of valid filter tags.
[yellow::b]Enter[-::-]
Execute the search based on the entered prompt.
- If the prompt is invalid, an error message will be displayed.
[yellow::b]Escape[-::-]
Exit the search prompt without performing a search.
[yellow::b]Down Arrow[-::-], [yellow::b]Up Arrow[-::-]
Scroll through the search history.
[green::b]Available Search Filters[-::-]
%s

View file

@ -0,0 +1,101 @@
package tui
import (
_ "embed"
"fmt"
"strings"
"github.com/gdamore/tcell/v2"
"github.com/rivo/tview"
)
var (
//go:embed help-pages/hotkeys.txt
hotkeysHelpText string
//go:embed help-pages/searching.txt
searchingHelpText string
)
type HelpPage struct {
*tview.Box
pages []*tview.TextView
currentPage int
filters []string
filterHints map[string]string
}
func NewHelpPage(filters []string, hints map[string]string) *HelpPage {
hp := &HelpPage{
Box: tview.NewBox(),
filters: filters,
filterHints: hints,
}
page := tview.NewTextView().
SetDynamicColors(true).
SetText(hotkeysHelpText)
hp.addPage(page)
page = tview.NewTextView().
SetDynamicColors(true).
SetText(fmt.Sprintf(searchingHelpText, hp.getFiltersText()))
hp.addPage(page)
return hp
}
func (hp *HelpPage) addPage(page *tview.TextView) {
hp.pages = append(hp.pages, page)
}
func (hp *HelpPage) getFiltersText() string {
if len(hp.filters) == 0 {
return "\t\tNo filters defined.\n"
}
filtersText := strings.Builder{}
gapSize := 4
tagMaxWidth := 3
for _, filter := range hp.filters {
tagMaxWidth = max(tagMaxWidth, len(filter))
}
filtersText.WriteString("\t\t[yellow::b]Tag")
filtersText.WriteString(strings.Repeat(" ", gapSize))
filtersText.WriteString("\tValue[-::-]\n\n")
for _, filter := range hp.filters {
filtersText.WriteString("\t\t")
filtersText.WriteString(filter)
filtersText.WriteString(strings.Repeat(" ", tagMaxWidth-len(filter)+gapSize))
filtersText.WriteString(hp.filterHints[filter])
filtersText.WriteRune('\n')
}
return filtersText.String()
}
func (hp *HelpPage) Draw(screen tcell.Screen) {
x, y, width, height := hp.GetInnerRect()
hp.pages[hp.currentPage].SetRect(x+1, y+1, width-2, height-2)
hp.pages[hp.currentPage].Draw(screen)
}
func (hp *HelpPage) InputHandler() func(event *tcell.EventKey, setFocus func(p tview.Primitive)) {
return hp.WrapInputHandler(func(event *tcell.EventKey, _ func(tview.Primitive)) {
if event.Key() == tcell.KeyEnter {
hp.currentPage++
hp.currentPage %= len(hp.pages)
return
}
hp.pages[hp.currentPage].InputHandler()(event, func(tview.Primitive) {})
})
}
func (hp *HelpPage) MouseHandler() func(action tview.MouseAction, event *tcell.EventMouse, setFocus func(p tview.Primitive)) (consumed bool, capture tview.Primitive) {
return hp.WrapMouseHandler(func(action tview.MouseAction, event *tcell.EventMouse, _ func(tview.Primitive)) (consumed bool, capture tview.Primitive) {
return hp.pages[hp.currentPage].MouseHandler()(action, event, func(tview.Primitive) {})
})
}

View file

@ -0,0 +1,77 @@
package tui
import (
"github.com/gdamore/tcell/v2"
"github.com/rivo/tview"
)
type InputFieldWithHistory struct {
*tview.InputField
history []string
historyLimit int
historyPointer int
currentContent string
}
func NewInputFieldWithHistory(historyLimit int) *InputFieldWithHistory {
return &InputFieldWithHistory{
InputField: tview.NewInputField(),
historyLimit: historyLimit,
}
}
func (f *InputFieldWithHistory) AddToHistory(s string) {
// Stop scrolling history on history change, need to start scrolling again.
defer func() { f.historyPointer = len(f.history) }()
// Used history data for search prompt, so just make that data recent.
if f.historyPointer != len(f.history) && s == f.history[f.historyPointer] {
f.history = append(f.history[:f.historyPointer], f.history[f.historyPointer+1:]...)
f.history = append(f.history, s)
}
if len(f.history) == f.historyLimit {
f.history = f.history[1:]
}
f.history = append(f.history, s)
}
func (f *InputFieldWithHistory) InputHandler() func(event *tcell.EventKey, setFocus func(p tview.Primitive)) {
return f.WrapInputHandler(func(event *tcell.EventKey, _ func(tview.Primitive)) {
switch event.Key() {
case tcell.KeyDown:
if len(f.history) == 0 {
return
}
// Need to start iterating before.
if f.historyPointer == len(f.history) {
return
}
// Iterate to most recent prompts.
f.historyPointer++
// Stop iterating over history.
if f.historyPointer == len(f.history) {
f.InputField.SetText(f.currentContent)
return
}
f.InputField.SetText(f.history[f.historyPointer])
case tcell.KeyUp:
if len(f.history) == 0 {
return
}
// Start iterating over history.
if f.historyPointer == len(f.history) {
f.currentContent = f.InputField.GetText()
}
// End of history.
if f.historyPointer == 0 {
return
}
// Iterate to least recent prompts.
f.historyPointer--
f.InputField.SetText(f.history[f.historyPointer])
default:
f.InputField.InputHandler()(event, func(tview.Primitive) {})
}
})
}

View file

@ -0,0 +1,72 @@
package tui
import (
"context"
"fmt"
"sync/atomic"
"time"
"github.com/gdamore/tcell/v2"
"github.com/rivo/tview"
)
type LoadingBar struct {
*tview.Box
view *tview.TextView
secondsElapsed atomic.Int64
needDrawFunc func()
reset func()
}
func NewLoadingBar(needDrawFunc func()) *LoadingBar {
b := &LoadingBar{
Box: tview.NewBox(),
view: tview.NewTextView(),
needDrawFunc: needDrawFunc,
}
b.view.SetBackgroundColor(tview.Styles.PrimaryTextColor)
b.view.SetTextColor(b.GetBackgroundColor())
return b
}
func (b *LoadingBar) Start(ctx context.Context) {
ctx, b.reset = context.WithCancel(ctx)
go func() {
ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()
b.secondsElapsed.Store(0)
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
b.secondsElapsed.Add(1)
b.needDrawFunc()
}
}
}()
}
func (b *LoadingBar) Stop() {
b.reset()
}
func (b *LoadingBar) Draw(screen tcell.Screen) {
seconds := b.secondsElapsed.Load()
var time string
switch {
case seconds < 60:
time = fmt.Sprintf("%ds", seconds)
default:
time = fmt.Sprintf("%dm%ds", seconds/60, seconds%60)
}
b.view.SetText(fmt.Sprintf(" Loading... %s (press Escape to cancel) ", time))
x, y, width, _ := b.GetInnerRect()
b.view.SetRect(x, y, width, 1)
b.view.Draw(screen)
}

View file

@ -0,0 +1,271 @@
package tui
import (
"context"
"errors"
"fmt"
"math"
"sync"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
"github.com/gdamore/tcell/v2"
"github.com/rivo/tview"
)
type updateType int
const (
other updateType = iota
moveToPrevPage
moveToNextPage
moveUp
moveDown
moveHome
moveEnd
)
type RecordsView struct {
*tview.Box
mu sync.RWMutex
onUnmount func()
bucket *Bucket
records []*Record
buffer chan *Record
firstRecordIndex int
lastRecordIndex int
selectedRecordIndex int
updateType updateType
ui *UI
filter *Filter
}
func NewRecordsView(ui *UI, bucket *Bucket, filter *Filter) *RecordsView {
return &RecordsView{
Box: tview.NewBox(),
bucket: bucket,
ui: ui,
filter: filter,
}
}
func (v *RecordsView) Mount(ctx context.Context) error {
if v.onUnmount != nil {
return errors.New("try to mount already mounted component")
}
ctx, v.onUnmount = context.WithCancel(ctx)
tempBuffer, err := LoadRecords(ctx, v.ui.db, v.bucket.Path, v.ui.loadBufferSize)
if err != nil {
return err
}
v.buffer = make(chan *Record, v.ui.loadBufferSize)
go func() {
defer close(v.buffer)
for item := range tempBuffer {
if item.err != nil {
v.ui.stopOnError(err)
break
}
record := item.val
record.Entry, _, err = v.bucket.NextParser(record.Key, record.Value)
if err != nil {
v.ui.stopOnError(err)
break
}
if v.filter.Apply(record.Entry).Result() != common.Yes {
continue
}
v.buffer <- record
}
}()
return nil
}
func (v *RecordsView) Unmount() {
if v.onUnmount == nil {
panic("try to unmount not mounted component")
}
v.onUnmount()
v.onUnmount = nil
}
func (v *RecordsView) Update(ctx context.Context) error {
_, _, _, recordsPerPage := v.GetInnerRect()
firstRecordIndex, lastRecordIndex, selectedRecordIndex := v.getNewIndexes()
loop:
for len(v.records) < lastRecordIndex {
select {
case <-ctx.Done():
return nil
case record, ok := <-v.buffer:
if !ok {
break loop
}
v.records = append(v.records, record)
}
}
// Set the update type to its default value after some specific key event
// has been handled.
v.updateType = other
firstRecordIndex = max(0, min(firstRecordIndex, len(v.records)-recordsPerPage))
lastRecordIndex = min(firstRecordIndex+recordsPerPage, len(v.records))
selectedRecordIndex = min(selectedRecordIndex, lastRecordIndex-1)
v.mu.Lock()
v.firstRecordIndex = firstRecordIndex
v.lastRecordIndex = lastRecordIndex
v.selectedRecordIndex = selectedRecordIndex
v.mu.Unlock()
return nil
}
func (v *RecordsView) getNewIndexes() (int, int, int) {
v.mu.RLock()
firstRecordIndex := v.firstRecordIndex
lastRecordIndex := v.lastRecordIndex
selectedRecordIndex := v.selectedRecordIndex
v.mu.RUnlock()
_, _, _, recordsPerPage := v.GetInnerRect()
switch v.updateType {
case moveUp:
if selectedRecordIndex != firstRecordIndex {
selectedRecordIndex--
break
}
firstRecordIndex = max(0, firstRecordIndex-1)
lastRecordIndex = min(firstRecordIndex+recordsPerPage, len(v.records))
selectedRecordIndex = firstRecordIndex
case moveToPrevPage:
if selectedRecordIndex != firstRecordIndex {
selectedRecordIndex = firstRecordIndex
break
}
firstRecordIndex = max(0, firstRecordIndex-recordsPerPage)
lastRecordIndex = firstRecordIndex + recordsPerPage
selectedRecordIndex = firstRecordIndex
case moveDown:
if selectedRecordIndex != lastRecordIndex-1 {
selectedRecordIndex++
break
}
firstRecordIndex++
lastRecordIndex++
selectedRecordIndex++
case moveToNextPage:
if selectedRecordIndex != lastRecordIndex-1 {
selectedRecordIndex = lastRecordIndex - 1
break
}
firstRecordIndex += recordsPerPage
lastRecordIndex = firstRecordIndex + recordsPerPage
selectedRecordIndex = lastRecordIndex - 1
case moveHome:
firstRecordIndex = 0
lastRecordIndex = firstRecordIndex + recordsPerPage
selectedRecordIndex = 0
case moveEnd:
lastRecordIndex = math.MaxInt32
firstRecordIndex = lastRecordIndex - recordsPerPage
selectedRecordIndex = lastRecordIndex - 1
default:
lastRecordIndex = firstRecordIndex + recordsPerPage
}
return firstRecordIndex, lastRecordIndex, selectedRecordIndex
}
func (v *RecordsView) GetInnerRect() (int, int, int, int) {
x, y, width, height := v.Box.GetInnerRect()
// Left padding.
x = min(x+3, x+width-1)
width = max(width-3, 0)
return x, y, width, height
}
func (v *RecordsView) Draw(screen tcell.Screen) {
v.mu.RLock()
firstRecordIndex := v.firstRecordIndex
lastRecordIndex := v.lastRecordIndex
selectedRecordIndex := v.selectedRecordIndex
records := v.records
v.mu.RUnlock()
v.DrawForSubclass(screen, v)
x, y, width, height := v.GetInnerRect()
if height == 0 {
return
}
// No records in that bucket.
if firstRecordIndex == lastRecordIndex {
tview.Print(
screen, "Empty Bucket", x, y, width, tview.AlignCenter, tview.Styles.PrimaryTextColor,
)
return
}
for index := firstRecordIndex; index < lastRecordIndex; index++ {
result := records[index].Entry
text := result.String()
if index == selectedRecordIndex {
text = fmt.Sprintf("[:white]%s[:-]", text)
tview.Print(screen, text, x, y, width, tview.AlignLeft, tview.Styles.PrimitiveBackgroundColor)
} else {
tview.Print(screen, text, x, y, width, tview.AlignLeft, tview.Styles.PrimaryTextColor)
}
y++
}
}
func (v *RecordsView) InputHandler() func(event *tcell.EventKey, _ func(p tview.Primitive)) {
return v.WrapInputHandler(func(event *tcell.EventKey, _ func(p tview.Primitive)) {
switch m, k := event.Modifiers(), event.Key(); {
case m == 0 && k == tcell.KeyPgUp:
v.updateType = moveToPrevPage
case m == 0 && k == tcell.KeyPgDn:
v.updateType = moveToNextPage
case m == 0 && k == tcell.KeyUp:
v.updateType = moveUp
case m == 0 && k == tcell.KeyDown:
v.updateType = moveDown
case m == 0 && k == tcell.KeyHome:
v.updateType = moveHome
case m == 0 && k == tcell.KeyEnd:
v.updateType = moveEnd
case k == tcell.KeyEnter:
v.mu.RLock()
selectedRecordIndex := v.selectedRecordIndex
records := v.records
v.mu.RUnlock()
if len(records) != 0 {
current := records[selectedRecordIndex]
v.ui.moveNextPage(NewDetailedView(current.Entry.DetailedString()))
}
}
})
}

View file

@ -0,0 +1,18 @@
package tui
import (
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
)
type Bucket struct {
Name []byte
Path [][]byte
Entry common.SchemaEntry
NextParser common.Parser
}
type Record struct {
Key, Value []byte
Path [][]byte
Entry common.SchemaEntry
}

View file

@ -0,0 +1,561 @@
package tui
import (
"context"
"errors"
"fmt"
"strings"
"sync/atomic"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/common"
"github.com/davecgh/go-spew/spew"
"github.com/gdamore/tcell/v2"
"github.com/rivo/tview"
"go.etcd.io/bbolt"
)
type Config struct {
LoadBufferSize int
SearchHistorySize int
LoadingIndicatorLag time.Duration
}
var DefaultConfig = Config{
LoadBufferSize: 100,
SearchHistorySize: 100,
LoadingIndicatorLag: 500 * time.Millisecond,
}
type Primitive interface {
tview.Primitive
Mount(ctx context.Context) error
Update(ctx context.Context) error
Unmount()
}
type UI struct {
*tview.Box
// Need to use context while updating pages those read data from a database.
// Context should be shared among all mount and updates. Current TUI library
// doesn't use contexts at all, so I do that feature by myself.
//nolint:containedctx
ctx context.Context
onStop func()
app *tview.Application
db *bbolt.DB
pageHistory []Primitive
mountedPage Primitive
pageToMount Primitive
pageStub tview.Primitive
infoBar *tview.TextView
searchBar *InputFieldWithHistory
loadingBar *LoadingBar
helpBar *tview.TextView
helpPage *HelpPage
searchErrorBar *tview.TextView
isSearching bool
isLoading atomic.Bool
isShowingError bool
isShowingHelp bool
loadBufferSize int
rootParser common.Parser
loadingIndicatorLag time.Duration
cancelLoading func()
filters map[string]func(string) (any, error)
compositeFilters map[string]func(string) (map[string]any, error)
filterHints map[string]string
}
func NewUI(
ctx context.Context,
app *tview.Application,
db *bbolt.DB,
rootParser common.Parser,
cfg *Config,
) *UI {
spew.Config.DisableMethods = true
if cfg == nil {
cfg = &DefaultConfig
}
ui := &UI{
Box: tview.NewBox(),
app: app,
db: db,
rootParser: rootParser,
filters: make(map[string]func(string) (any, error)),
compositeFilters: make(map[string]func(string) (map[string]any, error)),
filterHints: make(map[string]string),
loadBufferSize: cfg.LoadBufferSize,
loadingIndicatorLag: cfg.LoadingIndicatorLag,
}
ui.ctx, ui.onStop = context.WithCancel(ctx)
backgroundColor := ui.GetBackgroundColor()
textColor := tview.Styles.PrimaryTextColor
inverseBackgroundColor := textColor
inverseTextColor := backgroundColor
alertTextColor := tcell.ColorRed
ui.pageStub = tview.NewBox()
ui.infoBar = tview.NewTextView()
ui.infoBar.SetBackgroundColor(inverseBackgroundColor)
ui.infoBar.SetTextColor(inverseTextColor)
ui.infoBar.SetText(
fmt.Sprintf(" %s (press h for help, q to quit) ", db.Path()),
)
ui.searchBar = NewInputFieldWithHistory(cfg.SearchHistorySize)
ui.searchBar.SetFieldBackgroundColor(backgroundColor)
ui.searchBar.SetFieldTextColor(textColor)
ui.searchBar.SetLabelColor(textColor)
ui.searchBar.Focus(nil)
ui.searchBar.SetLabel("/")
ui.searchErrorBar = tview.NewTextView()
ui.searchErrorBar.SetBackgroundColor(backgroundColor)
ui.searchErrorBar.SetTextColor(alertTextColor)
ui.helpBar = tview.NewTextView()
ui.helpBar.SetBackgroundColor(inverseBackgroundColor)
ui.helpBar.SetTextColor(inverseTextColor)
ui.helpBar.SetText(" Press Enter for next page or Escape to exit help ")
ui.loadingBar = NewLoadingBar(ui.triggerDraw)
ui.pageToMount = NewBucketsView(ui, NewFilter(nil))
return ui
}
func (ui *UI) checkFilterExists(typ string) bool {
if _, ok := ui.filters[typ]; ok {
return true
}
if _, ok := ui.compositeFilters[typ]; ok {
return true
}
return false
}
func (ui *UI) AddFilter(
typ string,
parser func(string) (any, error),
helpHint string,
) error {
if ui.checkFilterExists(typ) {
return fmt.Errorf("filter %s already exists", typ)
}
ui.filters[typ] = parser
ui.filterHints[typ] = helpHint
return nil
}
func (ui *UI) AddCompositeFilter(
typ string,
parser func(string) (map[string]any, error),
helpHint string,
) error {
if ui.checkFilterExists(typ) {
return fmt.Errorf("filter %s already exists", typ)
}
ui.compositeFilters[typ] = parser
ui.filterHints[typ] = helpHint
return nil
}
func (ui *UI) stopOnError(err error) {
if err != nil {
ui.onStop()
ui.app.QueueEvent(tcell.NewEventError(err))
}
}
func (ui *UI) stop() {
ui.onStop()
ui.app.Stop()
}
func (ui *UI) movePrevPage() {
if len(ui.pageHistory) != 0 {
ui.mountedPage.Unmount()
ui.mountedPage = ui.pageHistory[len(ui.pageHistory)-1]
ui.pageHistory = ui.pageHistory[:len(ui.pageHistory)-1]
ui.triggerDraw()
}
}
func (ui *UI) moveNextPage(page Primitive) {
ui.pageToMount = page
ui.triggerDraw()
}
func (ui *UI) triggerDraw() {
go ui.app.QueueUpdateDraw(func() {})
}
func (ui *UI) Draw(screen tcell.Screen) {
if ui.isLoading.Load() {
ui.draw(screen)
return
}
ui.isLoading.Store(true)
ctx, cancel := context.WithCancel(ui.ctx)
ready := make(chan struct{})
go func() {
ui.load(ctx)
cancel()
close(ready)
ui.isLoading.Store(false)
}()
select {
case <-ready:
case <-time.After(ui.loadingIndicatorLag):
ui.loadingBar.Start(ui.ctx)
ui.cancelLoading = cancel
go func() {
<-ready
ui.loadingBar.Stop()
ui.triggerDraw()
}()
}
ui.draw(screen)
}
func (ui *UI) load(ctx context.Context) {
if ui.mountedPage == nil && ui.pageToMount == nil {
ui.stop()
return
}
if ui.pageToMount != nil {
ui.mountAndUpdate(ctx)
} else {
ui.update(ctx)
}
}
func (ui *UI) draw(screen tcell.Screen) {
ui.DrawForSubclass(screen, ui)
x, y, width, height := ui.GetInnerRect()
var (
pageToDraw tview.Primitive
barToDraw tview.Primitive
)
switch {
case ui.isShowingHelp:
if ui.helpPage == nil {
var filters []string
for f := range ui.filters {
filters = append(filters, f)
}
for f := range ui.compositeFilters {
filters = append(filters, f)
}
ui.helpPage = NewHelpPage(filters, ui.filterHints)
}
pageToDraw = ui.helpPage
case ui.mountedPage != nil:
pageToDraw = ui.mountedPage
default:
pageToDraw = ui.pageStub
}
pageToDraw.SetRect(x, y, width, height-1)
pageToDraw.Draw(screen)
// Search bar uses cursor and we need to hide it when another bar is drawn.
screen.HideCursor()
switch {
case ui.isLoading.Load():
barToDraw = ui.loadingBar
case ui.isSearching:
barToDraw = ui.searchBar
case ui.isShowingError:
barToDraw = ui.searchErrorBar
case ui.isShowingHelp:
barToDraw = ui.helpBar
default:
barToDraw = ui.infoBar
}
barToDraw.SetRect(x, y+height-1, width, 1)
barToDraw.Draw(screen)
}
func (ui *UI) mountAndUpdate(ctx context.Context) {
defer func() {
// Operation succeeded or was canceled, either way reset page to mount.
ui.pageToMount = nil
}()
// Mount should use app global context.
//nolint:contextcheck
err := ui.pageToMount.Mount(ui.ctx)
if err != nil {
ui.stopOnError(err)
return
}
x, y, width, height := ui.GetInnerRect()
ui.pageToMount.SetRect(x, y, width, height-1)
s := loadOp(ctx, ui.pageToMount.Update)
if s.err != nil {
ui.pageToMount.Unmount()
ui.stopOnError(s.err)
return
}
// Update was canceled.
if !s.done {
ui.pageToMount.Unmount()
return
}
if ui.mountedPage != nil {
ui.pageHistory = append(ui.pageHistory, ui.mountedPage)
}
ui.mountedPage = ui.pageToMount
}
func (ui *UI) update(ctx context.Context) {
x, y, width, height := ui.GetInnerRect()
ui.mountedPage.SetRect(x, y, width, height-1)
s := loadOp(ctx, ui.mountedPage.Update)
if s.err != nil {
ui.stopOnError(s.err)
return
}
}
type status struct {
done bool
err error
}
func loadOp(ctx context.Context, op func(ctx context.Context) error) status {
errCh := make(chan error)
go func() {
errCh <- op(ctx)
}()
select {
case <-ctx.Done():
return status{done: false, err: nil}
case err := <-errCh:
return status{done: true, err: err}
}
}
func (ui *UI) InputHandler() func(event *tcell.EventKey, setFocus func(p tview.Primitive)) {
return ui.WrapInputHandler(func(event *tcell.EventKey, _ func(tview.Primitive)) {
switch {
case ui.isLoading.Load():
ui.handleInputOnLoading(event)
case ui.isShowingHelp:
ui.handleInputOnShowingHelp(event)
case ui.isShowingError:
ui.handleInputOnShowingError()
case ui.isSearching:
ui.handleInputOnSearching(event)
default:
ui.handleInput(event)
}
})
}
func (ui *UI) handleInput(event *tcell.EventKey) {
m, k, r := event.Modifiers(), event.Key(), event.Rune()
switch {
case k == tcell.KeyEsc:
ui.movePrevPage()
case m == 0 && k == tcell.KeyRune && r == 'h':
ui.isShowingHelp = true
case m == 0 && k == tcell.KeyRune && r == '/':
ui.isSearching = true
case m == 0 && k == tcell.KeyRune && r == 'q':
ui.stop()
default:
if ui.mountedPage != nil {
ui.mountedPage.InputHandler()(event, func(tview.Primitive) {})
}
}
}
func (ui *UI) handleInputOnLoading(event *tcell.EventKey) {
switch k, r := event.Key(), event.Rune(); {
case k == tcell.KeyEsc:
ui.cancelLoading()
case k == tcell.KeyRune && r == 'q':
ui.stop()
}
}
func (ui *UI) handleInputOnShowingError() {
ui.isShowingError = false
ui.isSearching = true
}
func (ui *UI) handleInputOnShowingHelp(event *tcell.EventKey) {
k, r := event.Key(), event.Rune()
switch {
case k == tcell.KeyEsc:
ui.isShowingHelp = false
case k == tcell.KeyRune && r == 'q':
ui.stop()
default:
ui.helpPage.InputHandler()(event, func(tview.Primitive) {})
}
}
func (ui *UI) handleInputOnSearching(event *tcell.EventKey) {
m, k := event.Modifiers(), event.Key()
switch {
case k == tcell.KeyEnter:
prompt := ui.searchBar.GetText()
res, err := ui.processPrompt(prompt)
if err != nil {
ui.isShowingError = true
ui.isSearching = false
ui.searchErrorBar.SetText(err.Error() + " (press any key to continue)")
return
}
switch ui.mountedPage.(type) {
case *BucketsView:
ui.moveNextPage(NewBucketsView(ui, res))
case *RecordsView:
bucket := ui.mountedPage.(*RecordsView).bucket
ui.moveNextPage(NewRecordsView(ui, bucket, res))
}
if ui.searchBar.GetText() != "" {
ui.searchBar.AddToHistory(ui.searchBar.GetText())
}
ui.searchBar.SetText("")
ui.isSearching = false
case k == tcell.KeyEsc:
ui.isSearching = false
case (k == tcell.KeyBackspace2 || m&tcell.ModCtrl != 0 && k == tcell.KeyETB) && len(ui.searchBar.GetText()) == 0:
ui.isSearching = false
default:
ui.searchBar.InputHandler()(event, func(tview.Primitive) {})
}
ui.Box.MouseHandler()
}
func (ui *UI) WithPrompt(prompt string) error {
filter, err := ui.processPrompt(prompt)
if err != nil {
return err
}
ui.pageToMount = NewBucketsView(ui, filter)
if prompt != "" {
ui.searchBar.AddToHistory(prompt)
}
return nil
}
func (ui *UI) processPrompt(prompt string) (filter *Filter, err error) {
if prompt == "" {
return NewFilter(nil), nil
}
filterMap := make(map[string]any)
for _, filterString := range strings.Split(prompt, "+") {
parts := strings.Split(filterString, ":")
if len(parts) != 2 {
return nil, errors.New("expected 'tag:value [+ tag:value]...'")
}
filterTag := strings.TrimSpace(parts[0])
filterValueString := strings.TrimSpace(parts[1])
if _, exists := filterMap[filterTag]; exists {
return nil, fmt.Errorf("duplicate filter tag '%s'", filterTag)
}
parser, ok := ui.filters[filterTag]
if ok {
filterValue, err := parser(filterValueString)
if err != nil {
return nil, fmt.Errorf("can't parse '%s' filter value: %w", filterTag, err)
}
filterMap[filterTag] = filterValue
continue
}
compositeParser, ok := ui.compositeFilters[filterTag]
if ok {
compositeFilterValue, err := compositeParser(filterValueString)
if err != nil {
return nil, fmt.Errorf(
"can't parse '%s' filter value '%s': %w",
filterTag, filterValueString, err,
)
}
for tag, value := range compositeFilterValue {
if _, exists := filterMap[tag]; exists {
return nil, fmt.Errorf(
"found duplicate filter tag '%s' while processing composite filter with tag '%s'",
tag, filterTag,
)
}
filterMap[tag] = value
}
continue
}
return nil, fmt.Errorf("unknown filter tag '%s'", filterTag)
}
return NewFilter(filterMap), nil
}

View file

@ -0,0 +1,97 @@
package tui
import (
"errors"
"strings"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/mr-tron/base58"
)
func CIDParser(s string) (any, error) {
data, err := base58.Decode(s)
if err != nil {
return nil, err
}
var id cid.ID
if err = id.Decode(data); err != nil {
return nil, err
}
return id, nil
}
func OIDParser(s string) (any, error) {
data, err := base58.Decode(s)
if err != nil {
return nil, err
}
var id oid.ID
if err = id.Decode(data); err != nil {
return nil, err
}
return id, nil
}
func AddressParser(s string) (map[string]any, error) {
m := make(map[string]any)
parts := strings.Split(s, "/")
if len(parts) != 2 {
return nil, errors.New("expected <cid>/<oid>")
}
cnr, err := CIDParser(parts[0])
if err != nil {
return nil, err
}
obj, err := OIDParser(parts[1])
if err != nil {
return nil, err
}
m["cid"] = cnr
m["oid"] = obj
return m, nil
}
func keyParser(s string) (any, error) {
if s == "" {
return nil, errors.New("empty attribute key")
}
return s, nil
}
func valueParser(s string) (any, error) {
if s == "" {
return nil, errors.New("empty attribute value")
}
return s, nil
}
func AttributeParser(s string) (map[string]any, error) {
m := make(map[string]any)
parts := strings.Split(s, "/")
if len(parts) != 1 && len(parts) != 2 {
return nil, errors.New("expected <key> or <key>/<value>")
}
key, err := keyParser(parts[0])
if err != nil {
return nil, err
}
m["key"] = key
if len(parts) == 1 {
return m, nil
}
value, err := valueParser(parts[1])
if err != nil {
return nil, err
}
m["value"] = value
return m, nil
}

View file

@ -17,5 +17,5 @@ var Root = &cobra.Command{
}
func init() {
Root.AddCommand(listCMD, inspectCMD)
Root.AddCommand(listCMD, inspectCMD, tuiCMD)
}

View file

@ -0,0 +1,79 @@
package writecache
import (
"context"
"fmt"
common "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal"
schema "git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/schema/writecache"
"git.frostfs.info/TrueCloudLab/frostfs-node/cmd/frostfs-lens/internal/tui"
"github.com/rivo/tview"
"github.com/spf13/cobra"
"go.etcd.io/bbolt"
)
var tuiCMD = &cobra.Command{
Use: "explore",
Short: "Write cache exploration with a terminal UI",
Long: `Launch a terminal UI to explore write cache and search for data.
Available search filters:
- cid CID
- oid OID
- addr CID/OID
`,
Run: tuiFunc,
}
var initialPrompt string
func init() {
common.AddComponentPathFlag(tuiCMD, &vPath)
tuiCMD.Flags().StringVar(
&initialPrompt,
"filter",
"",
"Filter prompt to start with, format 'tag:value [+ tag:value]...'",
)
}
func tuiFunc(cmd *cobra.Command, _ []string) {
common.ExitOnErr(cmd, runTUI(cmd))
}
func runTUI(cmd *cobra.Command) error {
db, err := openDB(false)
if err != nil {
return fmt.Errorf("couldn't open database: %w", err)
}
defer db.Close()
ctx, cancel := context.WithCancel(cmd.Context())
defer cancel()
app := tview.NewApplication()
ui := tui.NewUI(ctx, app, db, schema.WritecacheParser, nil)
_ = ui.AddFilter("cid", tui.CIDParser, "CID")
_ = ui.AddFilter("oid", tui.OIDParser, "OID")
_ = ui.AddCompositeFilter("addr", tui.AddressParser, "CID/OID")
err = ui.WithPrompt(initialPrompt)
if err != nil {
return fmt.Errorf("invalid filter prompt: %w", err)
}
app.SetRoot(ui, true).SetFocus(ui)
return app.Run()
}
func openDB(writable bool) (*bbolt.DB, error) {
db, err := bbolt.Open(vPath, 0o600, &bbolt.Options{
ReadOnly: !writable,
})
if err != nil {
return nil, err
}
return db, nil
}

Some files were not shown because too many files have changed in this diff Show more