lens: Add ability to view raw data in `metabase` #1223

New issue

Closed

opened 2024-07-03 09:48:52 +00:00 by acid-ant · 3 comments

acid-ant commented

2024-07-03 09:48:52 +00:00

Member

Currently, lens allow viewing prepared data stored in metabase, like objects in graveyard or garbage buckets.
For development purposes, we need to view raw info for buckets and content in them.

It is impossible to use web or UI app on the server because it restricts by security reasons.

For development purposes we use dev-env, it will allow you to startup frostfs locally.
Follow instructions from README.md to start it.

In scope of this task:

Investigate how storage engine works
Investigate xk6-frostfs and use it to generate IO
Use frostfs-lens meta inspect\list-* to look at data stored in metabase
Implement command frostfs-lens meta tui
- Look at functionality of boltdbweb
- Suggested to use tview for TUI
  - Do not use violet color scheme if it is possible
- Implement raw mode - print the info about buckets and it content as is
  - Find by prefix
  - Remove bucket/by key/all
- Implement api mode - output data in humanreadable format
  - Use storage engine API
  - Group by graveyard, ecinfo, garbage, ... buckets
  - Ability to inspect selected object

Currently, `lens` allow viewing prepared data stored in `metabase`, like objects in `graveyard` or `garbage` buckets. For development purposes, we need to view raw info for buckets and content in them. It is impossible to use web or UI app on the server because it restricts by security reasons. For development purposes we use [dev-env](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env), it will allow you to startup `frostfs` locally. Follow instructions from [README.md](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env/src/branch/master/README.md) to start it. In scope of this task: - Investigate how [storage engine](https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/local_object_storage) works - Investigate [xk6-frostfs](https://git.frostfs.info/TrueCloudLab/xk6-frostfs) and use it to generate IO - Use `frostfs-lens meta inspect\list-*` to look at data stored in `metabase` - Implement command `frostfs-lens meta tui` - Look at functionality of [boltdbweb](https://github.com/evnix/boltdbweb) - Suggested to use [tview](https://github.com/rivo/tview/) for TUI - Do not use violet color scheme if it is possible - Implement `raw` mode - print the info about buckets and it content as is - Find by prefix - Remove bucket/by key/all - Implement `api` mode - output data in humanreadable format - Use storage engine API - Group by graveyard, ecinfo, garbage, ... buckets - Ability to `inspect` selected object

acid-ant added the

triage

label 2024-07-03 09:48:52 +00:00

acid-ant commented

2024-07-03 09:49:14 +00:00

Author

Member

cc @fyrchik, @dstepanov-yadro

a-savchuk was assigned by acid-ant

2024-07-03 09:49:42 +00:00

fyrchik added the

frostfs-lens

label 2024-07-03 11:03:25 +00:00

acid-ant commented

2024-07-04 07:40:45 +00:00

Author

Member

How to use xk6-frostfs:

Clone https://git.frostfs.info/TrueCloudLab/xk6-frostfs
Follow instructions from README.md to install and build
Use bellow script to preset and generate pressure:

$ cat loader_devenv.sh 
#!/usr/bin/env bash

preset() {
  rm -rf <PATH_TO_WORKSPACE>/xk6-frostfs/grpc.json
  ./scenarios/preset/preset_grpc.py --size 64 --containers 1 --out <PATH_TO_WORKSPACE>/xk6-frostfs/grpc.json \
    --endpoint s01.frostfs.devenv:8080 \
    --preload_obj 5 --policy "REP 2 IN X CBF 2 SELECT 2 FROM * AS X" \
    --config ../frostfs-dev-env/cfg.yml --wallet ../frostfs-dev-env/wallets/wallet.json
}

pressure() {
  rm -rf registry.bolt
  ./k6 run -e DURATION=300 -e WRITE_OBJ_SIZE=64 -e READERS=5 -e WRITERS=5 -e DELETERS=0 \
    -e REGISTRY_FILE=registry.bolt \
    -e GRPC_ENDPOINTS=s01.frostfs.devenv:8080,s02.frostfs.devenv:8080,s03.frostfs.devenv:8080,s04.frostfs.devenv:8080 \
    -e PREGEN_JSON=<PATH_TO_WORKSPACE>/xk6-frostfs/grpc.json \
    scenarios/grpc.js
}

$1

How to use xk6-frostfs: - Clone https://git.frostfs.info/TrueCloudLab/xk6-frostfs - Follow instructions from README.md to install and build - Use bellow script to preset and generate pressure: ``` $ cat loader_devenv.sh #!/usr/bin/env bash preset() { rm -rf <PATH_TO_WORKSPACE>/xk6-frostfs/grpc.json ./scenarios/preset/preset_grpc.py --size 64 --containers 1 --out <PATH_TO_WORKSPACE>/xk6-frostfs/grpc.json \ --endpoint s01.frostfs.devenv:8080 \ --preload_obj 5 --policy "REP 2 IN X CBF 2 SELECT 2 FROM * AS X" \ --config ../frostfs-dev-env/cfg.yml --wallet ../frostfs-dev-env/wallets/wallet.json } pressure() { rm -rf registry.bolt ./k6 run -e DURATION=300 -e WRITE_OBJ_SIZE=64 -e READERS=5 -e WRITERS=5 -e DELETERS=0 \ -e REGISTRY_FILE=registry.bolt \ -e GRPC_ENDPOINTS=s01.frostfs.devenv:8080,s02.frostfs.devenv:8080,s03.frostfs.devenv:8080,s04.frostfs.devenv:8080 \ -e PREGEN_JSON=<PATH_TO_WORKSPACE>/xk6-frostfs/grpc.json \ scenarios/grpc.js } $1 ```