Do not store split info for small objects in metabase

dstepanov-yadro commented

2024-02-12 13:30:23 +00:00

Collaborator

Now frostfs-storage stores split info for all the objects: simple and complex. See 3a41858a0f/pkg/local_object_storage/metabase/put.go (L251)

Describe the solution you'd like

Do not store split info for small objects. It should improve performance and save some place for metabase.

Describe alternatives you've considered

Leave as is. It works!

## Is your feature request related to a problem? Please describe. Now frostfs-storage stores split info for all the objects: simple and complex. See https://git.frostfs.info/TrueCloudLab/frostfs-node/src/commit/3a41858a0f2b02f426d37abe9d9a39ca1835436f/pkg/local_object_storage/metabase/put.go#L251 ## Describe the solution you'd like Do not store split info for small objects. It should improve performance and save some place for metabase. ## Describe alternatives you've considered Leave as is. It works!

dstepanov-yadro added the

labels 2024-02-12 13:30:23 +00:00

fyrchik added

bug

and removed

triage

labels 2024-02-13 05:06:03 +00:00

fyrchik added this to the v0.38.0 milestone 2024-02-13 05:06:14 +00:00

fyrchik removed the

bug

label 2024-02-13 05:08:01 +00:00

fyrchik modified the milestone from v0.38.0 to v0.39.0

2024-02-13 05:08:43 +00:00

dstepanov-yadro self-assigned this 2024-02-20 13:29:51 +00:00

dstepanov-yadro commented

2024-02-21 11:43:23 +00:00

Poster

Collaborator

This will change search with --root flag: small object will be not selected with this flag.

As is:

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli container create -r localhost:8080 --wallet ./dev/wallet.json --policy "REP 1 IN X CBF 1 SELECT 1 FROM * AS X" --basic-acl public-read-write --await
Enter password > 
CID: HAUxsiZzbGr3TnZXNwk1NXmBx33CYv9GZMeeH5NjHX2r
awaiting...
container has been persisted on sidechain

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli object put -r localhost:8080 --wallet ./dev/wallet.json --file ~/payload/128MB.bin --cid HAUxsiZzbGr3TnZXNwk1NXmBx33CYv9GZMeeH5NjHX2r
Enter password > 
 134217728 / 134217728 [=============================================================================================================================================================================] 100.00% 5s
[/home/dstepanov/payload/128MB.bin] Object successfully stored
  OID: 3kQMAYfpf4xKwYf5BadXCCco4r6JF1dcMQrhe5ifsDYy
  CID: HAUxsiZzbGr3TnZXNwk1NXmBx33CYv9GZMeeH5NjHX2r

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli object put -r localhost:8080 --wallet ./dev/wallet.json --file ~/payload/128KB.bin --cid HAUxsiZzbGr3TnZXNwk1NXmBx33CYv9GZMeeH5NjHX2r
Enter password > 
 131072 / 131072 [===================================================================================================================================================================================] 100.00% 0s
[/home/dstepanov/payload/128KB.bin] Object successfully stored
  OID: 9F4D6NVocUANZx7Gzzng95t2hYuckymwuo3u95QqqirA
  CID: HAUxsiZzbGr3TnZXNwk1NXmBx33CYv9GZMeeH5NjHX2r

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli object search -r localhost:8080 --wallet ./dev/wallet.json --cid HAUxsiZzbGr3TnZXNwk1NXmBx33CYv9GZMeeH5NjHX2r --root
Enter password > 
Found 2 objects.
3kQMAYfpf4xKwYf5BadXCCco4r6JF1dcMQrhe5ifsDYy
9F4D6NVocUANZx7Gzzng95t2hYuckymwuo3u95QqqirA

To be:

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli container create -r localhost:8080 --wallet ./dev/wallet.json --policy "REP 1 IN X CBF 1 SELECT 1 FROM * AS X" --basic-acl public-read-write --await
Enter password > 
CID: 8a3NhTk29P7A4v76xyqtfffqWZPkeoXsMHsbiZfDxhbm
awaiting...
container has been persisted on sidechain

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli object put -r localhost:8080 --wallet ./dev/wallet.json --file ~/payload/128KB.bin --cid 8a3NhTk29P7A4v76xyqtfffqWZPkeoXsMHsbiZfDxhbm
Enter password > 
 131072 / 131072 [===================================================================================================================================================================================] 100.00% 0s
[/home/dstepanov/payload/128KB.bin] Object successfully stored
  OID: HNAknJUh5cCJmqP5oAXf4jLNLH6zKh8MjdpsqeWDNrNu
  CID: 8a3NhTk29P7A4v76xyqtfffqWZPkeoXsMHsbiZfDxhbm

dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli object put -r localhost:8080 --wallet ./dev/wallet.json --file ~/payload/128MB.bin --cid 8a3NhTk29P7A4v76xyqtfffqWZPkeoXsMHsbiZfDxhbm
Enter password > 
 134217728 / 134217728 [=============================================================================================================================================================================] 100.00% 5s
[/home/dstepanov/payload/128MB.bin] Object successfully stored
  OID: GGrWDW47LeKMX8oyHkFK1iX8Awvk1Q9codH49PzwjVQu
  CID: 8a3NhTk29P7A4v76xyqtfffqWZPkeoXsMHsbiZfDxhbm
  
dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli object search -r localhost:8080 --wallet ./dev/wallet.json --cid 8a3NhTk29P7A4v76xyqtfffqWZPkeoXsMHsbiZfDxhbm --root
Enter password > 
Found 1 objects.
GGrWDW47LeKMX8oyHkFK1iX8Awvk1Q9codH49PzwjVQu

This will change search with `--root` flag: small object will be not selected with this flag. As is: ``` dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli container create -r localhost:8080 --wallet ./dev/wallet.json --policy "REP 1 IN X CBF 1 SELECT 1 FROM * AS X" --basic-acl public-read-write --await Enter password > CID: HAUxsiZzbGr3TnZXNwk1NXmBx33CYv9GZMeeH5NjHX2r awaiting... container has been persisted on sidechain dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli object put -r localhost:8080 --wallet ./dev/wallet.json --file ~/payload/128MB.bin --cid HAUxsiZzbGr3TnZXNwk1NXmBx33CYv9GZMeeH5NjHX2r Enter password > 134217728 / 134217728 [=============================================================================================================================================================================] 100.00% 5s [/home/dstepanov/payload/128MB.bin] Object successfully stored OID: 3kQMAYfpf4xKwYf5BadXCCco4r6JF1dcMQrhe5ifsDYy CID: HAUxsiZzbGr3TnZXNwk1NXmBx33CYv9GZMeeH5NjHX2r dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli object put -r localhost:8080 --wallet ./dev/wallet.json --file ~/payload/128KB.bin --cid HAUxsiZzbGr3TnZXNwk1NXmBx33CYv9GZMeeH5NjHX2r Enter password > 131072 / 131072 [===================================================================================================================================================================================] 100.00% 0s [/home/dstepanov/payload/128KB.bin] Object successfully stored OID: 9F4D6NVocUANZx7Gzzng95t2hYuckymwuo3u95QqqirA CID: HAUxsiZzbGr3TnZXNwk1NXmBx33CYv9GZMeeH5NjHX2r dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli object search -r localhost:8080 --wallet ./dev/wallet.json --cid HAUxsiZzbGr3TnZXNwk1NXmBx33CYv9GZMeeH5NjHX2r --root Enter password > Found 2 objects. 3kQMAYfpf4xKwYf5BadXCCco4r6JF1dcMQrhe5ifsDYy 9F4D6NVocUANZx7Gzzng95t2hYuckymwuo3u95QqqirA ``` To be: ``` dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli container create -r localhost:8080 --wallet ./dev/wallet.json --policy "REP 1 IN X CBF 1 SELECT 1 FROM * AS X" --basic-acl public-read-write --await Enter password > CID: 8a3NhTk29P7A4v76xyqtfffqWZPkeoXsMHsbiZfDxhbm awaiting... container has been persisted on sidechain dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli object put -r localhost:8080 --wallet ./dev/wallet.json --file ~/payload/128KB.bin --cid 8a3NhTk29P7A4v76xyqtfffqWZPkeoXsMHsbiZfDxhbm Enter password > 131072 / 131072 [===================================================================================================================================================================================] 100.00% 0s [/home/dstepanov/payload/128KB.bin] Object successfully stored OID: HNAknJUh5cCJmqP5oAXf4jLNLH6zKh8MjdpsqeWDNrNu CID: 8a3NhTk29P7A4v76xyqtfffqWZPkeoXsMHsbiZfDxhbm dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli object put -r localhost:8080 --wallet ./dev/wallet.json --file ~/payload/128MB.bin --cid 8a3NhTk29P7A4v76xyqtfffqWZPkeoXsMHsbiZfDxhbm Enter password > 134217728 / 134217728 [=============================================================================================================================================================================] 100.00% 5s [/home/dstepanov/payload/128MB.bin] Object successfully stored OID: GGrWDW47LeKMX8oyHkFK1iX8Awvk1Q9codH49PzwjVQu CID: 8a3NhTk29P7A4v76xyqtfffqWZPkeoXsMHsbiZfDxhbm dstepanov@dstepanov:~/src/frostfs-node$ ./bin/frostfs-cli object search -r localhost:8080 --wallet ./dev/wallet.json --cid 8a3NhTk29P7A4v76xyqtfffqWZPkeoXsMHsbiZfDxhbm --root Enter password > Found 1 objects. GGrWDW47LeKMX8oyHkFK1iX8Awvk1Q9codH49PzwjVQu ```

dstepanov-yadro referenced this issue

2024-02-26 08:59:04 +00:00

metabase: Do not store root info for small objects #1007

dstepanov-yadro commented

2024-04-22 09:31:52 +00:00

Poster

Collaborator

Work continues on a new implementation of the metabase. We will perform optimizations within the framework of changing the storage engine.

dstepanov-yadro closed this issue

2024-04-22 09:31:56 +00:00

Do not store split info for small objects in metabase #977

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered