Fix Search for EC-splitted objects #1144

Merged
fyrchik merged 1 commit from aarifullin/frostfs-node:feat/ec_search into master 2024-09-04 19:51:08 +00:00
Member
  • Support ec parent filter for Search

Proprely choose root OID for EC-splitted objects:

  • If EC-parent is a part of Split itself, then save to root bucket its parent (if it is not nil);
  • If EC-parent is not a part of Split itself, then save to root bucket OID of this EC-parent.

The container is created with EC-policy:

frostfs-cli container create -r s01.frostfs.devenv:8080 --wallet wallet.json --await --policy 'EC 3.1 CBF 1' --basic-acl 0

Let's make search with --root

frostfs-cli object search -r s01.frostfs.devenv:8080 --wallet wallet.json --cid G6zyDxdkyWhboZq12j8CbZBHVMRGPPjRnt6z3zLeaQQu --root

ROOT

Search with --root should return only user objects.
If an object is small, then EC-parent's OID must be listed.
If an object is big, then the object is Split-ted, and then the parts of Split are EC-splitted. That means, then no EC-parent is listed but only the last part of Split (user object)

frostfs-cli object put -r s01.frostfs.devenv:8080 --wallet wallet.json --cid 4SJMJNPC4jCVwJBFwzc1SKYqXJCpX8u8yCjmBdsyWQke --file big_object

Output:

OID: 79xGoKYwhyJQhrDNb7bHhY1WCvN6trHJPTjKkw24c6W9
CID: 4SJMJNPC4jCVwJBFwzc1SKYqXJCpX8u8yCjmBdsyWQke

Let's search with --root

frostfs-cli object search -r s01.frostfs.devenv:8080 --wallet wallet.json --cid 4SJMJNPC4jCVwJBFwzc1SKYqXJCpX8u8yCjmBdsyWQke --root

Output:

Found 1 objects.
79xGoKYwhyJQhrDNb7bHhY1WCvN6trHJPTjKkw24c6W9

Putting small object

frostfs-cli object put -r s01.frostfs.devenv:8080 --wallet wallet.json --cid 4SJMJNPC4jCVwJBFwzc1SKYqXJCpX8u8yCjmBdsyWQke --file random_binary_small
frostfs-cli object search -r s01.frostfs.devenv:8080 --wallet wallet.json --cid 4SJMJNPC4jCVwJBFwzc1SKYqXJCpX8u8yCjmBdsyWQke --root

Output

79xGoKYwhyJQhrDNb7bHhY1WCvN6trHJPTjKkw24c6W9 # Split
AM5Kx5QnsnbUHHLW37xWiPszn6eTd45irx28vcSqcZjE # EC-parent

Filter ec.parent

frostfs-cli object put -r s01.frostfs.devenv:8080 --wallet wallet.json --cid 7ygY2uqM6eVwBaaM2K3wNn9A434FhnUQEavRA2CDbgKj --file random_binary_small

Output:

OID: AgFKsq3AUbCRw7Dz6nuDBGUHZMkWw36sfBFH4V5YMxJK
CID: 7ygY2uqM6eVwBaaM2K3wNn9A434FhnUQEavRA2CDbgKj

Then

frostfs-cli object search -r s01.frostfs.devenv:8080 --wallet wallet.json --cid 7ygY2uqM6eVwBaaM2K3wNn9A434FhnUQEavRA2CDbgKj --filters '$Object:ec.parent EQ AgFKsq3AUbCRw7Dz6nuDBGUHZMkWw36sfBFH4V5YMxJK'

Output (EC-chunks with this EC-parent):

49bWaPXRVyc3pScj8FUmD83hre9oqbo9Gcm3VEFx2wah
4Q185J53EvL2BWNcK4juU4HKeChLKpmXVxm25orpzws7
4cPj4pg5JLYokTATpRyAH55nAbAAKUwPnCyE8ZXghJ8W
5cTZ7H2dUJs8KBKutENgT8SEtHv5Av8HPmGB2rLbDfop
* Support ec parent filter for `Search` Proprely choose root OID for EC-splitted objects: * If EC-parent is a part of Split itself, then save to root bucket its parent (if it is not nil); * If EC-parent is not a part of Split itself, then save to root bucket OID of this EC-parent. The container is created with `EC`-policy: ```bash frostfs-cli container create -r s01.frostfs.devenv:8080 --wallet wallet.json --await --policy 'EC 3.1 CBF 1' --basic-acl 0 ``` Let's make `search` with `--root` ```bash frostfs-cli object search -r s01.frostfs.devenv:8080 --wallet wallet.json --cid G6zyDxdkyWhboZq12j8CbZBHVMRGPPjRnt6z3zLeaQQu --root ``` ### ROOT Search with `--root` should return only user objects. If an object is small, then EC-parent's OID must be listed. If an object is big, then the object is **Split**-ted, and then the parts of Split are **EC-splitted**. That means, then no EC-parent is listed but only the last part of `Split` (user object) ```bash frostfs-cli object put -r s01.frostfs.devenv:8080 --wallet wallet.json --cid 4SJMJNPC4jCVwJBFwzc1SKYqXJCpX8u8yCjmBdsyWQke --file big_object ``` Output: ```bash OID: 79xGoKYwhyJQhrDNb7bHhY1WCvN6trHJPTjKkw24c6W9 CID: 4SJMJNPC4jCVwJBFwzc1SKYqXJCpX8u8yCjmBdsyWQke ``` Let's search with `--root` ```bash frostfs-cli object search -r s01.frostfs.devenv:8080 --wallet wallet.json --cid 4SJMJNPC4jCVwJBFwzc1SKYqXJCpX8u8yCjmBdsyWQke --root ``` Output: ```bash Found 1 objects. 79xGoKYwhyJQhrDNb7bHhY1WCvN6trHJPTjKkw24c6W9 ``` Putting small object ```bash frostfs-cli object put -r s01.frostfs.devenv:8080 --wallet wallet.json --cid 4SJMJNPC4jCVwJBFwzc1SKYqXJCpX8u8yCjmBdsyWQke --file random_binary_small ``` ```bash frostfs-cli object search -r s01.frostfs.devenv:8080 --wallet wallet.json --cid 4SJMJNPC4jCVwJBFwzc1SKYqXJCpX8u8yCjmBdsyWQke --root ``` Output ``` 79xGoKYwhyJQhrDNb7bHhY1WCvN6trHJPTjKkw24c6W9 # Split AM5Kx5QnsnbUHHLW37xWiPszn6eTd45irx28vcSqcZjE # EC-parent ``` ### Filter ec.parent ``` frostfs-cli object put -r s01.frostfs.devenv:8080 --wallet wallet.json --cid 7ygY2uqM6eVwBaaM2K3wNn9A434FhnUQEavRA2CDbgKj --file random_binary_small ``` Output: ``` OID: AgFKsq3AUbCRw7Dz6nuDBGUHZMkWw36sfBFH4V5YMxJK CID: 7ygY2uqM6eVwBaaM2K3wNn9A434FhnUQEavRA2CDbgKj ``` Then ```bash frostfs-cli object search -r s01.frostfs.devenv:8080 --wallet wallet.json --cid 7ygY2uqM6eVwBaaM2K3wNn9A434FhnUQEavRA2CDbgKj --filters '$Object:ec.parent EQ AgFKsq3AUbCRw7Dz6nuDBGUHZMkWw36sfBFH4V5YMxJK' ``` Output (EC-chunks with this EC-parent): ``` 49bWaPXRVyc3pScj8FUmD83hre9oqbo9Gcm3VEFx2wah 4Q185J53EvL2BWNcK4juU4HKeChLKpmXVxm25orpzws7 4cPj4pg5JLYokTATpRyAH55nAbAAKUwPnCyE8ZXghJ8W 5cTZ7H2dUJs8KBKutENgT8SEtHv5Av8HPmGB2rLbDfop ```
aarifullin force-pushed feat/ec_search from ed49239407 to e7f09aea0a 2024-05-21 11:32:50 +00:00 Compare
aarifullin requested review from storage-core-committers 2024-05-21 11:32:57 +00:00
aarifullin requested review from storage-core-developers 2024-05-21 11:34:16 +00:00
aarifullin force-pushed feat/ec_search from e7f09aea0a to dac3cc891c 2024-05-21 12:39:18 +00:00 Compare
aarifullin force-pushed feat/ec_search from dac3cc891c to 59fa9bafd4 2024-05-21 12:47:23 +00:00 Compare
aarifullin force-pushed feat/ec_search from 59fa9bafd4 to a2894350a4 2024-05-23 07:47:33 +00:00 Compare
acid-ant approved these changes 2024-05-31 08:30:38 +00:00
@ -325,1 +344,4 @@
if ech := obj.ECHeader(); ech != nil {
err := f(tx, namedBucketItem{
name: ecParentToChunksBucketName(cnr, bucketName),
Member

Am I right that this bucket will be removed automatically here?

Am I right that this bucket will be removed automatically [here](https://git.frostfs.info/TrueCloudLab/frostfs-node/src/commit/92e19feb577242aea36639e554b3cf71fc7085d1/pkg/local_object_storage/metabase/delete.go#L335)?
Author
Member

Yes, it should be

Yes, it should be
aarifullin changed title from Fix Search for EC-splitted objects to WIP: Fix Search for EC-splitted objects 2024-05-31 10:55:29 +00:00
aarifullin force-pushed feat/ec_search from ef3ca4dc36 to 5aacb8fc86 2024-05-31 16:56:21 +00:00 Compare
aarifullin changed title from WIP: Fix Search for EC-splitted objects to Fix Search for EC-splitted objects 2024-05-31 16:56:43 +00:00
aarifullin requested review from acid-ant 2024-05-31 17:02:25 +00:00
acid-ant approved these changes 2024-05-31 17:22:21 +00:00
fyrchik approved these changes 2024-05-31 18:43:07 +00:00
@ -346,0 +374,4 @@
var attrs []objectSDK.Attribute
if obj.ECHeader() != nil {
attrs = obj.ECHeader().ParentAttributes()
objKey = objectKey(obj.ECHeader().Parent(), make([]byte, objectKeySize))
Owner

We can probably reuse the slice in the second argument, no?

We can probably reuse the slice in the second argument, no?
fyrchik merged commit 5aacb8fc86 into master 2024-05-31 18:43:18 +00:00
Sign in to join this conversation.
No reviewers
TrueCloudLab/storage-core-developers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#1144
No description provided.