bugfix/split_tree #437

Merged
alexvanin merged 1 commit from dkirillov/frostfs-s3-gw:bugfix/split_tree into support/v0.30 2024-09-04 19:51:14 +00:00
Member

Continuation of #430

[#437] tree: Support removing old split system nodes

It's need to fit user expectation on deleting CORs for example.
Previously after removing cors (that was uploaded in split manner)
we can still get some data (from other node)
because deletion worked only for latest node version.

Continuation of #430 [#437] tree: Support removing old split system nodes It's need to fit user expectation on deleting CORs for example. Previously after removing cors (that was uploaded in split manner) we can still get some data (from other node) because deletion worked only for latest node version.
dkirillov self-assigned this 2024-07-22 06:34:32 +00:00
dkirillov added 2 commits 2024-07-22 06:34:41 +00:00
It's need to fit user expectation on deleting CORs for example.
Previously after removing cors (that was uploaded in split manner)
we can still get some data (from other node)
because deletion worked only for latest node version.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
[#430] multipart: Support removing duplicated parts
Some checks failed
/ DCO (pull_request) Successful in 1m38s
/ Vulncheck (pull_request) Successful in 2m56s
/ Lint (pull_request) Failing after 11s
/ Tests (1.21) (pull_request) Successful in 3m9s
/ Tests (1.22) (pull_request) Successful in 3m2s
/ Builds (1.21) (pull_request) Successful in 3m0s
/ Builds (1.22) (pull_request) Successful in 3m6s
1dd43924cc
Previously after tree split we can have duplicated parts
(several objects and tree node referred to the same part number).
Some of them couldn't be deleted after abort or compete action.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
dkirillov requested review from storage-services-committers 2024-07-22 06:34:53 +00:00
dkirillov requested review from storage-services-developers 2024-07-22 06:34:58 +00:00
dkirillov force-pushed bugfix/split_tree from 1dd43924cc to 36361ee535 2024-07-22 06:35:31 +00:00 Compare
Owner

I suggest to update link in commits to [#437] . Also, please, describe solving issues in the PR description.

I suggest to update link in commits to `[#437]` . Also, please, describe solving issues in the PR description.
dkirillov force-pushed bugfix/split_tree from 36361ee535 to 47b8202c86 2024-07-22 07:43:55 +00:00 Compare
dkirillov force-pushed bugfix/split_tree from 47b8202c86 to 689f7ee818 2024-07-22 07:51:34 +00:00 Compare
alexvanin added this to the v0.30.1 milestone 2024-07-23 12:34:02 +00:00
alexvanin changed title from bugfix/split_tree to bugfix/split_tree 2024-07-23 12:36:11 +00:00
alexvanin changed target branch from master to support/v0.30 2024-07-23 12:36:12 +00:00
alexvanin approved these changes 2024-07-23 13:14:29 +00:00
alexvanin left a comment
Owner

Looks good. As for me, I would rather keep tree cleaning stuff from tree.go in a separate function with proper comment on why we have to do that.

Let me know what you think about it.
Also I would like some feedback other whether it makes code simpler to read or not
/cc @mbiryukova, @r.loginov

We can define function like this

type cleanOldMultiNodePrm struct {
	mn             *multiSystemNode
	bktInfo        *data.BucketInfo
	splitLogError  string
	removeLogError string
}

func (c *Tree) cleanOldMultiNodes(ctx context.Context, prm cleanOldMultiNodePrm, removeCb ...func(oid.ID)) {
	for _, node := range prm.mn.Old() {
		ind := node.GetLatestNodeIndex()
		if node.IsSplit() {
			c.reqLogger(ctx).Error(prm.splitLogError, zap.Uint64s("ids", node.ID))
		}
		if err := c.service.RemoveNode(ctx, prm.bktInfo, systemTree, node.ID[ind]); err != nil {
			c.reqLogger(ctx).Warn(prm.removeLogError, zap.Uint64("id", node.ID[ind]))
		} else {
			for _, cb := range removeCb {
				cb(node.ObjID)
			}
		}
	}
}

And use it like this

	c.cleanOldMultiNodes(ctx, cleanOldMultiNodePrm{
		mn:             multiNode,
		bktInfo:        bktInfo,
		splitLogError:  logs.BucketCORSNodeHasMultipleIDs,
		removeLogError: logs.FailedToRemoveOldBucketCORSNode,
	}, func(id oid.ID) {
		objToDelete = append(objToDelete, id)
	})
Looks good. As for me, I would rather keep tree cleaning stuff from `tree.go` in a separate function with proper comment on why we have to do that. Let me know what you think about it. Also I would like some feedback other whether it makes code simpler to read or not /cc @mbiryukova, @r.loginov We can define function like this ```go type cleanOldMultiNodePrm struct { mn *multiSystemNode bktInfo *data.BucketInfo splitLogError string removeLogError string } func (c *Tree) cleanOldMultiNodes(ctx context.Context, prm cleanOldMultiNodePrm, removeCb ...func(oid.ID)) { for _, node := range prm.mn.Old() { ind := node.GetLatestNodeIndex() if node.IsSplit() { c.reqLogger(ctx).Error(prm.splitLogError, zap.Uint64s("ids", node.ID)) } if err := c.service.RemoveNode(ctx, prm.bktInfo, systemTree, node.ID[ind]); err != nil { c.reqLogger(ctx).Warn(prm.removeLogError, zap.Uint64("id", node.ID[ind])) } else { for _, cb := range removeCb { cb(node.ObjID) } } } } ``` And use it like this ```go c.cleanOldMultiNodes(ctx, cleanOldMultiNodePrm{ mn: multiNode, bktInfo: bktInfo, splitLogError: logs.BucketCORSNodeHasMultipleIDs, removeLogError: logs.FailedToRemoveOldBucketCORSNode, }, func(id oid.ID) { objToDelete = append(objToDelete, id) }) ```
dkirillov added 1 commit 2024-07-23 13:47:21 +00:00
[#437] tree: Add cleanOldNodes method
All checks were successful
/ Vulncheck (pull_request) Successful in 59s
/ DCO (pull_request) Successful in 56s
/ Builds (1.21) (pull_request) Successful in 1m26s
/ Builds (1.22) (pull_request) Successful in 1m18s
/ Lint (pull_request) Successful in 2m11s
/ Tests (1.21) (pull_request) Successful in 1m43s
/ Tests (1.22) (pull_request) Successful in 1m36s
2bae704d3e
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
alexvanin approved these changes 2024-07-23 13:50:50 +00:00
alexvanin merged commit 2bae704d3e into support/v0.30 2024-07-24 12:10:32 +00:00
alexvanin deleted branch bugfix/split_tree 2024-07-24 12:10:34 +00:00
alexvanin referenced this pull request from a commit 2024-07-26 06:54:12 +00:00
Sign in to join this conversation.
No reviewers
TrueCloudLab/storage-services-developers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-s3-gw#437
No description provided.