bugfix/split_tree #437

Merged

alexvanin merged 1 commit from dkirillov/frostfs-s3-gw:bugfix/split_tree into support/v0.30 2024-09-04 19:51:14 +00:00

Conversation 1 Commits 1 Files changed 2 +26 -46

dkirillov commented 2024-07-22 06:34:31 +00:00

Member

Copy link

Continuation of #430

[#437] tree: Support removing old split system nodes

It's need to fit user expectation on deleting CORs for example.
Previously after removing cors (that was uploaded in split manner)
we can still get some data (from other node)
because deletion worked only for latest node version.

Continuation of #430 [#437] tree: Support removing old split system nodes It's need to fit user expectation on deleting CORs for example. Previously after removing cors (that was uploaded in split manner) we can still get some data (from other node) because deletion worked only for latest node version.

dkirillov self-assigned this 2024-07-22 06:34:32 +00:00

dkirillov added 2 commits 2024-07-22 06:34:41 +00:00

[#430] tree: Support removing old split system nodes ... f3dd9c86a6

It's need to fit user expectation on deleting CORs for example.
Previously after removing cors (that was uploaded in split manner)
we can still get some data (from other node)
because deletion worked only for latest node version.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>

[#430] multipart: Support removing duplicated parts ... 1dd43924cc

Previously after tree split we can have duplicated parts
(several objects and tree node referred to the same part number).
Some of them couldn't be deleted after abort or compete action.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>

dkirillov requested review from storage-services-committers 2024-07-22 06:34:53 +00:00

dkirillov requested review from storage-services-developers 2024-07-22 06:34:58 +00:00

dkirillov force-pushed bugfix/split_tree from 1dd43924cc to 36361ee535 2024-07-22 06:35:31 +00:00 Compare

alexvanin commented 2024-07-22 07:31:30 +00:00

Owner

Copy link

I suggest to update link in commits to [#437] . Also, please, describe solving issues in the PR description.

I suggest to update link in commits to `[#437]` . Also, please, describe solving issues in the PR description.

dkirillov force-pushed bugfix/split_tree from 36361ee535 to 47b8202c86 2024-07-22 07:43:55 +00:00 Compare

dkirillov force-pushed bugfix/split_tree from 47b8202c86 to 689f7ee818 2024-07-22 07:51:34 +00:00 Compare

alexvanin added this to the v0.30.1 milestone 2024-07-23 12:34:02 +00:00

alexvanin changed title from bugfix/split_tree to bugfix/split_tree 2024-07-23 12:36:11 +00:00

alexvanin changed target branch from master to support/v0.30 2024-07-23 12:36:12 +00:00

alexvanin approved these changes 2024-07-23 13:14:29 +00:00

alexvanin left a comment

Owner

Copy link

Looks good. As for me, I would rather keep tree cleaning stuff from tree.go in a separate function with proper comment on why we have to do that.

Let me know what you think about it.
Also I would like some feedback other whether it makes code simpler to read or not
/cc @mbiryukova, @r.loginov

We can define function like this

type cleanOldMultiNodePrm struct {
	mn             *multiSystemNode
	bktInfo        *data.BucketInfo
	splitLogError  string
	removeLogError string
}

func (c *Tree) cleanOldMultiNodes(ctx context.Context, prm cleanOldMultiNodePrm, removeCb ...func(oid.ID)) {
	for _, node := range prm.mn.Old() {
		ind := node.GetLatestNodeIndex()
		if node.IsSplit() {
			c.reqLogger(ctx).Error(prm.splitLogError, zap.Uint64s("ids", node.ID))
		}
		if err := c.service.RemoveNode(ctx, prm.bktInfo, systemTree, node.ID[ind]); err != nil {
			c.reqLogger(ctx).Warn(prm.removeLogError, zap.Uint64("id", node.ID[ind]))
		} else {
			for _, cb := range removeCb {
				cb(node.ObjID)
			}
		}
	}
}

And use it like this

	c.cleanOldMultiNodes(ctx, cleanOldMultiNodePrm{
		mn:             multiNode,
		bktInfo:        bktInfo,
		splitLogError:  logs.BucketCORSNodeHasMultipleIDs,
		removeLogError: logs.FailedToRemoveOldBucketCORSNode,
	}, func(id oid.ID) {
		objToDelete = append(objToDelete, id)
	})

Looks good. As for me, I would rather keep tree cleaning stuff from `tree.go` in a separate function with proper comment on why we have to do that. Let me know what you think about it. Also I would like some feedback other whether it makes code simpler to read or not /cc @mbiryukova, @r.loginov We can define function like this ```go type cleanOldMultiNodePrm struct { mn *multiSystemNode bktInfo *data.BucketInfo splitLogError string removeLogError string } func (c *Tree) cleanOldMultiNodes(ctx context.Context, prm cleanOldMultiNodePrm, removeCb ...func(oid.ID)) { for _, node := range prm.mn.Old() { ind := node.GetLatestNodeIndex() if node.IsSplit() { c.reqLogger(ctx).Error(prm.splitLogError, zap.Uint64s("ids", node.ID)) } if err := c.service.RemoveNode(ctx, prm.bktInfo, systemTree, node.ID[ind]); err != nil { c.reqLogger(ctx).Warn(prm.removeLogError, zap.Uint64("id", node.ID[ind])) } else { for _, cb := range removeCb { cb(node.ObjID) } } } } ``` And use it like this ```go c.cleanOldMultiNodes(ctx, cleanOldMultiNodePrm{ mn: multiNode, bktInfo: bktInfo, splitLogError: logs.BucketCORSNodeHasMultipleIDs, removeLogError: logs.FailedToRemoveOldBucketCORSNode, }, func(id oid.ID) { objToDelete = append(objToDelete, id) }) ```

dkirillov added 1 commit 2024-07-23 13:47:21 +00:00

[#437] tree: Add cleanOldNodes method ... 2bae704d3e

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>

alexvanin approved these changes 2024-07-23 13:50:50 +00:00

alexvanin merged commit 2bae704d3e into support/v0.30 2024-07-24 12:10:32 +00:00

alexvanin referenced this pull request from a commit 2024-07-24 12:10:34 +00:00

[#437] tree: Support removing old split system nodes

alexvanin referenced this pull request from a commit 2024-07-24 12:10:34 +00:00

[#437] tree: Add cleanOldNodes method

alexvanin deleted branch bugfix/split_tree 2024-07-24 12:10:34 +00:00

alexvanin referenced this pull request 2024-07-24 13:21:35 +00:00

port: tree: Support removing old split system nodes #444

alexvanin referenced this pull request from a commit 2024-07-26 06:54:12 +00:00

[#437] tree: Support removing old split system nodes

alexvanin referenced this pull request from a commit 2024-07-26 06:54:12 +00:00

[#437] Rename getCORSAddress

Sign in to join this conversation.

Reviewers

No reviewers

alexvanin

TrueCloudLab/storage-services-developers

Labels

Clear labels   

P0

Highest

  

P1

High

  

P2

Medium

  

P3

Low

  

good first issue

Good for newcomers

  

Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  

blocked

The issue or PR is blocked by something

  

bug

Something is not working

  

config

Configuration format update or breaking change

  

discussion

Talking about something in order to reach a decision or to exchange ideas

  

documentation

Documentation related

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

go

Language features adoption

  

help wanted

Extra attention is needed

  

internal

  

invalid

Something is wrong

  

kludge

This is a kludge and we know it

  

observability

Related to metrics, tracing and logs

  

perfomance

Perfomance related

  

question

More information is needed

  

refactoring

Refactoring

  

wontfix

This won't be fixed

No labels

P0

P1

P2

P3

good first issue

Infrastructure

blocked

bug

config

discussion

documentation

duplicate

enhancement

go

help wanted

internal

invalid

kludge

observability

perfomance

question

refactoring

wontfix

Milestone

Clear milestone

No items

No milestone

v0.30.1

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

dkirillov

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-s3-gw#437

No description provided.