[#275] Change logic abort multipart upload #275
No reviewers
Labels
No labels
P0
P1
P2
P3
good first issue
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: TrueCloudLab/frostfs-s3-gw#275
Loading…
Reference in a new issue
No description provided.
Delete branch ":feature/change_logic_abort_multipart"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
When abort multipart upload, the following situation may occur: when deleting multipart upload nodes from the wooden service, they may not immediately synchronize with other nodes. Because of this, for example, the ListParts operation can return a list of downloaded parts, although the multipart download was interrupted and should no longer exist.
The solution to the problem is to change the logic of node deletion during Abort Multipart. It is proposed not to delete all nodes of the tree associated with multipart upload, but to leave the root node with the interrupt flag enabled. When reading the multipart upload root node, this flag is checked: if it is enabled, then we do not go to poll the nodes of the tree service to receive parts upload.
Signed-off-by: Roman Loginov r.loginov@yadro.com
37c9c5b11d
to51acb2462e
[##] Change logic abort multipart uploadto [#275] Change logic abort multipart upload[#275] Change logic abort multipart uploadto WIP: [#275] Change logic abort multipart upload51acb2462e
to86f29fff60
WIP: [#275] Change logic abort multipart uploadto [#275] Change logic abort multipart uploadDo we have the same problem with
CompleteMultipartUpload
?@ -195,2 +195,4 @@
}
if multipartInfo.Aborted {
return "", fmt.Errorf("%w", s3errors.GetAPIError(s3errors.ErrNoSuchUpload))
Can we encapsulate this check into
GetMultipartUpload
method inTreeService
interface?@ -566,3 +574,3 @@
}
return n.treeService.DeleteMultipartUpload(ctx, p.Bkt, multipartInfo.ID)
if err = n.treeService.DeleteMultipartUpload(ctx, p.Bkt, multipartInfo.ID); err != nil {
Probably we can pass
multipartInfo
toDeleteMultiaprtUpload
and encapsulate creating new node with aborted attribute there86f29fff60
to267a8f4996
Yes, this problem is also present for CompleteMultipartUpload.
@ -1259,6 +1271,7 @@ func metaFromMultipart(info *data.MultipartInfo, fileName string) map[string]str
info.Meta[uploadIDKV] = info.UploadID
info.Meta[ownerKV] = info.Owner.EncodeToString()
info.Meta[createdKV] = strconv.FormatInt(info.Created.UTC().UnixMilli(), 10)
info.Meta[finished] = strconv.FormatBool(info.Finished)
Actually we can set this attribute only if it's true
@ -1010,3 +1015,3 @@
continue
}
if info.UploadID == uploadID {
if info.UploadID == uploadID && !info.Finished {
Probably we can return
layer.ErrNodeNotFound
ifuploadID
matched but MultipartUpload has already finished267a8f4996
to83f9d03b4b
@ -60,6 +60,7 @@ This document outlines major changes between releases.
- Generalise config param `use_default_xmlns_for_complete_multipart` to `use_default_xmlns` so that use default xmlns for all requests (#221)
- Set server IdleTimeout and ReadHeaderTimeout to `30s` and allow to configure them (#220)
- Return `ETag` value in quotes (#219)
- Change the logic of working with a tree service in AbortMultipartUpload (#275)
in AbortMultipartUpload/CompleteMultipartUpload?
Or probably we can name this as using tombstone when delete multipart upload
LGTM
83f9d03b4b
tof0186b7e23
f0186b7e23
toa6ef98c4ce
a6ef98c4ce
to42d96d3a12
@ -992,6 +997,9 @@ func (c *Tree) GetMultipartUpload(ctx context.Context, bktInfo *data.BucketInfo,
continue
}
if info.UploadID == uploadID {
if info.Finished {
Maybe we should do this check in
GetMultipartUploadsByPrefix
too[#275] Change logic abort multipart uploadto WIP: [#275] Change logic abort multipart upload42d96d3a12
to1184b83e8a
WIP: [#275] Change logic abort multipart uploadto [#275] Change logic abort multipart upload1184b83e8a
to94017462ad
@ -248,0 +249,4 @@
finished, _ := treeNode.Get(finishedKV)
if flag, err := strconv.ParseBool(finished); err == nil {
multipartInfo.Finished = flag
}
What happens when this code is running during multipart upload started by previous version of the gateway?
Previous version won't set
finishedKV
flag, soParseBool
returns error andmultipartInfo.Finished
staysfalse
. Shouldn't it betrue
in such cases?As we found out, this code isn't triggered when accessing multipart upload finished by previous version of the gateway, so it works as expected.
94017462ad
to6f9ee3da76