[#559] Remove multipart objects using tombstones #560

Merged
alexvanin merged 1 commit from mbiryukova/frostfs-s3-gw:feature/multipart_tombstone into master 2024-12-04 08:16:11 +00:00
Member

Closes #559

Signed-off-by: Marina Biryukova m.biryukova@yadro.com

Closes #559 Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
mbiryukova self-assigned this 2024-11-22 09:51:46 +00:00
mbiryukova added 1 commit 2024-11-22 09:51:46 +00:00
[#559] Remove multipart objects using tombstones
All checks were successful
/ DCO (pull_request) Successful in 1m44s
/ Vulncheck (pull_request) Successful in 1m52s
/ Builds (pull_request) Successful in 2m1s
/ Lint (pull_request) Successful in 2m30s
/ Tests (pull_request) Successful in 2m2s
4305c8507f
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
mbiryukova requested review from alexvanin 2024-11-22 09:51:47 +00:00
mbiryukova requested review from dkirillov 2024-11-22 09:51:47 +00:00
mbiryukova requested review from pogpp 2024-11-22 09:53:50 +00:00
mbiryukova requested review from r.loginov 2024-11-22 09:53:51 +00:00
mbiryukova requested review from nzinkevich 2024-11-22 09:53:52 +00:00
dkirillov reviewed 2024-11-26 10:05:05 +00:00
@ -636,3 +636,3 @@
}
if err = h.obj.AbortMultipartUpload(ctx, p); err != nil {
networkInfo, err := h.obj.GetNetworkInfo(ctx)
Member

It seems we can get network info inside h.obj.AbortMultipartUpload handler

It seems we can get network info inside `h.obj.AbortMultipartUpload` handler
dkirillov marked this conversation as resolved
@ -218,2 +226,4 @@
AttributeFrostfsCopiesNumber = "frostfs-copies-number" // such format to match X-Amz-Meta-Frostfs-Copies-Number header
tombstoneLifetime = uint64(10)
Member

Can we make this configurable?

Can we make this configurable?
dkirillov marked this conversation as resolved
@ -772,0 +779,4 @@
var wg sync.WaitGroup
i := 0
tombstoneMembersSize := n.features.TombstoneMembersSize()
for ; i < len(members)/tombstoneMembersSize; i++ {
Member

What about something like this:

diff --git a/api/layer/layer.go b/api/layer/layer.go
index 51e7edd8..d5bdca41 100644
--- a/api/layer/layer.go
+++ b/api/layer/layer.go
@@ -777,14 +777,14 @@ func (n *Layer) removeCombinedObject(ctx context.Context, bkt *data.BucketInfo,
 
 func (n *Layer) putTombstones(ctx context.Context, bkt *data.BucketInfo, networkInfo netmap.NetworkInfo, members []oid.ID) {
 	var wg sync.WaitGroup
-	i := 0
 	tombstoneMembersSize := n.features.TombstoneMembersSize()
-	for ; i < len(members)/tombstoneMembersSize; i++ {
-		n.submitPutTombstone(ctx, bkt, members[tombstoneMembersSize*i:tombstoneMembersSize*(i+1)], networkInfo.CurrentEpoch()+tombstoneLifetime, &wg)
-	}
 
-	if len(members)%tombstoneMembersSize != 0 {
-		n.submitPutTombstone(ctx, bkt, members[tombstoneMembersSize*i:], networkInfo.CurrentEpoch()+tombstoneLifetime, &wg)
+	for i := 0; i < len(members); i += tombstoneMembersSize {
+		end := tombstoneMembersSize * (i + 1)
+		if end > len(members) {
+			end = len(members)
+		}
+		n.submitPutTombstone(ctx, bkt, members[i:end], networkInfo.CurrentEpoch()+tombstoneLifetime, &wg)
 	}
 
 	wg.Wait()

What about something like this: ```diff diff --git a/api/layer/layer.go b/api/layer/layer.go index 51e7edd8..d5bdca41 100644 --- a/api/layer/layer.go +++ b/api/layer/layer.go @@ -777,14 +777,14 @@ func (n *Layer) removeCombinedObject(ctx context.Context, bkt *data.BucketInfo, func (n *Layer) putTombstones(ctx context.Context, bkt *data.BucketInfo, networkInfo netmap.NetworkInfo, members []oid.ID) { var wg sync.WaitGroup - i := 0 tombstoneMembersSize := n.features.TombstoneMembersSize() - for ; i < len(members)/tombstoneMembersSize; i++ { - n.submitPutTombstone(ctx, bkt, members[tombstoneMembersSize*i:tombstoneMembersSize*(i+1)], networkInfo.CurrentEpoch()+tombstoneLifetime, &wg) - } - if len(members)%tombstoneMembersSize != 0 { - n.submitPutTombstone(ctx, bkt, members[tombstoneMembersSize*i:], networkInfo.CurrentEpoch()+tombstoneLifetime, &wg) + for i := 0; i < len(members); i += tombstoneMembersSize { + end := tombstoneMembersSize * (i + 1) + if end > len(members) { + end = len(members) + } + n.submitPutTombstone(ctx, bkt, members[i:end], networkInfo.CurrentEpoch()+tombstoneLifetime, &wg) } wg.Wait() ```
dkirillov marked this conversation as resolved
@ -772,0 +809,4 @@
}
}
func (n *Layer) getMembers(ctx context.Context, bkt *data.BucketInfo, objID oid.ID) []oid.ID {
Member

Did you consider using relations.Relations from sdk?
Something like this:

diff --git a/api/layer/frostfs/frostfs.go b/api/layer/frostfs/frostfs.go
index 8dcc7bef..384f1a85 100644
--- a/api/layer/frostfs/frostfs.go
+++ b/api/layer/frostfs/frostfs.go
@@ -13,6 +13,7 @@ import (
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
 	oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
+	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/relations"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
 	"git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain"
@@ -350,4 +351,6 @@ type FrostFS interface {
 
 	// NetworkInfo returns parameters of FrostFS network.
 	NetworkInfo(context.Context) (netmap.NetworkInfo, error)
+
+	Relations() relations.Relations
 }
diff --git a/api/layer/multipart_upload.go b/api/layer/multipart_upload.go
index ca246678..16edc9cc 100644
--- a/api/layer/multipart_upload.go
+++ b/api/layer/multipart_upload.go
@@ -24,6 +24,7 @@ import (
 	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
 	oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
+	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/relations"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
 	"github.com/minio/sio"
 	"go.uber.org/zap"
@@ -569,7 +570,11 @@ func (n *Layer) deleteUploadedParts(ctx context.Context, bkt *data.BucketInfo, p
 	members := make([]oid.ID, 0)
 	for _, infos := range parts {
 		for _, info := range infos {
-			members = append(members, n.getMembers(ctx, bkt, info.OID)...)
+			oids, err := relations.ListAllRelations(ctx, n.frostFS.Relations(), bkt.CID, info.OID, relations.Tokens{})
+			if err != nil {
+
+			}
+			members = append(members, oids...)
 		}
 	}
 
diff --git a/internal/frostfs/frostfs.go b/internal/frostfs/frostfs.go
index 219a1df4..59b3161f 100644
--- a/internal/frostfs/frostfs.go
+++ b/internal/frostfs/frostfs.go
@@ -19,6 +19,7 @@ import (
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
 	oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
+	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/relations"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/pool"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
@@ -297,6 +298,10 @@ func (x *FrostFS) HeadObject(ctx context.Context, prm frostfs.PrmObjectHead) (*o
 	return &res, nil
 }
 
+func (x *FrostFS) Relations() relations.Relations {
+	return x.pool
+}
+
 // GetObject implements layer.FrostFS interface method.
 func (x *FrostFS) GetObject(ctx context.Context, prm frostfs.PrmObjectGet) (*frostfs.Object, error) {
 	var addr oid.Address

Did you consider using `relations.Relations` from sdk? Something like this: ```diff diff --git a/api/layer/frostfs/frostfs.go b/api/layer/frostfs/frostfs.go index 8dcc7bef..384f1a85 100644 --- a/api/layer/frostfs/frostfs.go +++ b/api/layer/frostfs/frostfs.go @@ -13,6 +13,7 @@ import ( "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" + "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/relations" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain" @@ -350,4 +351,6 @@ type FrostFS interface { // NetworkInfo returns parameters of FrostFS network. NetworkInfo(context.Context) (netmap.NetworkInfo, error) + + Relations() relations.Relations } diff --git a/api/layer/multipart_upload.go b/api/layer/multipart_upload.go index ca246678..16edc9cc 100644 --- a/api/layer/multipart_upload.go +++ b/api/layer/multipart_upload.go @@ -24,6 +24,7 @@ import ( "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" + "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/relations" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/minio/sio" "go.uber.org/zap" @@ -569,7 +570,11 @@ func (n *Layer) deleteUploadedParts(ctx context.Context, bkt *data.BucketInfo, p members := make([]oid.ID, 0) for _, infos := range parts { for _, info := range infos { - members = append(members, n.getMembers(ctx, bkt, info.OID)...) + oids, err := relations.ListAllRelations(ctx, n.frostFS.Relations(), bkt.CID, info.OID, relations.Tokens{}) + if err != nil { + + } + members = append(members, oids...) } } diff --git a/internal/frostfs/frostfs.go b/internal/frostfs/frostfs.go index 219a1df4..59b3161f 100644 --- a/internal/frostfs/frostfs.go +++ b/internal/frostfs/frostfs.go @@ -19,6 +19,7 @@ import ( "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" + "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/relations" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/pool" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" @@ -297,6 +298,10 @@ func (x *FrostFS) HeadObject(ctx context.Context, prm frostfs.PrmObjectHead) (*o return &res, nil } +func (x *FrostFS) Relations() relations.Relations { + return x.pool +} + // GetObject implements layer.FrostFS interface method. func (x *FrostFS) GetObject(ctx context.Context, prm frostfs.PrmObjectGet) (*frostfs.Object, error) { var addr oid.Address ```
dkirillov marked this conversation as resolved
@ -82,6 +82,18 @@ func (n *Layer) objectHead(ctx context.Context, bktInfo *data.BucketInfo, idObj
return n.frostFS.HeadObject(ctx, prm)
}
func (n *Layer) objectHeadRaw(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID) (*object.Object, error) {
Member

What about

diff --git a/api/layer/object.go b/api/layer/object.go
index b997dd89..90b39e56 100644
--- a/api/layer/object.go
+++ b/api/layer/object.go
@@ -72,21 +72,18 @@ func newAddress(cnr cid.ID, obj oid.ID) oid.Address {
 
 // objectHead returns all object's headers.
 func (n *Layer) objectHead(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID) (*object.Object, error) {
-       prm := frostfs.PrmObjectHead{
-               Container: bktInfo.CID,
-               Object:    idObj,
-       }
-
-       n.prepareAuthParameters(ctx, &prm.PrmAuth, bktInfo.Owner)
-
-       return n.frostFS.HeadObject(ctx, prm)
+       return n.objectHeadBase(ctx, bktInfo, idObj, false)
 }
 
 func (n *Layer) objectHeadRaw(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID) (*object.Object, error) {
+       return n.objectHeadBase(ctx, bktInfo, idObj, true)
+}
+
+func (n *Layer) objectHeadBase(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID, raw bool) (*object.Object, error) {
        prm := frostfs.PrmObjectHead{
                Container: bktInfo.CID,
                Object:    idObj,
-               Raw:       true,
+               Raw:       raw,
        }
 
        n.prepareAuthParameters(ctx, &prm.PrmAuth, bktInfo.Owner)

What about ```diff diff --git a/api/layer/object.go b/api/layer/object.go index b997dd89..90b39e56 100644 --- a/api/layer/object.go +++ b/api/layer/object.go @@ -72,21 +72,18 @@ func newAddress(cnr cid.ID, obj oid.ID) oid.Address { // objectHead returns all object's headers. func (n *Layer) objectHead(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID) (*object.Object, error) { - prm := frostfs.PrmObjectHead{ - Container: bktInfo.CID, - Object: idObj, - } - - n.prepareAuthParameters(ctx, &prm.PrmAuth, bktInfo.Owner) - - return n.frostFS.HeadObject(ctx, prm) + return n.objectHeadBase(ctx, bktInfo, idObj, false) } func (n *Layer) objectHeadRaw(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID) (*object.Object, error) { + return n.objectHeadBase(ctx, bktInfo, idObj, true) +} + +func (n *Layer) objectHeadBase(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID, raw bool) (*object.Object, error) { prm := frostfs.PrmObjectHead{ Container: bktInfo.CID, Object: idObj, - Raw: true, + Raw: raw, } n.prepareAuthParameters(ctx, &prm.PrmAuth, bktInfo.Owner) ```
dkirillov marked this conversation as resolved
cmd/s3-gw/app.go Outdated
@ -239,6 +251,7 @@ func newAppSettings(log *Logger, v *viper.Viper) *appSettings {
reconnectInterval: fetchReconnectInterval(v),
frostfsidValidation: v.GetBool(cfgFrostfsIDValidationEnabled),
dialerSource: getDialerSource(log.logger, v),
workerPoolSize: v.GetInt(cfgWorkerPoolSize),
Member

Can we use fetch... function that check config provided value and will return default if it's 0 or less?

Can we use `fetch...` function that check config provided value and will return default if it's 0 or less?
dkirillov marked this conversation as resolved
@ -228,1 +228,4 @@
source_ip_header: "Source-Ip"
worker_pool_size: 100
tombstone_members_size: 100
Member

Maybe create new section frostfs for these parameters?

Maybe create new section `frostfs` for these parameters?
Author
Member

frostfs section is already in config, I can move these parameters there. Maybe it's worth creating subsection for all tombstone-related parameters in frostfs section?

`frostfs` section is already in config, I can move these parameters there. Maybe it's worth creating subsection for all tombstone-related parameters in `frostfs` section?
Member

It's up to you. I'm fine with both options

It's up to you. I'm fine with both options
dkirillov marked this conversation as resolved
@ -243,6 +246,8 @@ source_ip_header: "Source-Ip"
| `allowed_access_key_id_prefixes` | `[]string` | no | | List of allowed `AccessKeyID` prefixes which S3 GW serve. If the parameter is omitted, all `AccessKeyID` will be accepted. |
| `reconnect_interval` | `duration` | no | `1m` | Listeners reconnection interval. |
| `source_ip_header` | `string` | yes | | Custom header to retrieve Source IP. |
| `worker_pool_size` | `int` | no | `100` | Maximum worker count in layer's worker pool. |
Member

Probably we should use more specific name for this parameter. I mean mention that this worker pool relates to object removal / putting tombstone

Probably we should use more specific name for this parameter. I mean mention that this worker pool relates to object removal / putting tombstone
Member

question: What about using generic worker-pools layer- or app-wide? For example, there is already a worker pool in index pages. Couldn we reuse it? Or it may cause too large job queue?

question: What about using generic worker-pools `layer-` or `app-wide`? For example, there is already a worker pool in index pages. Couldn we reuse it? Or it may cause too large job queue?
Author
Member

Index pages are only in http-gw, no?

Index pages are only in http-gw, no?
Member

Oh, yes, I got it wrong

Oh, yes, I got it wrong
dkirillov marked this conversation as resolved
@ -216,2 +216,4 @@
obj.SetPayloadSize(prm.PayloadSize)
if prm.Type > 0 {
obj.SetType(prm.Type)
Member

It seems we can unconditionally set type (0 means regular object)

It seems we can unconditionally set type (0 means regular object)
dkirillov marked this conversation as resolved
mbiryukova force-pushed feature/multipart_tombstone from 4305c8507f to 037e972424 2024-11-27 14:31:35 +00:00 Compare
dkirillov reviewed 2024-11-28 14:50:33 +00:00
@ -75,2 +75,4 @@
defaultRetryMaxBackoff = 30 * time.Second
defaultRetryStrategy = handler.RetryStrategyExponential
defaultTombstoneLifetime = uint64(10)
Member

We can write just

defaultTombstoneLifetime       = 10
We can write just ```golang defaultTombstoneLifetime = 10 ```
dkirillov marked this conversation as resolved
dkirillov reviewed 2024-11-28 14:51:42 +00:00
@ -166,0 +166,4 @@
# Tombstone's lifetime in epochs.
S3_GW_FROSTFS_TOMBSTONE_LIFETIME=10
# Maximum number of object IDs in one tombstone.
S3_GW_FROSTFS_TOMBSTONE_MEMBERS_COUNT=100
Member

It seems the variable must be

S3_GW_FROSTFS_TOMBSTONE_MEMBERS_SIZE=100
It seems the variable must be ``` S3_GW_FROSTFS_TOMBSTONE_MEMBERS_SIZE=100 ```
dkirillov marked this conversation as resolved
dkirillov reviewed 2024-11-28 14:52:26 +00:00
@ -202,0 +203,4 @@
# Tombstone's lifetime in epochs.
lifetime: 10
# Maximum number of object IDs in one tombstone.
members_count: 100
Member

It seems the variable must be

tombstone:
  members_size: 100
It seems the variable must be ```yaml tombstone: members_size: 100 ```
dkirillov marked this conversation as resolved
dkirillov approved these changes 2024-11-28 14:53:34 +00:00
Dismissed
dkirillov left a comment
Member

LGTM, but correct (in code or in config examples) config value name members_count or members_size

LGTM, but correct (in code or in config examples) config value name `members_count` or `members_size`
mbiryukova force-pushed feature/multipart_tombstone from 037e972424 to d225d24928 2024-11-29 10:05:55 +00:00 Compare
mbiryukova dismissed dkirillov's review 2024-11-29 10:05:55 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

mbiryukova requested review from dkirillov 2024-11-29 10:10:30 +00:00
r.loginov approved these changes 2024-12-02 06:39:29 +00:00
Dismissed
dkirillov approved these changes 2024-12-02 13:32:13 +00:00
Dismissed
alexvanin approved these changes 2024-12-03 14:06:51 +00:00
Dismissed
alexvanin left a comment
Owner

Overall LGTM, consider splitting layer chainges into separate file. But this is not required.

Overall LGTM, consider splitting layer chainges into separate file. But this is not required.
@ -264,2 +273,3 @@
func (t *TestFrostFS) CreateObject(_ context.Context, prm frostfs.PrmObjectCreate) (*frostfs.CreateObjectResult, error) {
func (t *TestFrostFS) CreateObject(ctx context.Context, prm frostfs.PrmObjectCreate) (*frostfs.CreateObjectResult, error) {
if prm.Type == object.TypeTombstone {
Owner

nitpick: I think this looks better in separate private

createTombstone(ctx context.Context, prm frostfs.PrmObjectCreate) {
}

This condition does not reuse any code from CreateObject so there is no point to inline this code.

nitpick: I think this looks better in separate private ``` createTombstone(ctx context.Context, prm frostfs.PrmObjectCreate) { } ``` This condition does not reuse any code from `CreateObject` so there is no point to inline this code.
alexvanin marked this conversation as resolved
@ -772,0 +788,4 @@
return nil
}
func (n *Layer) putTombstones(ctx context.Context, bkt *data.BucketInfo, networkInfo netmap.NetworkInfo, members []oid.ID) {
Owner

What you think about having all these tombstone collecting function (putTombstones, submitPutTombstone, putTombstoneObject, prepareTokensParameter) in separate file, e.g. tombstones.go? It would be much easier to follow and update this code, on my opinion.

What you think about having all these tombstone collecting function (`putTombstones`, `submitPutTombstone`, `putTombstoneObject`, `prepareTokensParameter`) in separate file, e.g. `tombstones.go`? It would be much easier to follow and update this code, on my opinion.
alexvanin marked this conversation as resolved
@ -214,6 +215,7 @@ func (x *FrostFS) CreateObject(ctx context.Context, prm frostfs.PrmObjectCreate)
obj.SetOwnerID(x.owner)
obj.SetAttributes(attrs...)
obj.SetPayloadSize(prm.PayloadSize)
obj.SetType(prm.Type)
Owner

Is default type a regular object? As far as I understand, prm.Type is optional.

Is default type a regular object? As far as I understand, `prm.Type` is optional.
Author
Member

Yes, default type is regular

Yes, default type is regular
alexvanin marked this conversation as resolved
mbiryukova force-pushed feature/multipart_tombstone from d225d24928 to 309b53f1fb 2024-12-04 08:02:46 +00:00 Compare
mbiryukova dismissed r.loginov's review 2024-12-04 08:02:46 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

mbiryukova dismissed dkirillov's review 2024-12-04 08:02:46 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

mbiryukova dismissed alexvanin's review 2024-12-04 08:02:46 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

mbiryukova force-pushed feature/multipart_tombstone from 309b53f1fb to f215d200e8 2024-12-04 08:04:04 +00:00 Compare
alexvanin approved these changes 2024-12-04 08:12:26 +00:00
alexvanin merged commit f215d200e8 into master 2024-12-04 08:16:11 +00:00
alexvanin deleted branch feature/multipart_tombstone 2024-12-04 08:16:16 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-s3-gw#560
No description provided.