TrueCloudLab/frostfs-s3-gw
18
3
Fork 20
Code Issues 29 Pull requests 7 Projects Releases 23 Packages 1 Activity Actions

feature/lifecycle_convert_date #516

Merged
alexvanin merged 2 commits from mbiryukova/frostfs-s3-gw:feature/lifecycle_convert_date into master 2024-10-22 14:21:50 +00:00
Conversation 0 Commits 2 Files changed 9 +179 -63
mbiryukova commented 2024-10-15 11:34:11 +00:00
Member
Copy link
No description provided.
mbiryukova self-assigned this 2024-10-15 11:34:11 +00:00
mbiryukova added 2 commits 2024-10-15 11:34:11 +00:00 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
[#xxx] Check Content-Md5 of lifecycle configuration
Some checks failed
/ DCO (pull_request) Failing after 1m4s
Details
/ Vulncheck (pull_request) Successful in 1m32s
Details
/ Builds (pull_request) Successful in 1m40s
Details
/ Lint (pull_request) Successful in 3m25s
Details
/ Tests (pull_request) Successful in 1m46s
Details
097c0b6e36 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
mbiryukova force-pushed feature/lifecycle_convert_date from 097c0b6e36 to ff57dbb1dc 2024-10-15 11:35:49 +00:00 Compare
mbiryukova requested review from storage-services-committers 2024-10-15 13:18:08 +00:00
mbiryukova requested review from storage-services-developers 2024-10-15 13:18:09 +00:00
nzinkevich reviewed 2024-10-16 06:29:07 +00:00
api/handler/lifecycle.go Outdated
@ -236,0 +312,4 @@
for {
n, err := reader.Read(buf)
if n > 0 {
println(string(buf[:n]))
nzinkevich commented 2024-10-16 06:29:07 +00:00
Member
Copy link

Delete this line

Delete this line
👀 1
dkirillov reviewed 2024-10-16 06:32:53 +00:00
api/handler/lifecycle.go Outdated
@ -236,0 +312,4 @@
for {
n, err := reader.Read(buf)
if n > 0 {
println(string(buf[:n]))
dkirillov commented 2024-10-16 06:32:53 +00:00
Member
Copy link

Debug?

Debug?
👀 1
dkirillov marked this conversation as resolved
dkirillov reviewed 2024-10-16 06:36:15 +00:00
api/handler/lifecycle.go Outdated
@ -236,0 +310,4 @@
buf := make([]byte, 64*1024)
for {
n, err := reader.Read(buf)
dkirillov commented 2024-10-16 06:36:15 +00:00
Member
Copy link

Why don't we use io.Copy or io.CopyBuffer?

Why don't we use `io.Copy` or `io.CopyBuffer`?
👍 1
dkirillov marked this conversation as resolved
dkirillov reviewed 2024-10-16 06:38:05 +00:00
api/data/lifecycle.go Outdated
@ -32,1 +30,3 @@
ExpiredObjectDeleteMarker *bool `xml:"ExpiredObjectDeleteMarker,omitempty"`
Date string `xml:"Date,omitempty"`
Days *int `xml:"Days,omitempty"`
Epoch *uint64 `xml:"Epoch,omitempty"`
dkirillov commented 2024-10-16 06:38:05 +00:00
Member
Copy link

Should we add the similar field (Epochs) to AbortIncompleteMultipartUpload and NonCurrentVersionExpiration

Should we add the similar field (`Epochs`) to `AbortIncompleteMultipartUpload` and `NonCurrentVersionExpiration`
mbiryukova commented 2024-10-18 11:43:51 +00:00
Author
Member
Copy link

I’m not sure if this is necessary, current date is not involved in days conversion

I’m not sure if this is necessary, current date is not involved in days conversion
dkirillov commented 2024-10-18 14:38:54 +00:00
Member
Copy link

Probably it's better use duration not in days but in epochs (abort multipart not after 2 days but after N epochs after initiation)

Probably it's better use duration not in days but in epochs (abort multipart not after 2 days but after N epochs after initiation)
dkirillov commented 2024-10-21 09:56:10 +00:00
Member
Copy link

Otherwise, it seems we will have some inconsistency. For example it we tick some epochs (e.g. in tests) we will expire objects that have specific expiration date. But we won't expire object if they have Days expiration

@alexvanin

Otherwise, it seems we will have some inconsistency. For example it we tick some epochs (e.g. in tests) we will expire objects that have specific expiration date. But we won't expire object if they have `Days` expiration @alexvanin
dkirillov marked this conversation as resolved
dkirillov reviewed 2024-10-16 08:01:05 +00:00
api/handler/lifecycle.go Outdated
@ -234,2 +269,4 @@
return nil
}
func timeToEpoch(ni *netmap.NetworkInfo, t time.Time) (uint64, error) {
dkirillov commented 2024-10-16 08:01:05 +00:00
Member
Copy link

Consider reusing this

diff --git a/api/handler/lifecycle.go b/api/handler/lifecycle.go
index bd89fe9f..720e37e2 100644
--- a/api/handler/lifecycle.go
+++ b/api/handler/lifecycle.go
@@ -2,11 +2,11 @@ package handler
 
 import (
 	"bytes"
+	"context"
 	"crypto/md5"
 	"encoding/base64"
 	"fmt"
 	"io"
-	"math"
 	"net/http"
 	"time"
 
@@ -15,6 +15,7 @@ import (
 	apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
 	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
 	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
+	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/util"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
 )
 
@@ -95,7 +96,7 @@ func (h *handler) PutBucketLifecycleHandler(w http.ResponseWriter, r *http.Reque
 		return
 	}
 
-	if err = checkLifecycleConfiguration(cfg, &networkInfo); err != nil {
+	if err = checkLifecycleConfiguration(ctx, cfg, &networkInfo); err != nil {
 		h.logAndSendError(w, "invalid lifecycle configuration", reqInfo, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrMalformedXML), err.Error()))
 		return
 	}
@@ -135,7 +136,8 @@ func (h *handler) DeleteBucketLifecycleHandler(w http.ResponseWriter, r *http.Re
 	w.WriteHeader(http.StatusNoContent)
 }
 
-func checkLifecycleConfiguration(cfg *data.LifecycleConfiguration, ni *netmap.NetworkInfo) error {
+func checkLifecycleConfiguration(ctx context.Context, cfg *data.LifecycleConfiguration, ni *netmap.NetworkInfo) error {
+	now := layer.TimeNow(ctx)
 	if len(cfg.Rules) > maxRules {
 		return fmt.Errorf("number of rules cannot be greater than %d", maxRules)
 	}
@@ -191,7 +193,7 @@ func checkLifecycleConfiguration(cfg *data.LifecycleConfiguration, ni *netmap.Ne
 					return fmt.Errorf("invalid value of expiration date: %s", rule.Expiration.Date)
 				}
 
-				epoch, err := timeToEpoch(ni, parsedTime)
+				epoch, err := util.TimeToEpoch(ni, now, parsedTime)
 				if err != nil {
 					return fmt.Errorf("convert time to epoch: %w", err)
 				}
@@ -269,42 +271,6 @@ func checkLifecycleRuleFilter(filter *data.LifecycleRuleFilter) error {
 	return nil
 }
 
-func timeToEpoch(ni *netmap.NetworkInfo, t time.Time) (uint64, error) {
-	duration := t.Sub(time.Now())
-	durationAbs := duration.Abs()
-
-	durEpoch := ni.EpochDuration()
-	if durEpoch == 0 {
-		return 0, fmt.Errorf("epoch duration is missing or zero")
-	}
-
-	msPerEpoch := durEpoch * uint64(ni.MsPerBlock())
-	epochLifetime := uint64(durationAbs.Milliseconds()) / msPerEpoch
-
-	if uint64(durationAbs.Milliseconds())%msPerEpoch != 0 {
-		epochLifetime++
-	}
-
-	curr := ni.CurrentEpoch()
-
-	var epoch uint64
-	if duration > 0 {
-		if epochLifetime >= math.MaxUint64-curr {
-			epoch = math.MaxUint64
-		} else {
-			epoch = curr + epochLifetime
-		}
-	} else {
-		if epochLifetime >= curr {
-			epoch = 0
-		} else {
-			epoch = curr - epochLifetime
-		}
-	}
-
-	return epoch, nil
-}
-
 func getContentMD5(reader io.Reader) ([]byte, error) {
 	hash := md5.New()
 	buf := make([]byte, 64*1024)
diff --git a/internal/frostfs/frostfs.go b/internal/frostfs/frostfs.go
index 95ac17bf..03f8a7f6 100644
--- a/internal/frostfs/frostfs.go
+++ b/internal/frostfs/frostfs.go
@@ -5,13 +5,13 @@ import (
 	"errors"
 	"fmt"
 	"io"
-	"math"
 	"strconv"
 	"time"
 
 	objectv2 "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object"
 	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
 	frosterr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
+	"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/util"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
 	cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
@@ -54,8 +54,7 @@ func NewFrostFS(p *pool.Pool, key *keys.PrivateKey) *FrostFS {
 
 // TimeToEpoch implements layer.FrostFS interface method.
 func (x *FrostFS) TimeToEpoch(ctx context.Context, now, futureTime time.Time) (uint64, uint64, error) {
-	dur := futureTime.Sub(now)
-	if dur < 0 {
+	if futureTime.Before(now) {
 		return 0, 0, fmt.Errorf("time '%s' must be in the future (after %s)",
 			futureTime.Format(time.RFC3339), now.Format(time.RFC3339))
 	}
@@ -65,27 +64,12 @@ func (x *FrostFS) TimeToEpoch(ctx context.Context, now, futureTime time.Time) (u
 		return 0, 0, handleObjectError("get network info via client", err)
 	}
 
-	durEpoch := networkInfo.EpochDuration()
-	if durEpoch == 0 {
-		return 0, 0, errors.New("epoch duration is missing or zero")
-	}
-
-	curr := networkInfo.CurrentEpoch()
-	msPerEpoch := durEpoch * uint64(networkInfo.MsPerBlock())
-
-	epochLifetime := uint64(dur.Milliseconds()) / msPerEpoch
-	if uint64(dur.Milliseconds())%msPerEpoch != 0 {
-		epochLifetime++
-	}
-
-	var epoch uint64
-	if epochLifetime >= math.MaxUint64-curr {
-		epoch = math.MaxUint64
-	} else {
-		epoch = curr + epochLifetime
+	epoch, err := util.TimeToEpoch(&networkInfo, now, futureTime)
+	if err != nil {
+		return 0, 0, err
 	}
 
-	return curr, epoch, nil
+	return networkInfo.CurrentEpoch(), epoch, nil
 }
 
 // Container implements layer.FrostFS interface method.
diff --git a/internal/frostfs/util/util.go b/internal/frostfs/util/util.go
index 444504b1..00a99059 100644
--- a/internal/frostfs/util/util.go
+++ b/internal/frostfs/util/util.go
@@ -2,9 +2,12 @@ package util
 
 import (
 	"fmt"
+	"math"
 	"strings"
+	"time"
 
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
+	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
 	"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/ns"
 	"github.com/nspcc-dev/neo-go/pkg/util"
 )
@@ -32,3 +35,39 @@ func ResolveContractHash(contractHash, rpcAddress string) (util.Uint160, error)
 
 	return nns.ResolveContractHash(domain)
 }
+
+func TimeToEpoch(ni *netmap.NetworkInfo, now, t time.Time) (uint64, error) {
+	duration := t.Sub(now)
+	durationAbs := duration.Abs()
+
+	durEpoch := ni.EpochDuration()
+	if durEpoch == 0 {
+		return 0, fmt.Errorf("epoch duration is missing or zero")
+	}
+
+	msPerEpoch := durEpoch * uint64(ni.MsPerBlock())
+	epochLifetime := uint64(durationAbs.Milliseconds()) / msPerEpoch
+
+	if uint64(durationAbs.Milliseconds())%msPerEpoch != 0 {
+		epochLifetime++
+	}
+
+	curr := ni.CurrentEpoch()
+
+	var epoch uint64
+	if duration > 0 {
+		if epochLifetime >= math.MaxUint64-curr {
+			epoch = math.MaxUint64
+		} else {
+			epoch = curr + epochLifetime
+		}
+	} else {
+		if epochLifetime >= curr {
+			epoch = 0
+		} else {
+			epoch = curr - epochLifetime
+		}
+	}
+
+	return epoch, nil
+}
Consider reusing this ```diff diff --git a/api/handler/lifecycle.go b/api/handler/lifecycle.go index bd89fe9f..720e37e2 100644 --- a/api/handler/lifecycle.go +++ b/api/handler/lifecycle.go @@ -2,11 +2,11 @@ package handler import ( "bytes" + "context" "crypto/md5" "encoding/base64" "fmt" "io" - "math" "net/http" "time" @@ -15,6 +15,7 @@ import ( apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" + "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/util" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" ) @@ -95,7 +96,7 @@ func (h *handler) PutBucketLifecycleHandler(w http.ResponseWriter, r *http.Reque return } - if err = checkLifecycleConfiguration(cfg, &networkInfo); err != nil { + if err = checkLifecycleConfiguration(ctx, cfg, &networkInfo); err != nil { h.logAndSendError(w, "invalid lifecycle configuration", reqInfo, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrMalformedXML), err.Error())) return } @@ -135,7 +136,8 @@ func (h *handler) DeleteBucketLifecycleHandler(w http.ResponseWriter, r *http.Re w.WriteHeader(http.StatusNoContent) } -func checkLifecycleConfiguration(cfg *data.LifecycleConfiguration, ni *netmap.NetworkInfo) error { +func checkLifecycleConfiguration(ctx context.Context, cfg *data.LifecycleConfiguration, ni *netmap.NetworkInfo) error { + now := layer.TimeNow(ctx) if len(cfg.Rules) > maxRules { return fmt.Errorf("number of rules cannot be greater than %d", maxRules) } @@ -191,7 +193,7 @@ func checkLifecycleConfiguration(cfg *data.LifecycleConfiguration, ni *netmap.Ne return fmt.Errorf("invalid value of expiration date: %s", rule.Expiration.Date) } - epoch, err := timeToEpoch(ni, parsedTime) + epoch, err := util.TimeToEpoch(ni, now, parsedTime) if err != nil { return fmt.Errorf("convert time to epoch: %w", err) } @@ -269,42 +271,6 @@ func checkLifecycleRuleFilter(filter *data.LifecycleRuleFilter) error { return nil } -func timeToEpoch(ni *netmap.NetworkInfo, t time.Time) (uint64, error) { - duration := t.Sub(time.Now()) - durationAbs := duration.Abs() - - durEpoch := ni.EpochDuration() - if durEpoch == 0 { - return 0, fmt.Errorf("epoch duration is missing or zero") - } - - msPerEpoch := durEpoch * uint64(ni.MsPerBlock()) - epochLifetime := uint64(durationAbs.Milliseconds()) / msPerEpoch - - if uint64(durationAbs.Milliseconds())%msPerEpoch != 0 { - epochLifetime++ - } - - curr := ni.CurrentEpoch() - - var epoch uint64 - if duration > 0 { - if epochLifetime >= math.MaxUint64-curr { - epoch = math.MaxUint64 - } else { - epoch = curr + epochLifetime - } - } else { - if epochLifetime >= curr { - epoch = 0 - } else { - epoch = curr - epochLifetime - } - } - - return epoch, nil -} - func getContentMD5(reader io.Reader) ([]byte, error) { hash := md5.New() buf := make([]byte, 64*1024) diff --git a/internal/frostfs/frostfs.go b/internal/frostfs/frostfs.go index 95ac17bf..03f8a7f6 100644 --- a/internal/frostfs/frostfs.go +++ b/internal/frostfs/frostfs.go @@ -5,13 +5,13 @@ import ( "errors" "fmt" "io" - "math" "strconv" "time" objectv2 "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs" frosterr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors" + "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/util" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" @@ -54,8 +54,7 @@ func NewFrostFS(p *pool.Pool, key *keys.PrivateKey) *FrostFS { // TimeToEpoch implements layer.FrostFS interface method. func (x *FrostFS) TimeToEpoch(ctx context.Context, now, futureTime time.Time) (uint64, uint64, error) { - dur := futureTime.Sub(now) - if dur < 0 { + if futureTime.Before(now) { return 0, 0, fmt.Errorf("time '%s' must be in the future (after %s)", futureTime.Format(time.RFC3339), now.Format(time.RFC3339)) } @@ -65,27 +64,12 @@ func (x *FrostFS) TimeToEpoch(ctx context.Context, now, futureTime time.Time) (u return 0, 0, handleObjectError("get network info via client", err) } - durEpoch := networkInfo.EpochDuration() - if durEpoch == 0 { - return 0, 0, errors.New("epoch duration is missing or zero") - } - - curr := networkInfo.CurrentEpoch() - msPerEpoch := durEpoch * uint64(networkInfo.MsPerBlock()) - - epochLifetime := uint64(dur.Milliseconds()) / msPerEpoch - if uint64(dur.Milliseconds())%msPerEpoch != 0 { - epochLifetime++ - } - - var epoch uint64 - if epochLifetime >= math.MaxUint64-curr { - epoch = math.MaxUint64 - } else { - epoch = curr + epochLifetime + epoch, err := util.TimeToEpoch(&networkInfo, now, futureTime) + if err != nil { + return 0, 0, err } - return curr, epoch, nil + return networkInfo.CurrentEpoch(), epoch, nil } // Container implements layer.FrostFS interface method. diff --git a/internal/frostfs/util/util.go b/internal/frostfs/util/util.go index 444504b1..00a99059 100644 --- a/internal/frostfs/util/util.go +++ b/internal/frostfs/util/util.go @@ -2,9 +2,12 @@ package util import ( "fmt" + "math" "strings" + "time" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container" + "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/ns" "github.com/nspcc-dev/neo-go/pkg/util" ) @@ -32,3 +35,39 @@ func ResolveContractHash(contractHash, rpcAddress string) (util.Uint160, error) return nns.ResolveContractHash(domain) } + +func TimeToEpoch(ni *netmap.NetworkInfo, now, t time.Time) (uint64, error) { + duration := t.Sub(now) + durationAbs := duration.Abs() + + durEpoch := ni.EpochDuration() + if durEpoch == 0 { + return 0, fmt.Errorf("epoch duration is missing or zero") + } + + msPerEpoch := durEpoch * uint64(ni.MsPerBlock()) + epochLifetime := uint64(durationAbs.Milliseconds()) / msPerEpoch + + if uint64(durationAbs.Milliseconds())%msPerEpoch != 0 { + epochLifetime++ + } + + curr := ni.CurrentEpoch() + + var epoch uint64 + if duration > 0 { + if epochLifetime >= math.MaxUint64-curr { + epoch = math.MaxUint64 + } else { + epoch = curr + epochLifetime + } + } else { + if epochLifetime >= curr { + epoch = 0 + } else { + epoch = curr - epochLifetime + } + } + + return epoch, nil +} ```
dkirillov marked this conversation as resolved
mbiryukova force-pushed feature/lifecycle_convert_date from ff57dbb1dc to a6372f252f 2024-10-18 15:47:15 +00:00 Compare
alexvanin approved these changes 2024-10-22 14:06:44 +00:00
dkirillov approved these changes 2024-10-22 14:15:18 +00:00
alexvanin merged commit 09c11262c6 into master 2024-10-22 14:21:50 +00:00
alexvanin deleted branch feature/lifecycle_convert_date 2024-10-22 14:21:51 +00:00
alexvanin added this to the v0.31.0 milestone 2024-11-20 12:09:53 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
alexvanin
dkirillov
TrueCloudLab/storage-services-developers
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
good first issue

Good for newcomers

  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
good first issue
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
v0.31.0
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
mbiryukova
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-s3-gw#516
No description provided.
Delete branch "mbiryukova/frostfs-s3-gw:feature/lifecycle_convert_date"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?