Compare commits

...

41 commits

Author SHA1 Message Date
84bbd767ae Use netmap in tree pool requests
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-10-28 18:03:56 +03:00
424038de6c [#524] Update pool to treat maintenance mode differently
Contains these changes for pool component in SDK:
* frostfs-sdk-go#279 fix mm error counting during search operation
* frostfs-sdk-go#283 immediately mark mm node as unhealthy
* frostfs-sdk-go#278 do not reconnect to mm node

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-10-23 12:17:20 +00:00
3cf27d281d [#509] Support fallback address when getting box
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
3c7cb82553 [#509] Init resolvers before first resolving
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
57b7e83380 [#509] Save isCustom flag into accessbox
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
6a90f4e624 [#509] Update docs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
cb3753f286 [#509] Add tests for custom credentials
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
81209e308c [#509] Fix tests
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
b78e55e101 [#509] Support custom AWS credentials
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
25c24f5ce6 [#522] Add waiter to contract clients
Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2024-10-23 09:22:19 +03:00
09c11262c6 [#516] Check Content-Md5 of lifecycle configuration
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-10-22 14:21:49 +00:00
f120715a37 [#516] Convert expiration date to epoch
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-10-22 14:21:49 +00:00
aaed083d82 [#520] Support the continuous use of interceptors
We can always add interceptors to the grpc
connection to the storage, since the actual
use will be controlled by the configuration
from the frostfs-observability library.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-10-21 11:49:22 +03:00
e35b582fe2 [#506] Deny bucket names with dot
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-10-08 12:50:22 +03:00
39fc7aa3ee [#404] Fix using encoding type in multipart upload listing
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-10-08 09:44:05 +00:00
da41f47826 [#404] Add check of encoding type in object listing
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-10-08 09:44:05 +00:00
9e5fb4be95 [#507] Return not implemented by default in bucket router
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-10-07 05:38:35 +00:00
346243b159 [#450] Fix getPutPayloadSize
If X-Amz-Decoded-Content-Length explicitly set use it even if value is 0

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-03 14:23:06 +03:00
03481274f0 [#467] authmate: Add sign command
Support singing arbitrary data using aws sigv4 algorithm

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-02 13:42:22 +00:00
c2adbd758a [#488] middleware/auth: Add frostfs-to-s3 error transformation
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-10-02 13:18:25 +03:00
bc17ab5e47 [#488] middleware/policy: Add frostfs-to-s3 error transformation
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-10-02 12:35:04 +03:00
9fadfbbc2f [#488] Renamed api/errors, layer/frostfs and layer/tree package names
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-10-02 12:35:04 +03:00
827ea1a41e [#488] Move layer/frostfs.go to layer/frostfs/frostfs.go
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-10-02 12:35:04 +03:00
968f10a72f [#488] Move layer/tree_service.go to layer/tree/tree_service.go
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-10-02 12:35:04 +03:00
582e6ac642 [#503] Update SDK to fix error counting
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-09-27 10:36:48 +03:00
99f273f9af [#461] Configure logger sampling policy
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2024-09-26 10:34:44 +03:00
cd96adef36 [#499] Fix of sighup traicing docs
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-09-25 14:34:18 +03:00
738ce14f50 [#434] Remove container on failed bucket creation
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-09-25 07:15:24 +00:00
5358e39f71 [#496] Update frostfs-go-sdk for in-flight requests support
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-09-23 08:57:40 +03:00
34c1426b9f [#484] Add root ca cert for telemetry configuration
Signed-off-by: Aleksey Savaitan <a.savaitan@yadro.com>
2024-09-19 11:07:13 +00:00
8ca73e2079 [#493] Fix of receiving VHS namespaces map
In the process of forming a map with namespaces
for which VHS is enabled, we resolve the alias
of the namespace. The problem is that to resolve,
we need default namespace names, which in turn do
not have time to decide by this time. Therefore,
now the check for the default name takes place
directly in the prepareVHSNamespaces function
based on previously read default names.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-09-17 16:57:05 +03:00
a87c636b4c [#493] Fix X-Frostfs-S3-VHS header processing
It is assumed that the X-Frostfs-S3-VHS header
will have the value enabled/disabled instead
of true/false.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-09-16 14:28:19 +03:00
26baf8a94e [#492] Add panic catchers to fuzzing tests
Signed-off-by: Roman Ognev <r.ognev@yadro.com>
2024-09-16 10:36:30 +00:00
f187141ae5 [#486] Fix PUT object with negative Content-Length
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-09-16 08:45:46 +00:00
3cffc782e9 [#486] Fix PUT encrypted object
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-09-16 08:45:46 +00:00
d0e4d55772 [#460] Add network info cache
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-09-13 09:56:24 +00:00
42e72889a5 [#450] Add test for chunk-encoded object size
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-09-13 12:00:24 +03:00
98815d5473 [#450] Fix aws-chunked header workflow
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2024-09-13 11:59:07 +03:00
62615d7ab7 [#369] Request reproducer
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-09-11 15:25:09 +03:00
575ab4d294 [#369] Enhanced http requests logging
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-09-11 15:25:09 +03:00
d919e6cce2 [#482] Fix containers resolving
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-09-05 12:33:14 +03:00
116 changed files with 4172 additions and 1530 deletions

View file

@ -3,11 +3,12 @@ FROM golang:1.22 AS builder
ARG BUILD=now
ARG REPO=git.frostfs.info/TrueCloudLab/frostfs-s3-gw
ARG VERSION=dev
ARG GOFLAGS=""
WORKDIR /src
COPY . /src
RUN make
RUN make GOFLAGS=${GOFLAGS}
# Executable image
FROM alpine AS frostfs-s3-gw

View file

@ -8,9 +8,12 @@ This document outlines major changes between releases.
- Add support for virtual hosted style addressing (#446, #449)
- Support new param `frostfs.graceful_close_on_switch_timeout` (#475)
- Support patch object method (#479)
- Add `sign` command to `frostfs-s3-authmate` (#467)
- Support custom aws credentials (#509)
### Changed
- Update go version to go1.19 (#470)
- Avoid maintenance mode storage node during object operations (#524)
## [0.30.0] - Kangshung -2024-07-19

View file

@ -14,6 +14,8 @@ METRICS_DUMP_OUT ?= ./metrics-dump.json
CMDS = $(addprefix frostfs-, $(notdir $(wildcard cmd/*)))
BINS = $(addprefix $(BINDIR)/, $(CMDS))
GOFLAGS ?=
# Variables for docker
REPO_BASENAME = $(shell basename `go list -m`)
HUB_IMAGE ?= "truecloudlab/$(REPO_BASENAME)"
@ -44,6 +46,7 @@ all: $(BINS)
$(BINS): $(BINDIR) dep
@echo "⇒ Build $@"
CGO_ENABLED=0 \
GOFLAGS=$(GOFLAGS) \
go build -v -trimpath \
-ldflags "-X $(REPO)/internal/version.Version=$(VERSION)" \
-o $@ ./cmd/$(subst frostfs-,,$(notdir $@))
@ -70,7 +73,7 @@ docker/%:
-w /src \
-u `stat -c "%u:%g" .` \
--env HOME=/src \
golang:$(GO_VERSION) make $*,\
golang:$(GO_VERSION) make GOFLAGS=$(GOFLAGS) $*,\
@echo "supported docker targets: all $(BINS) lint")
# Run tests
@ -121,6 +124,7 @@ image:
@docker build \
--build-arg REPO=$(REPO) \
--build-arg VERSION=$(VERSION) \
--build-arg GOFLAGS=$(GOFLAGS) \
--rm \
-f .docker/Dockerfile \
-t $(HUB_IMAGE):$(HUB_TAG) .

View file

@ -14,16 +14,17 @@ import (
"time"
v4 "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth/signer/v4"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/tokens"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/aws/aws-sdk-go/aws/credentials"
)
// authorizationFieldRegexp -- is regexp for credentials with Base58 encoded cid and oid and '0' (zero) as delimiter.
var authorizationFieldRegexp = regexp.MustCompile(`AWS4-HMAC-SHA256 Credential=(?P<access_key_id>[^/]+)/(?P<date>[^/]+)/(?P<region>[^/]*)/(?P<service>[^/]+)/aws4_request,\s*SignedHeaders=(?P<signed_header_fields>.+),\s*Signature=(?P<v4_signature>.+)`)
// AuthorizationFieldRegexp -- is regexp for credentials with Base58 encoded cid and oid and '0' (zero) as delimiter.
var AuthorizationFieldRegexp = regexp.MustCompile(`AWS4-HMAC-SHA256 Credential=(?P<access_key_id>[^/]+)/(?P<date>[^/]+)/(?P<region>[^/]*)/(?P<service>[^/]+)/aws4_request,\s*SignedHeaders=(?P<signed_header_fields>.+),\s*Signature=(?P<v4_signature>.+)`)
// postPolicyCredentialRegexp -- is regexp for credentials when uploading file using POST with policy.
var postPolicyCredentialRegexp = regexp.MustCompile(`(?P<access_key_id>[^/]+)/(?P<date>[^/]+)/(?P<region>[^/]*)/(?P<service>[^/]+)/aws4_request`)
@ -34,6 +35,11 @@ type (
postReg *RegexpSubmatcher
cli tokens.Credentials
allowedAccessKeyIDPrefixes []string // empty slice means all access key ids are allowed
settings CenterSettings
}
CenterSettings interface {
AccessBoxContainer() (cid.ID, bool)
}
//nolint:revive
@ -50,7 +56,6 @@ type (
)
const (
accessKeyPartsNum = 2
authHeaderPartsNum = 6
maxFormSizeMemory = 50 * 1048576 // 50 MB
@ -82,24 +87,20 @@ var ContentSHA256HeaderStandardValue = map[string]struct{}{
}
// New creates an instance of AuthCenter.
func New(creds tokens.Credentials, prefixes []string) *Center {
func New(creds tokens.Credentials, prefixes []string, settings CenterSettings) *Center {
return &Center{
cli: creds,
reg: NewRegexpMatcher(authorizationFieldRegexp),
reg: NewRegexpMatcher(AuthorizationFieldRegexp),
postReg: NewRegexpMatcher(postPolicyCredentialRegexp),
allowedAccessKeyIDPrefixes: prefixes,
settings: settings,
}
}
func (c *Center) parseAuthHeader(header string) (*AuthHeader, error) {
submatches := c.reg.GetSubmatches(header)
if len(submatches) != authHeaderPartsNum {
return nil, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrAuthorizationHeaderMalformed), header)
}
accessKey := strings.Split(submatches["access_key_id"], "0")
if len(accessKey) != accessKeyPartsNum {
return nil, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrInvalidAccessKeyID), accessKey)
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrAuthorizationHeaderMalformed), header)
}
signedFields := strings.Split(submatches["signed_header_fields"], ";")
@ -114,15 +115,6 @@ func (c *Center) parseAuthHeader(header string) (*AuthHeader, error) {
}, nil
}
func getAddress(accessKeyID string) (oid.Address, error) {
var addr oid.Address
if err := addr.DecodeString(strings.ReplaceAll(accessKeyID, "0", "/")); err != nil {
return addr, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrInvalidAccessKeyID), accessKeyID)
}
return addr, nil
}
func IsStandardContentSHA256(key string) bool {
_, ok := ContentSHA256HeaderStandardValue[key]
return ok
@ -181,14 +173,14 @@ func (c *Center) Authenticate(r *http.Request) (*middleware.Box, error) {
return nil, err
}
addr, err := getAddress(authHdr.AccessKeyID)
cnrID, err := c.getAccessBoxContainer(authHdr.AccessKeyID)
if err != nil {
return nil, err
}
box, attrs, err := c.cli.GetBox(r.Context(), addr)
box, attrs, err := c.cli.GetBox(r.Context(), cnrID, authHdr.AccessKeyID)
if err != nil {
return nil, fmt.Errorf("get box '%s': %w", addr, err)
return nil, fmt.Errorf("get box by access key '%s': %w", authHdr.AccessKeyID, err)
}
if err = checkFormatHashContentSHA256(r.Header.Get(AmzContentSHA256)); err != nil {
@ -216,15 +208,29 @@ func (c *Center) Authenticate(r *http.Request) (*middleware.Box, error) {
return result, nil
}
func (c *Center) getAccessBoxContainer(accessKeyID string) (cid.ID, error) {
var addr oid.Address
if err := addr.DecodeString(strings.ReplaceAll(accessKeyID, "0", "/")); err == nil {
return addr.Container(), nil
}
cnrID, ok := c.settings.AccessBoxContainer()
if ok {
return cnrID, nil
}
return cid.ID{}, fmt.Errorf("%w: unknown container for creds '%s'", apierr.GetAPIError(apierr.ErrInvalidAccessKeyID), accessKeyID)
}
func checkFormatHashContentSHA256(hash string) error {
if !IsStandardContentSHA256(hash) {
hashBinary, err := hex.DecodeString(hash)
if err != nil {
return fmt.Errorf("%w: decode hash: %s: %s", apiErrors.GetAPIError(apiErrors.ErrContentSHA256Mismatch),
return fmt.Errorf("%w: decode hash: %s: %s", apierr.GetAPIError(apierr.ErrContentSHA256Mismatch),
hash, err.Error())
}
if len(hashBinary) != sha256.Size && len(hash) != 0 {
return fmt.Errorf("%w: invalid hash size %d", apiErrors.GetAPIError(apiErrors.ErrContentSHA256Mismatch), len(hashBinary))
return fmt.Errorf("%w: invalid hash size %d", apierr.GetAPIError(apierr.ErrContentSHA256Mismatch), len(hashBinary))
}
}
@ -242,12 +248,12 @@ func (c Center) checkAccessKeyID(accessKeyID string) error {
}
}
return fmt.Errorf("%w: accesskeyID prefix isn't allowed", apiErrors.GetAPIError(apiErrors.ErrAccessDenied))
return fmt.Errorf("%w: accesskeyID prefix isn't allowed", apierr.GetAPIError(apierr.ErrAccessDenied))
}
func (c *Center) checkFormData(r *http.Request) (*middleware.Box, error) {
if err := r.ParseMultipartForm(maxFormSizeMemory); err != nil {
return nil, fmt.Errorf("%w: parse multipart form with max size %d", apiErrors.GetAPIError(apiErrors.ErrInvalidArgument), maxFormSizeMemory)
return nil, fmt.Errorf("%w: parse multipart form with max size %d", apierr.GetAPIError(apierr.ErrInvalidArgument), maxFormSizeMemory)
}
if err := prepareForm(r.MultipartForm); err != nil {
@ -262,7 +268,7 @@ func (c *Center) checkFormData(r *http.Request) (*middleware.Box, error) {
creds := MultipartFormValue(r, "x-amz-credential")
submatches := c.postReg.GetSubmatches(creds)
if len(submatches) != 4 {
return nil, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrAuthorizationHeaderMalformed), creds)
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrAuthorizationHeaderMalformed), creds)
}
signatureDateTime, err := time.Parse("20060102T150405Z", MultipartFormValue(r, "x-amz-date"))
@ -272,14 +278,14 @@ func (c *Center) checkFormData(r *http.Request) (*middleware.Box, error) {
accessKeyID := submatches["access_key_id"]
addr, err := getAddress(accessKeyID)
cnrID, err := c.getAccessBoxContainer(accessKeyID)
if err != nil {
return nil, err
}
box, attrs, err := c.cli.GetBox(r.Context(), addr)
box, attrs, err := c.cli.GetBox(r.Context(), cnrID, accessKeyID)
if err != nil {
return nil, fmt.Errorf("get box '%s': %w", addr, err)
return nil, fmt.Errorf("get box by accessKeyID '%s': %w", accessKeyID, err)
}
secret := box.Gate.SecretKey
@ -288,7 +294,7 @@ func (c *Center) checkFormData(r *http.Request) (*middleware.Box, error) {
signature := SignStr(secret, service, region, signatureDateTime, policy)
reqSignature := MultipartFormValue(r, "x-amz-signature")
if signature != reqSignature {
return nil, fmt.Errorf("%w: %s != %s", apiErrors.GetAPIError(apiErrors.ErrSignatureDoesNotMatch),
return nil, fmt.Errorf("%w: %s != %s", apierr.GetAPIError(apierr.ErrSignatureDoesNotMatch),
reqSignature, signature)
}
@ -333,11 +339,11 @@ func (c *Center) checkSign(authHeader *AuthHeader, box *accessbox.Box, request *
if authHeader.IsPresigned {
now := time.Now()
if signatureDateTime.Add(authHeader.Expiration).Before(now) {
return fmt.Errorf("%w: expired: now %s, signature %s", apiErrors.GetAPIError(apiErrors.ErrExpiredPresignRequest),
return fmt.Errorf("%w: expired: now %s, signature %s", apierr.GetAPIError(apierr.ErrExpiredPresignRequest),
now.Format(time.RFC3339), signatureDateTime.Format(time.RFC3339))
}
if now.Before(signatureDateTime) {
return fmt.Errorf("%w: signature time from the future: now %s, signature %s", apiErrors.GetAPIError(apiErrors.ErrBadRequest),
return fmt.Errorf("%w: signature time from the future: now %s, signature %s", apierr.GetAPIError(apierr.ErrBadRequest),
now.Format(time.RFC3339), signatureDateTime.Format(time.RFC3339))
}
if _, err := signer.Presign(request, nil, authHeader.Service, authHeader.Region, authHeader.Expiration, signatureDateTime); err != nil {
@ -352,7 +358,7 @@ func (c *Center) checkSign(authHeader *AuthHeader, box *accessbox.Box, request *
}
if authHeader.SignatureV4 != signature {
return fmt.Errorf("%w: %s != %s: headers %v", apiErrors.GetAPIError(apiErrors.ErrSignatureDoesNotMatch),
return fmt.Errorf("%w: %s != %s: headers %v", apierr.GetAPIError(apierr.ErrSignatureDoesNotMatch),
authHeader.SignatureV4, signature, authHeader.SignedFields)
}

View file

@ -72,7 +72,7 @@ func DoFuzzAuthenticate(input []byte) int {
c := &Center{
cli: mock,
reg: NewRegexpMatcher(authorizationFieldRegexp),
reg: NewRegexpMatcher(AuthorizationFieldRegexp),
postReg: NewRegexpMatcher(postPolicyCredentialRegexp),
}

View file

@ -17,8 +17,9 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/tokens"
frostfsErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
frosterr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
oidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id/test"
@ -28,11 +29,23 @@ import (
"go.uber.org/zap/zaptest"
)
type centerSettingsMock struct {
accessBoxContainer *cid.ID
}
func (c *centerSettingsMock) AccessBoxContainer() (cid.ID, bool) {
if c.accessBoxContainer == nil {
return cid.ID{}, false
}
return *c.accessBoxContainer, true
}
func TestAuthHeaderParse(t *testing.T) {
defaultHeader := "AWS4-HMAC-SHA256 Credential=oid0cid/20210809/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2811ccb9e242f41426738fb1f"
center := &Center{
reg: NewRegexpMatcher(authorizationFieldRegexp),
reg: NewRegexpMatcher(AuthorizationFieldRegexp),
settings: &centerSettingsMock{},
}
for _, tc := range []struct {
@ -57,11 +70,6 @@ func TestAuthHeaderParse(t *testing.T) {
err: errors.GetAPIError(errors.ErrAuthorizationHeaderMalformed),
expected: nil,
},
{
header: strings.ReplaceAll(defaultHeader, "oid0cid", "oidcid"),
err: errors.GetAPIError(errors.ErrInvalidAccessKeyID),
expected: nil,
},
} {
authHeader, err := center.parseAuthHeader(tc.header)
require.ErrorIs(t, err, tc.err, tc.header)
@ -69,43 +77,6 @@ func TestAuthHeaderParse(t *testing.T) {
}
}
func TestAuthHeaderGetAddress(t *testing.T) {
defaulErr := errors.GetAPIError(errors.ErrInvalidAccessKeyID)
for _, tc := range []struct {
authHeader *AuthHeader
err error
}{
{
authHeader: &AuthHeader{
AccessKeyID: "vWqF8cMDRbJcvnPLALoQGnABPPhw8NyYMcGsfDPfZJM0HrgjonN8CgFvCZ3kh9BUXw4W2tJ5E7EAGhueSF122HB",
},
err: nil,
},
{
authHeader: &AuthHeader{
AccessKeyID: "vWqF8cMDRbJcvnPLALoQGnABPPhw8NyYMcGsfDPfZJMHrgjonN8CgFvCZ3kh9BUXw4W2tJ5E7EAGhueSF122HB",
},
err: defaulErr,
},
{
authHeader: &AuthHeader{
AccessKeyID: "oid0cid",
},
err: defaulErr,
},
{
authHeader: &AuthHeader{
AccessKeyID: "oidcid",
},
err: defaulErr,
},
} {
_, err := getAddress(tc.authHeader.AccessKeyID)
require.ErrorIs(t, err, tc.err, tc.authHeader.AccessKeyID)
}
}
func TestSignature(t *testing.T) {
secret := "66be461c3cd429941c55daf42fad2b8153e5a2016ba89c9494d97677cc9d3872"
strToSign := "eyAiZXhwaXJhdGlvbiI6ICIyMDE1LTEyLTMwVDEyOjAwOjAwLjAwMFoiLAogICJjb25kaXRpb25zIjogWwogICAgeyJidWNrZXQiOiAiYWNsIn0sCiAgICBbInN0YXJ0cy13aXRoIiwgIiRrZXkiLCAidXNlci91c2VyMS8iXSwKICAgIHsic3VjY2Vzc19hY3Rpb25fcmVkaXJlY3QiOiAiaHR0cDovL2xvY2FsaG9zdDo4MDg0L2FjbCJ9LAogICAgWyJzdGFydHMtd2l0aCIsICIkQ29udGVudC1UeXBlIiwgImltYWdlLyJdLAogICAgeyJ4LWFtei1tZXRhLXV1aWQiOiAiMTQzNjUxMjM2NTEyNzQifSwKICAgIFsic3RhcnRzLXdpdGgiLCAiJHgtYW16LW1ldGEtdGFnIiwgIiJdLAoKICAgIHsiWC1BbXotQ3JlZGVudGlhbCI6ICI4Vmk0MVBIbjVGMXNzY2J4OUhqMXdmMUU2aERUYURpNndxOGhxTU05NllKdTA1QzVDeUVkVlFoV1E2aVZGekFpTkxXaTlFc3BiUTE5ZDRuR3pTYnZVZm10TS8yMDE1MTIyOS91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0sCiAgICB7IngtYW16LWFsZ29yaXRobSI6ICJBV1M0LUhNQUMtU0hBMjU2In0sCiAgICB7IlgtQW16LURhdGUiOiAiMjAxNTEyMjlUMDAwMDAwWiIgfSwKICAgIHsieC1pZ25vcmUtdG1wIjogInNvbWV0aGluZyIgfQogIF0KfQ=="
@ -171,17 +142,17 @@ func TestCheckFormatContentSHA256(t *testing.T) {
}
type frostFSMock struct {
objects map[oid.Address]*object.Object
objects map[string]*object.Object
}
func newFrostFSMock() *frostFSMock {
return &frostFSMock{
objects: map[oid.Address]*object.Object{},
objects: map[string]*object.Object{},
}
}
func (f *frostFSMock) GetCredsObject(_ context.Context, address oid.Address) (*object.Object, error) {
obj, ok := f.objects[address]
func (f *frostFSMock) GetCredsObject(_ context.Context, prm tokens.PrmGetCredsObject) (*object.Object, error) {
obj, ok := f.objects[prm.AccessKeyID]
if !ok {
return nil, fmt.Errorf("not found")
}
@ -208,7 +179,7 @@ func TestAuthenticate(t *testing.T) {
GateKey: key.PublicKey(),
}}
accessBox, secret, err := accessbox.PackTokens(gateData, []byte("secret"))
accessBox, secret, err := accessbox.PackTokens(gateData, []byte("secret"), false)
require.NoError(t, err)
data, err := accessBox.Marshal()
require.NoError(t, err)
@ -219,10 +190,10 @@ func TestAuthenticate(t *testing.T) {
obj.SetContainerID(addr.Container())
obj.SetID(addr.Object())
frostfs := newFrostFSMock()
frostfs.objects[addr] = &obj
accessKeyID := getAccessKeyID(addr)
accessKeyID := addr.Container().String() + "0" + addr.Object().String()
frostfs := newFrostFSMock()
frostfs.objects[accessKeyID] = &obj
awsCreds := credentials.NewStaticCredentials(accessKeyID, secret.SecretKey, "")
defaultSigner := v4.NewSigner(awsCreds)
@ -413,13 +384,13 @@ func TestAuthenticate(t *testing.T) {
} {
t.Run(tc.name, func(t *testing.T) {
creds := tokens.New(bigConfig)
cntr := New(creds, tc.prefixes)
cntr := New(creds, tc.prefixes, &centerSettingsMock{})
box, err := cntr.Authenticate(tc.request)
if tc.err {
require.Error(t, err)
if tc.errCode > 0 {
err = frostfsErrors.UnwrapErr(err)
err = frosterr.UnwrapErr(err)
require.Equal(t, errors.GetAPIError(tc.errCode), err)
}
} else {
@ -455,7 +426,7 @@ func TestHTTPPostAuthenticate(t *testing.T) {
GateKey: key.PublicKey(),
}}
accessBox, secret, err := accessbox.PackTokens(gateData, []byte("secret"))
accessBox, secret, err := accessbox.PackTokens(gateData, []byte("secret"), false)
require.NoError(t, err)
data, err := accessBox.Marshal()
require.NoError(t, err)
@ -466,10 +437,11 @@ func TestHTTPPostAuthenticate(t *testing.T) {
obj.SetContainerID(addr.Container())
obj.SetID(addr.Object())
frostfs := newFrostFSMock()
frostfs.objects[addr] = &obj
accessKeyID := getAccessKeyID(addr)
frostfs := newFrostFSMock()
frostfs.objects[accessKeyID] = &obj
accessKeyID := addr.Container().String() + "0" + addr.Object().String()
invalidAccessKeyID := oidtest.Address().String() + "0" + oidtest.Address().Object().String()
timeToSign := time.Now()
@ -590,13 +562,13 @@ func TestHTTPPostAuthenticate(t *testing.T) {
} {
t.Run(tc.name, func(t *testing.T) {
creds := tokens.New(bigConfig)
cntr := New(creds, tc.prefixes)
cntr := New(creds, tc.prefixes, &centerSettingsMock{})
box, err := cntr.Authenticate(tc.request)
if tc.err {
require.Error(t, err)
if tc.errCode > 0 {
err = frostfsErrors.UnwrapErr(err)
err = frosterr.UnwrapErr(err)
require.Equal(t, errors.GetAPIError(tc.errCode), err)
}
} else {
@ -633,3 +605,7 @@ func getRequestWithMultipartForm(t *testing.T, policy, creds, date, sign, fieldN
return req
}
func getAccessKeyID(addr oid.Address) string {
return strings.ReplaceAll(addr.EncodeToString(), "/", "0")
}

View file

@ -29,11 +29,11 @@ func newTokensFrostfsMock() *credentialsMock {
}
func (m credentialsMock) addBox(addr oid.Address, box *accessbox.Box) {
m.boxes[addr.String()] = box
m.boxes[getAccessKeyID(addr)] = box
}
func (m credentialsMock) GetBox(_ context.Context, addr oid.Address) (*accessbox.Box, []object.Attribute, error) {
box, ok := m.boxes[addr.String()]
func (m credentialsMock) GetBox(_ context.Context, _ cid.ID, accessKeyID string) (*accessbox.Box, []object.Attribute, error) {
box, ok := m.boxes[accessKeyID]
if !ok {
return nil, nil, &apistatus.ObjectNotFound{}
}
@ -41,11 +41,11 @@ func (m credentialsMock) GetBox(_ context.Context, addr oid.Address) (*accessbox
return box, nil, nil
}
func (m credentialsMock) Put(context.Context, cid.ID, tokens.CredentialsParam) (oid.Address, error) {
func (m credentialsMock) Put(context.Context, tokens.CredentialsParam) (oid.Address, error) {
return oid.Address{}, nil
}
func (m credentialsMock) Update(context.Context, oid.Address, tokens.CredentialsParam) (oid.Address, error) {
func (m credentialsMock) Update(context.Context, tokens.CredentialsParam) (oid.Address, error) {
return oid.Address{}, nil
}
@ -84,9 +84,10 @@ func TestCheckSign(t *testing.T) {
mock.addBox(accessKeyAddr, expBox)
c := &Center{
cli: mock,
reg: NewRegexpMatcher(authorizationFieldRegexp),
postReg: NewRegexpMatcher(postPolicyCredentialRegexp),
cli: mock,
reg: NewRegexpMatcher(AuthorizationFieldRegexp),
postReg: NewRegexpMatcher(postPolicyCredentialRegexp),
settings: &centerSettingsMock{},
}
box, err := c.Authenticate(req)
require.NoError(t, err)

View file

@ -30,6 +30,7 @@ type (
Box *accessbox.Box
Attributes []object.Attribute
PutTime time.Time
Address *oid.Address
}
)
@ -57,8 +58,8 @@ func NewAccessBoxCache(config *Config) *AccessBoxCache {
}
// Get returns a cached accessbox.
func (o *AccessBoxCache) Get(address oid.Address) *AccessBoxCacheValue {
entry, err := o.cache.Get(address)
func (o *AccessBoxCache) Get(accessKeyID string) *AccessBoxCacheValue {
entry, err := o.cache.Get(accessKeyID)
if err != nil {
return nil
}
@ -74,16 +75,11 @@ func (o *AccessBoxCache) Get(address oid.Address) *AccessBoxCacheValue {
}
// Put stores an accessbox to cache.
func (o *AccessBoxCache) Put(address oid.Address, box *accessbox.Box, attrs []object.Attribute) error {
val := &AccessBoxCacheValue{
Box: box,
Attributes: attrs,
PutTime: time.Now(),
}
return o.cache.Set(address, val)
func (o *AccessBoxCache) Put(accessKeyID string, val *AccessBoxCacheValue) error {
return o.cache.Set(accessKeyID, val)
}
// Delete removes an accessbox from cache.
func (o *AccessBoxCache) Delete(address oid.Address) {
o.cache.Remove(address)
func (o *AccessBoxCache) Delete(accessKeyID string) {
o.cache.Remove(accessKeyID)
}

View file

@ -1,13 +1,14 @@
package cache
import (
"strings"
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-contract/frostfsid/client"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
cidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id/test"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
oidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id/test"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/util"
@ -22,18 +23,21 @@ func TestAccessBoxCacheType(t *testing.T) {
addr := oidtest.Address()
box := &accessbox.Box{}
var attrs []object.Attribute
val := &AccessBoxCacheValue{
Box: box,
}
err := cache.Put(addr, box, attrs)
accessKeyID := getAccessKeyID(addr)
err := cache.Put(accessKeyID, val)
require.NoError(t, err)
val := cache.Get(addr)
require.Equal(t, box, val.Box)
require.Equal(t, attrs, val.Attributes)
resVal := cache.Get(accessKeyID)
require.Equal(t, box, resVal.Box)
require.Equal(t, 0, observedLog.Len())
err = cache.cache.Set(addr, "tmp")
err = cache.cache.Set(accessKeyID, "tmp")
require.NoError(t, err)
assertInvalidCacheEntry(t, cache.Get(addr), observedLog)
assertInvalidCacheEntry(t, cache.Get(accessKeyID), observedLog)
}
func TestBucketsCacheType(t *testing.T) {
@ -230,3 +234,7 @@ func getObservedLogger() (*zap.Logger, *observer.ObservedLogs) {
loggerCore, observedLog := observer.New(zap.WarnLevel)
return zap.New(loggerCore), observedLog
}
func getAccessKeyID(addr oid.Address) string {
return strings.ReplaceAll(addr.EncodeToString(), "/", "0")
}

65
api/cache/network_info.go vendored Normal file
View file

@ -0,0 +1,65 @@
package cache
import (
"fmt"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/bluele/gcache"
"go.uber.org/zap"
)
type (
// NetworkInfoCache provides cache for network info.
NetworkInfoCache struct {
cache gcache.Cache
logger *zap.Logger
}
// NetworkInfoCacheConfig stores expiration params for cache.
NetworkInfoCacheConfig struct {
Lifetime time.Duration
Logger *zap.Logger
}
)
const (
DefaultNetworkInfoCacheLifetime = 1 * time.Minute
networkInfoCacheSize = 1
networkInfoKey = "network_info"
)
// DefaultNetworkInfoConfig returns new default cache expiration values.
func DefaultNetworkInfoConfig(logger *zap.Logger) *NetworkInfoCacheConfig {
return &NetworkInfoCacheConfig{
Lifetime: DefaultNetworkInfoCacheLifetime,
Logger: logger,
}
}
// NewNetworkInfoCache creates an object of NetworkInfoCache.
func NewNetworkInfoCache(config *NetworkInfoCacheConfig) *NetworkInfoCache {
gc := gcache.New(networkInfoCacheSize).LRU().Expiration(config.Lifetime).Build()
return &NetworkInfoCache{cache: gc, logger: config.Logger}
}
func (c *NetworkInfoCache) Get() *netmap.NetworkInfo {
entry, err := c.cache.Get(networkInfoKey)
if err != nil {
return nil
}
result, ok := entry.(netmap.NetworkInfo)
if !ok {
c.logger.Warn(logs.InvalidCacheEntryType, zap.String("actual", fmt.Sprintf("%T", entry)),
zap.String("expected", fmt.Sprintf("%T", result)))
return nil
}
return &result
}
func (c *NetworkInfoCache) Put(info netmap.NetworkInfo) error {
return c.cache.Set(networkInfoKey, info)
}

View file

@ -6,6 +6,7 @@ import (
"time"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
@ -32,6 +33,7 @@ type (
LocationConstraint string
ObjectLockEnabled bool
HomomorphicHashDisabled bool
PlacementPolicy netmap.PlacementPolicy
}
// ObjectInfo holds S3 object data.

View file

@ -27,9 +27,10 @@ type (
}
LifecycleExpiration struct {
Date string `xml:"Date,omitempty"`
Days *int `xml:"Days,omitempty"`
ExpiredObjectDeleteMarker *bool `xml:"ExpiredObjectDeleteMarker,omitempty"`
Date string `xml:"Date,omitempty"`
Days *int `xml:"Days,omitempty"`
Epoch *uint64 `xml:"Epoch,omitempty"`
ExpiredObjectDeleteMarker *bool `xml:"ExpiredObjectDeleteMarker,omitempty"`
}
LifecycleRuleFilter struct {

View file

@ -1,10 +1,13 @@
package errors
import (
"errors"
"fmt"
"net/http"
frosterrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
frosterr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
)
type (
@ -1765,7 +1768,7 @@ var errorCodes = errorCodeMap{
// IsS3Error checks if the provided error is a specific s3 error.
func IsS3Error(err error, code ErrorCode) bool {
err = frosterrors.UnwrapErr(err)
err = frosterr.UnwrapErr(err)
e, ok := err.(Error)
return ok && e.ErrCode == code
}
@ -1802,6 +1805,26 @@ func GetAPIErrorWithError(code ErrorCode, err error) Error {
return errorCodes.toAPIErrWithErr(code, err)
}
// TransformToS3Error converts FrostFS error to the corresponding S3 error type.
func TransformToS3Error(err error) error {
err = frosterr.UnwrapErr(err) // this wouldn't work with errors.Join
var s3err Error
if errors.As(err, &s3err) {
return err
}
if errors.Is(err, frostfs.ErrAccessDenied) ||
errors.Is(err, tree.ErrNodeAccessDenied) {
return GetAPIError(ErrAccessDenied)
}
if errors.Is(err, frostfs.ErrGatewayTimeout) {
return GetAPIError(ErrGatewayTimeout)
}
return GetAPIError(ErrInternalError)
}
// ObjectError -- error that is linked to a specific object.
type ObjectError struct {
Err error

View file

@ -2,7 +2,12 @@ package errors
import (
"errors"
"fmt"
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"github.com/stretchr/testify/require"
)
func BenchmarkErrCode(b *testing.B) {
@ -24,3 +29,56 @@ func BenchmarkErrorsIs(b *testing.B) {
}
}
}
func TestTransformS3Errors(t *testing.T) {
for _, tc := range []struct {
name string
err error
expected ErrorCode
}{
{
name: "simple std error to internal error",
err: errors.New("some error"),
expected: ErrInternalError,
},
{
name: "layer access denied error to s3 access denied error",
err: frostfs.ErrAccessDenied,
expected: ErrAccessDenied,
},
{
name: "wrapped layer access denied error to s3 access denied error",
err: fmt.Errorf("wrap: %w", frostfs.ErrAccessDenied),
expected: ErrAccessDenied,
},
{
name: "layer node access denied error to s3 access denied error",
err: tree.ErrNodeAccessDenied,
expected: ErrAccessDenied,
},
{
name: "layer gateway timeout error to s3 gateway timeout error",
err: frostfs.ErrGatewayTimeout,
expected: ErrGatewayTimeout,
},
{
name: "s3 error to s3 error",
err: GetAPIError(ErrInvalidPart),
expected: ErrInvalidPart,
},
{
name: "wrapped s3 error to s3 error",
err: fmt.Errorf("wrap: %w", GetAPIError(ErrInvalidPart)),
expected: ErrInvalidPart,
},
} {
t.Run(tc.name, func(t *testing.T) {
err := TransformToS3Error(tc.err)
s3err, ok := err.(Error)
require.True(t, ok, "error must be s3 error")
require.Equalf(t, tc.expected, s3err.ErrCode,
"expected: '%s', got: '%s'",
GetAPIError(tc.expected).Code, GetAPIError(s3err.ErrCode).Code)
})
}
}

View file

@ -11,7 +11,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
@ -28,12 +28,12 @@ func TestPutObjectACLErrorAPE(t *testing.T) {
info := createBucket(hc, bktName)
putObjectWithHeadersAssertS3Error(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPublic}, s3errors.ErrAccessControlListNotSupported)
putObjectWithHeadersAssertS3Error(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPublic}, apierr.ErrAccessControlListNotSupported)
putObjectWithHeaders(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPrivate}) // only `private` canned acl is allowed, that is actually ignored
putObjectWithHeaders(hc, bktName, objName, nil)
aclBody := &AccessControlPolicy{}
putObjectACLAssertS3Error(hc, bktName, objName, info.Box, nil, aclBody, s3errors.ErrAccessControlListNotSupported)
putObjectACLAssertS3Error(hc, bktName, objName, info.Box, nil, aclBody, apierr.ErrAccessControlListNotSupported)
aclRes := getObjectACL(hc, bktName, objName)
checkPrivateACL(t, aclRes, info.Key.PublicKey())
@ -49,7 +49,7 @@ func TestCreateObjectACLErrorAPE(t *testing.T) {
copyObject(hc, bktName, objName, objNameCopy, CopyMeta{Headers: map[string]string{api.AmzACL: basicACLPublic}}, http.StatusBadRequest)
copyObject(hc, bktName, objName, objNameCopy, CopyMeta{Headers: map[string]string{api.AmzACL: basicACLPrivate}}, http.StatusOK)
createMultipartUploadAssertS3Error(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPublic}, s3errors.ErrAccessControlListNotSupported)
createMultipartUploadAssertS3Error(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPublic}, apierr.ErrAccessControlListNotSupported)
createMultipartUpload(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPrivate})
}
@ -60,7 +60,7 @@ func TestBucketACLAPE(t *testing.T) {
info := createBucket(hc, bktName)
aclBody := &AccessControlPolicy{}
putBucketACLAssertS3Error(hc, bktName, info.Box, nil, aclBody, s3errors.ErrAccessControlListNotSupported)
putBucketACLAssertS3Error(hc, bktName, info.Box, nil, aclBody, apierr.ErrAccessControlListNotSupported)
aclRes := getBucketACL(hc, bktName)
checkPrivateACL(t, aclRes, info.Key.PublicKey())
@ -113,7 +113,7 @@ func TestBucketPolicy(t *testing.T) {
createTestBucket(hc, bktName)
getBucketPolicy(hc, bktName, s3errors.ErrNoSuchBucketPolicy)
getBucketPolicy(hc, bktName, apierr.ErrNoSuchBucketPolicy)
newPolicy := engineiam.Policy{
Version: "2012-10-17",
@ -125,7 +125,7 @@ func TestBucketPolicy(t *testing.T) {
}},
}
putBucketPolicy(hc, bktName, newPolicy, s3errors.ErrMalformedPolicy)
putBucketPolicy(hc, bktName, newPolicy, apierr.ErrMalformedPolicy)
newPolicy.Statement[0].Resource[0] = arnAwsPrefix + bktName + "/*"
putBucketPolicy(hc, bktName, newPolicy)
@ -140,7 +140,7 @@ func TestBucketPolicyStatus(t *testing.T) {
createTestBucket(hc, bktName)
getBucketPolicy(hc, bktName, s3errors.ErrNoSuchBucketPolicy)
getBucketPolicy(hc, bktName, apierr.ErrNoSuchBucketPolicy)
newPolicy := engineiam.Policy{
Version: "2012-10-17",
@ -152,7 +152,7 @@ func TestBucketPolicyStatus(t *testing.T) {
}},
}
putBucketPolicy(hc, bktName, newPolicy, s3errors.ErrMalformedPolicyNotPrincipal)
putBucketPolicy(hc, bktName, newPolicy, apierr.ErrMalformedPolicyNotPrincipal)
newPolicy.Statement[0].NotPrincipal = nil
newPolicy.Statement[0].Principal = map[engineiam.PrincipalType][]string{engineiam.Wildcard: {}}
@ -221,7 +221,7 @@ func TestPutBucketPolicy(t *testing.T) {
assertStatus(hc.t, w, http.StatusOK)
}
func getBucketPolicy(hc *handlerContext, bktName string, errCode ...s3errors.ErrorCode) engineiam.Policy {
func getBucketPolicy(hc *handlerContext, bktName string, errCode ...apierr.ErrorCode) engineiam.Policy {
w, r := prepareTestRequest(hc, bktName, "", nil)
hc.Handler().GetBucketPolicyHandler(w, r)
@ -231,13 +231,13 @@ func getBucketPolicy(hc *handlerContext, bktName string, errCode ...s3errors.Err
err := json.NewDecoder(w.Result().Body).Decode(&policy)
require.NoError(hc.t, err)
} else {
assertS3Error(hc.t, w, s3errors.GetAPIError(errCode[0]))
assertS3Error(hc.t, w, apierr.GetAPIError(errCode[0]))
}
return policy
}
func getBucketPolicyStatus(hc *handlerContext, bktName string, errCode ...s3errors.ErrorCode) PolicyStatus {
func getBucketPolicyStatus(hc *handlerContext, bktName string, errCode ...apierr.ErrorCode) PolicyStatus {
w, r := prepareTestRequest(hc, bktName, "", nil)
hc.Handler().GetBucketPolicyStatusHandler(w, r)
@ -247,13 +247,13 @@ func getBucketPolicyStatus(hc *handlerContext, bktName string, errCode ...s3erro
err := xml.NewDecoder(w.Result().Body).Decode(&policyStatus)
require.NoError(hc.t, err)
} else {
assertS3Error(hc.t, w, s3errors.GetAPIError(errCode[0]))
assertS3Error(hc.t, w, apierr.GetAPIError(errCode[0]))
}
return policyStatus
}
func putBucketPolicy(hc *handlerContext, bktName string, bktPolicy engineiam.Policy, errCode ...s3errors.ErrorCode) {
func putBucketPolicy(hc *handlerContext, bktName string, bktPolicy engineiam.Policy, errCode ...apierr.ErrorCode) {
body, err := json.Marshal(bktPolicy)
require.NoError(hc.t, err)
@ -263,7 +263,7 @@ func putBucketPolicy(hc *handlerContext, bktName string, bktPolicy engineiam.Pol
if len(errCode) == 0 {
assertStatus(hc.t, w, http.StatusOK)
} else {
assertS3Error(hc.t, w, s3errors.GetAPIError(errCode[0]))
assertS3Error(hc.t, w, apierr.GetAPIError(errCode[0]))
}
}
@ -312,9 +312,9 @@ func createBucket(hc *handlerContext, bktName string) *createBucketInfo {
}
}
func createBucketAssertS3Error(hc *handlerContext, bktName string, box *accessbox.Box, code s3errors.ErrorCode) {
func createBucketAssertS3Error(hc *handlerContext, bktName string, box *accessbox.Box, code apierr.ErrorCode) {
w := createBucketBase(hc, bktName, box)
assertS3Error(hc.t, w, s3errors.GetAPIError(code))
assertS3Error(hc.t, w, apierr.GetAPIError(code))
}
func createBucketBase(hc *handlerContext, bktName string, box *accessbox.Box) *httptest.ResponseRecorder {
@ -330,9 +330,9 @@ func putBucketACL(hc *handlerContext, bktName string, box *accessbox.Box, header
assertStatus(hc.t, w, http.StatusOK)
}
func putBucketACLAssertS3Error(hc *handlerContext, bktName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy, code s3errors.ErrorCode) {
func putBucketACLAssertS3Error(hc *handlerContext, bktName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy, code apierr.ErrorCode) {
w := putBucketACLBase(hc, bktName, box, header, body)
assertS3Error(hc.t, w, s3errors.GetAPIError(code))
assertS3Error(hc.t, w, apierr.GetAPIError(code))
}
func putBucketACLBase(hc *handlerContext, bktName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy) *httptest.ResponseRecorder {
@ -360,9 +360,9 @@ func getBucketACLBase(hc *handlerContext, bktName string) *httptest.ResponseReco
return w
}
func putObjectACLAssertS3Error(hc *handlerContext, bktName, objName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy, code s3errors.ErrorCode) {
func putObjectACLAssertS3Error(hc *handlerContext, bktName, objName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy, code apierr.ErrorCode) {
w := putObjectACLBase(hc, bktName, objName, box, header, body)
assertS3Error(hc.t, w, s3errors.GetAPIError(code))
assertS3Error(hc.t, w, apierr.GetAPIError(code))
}
func putObjectACLBase(hc *handlerContext, bktName, objName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy) *httptest.ResponseRecorder {
@ -396,9 +396,9 @@ func putObjectWithHeaders(hc *handlerContext, bktName, objName string, headers m
return w.Header()
}
func putObjectWithHeadersAssertS3Error(hc *handlerContext, bktName, objName string, headers map[string]string, code s3errors.ErrorCode) {
func putObjectWithHeadersAssertS3Error(hc *handlerContext, bktName, objName string, headers map[string]string, code apierr.ErrorCode) {
w := putObjectWithHeadersBase(hc, bktName, objName, headers, nil, nil)
assertS3Error(hc.t, w, s3errors.GetAPIError(code))
assertS3Error(hc.t, w, apierr.GetAPIError(code))
}
func putObjectWithHeadersBase(hc *handlerContext, bktName, objName string, headers map[string]string, box *accessbox.Box, data []byte) *httptest.ResponseRecorder {

View file

@ -10,7 +10,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
apistatus "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client/status"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
@ -131,7 +131,7 @@ func TestDeleteObjectsError(t *testing.T) {
addr.SetContainer(bktInfo.CID)
addr.SetObject(nodeVersion.OID)
expectedError := apiErrors.GetAPIError(apiErrors.ErrAccessDenied)
expectedError := apierr.GetAPIError(apierr.ErrAccessDenied)
hc.tp.SetObjectError(addr, expectedError)
w := deleteObjectsBase(hc, bktName, [][2]string{{objName, nodeVersion.OID.EncodeToString()}})
@ -553,9 +553,9 @@ func checkNotFound(t *testing.T, hc *handlerContext, bktName, objName, version s
assertStatus(t, w, http.StatusNotFound)
}
func headObjectAssertS3Error(hc *handlerContext, bktName, objName, version string, code apiErrors.ErrorCode) {
func headObjectAssertS3Error(hc *handlerContext, bktName, objName, version string, code apierr.ErrorCode) {
w := headObjectBase(hc, bktName, objName, version)
assertS3Error(hc.t, w, apiErrors.GetAPIError(code))
assertS3Error(hc.t, w, apierr.GetAPIError(code))
}
func checkFound(t *testing.T, hc *handlerContext, bktName, objName, version string) {

View file

@ -16,6 +16,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"github.com/stretchr/testify/require"
)
@ -37,7 +38,7 @@ func TestSimpleGetEncrypted(t *testing.T) {
objInfo, err := tc.Layer().GetObjectInfo(tc.Context(), &layer.HeadObjectParams{BktInfo: bktInfo, Object: objName})
require.NoError(t, err)
obj, err := tc.MockedPool().GetObject(tc.Context(), layer.PrmObjectGet{Container: bktInfo.CID, Object: objInfo.ID})
obj, err := tc.MockedPool().GetObject(tc.Context(), frostfs.PrmObjectGet{Container: bktInfo.CID, Object: objInfo.ID})
require.NoError(t, err)
encryptedContent, err := io.ReadAll(obj.Payload)
require.NoError(t, err)

View file

@ -2,7 +2,7 @@ package handler
import (
"bytes"
stderrors "errors"
"errors"
"fmt"
"io"
"net/http"
@ -13,7 +13,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
apistatus "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client/status"
"github.com/stretchr/testify/require"
@ -89,7 +89,7 @@ func TestPreconditions(t *testing.T) {
name: "IfMatch false",
info: newInfo(etag, today),
args: &conditionalArgs{IfMatch: etag2},
expected: errors.GetAPIError(errors.ErrPreconditionFailed)},
expected: apierr.GetAPIError(apierr.ErrPreconditionFailed)},
{
name: "IfNoneMatch true",
info: newInfo(etag, today),
@ -99,7 +99,7 @@ func TestPreconditions(t *testing.T) {
name: "IfNoneMatch false",
info: newInfo(etag, today),
args: &conditionalArgs{IfNoneMatch: etag},
expected: errors.GetAPIError(errors.ErrNotModified)},
expected: apierr.GetAPIError(apierr.ErrNotModified)},
{
name: "IfModifiedSince true",
info: newInfo(etag, today),
@ -109,7 +109,7 @@ func TestPreconditions(t *testing.T) {
name: "IfModifiedSince false",
info: newInfo(etag, yesterday),
args: &conditionalArgs{IfModifiedSince: &today},
expected: errors.GetAPIError(errors.ErrNotModified)},
expected: apierr.GetAPIError(apierr.ErrNotModified)},
{
name: "IfUnmodifiedSince true",
info: newInfo(etag, yesterday),
@ -119,7 +119,7 @@ func TestPreconditions(t *testing.T) {
name: "IfUnmodifiedSince false",
info: newInfo(etag, today),
args: &conditionalArgs{IfUnmodifiedSince: &yesterday},
expected: errors.GetAPIError(errors.ErrPreconditionFailed)},
expected: apierr.GetAPIError(apierr.ErrPreconditionFailed)},
{
name: "IfMatch true, IfUnmodifiedSince false",
@ -131,19 +131,19 @@ func TestPreconditions(t *testing.T) {
name: "IfMatch false, IfUnmodifiedSince true",
info: newInfo(etag, yesterday),
args: &conditionalArgs{IfMatch: etag2, IfUnmodifiedSince: &today},
expected: errors.GetAPIError(errors.ErrPreconditionFailed),
expected: apierr.GetAPIError(apierr.ErrPreconditionFailed),
},
{
name: "IfNoneMatch false, IfModifiedSince true",
info: newInfo(etag, today),
args: &conditionalArgs{IfNoneMatch: etag, IfModifiedSince: &yesterday},
expected: errors.GetAPIError(errors.ErrNotModified),
expected: apierr.GetAPIError(apierr.ErrNotModified),
},
{
name: "IfNoneMatch true, IfModifiedSince false",
info: newInfo(etag, yesterday),
args: &conditionalArgs{IfNoneMatch: etag2, IfModifiedSince: &today},
expected: errors.GetAPIError(errors.ErrNotModified),
expected: apierr.GetAPIError(apierr.ErrNotModified),
},
} {
t.Run(tc.name, func(t *testing.T) {
@ -151,7 +151,7 @@ func TestPreconditions(t *testing.T) {
if tc.expected == nil {
require.NoError(t, actual)
} else {
require.True(t, stderrors.Is(actual, tc.expected), tc.expected, actual)
require.True(t, errors.Is(actual, tc.expected), tc.expected, actual)
}
})
}
@ -193,8 +193,8 @@ func TestGetObject(t *testing.T) {
hc.tp.SetObjectError(addr, &apistatus.ObjectNotFound{})
hc.tp.SetObjectError(objInfo.Address(), &apistatus.ObjectNotFound{})
getObjectAssertS3Error(hc, bktName, objName, objInfo.VersionID(), errors.ErrNoSuchVersion)
getObjectAssertS3Error(hc, bktName, objName, emptyVersion, errors.ErrNoSuchKey)
getObjectAssertS3Error(hc, bktName, objName, objInfo.VersionID(), apierr.ErrNoSuchVersion)
getObjectAssertS3Error(hc, bktName, objName, emptyVersion, apierr.ErrNoSuchKey)
}
func TestGetObjectEnabledMD5(t *testing.T) {
@ -236,9 +236,9 @@ func getObjectVersion(tc *handlerContext, bktName, objName, version string) []by
return content
}
func getObjectAssertS3Error(hc *handlerContext, bktName, objName, version string, code errors.ErrorCode) {
func getObjectAssertS3Error(hc *handlerContext, bktName, objName, version string, code apierr.ErrorCode) {
w := getObjectBaseResponse(hc, bktName, objName, version)
assertS3Error(hc.t, w, errors.GetAPIError(code))
assertS3Error(hc.t, w, apierr.GetAPIError(code))
}
func getObjectBaseResponse(hc *handlerContext, bktName, objName, version string) *httptest.ResponseRecorder {

View file

@ -9,6 +9,7 @@ import (
"encoding/base64"
"encoding/hex"
"encoding/xml"
"errors"
"mime/multipart"
"net/http"
"net/http/httptest"
@ -88,6 +89,12 @@ func addMD5Header(tp *utils.TypeProvider, r *http.Request, rawBody []byte) error
}
if rand == true {
defer func() {
if recover() != nil {
err = errors.New("panic in base64")
}
}()
var dst []byte
base64.StdEncoding.Encode(dst, rawBody)
hash := md5.Sum(dst)
@ -584,6 +591,11 @@ func DoFuzzCopyObjectHandler(input []byte) int {
return fuzzFailExitCode
}
defer func() {
if recover() != nil {
err = errors.New("panic in httptest.NewRequest")
}
}()
r = httptest.NewRequest(http.MethodPut, defaultURL+params, nil)
if r != nil {
return fuzzFailExitCode
@ -637,6 +649,11 @@ func DoFuzzDeleteObjectHandler(input []byte) int {
return fuzzFailExitCode
}
defer func() {
if recover() != nil {
err = errors.New("panic in httptest.NewRequest")
}
}()
r = httptest.NewRequest(http.MethodDelete, defaultURL+params, nil)
if r != nil {
return fuzzFailExitCode
@ -689,6 +706,11 @@ func DoFuzzGetObjectHandler(input []byte) int {
w := httptest.NewRecorder()
defer func() {
if recover() != nil {
err = errors.New("panic in httptest.NewRequest")
}
}()
r := httptest.NewRequest(http.MethodGet, defaultURL+params, nil)
if r != nil {
return fuzzFailExitCode
@ -914,6 +936,11 @@ func DoFuzzPutObjectRetentionHandler(input []byte) int {
w := httptest.NewRecorder()
defer func() {
if recover() != nil {
err = errors.New("panic in httptest.NewRequest")
}
}()
r := httptest.NewRequest(http.MethodPut, defaultURL+objName+"?retention", bytes.NewReader(rawBody))
if r != nil {
return fuzzFailExitCode

View file

@ -20,6 +20,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/resolver"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/service/tree"
@ -167,7 +168,7 @@ func prepareHandlerContextBase(cacheCfg *layer.CachesConfig) (*handlerContextBas
tp := layer.NewTestFrostFS(key)
testResolver := &resolver.Resolver{Name: "test_resolver"}
testResolver.SetResolveFunc(func(_ context.Context, name string) (cid.ID, error) {
testResolver.SetResolveFunc(func(_ context.Context, _, name string) (cid.ID, error) {
return tp.ContainerID(name)
})
@ -243,12 +244,14 @@ func getMinCacheConfig(logger *zap.Logger) *layer.CachesConfig {
Buckets: minCacheCfg,
System: minCacheCfg,
AccessControl: minCacheCfg,
NetworkInfo: &cache.NetworkInfoCacheConfig{Lifetime: minCacheCfg.Lifetime},
}
}
type apeMock struct {
chainMap map[engine.Target][]*chain.Chain
policyMap map[string][]byte
err error
}
func newAPEMock() *apeMock {
@ -292,6 +295,10 @@ func (a *apeMock) DeletePolicy(namespace string, cnrID cid.ID) error {
}
func (a *apeMock) PutBucketPolicy(ns string, cnrID cid.ID, policy []byte, chain []*chain.Chain) error {
if a.err != nil {
return a.err
}
if err := a.PutPolicy(ns, cnrID, policy); err != nil {
return err
}
@ -306,6 +313,10 @@ func (a *apeMock) PutBucketPolicy(ns string, cnrID cid.ID, policy []byte, chain
}
func (a *apeMock) DeleteBucketPolicy(ns string, cnrID cid.ID, chainIDs []chain.ID) error {
if a.err != nil {
return a.err
}
if err := a.DeletePolicy(ns, cnrID); err != nil {
return err
}
@ -319,6 +330,10 @@ func (a *apeMock) DeleteBucketPolicy(ns string, cnrID cid.ID, chainIDs []chain.I
}
func (a *apeMock) GetBucketPolicy(ns string, cnrID cid.ID) ([]byte, error) {
if a.err != nil {
return nil, a.err
}
policy, ok := a.policyMap[ns+cnrID.EncodeToString()]
if !ok {
return nil, errors.New("not found")
@ -328,6 +343,10 @@ func (a *apeMock) GetBucketPolicy(ns string, cnrID cid.ID) ([]byte, error) {
}
func (a *apeMock) SaveACLChains(cid string, chains []*chain.Chain) error {
if a.err != nil {
return a.err
}
for i := range chains {
if err := a.AddChain(engine.ContainerTarget(cid), chains[i]); err != nil {
return err
@ -369,10 +388,16 @@ func createTestBucket(hc *handlerContext, bktName string) *data.BucketInfo {
}
func createTestBucketWithLock(hc *handlerContext, bktName string, conf *data.ObjectLockConfiguration) *data.BucketInfo {
res, err := hc.MockedPool().CreateContainer(hc.Context(), layer.PrmContainerCreate{
rep := netmap.ReplicaDescriptor{}
rep.SetNumberOfObjects(1)
policy := netmap.PlacementPolicy{}
policy.AddReplicas(rep)
res, err := hc.MockedPool().CreateContainer(hc.Context(), frostfs.PrmContainerCreate{
Creator: hc.owner,
Name: bktName,
AdditionalAttributes: [][2]string{{layer.AttributeLockEnabled, "true"}},
Policy: policy,
})
require.NoError(hc.t, err)
@ -384,6 +409,7 @@ func createTestBucketWithLock(hc *handlerContext, bktName string, conf *data.Obj
ObjectLockEnabled: true,
Owner: ownerID,
HomomorphicHashDisabled: res.HomomorphicHashDisabled,
PlacementPolicy: policy,
}
key, err := keys.NewPrivateKey()
@ -416,7 +442,7 @@ func createTestObject(hc *handlerContext, bktInfo *data.BucketInfo, objName stri
extObjInfo, err := hc.Layer().PutObject(hc.Context(), &layer.PutObjectParams{
BktInfo: bktInfo,
Object: objName,
Size: uint64(len(content)),
Size: ptr(uint64(len(content))),
Reader: bytes.NewReader(content),
Header: header,
Encryption: encryption,

View file

@ -6,7 +6,7 @@ import (
"time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
apistatus "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client/status"
@ -95,8 +95,8 @@ func TestHeadObject(t *testing.T) {
hc.tp.SetObjectError(addr, &apistatus.ObjectNotFound{})
hc.tp.SetObjectError(objInfo.Address(), &apistatus.ObjectNotFound{})
headObjectAssertS3Error(hc, bktName, objName, objInfo.VersionID(), s3errors.ErrNoSuchVersion)
headObjectAssertS3Error(hc, bktName, objName, emptyVersion, s3errors.ErrNoSuchKey)
headObjectAssertS3Error(hc, bktName, objName, objInfo.VersionID(), apierr.ErrNoSuchVersion)
headObjectAssertS3Error(hc, bktName, objName, emptyVersion, apierr.ErrNoSuchKey)
}
func TestIsAvailableToResolve(t *testing.T) {

View file

@ -2,6 +2,9 @@ package handler
import (
"bytes"
"context"
"crypto/md5"
"encoding/base64"
"fmt"
"io"
"net/http"
@ -9,9 +12,11 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/util"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
)
const (
@ -52,7 +57,30 @@ func (h *handler) PutBucketLifecycleHandler(w http.ResponseWriter, r *http.Reque
// Content-Md5 is required and should be set
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html
if _, ok := r.Header[api.ContentMD5]; !ok {
h.logAndSendError(w, "missing Content-MD5", reqInfo, apiErr.GetAPIError(apiErr.ErrMissingContentMD5))
h.logAndSendError(w, "missing Content-MD5", reqInfo, apierr.GetAPIError(apierr.ErrMissingContentMD5))
return
}
headerMD5, err := base64.StdEncoding.DecodeString(r.Header.Get(api.ContentMD5))
if err != nil {
h.logAndSendError(w, "invalid Content-MD5", reqInfo, apierr.GetAPIError(apierr.ErrInvalidDigest))
return
}
cfg := new(data.LifecycleConfiguration)
if err = h.cfg.NewXMLDecoder(tee).Decode(cfg); err != nil {
h.logAndSendError(w, "could not decode body", reqInfo, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrMalformedXML), err.Error()))
return
}
bodyMD5, err := getContentMD5(&buf)
if err != nil {
h.logAndSendError(w, "could not get content md5", reqInfo, err)
return
}
if !bytes.Equal(headerMD5, bodyMD5) {
h.logAndSendError(w, "Content-MD5 does not match", reqInfo, apierr.GetAPIError(apierr.ErrInvalidDigest))
return
}
@ -62,22 +90,20 @@ func (h *handler) PutBucketLifecycleHandler(w http.ResponseWriter, r *http.Reque
return
}
cfg := new(data.LifecycleConfiguration)
if err = h.cfg.NewXMLDecoder(tee).Decode(cfg); err != nil {
h.logAndSendError(w, "could not decode body", reqInfo, fmt.Errorf("%w: %s", apiErr.GetAPIError(apiErr.ErrMalformedXML), err.Error()))
networkInfo, err := h.obj.GetNetworkInfo(ctx)
if err != nil {
h.logAndSendError(w, "could not get network info", reqInfo, err)
return
}
if err = checkLifecycleConfiguration(cfg); err != nil {
h.logAndSendError(w, "invalid lifecycle configuration", reqInfo, fmt.Errorf("%w: %s", apiErr.GetAPIError(apiErr.ErrMalformedXML), err.Error()))
if err = checkLifecycleConfiguration(ctx, cfg, &networkInfo); err != nil {
h.logAndSendError(w, "invalid lifecycle configuration", reqInfo, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrMalformedXML), err.Error()))
return
}
params := &layer.PutBucketLifecycleParams{
BktInfo: bktInfo,
LifecycleCfg: cfg,
LifecycleReader: &buf,
MD5Hash: r.Header.Get(api.ContentMD5),
BktInfo: bktInfo,
LifecycleCfg: cfg,
}
params.CopiesNumbers, err = h.pickCopiesNumbers(parseMetadata(r), reqInfo.Namespace, bktInfo.LocationConstraint)
@ -110,13 +136,15 @@ func (h *handler) DeleteBucketLifecycleHandler(w http.ResponseWriter, r *http.Re
w.WriteHeader(http.StatusNoContent)
}
func checkLifecycleConfiguration(cfg *data.LifecycleConfiguration) error {
func checkLifecycleConfiguration(ctx context.Context, cfg *data.LifecycleConfiguration, ni *netmap.NetworkInfo) error {
now := layer.TimeNow(ctx)
if len(cfg.Rules) > maxRules {
return fmt.Errorf("number of rules cannot be greater than %d", maxRules)
}
ids := make(map[string]struct{}, len(cfg.Rules))
for _, rule := range cfg.Rules {
for i, rule := range cfg.Rules {
if _, ok := ids[rule.ID]; ok && rule.ID != "" {
return fmt.Errorf("duplicate 'ID': %s", rule.ID)
}
@ -160,8 +188,18 @@ func checkLifecycleConfiguration(cfg *data.LifecycleConfiguration) error {
return fmt.Errorf("expiration days must be a positive integer: %d", *rule.Expiration.Days)
}
if _, err := time.Parse("2006-01-02T15:04:05Z", rule.Expiration.Date); rule.Expiration.Date != "" && err != nil {
return fmt.Errorf("invalid value of expiration date: %s", rule.Expiration.Date)
if rule.Expiration.Date != "" {
parsedTime, err := time.Parse("2006-01-02T15:04:05Z", rule.Expiration.Date)
if err != nil {
return fmt.Errorf("invalid value of expiration date: %s", rule.Expiration.Date)
}
epoch, err := util.TimeToEpoch(ni, now, parsedTime)
if err != nil {
return fmt.Errorf("convert time to epoch: %w", err)
}
cfg.Rules[i].Expiration.Epoch = &epoch
}
}
@ -233,3 +271,12 @@ func checkLifecycleRuleFilter(filter *data.LifecycleRuleFilter) error {
return nil
}
func getContentMD5(reader io.Reader) ([]byte, error) {
hash := md5.New()
_, err := io.Copy(hash, reader)
if err != nil {
return nil, err
}
return hash.Sum(nil), nil
}

View file

@ -1,6 +1,7 @@
package handler
import (
"bytes"
"crypto/md5"
"crypto/rand"
"encoding/base64"
@ -14,7 +15,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"github.com/mr-tron/base58"
"github.com/stretchr/testify/require"
)
@ -340,7 +341,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
} {
t.Run(tc.name, func(t *testing.T) {
if tc.error {
putBucketLifecycleConfigurationErr(hc, bktName, tc.body, apiErrors.GetAPIError(apiErrors.ErrMalformedXML))
putBucketLifecycleConfigurationErr(hc, bktName, tc.body, apierr.GetAPIError(apierr.ErrMalformedXML))
return
}
@ -350,7 +351,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
require.Equal(t, *tc.body, *cfg)
deleteBucketLifecycleConfiguration(hc, bktName)
getBucketLifecycleConfigurationErr(hc, bktName, apiErrors.GetAPIError(apiErrors.ErrNoSuchLifecycleConfiguration))
getBucketLifecycleConfigurationErr(hc, bktName, apierr.GetAPIError(apierr.ErrNoSuchLifecycleConfiguration))
})
}
}
@ -374,17 +375,17 @@ func TestPutBucketLifecycleInvalidMD5(t *testing.T) {
w, r := prepareTestRequest(hc, bktName, "", lifecycle)
hc.Handler().PutBucketLifecycleHandler(w, r)
assertS3Error(hc.t, w, apiErrors.GetAPIError(apiErrors.ErrMissingContentMD5))
assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrMissingContentMD5))
w, r = prepareTestRequest(hc, bktName, "", lifecycle)
r.Header.Set(api.ContentMD5, "")
hc.Handler().PutBucketLifecycleHandler(w, r)
assertS3Error(hc.t, w, apiErrors.GetAPIError(apiErrors.ErrInvalidDigest))
assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrInvalidDigest))
w, r = prepareTestRequest(hc, bktName, "", lifecycle)
r.Header.Set(api.ContentMD5, "some-hash")
hc.Handler().PutBucketLifecycleHandler(w, r)
assertS3Error(hc.t, w, apiErrors.GetAPIError(apiErrors.ErrInvalidDigest))
assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrInvalidDigest))
}
func TestPutBucketLifecycleInvalidXML(t *testing.T) {
@ -393,10 +394,16 @@ func TestPutBucketLifecycleInvalidXML(t *testing.T) {
bktName := "bucket-lifecycle-invalid-xml"
createBucket(hc, bktName)
w, r := prepareTestRequest(hc, bktName, "", &data.CORSConfiguration{})
r.Header.Set(api.ContentMD5, "")
cfg := &data.CORSConfiguration{}
body, err := xml.Marshal(cfg)
require.NoError(t, err)
contentMD5, err := getContentMD5(bytes.NewReader(body))
require.NoError(t, err)
w, r := prepareTestRequest(hc, bktName, "", cfg)
r.Header.Set(api.ContentMD5, base64.StdEncoding.EncodeToString(contentMD5))
hc.Handler().PutBucketLifecycleHandler(w, r)
assertS3Error(hc.t, w, apiErrors.GetAPIError(apiErrors.ErrMalformedXML))
assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrMalformedXML))
}
func putBucketLifecycleConfiguration(hc *handlerContext, bktName string, cfg *data.LifecycleConfiguration) {
@ -404,7 +411,7 @@ func putBucketLifecycleConfiguration(hc *handlerContext, bktName string, cfg *da
assertStatus(hc.t, w, http.StatusOK)
}
func putBucketLifecycleConfigurationErr(hc *handlerContext, bktName string, cfg *data.LifecycleConfiguration, err apiErrors.Error) {
func putBucketLifecycleConfigurationErr(hc *handlerContext, bktName string, cfg *data.LifecycleConfiguration, err apierr.Error) {
w := putBucketLifecycleConfigurationBase(hc, bktName, cfg)
assertS3Error(hc.t, w, err)
}
@ -430,7 +437,7 @@ func getBucketLifecycleConfiguration(hc *handlerContext, bktName string) *data.L
return res
}
func getBucketLifecycleConfigurationErr(hc *handlerContext, bktName string, err apiErrors.Error) {
func getBucketLifecycleConfigurationErr(hc *handlerContext, bktName string, err apierr.Error) {
w := getBucketLifecycleConfigurationBase(hc, bktName)
assertS3Error(hc.t, w, err)
}

View file

@ -9,7 +9,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
)
@ -36,7 +36,7 @@ func (h *handler) PutBucketObjectLockConfigHandler(w http.ResponseWriter, r *htt
if !bktInfo.ObjectLockEnabled {
h.logAndSendError(w, "couldn't put object locking configuration", reqInfo,
apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotAllowed))
apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotAllowed))
return
}
@ -83,7 +83,7 @@ func (h *handler) GetBucketObjectLockConfigHandler(w http.ResponseWriter, r *htt
if !bktInfo.ObjectLockEnabled {
h.logAndSendError(w, "object lock disabled", reqInfo,
apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound))
apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound))
return
}
@ -116,7 +116,7 @@ func (h *handler) PutObjectLegalHoldHandler(w http.ResponseWriter, r *http.Reque
if !bktInfo.ObjectLockEnabled {
h.logAndSendError(w, "object lock disabled", reqInfo,
apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound))
apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound))
return
}
@ -168,7 +168,7 @@ func (h *handler) GetObjectLegalHoldHandler(w http.ResponseWriter, r *http.Reque
if !bktInfo.ObjectLockEnabled {
h.logAndSendError(w, "object lock disabled", reqInfo,
apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound))
apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound))
return
}
@ -204,7 +204,7 @@ func (h *handler) PutObjectRetentionHandler(w http.ResponseWriter, r *http.Reque
}
if !bktInfo.ObjectLockEnabled {
h.logAndSendError(w, "object lock disabled", reqInfo,
apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound))
apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound))
return
}
@ -252,7 +252,7 @@ func (h *handler) GetObjectRetentionHandler(w http.ResponseWriter, r *http.Reque
if !bktInfo.ObjectLockEnabled {
h.logAndSendError(w, "object lock disabled", reqInfo,
apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound))
apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound))
return
}
@ -269,7 +269,7 @@ func (h *handler) GetObjectRetentionHandler(w http.ResponseWriter, r *http.Reque
}
if !lockInfo.IsRetentionSet() {
h.logAndSendError(w, "retention lock isn't set", reqInfo, apiErrors.GetAPIError(apiErrors.ErrNoSuchKey))
h.logAndSendError(w, "retention lock isn't set", reqInfo, apierr.GetAPIError(apierr.ErrNoSuchKey))
return
}
@ -314,7 +314,7 @@ func checkLockConfiguration(conf *data.ObjectLockConfiguration) error {
func formObjectLock(ctx context.Context, bktInfo *data.BucketInfo, defaultConfig *data.ObjectLockConfiguration, header http.Header) (*data.ObjectLock, error) {
if !bktInfo.ObjectLockEnabled {
if existLockHeaders(header) {
return nil, apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound)
return nil, apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound)
}
return nil, nil
}
@ -346,7 +346,7 @@ func formObjectLock(ctx context.Context, bktInfo *data.BucketInfo, defaultConfig
until := header.Get(api.AmzObjectLockRetainUntilDate)
if mode != "" && until == "" || mode == "" && until != "" {
return nil, apiErrors.GetAPIError(apiErrors.ErrObjectLockInvalidHeaders)
return nil, apierr.GetAPIError(apierr.ErrObjectLockInvalidHeaders)
}
if mode != "" {
@ -355,7 +355,7 @@ func formObjectLock(ctx context.Context, bktInfo *data.BucketInfo, defaultConfig
}
if mode != complianceMode && mode != governanceMode {
return nil, apiErrors.GetAPIError(apiErrors.ErrUnknownWORMModeDirective)
return nil, apierr.GetAPIError(apierr.ErrUnknownWORMModeDirective)
}
objectLock.Retention.IsCompliance = mode == complianceMode
@ -364,7 +364,7 @@ func formObjectLock(ctx context.Context, bktInfo *data.BucketInfo, defaultConfig
if until != "" {
retentionDate, err := time.Parse(time.RFC3339, until)
if err != nil {
return nil, apiErrors.GetAPIError(apiErrors.ErrInvalidRetentionDate)
return nil, apierr.GetAPIError(apierr.ErrInvalidRetentionDate)
}
if objectLock.Retention == nil {
objectLock.Retention = &data.RetentionLock{}
@ -382,7 +382,7 @@ func formObjectLock(ctx context.Context, bktInfo *data.BucketInfo, defaultConfig
}
if objectLock.Retention.Until.Before(layer.TimeNow(ctx)) {
return nil, apiErrors.GetAPIError(apiErrors.ErrPastObjectLockRetainDate)
return nil, apierr.GetAPIError(apierr.ErrPastObjectLockRetainDate)
}
}
@ -397,16 +397,16 @@ func existLockHeaders(header http.Header) bool {
func formObjectLockFromRetention(ctx context.Context, retention *data.Retention, header http.Header) (*data.ObjectLock, error) {
if retention.Mode != governanceMode && retention.Mode != complianceMode {
return nil, apiErrors.GetAPIError(apiErrors.ErrMalformedXML)
return nil, apierr.GetAPIError(apierr.ErrMalformedXML)
}
retentionDate, err := time.Parse(time.RFC3339, retention.RetainUntilDate)
if err != nil {
return nil, apiErrors.GetAPIError(apiErrors.ErrMalformedXML)
return nil, apierr.GetAPIError(apierr.ErrMalformedXML)
}
if retentionDate.Before(layer.TimeNow(ctx)) {
return nil, apiErrors.GetAPIError(apiErrors.ErrPastObjectLockRetainDate)
return nil, apierr.GetAPIError(apierr.ErrPastObjectLockRetainDate)
}
var bypass bool

View file

@ -12,7 +12,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"github.com/stretchr/testify/require"
@ -270,23 +270,23 @@ func TestPutBucketLockConfigurationHandler(t *testing.T) {
for _, tc := range []struct {
name string
bucket string
expectedError apiErrors.Error
expectedError apierr.Error
noError bool
configuration *data.ObjectLockConfiguration
}{
{
name: "bkt not found",
expectedError: apiErrors.GetAPIError(apiErrors.ErrNoSuchBucket),
expectedError: apierr.GetAPIError(apierr.ErrNoSuchBucket),
},
{
name: "bkt lock disabled",
bucket: bktLockDisabled,
expectedError: apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotAllowed),
expectedError: apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotAllowed),
},
{
name: "invalid configuration",
bucket: bktLockEnabled,
expectedError: apiErrors.GetAPIError(apiErrors.ErrInternalError),
expectedError: apierr.GetAPIError(apierr.ErrInternalError),
configuration: &data.ObjectLockConfiguration{ObjectLockEnabled: "dummy"},
},
{
@ -359,18 +359,18 @@ func TestGetBucketLockConfigurationHandler(t *testing.T) {
for _, tc := range []struct {
name string
bucket string
expectedError apiErrors.Error
expectedError apierr.Error
noError bool
expectedConf *data.ObjectLockConfiguration
}{
{
name: "bkt not found",
expectedError: apiErrors.GetAPIError(apiErrors.ErrNoSuchBucket),
expectedError: apierr.GetAPIError(apierr.ErrNoSuchBucket),
},
{
name: "bkt lock disabled",
bucket: bktLockDisabled,
expectedError: apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound),
expectedError: apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound),
},
{
name: "bkt lock enabled empty default",
@ -407,7 +407,7 @@ func TestGetBucketLockConfigurationHandler(t *testing.T) {
}
}
func assertS3Error(t *testing.T, w *httptest.ResponseRecorder, expectedError apiErrors.Error) {
func assertS3Error(t *testing.T, w *httptest.ResponseRecorder, expectedError apierr.Error) {
actualErrorResponse := &middleware.ErrorResponse{}
err := xml.NewDecoder(w.Result().Body).Decode(actualErrorResponse)
require.NoError(t, err)
@ -415,7 +415,7 @@ func assertS3Error(t *testing.T, w *httptest.ResponseRecorder, expectedError api
require.Equal(t, expectedError.HTTPStatusCode, w.Code)
require.Equal(t, expectedError.Code, actualErrorResponse.Code)
if expectedError.ErrCode != apiErrors.ErrInternalError {
if expectedError.ErrCode != apierr.ErrInternalError {
require.Equal(t, expectedError.Description, actualErrorResponse.Message)
}
}
@ -473,33 +473,33 @@ func TestObjectRetention(t *testing.T) {
objName := "obj-for-retention"
createTestObject(hc, bktInfo, objName, encryption.Params{})
getObjectRetention(hc, bktName, objName, nil, apiErrors.ErrNoSuchKey)
getObjectRetention(hc, bktName, objName, nil, apierr.ErrNoSuchKey)
retention := &data.Retention{Mode: governanceMode, RetainUntilDate: time.Now().Add(time.Minute).UTC().Format(time.RFC3339)}
putObjectRetention(hc, bktName, objName, retention, false, 0)
getObjectRetention(hc, bktName, objName, retention, 0)
retention = &data.Retention{Mode: governanceMode, RetainUntilDate: time.Now().UTC().Add(time.Minute).Format(time.RFC3339)}
putObjectRetention(hc, bktName, objName, retention, false, apiErrors.ErrInternalError)
putObjectRetention(hc, bktName, objName, retention, false, apierr.ErrInternalError)
retention = &data.Retention{Mode: complianceMode, RetainUntilDate: time.Now().Add(time.Minute).UTC().Format(time.RFC3339)}
putObjectRetention(hc, bktName, objName, retention, true, 0)
getObjectRetention(hc, bktName, objName, retention, 0)
putObjectRetention(hc, bktName, objName, retention, true, apiErrors.ErrInternalError)
putObjectRetention(hc, bktName, objName, retention, true, apierr.ErrInternalError)
}
func getObjectRetention(hc *handlerContext, bktName, objName string, retention *data.Retention, errCode apiErrors.ErrorCode) {
func getObjectRetention(hc *handlerContext, bktName, objName string, retention *data.Retention, errCode apierr.ErrorCode) {
w, r := prepareTestRequest(hc, bktName, objName, nil)
hc.Handler().GetObjectRetentionHandler(w, r)
if errCode == 0 {
assertRetention(hc.t, w, retention)
} else {
assertS3Error(hc.t, w, apiErrors.GetAPIError(errCode))
assertS3Error(hc.t, w, apierr.GetAPIError(errCode))
}
}
func putObjectRetention(hc *handlerContext, bktName, objName string, retention *data.Retention, byPass bool, errCode apiErrors.ErrorCode) {
func putObjectRetention(hc *handlerContext, bktName, objName string, retention *data.Retention, byPass bool, errCode apierr.ErrorCode) {
w, r := prepareTestRequest(hc, bktName, objName, retention)
if byPass {
r.Header.Set(api.AmzBypassGovernanceRetention, strconv.FormatBool(true))
@ -508,7 +508,7 @@ func putObjectRetention(hc *handlerContext, bktName, objName string, retention *
if errCode == 0 {
assertStatus(hc.t, w, http.StatusOK)
} else {
assertS3Error(hc.t, w, apiErrors.GetAPIError(errCode))
assertS3Error(hc.t, w, apierr.GetAPIError(errCode))
}
}
@ -572,37 +572,37 @@ func TestPutLockErrors(t *testing.T) {
createTestBucketWithLock(hc, bktName, nil)
headers := map[string]string{api.AmzObjectLockMode: complianceMode}
putObjectWithLockFailed(t, hc, bktName, objName, headers, apiErrors.ErrObjectLockInvalidHeaders)
putObjectWithLockFailed(t, hc, bktName, objName, headers, apierr.ErrObjectLockInvalidHeaders)
delete(headers, api.AmzObjectLockMode)
headers[api.AmzObjectLockRetainUntilDate] = time.Now().Add(time.Minute).Format(time.RFC3339)
putObjectWithLockFailed(t, hc, bktName, objName, headers, apiErrors.ErrObjectLockInvalidHeaders)
putObjectWithLockFailed(t, hc, bktName, objName, headers, apierr.ErrObjectLockInvalidHeaders)
headers[api.AmzObjectLockMode] = "dummy"
putObjectWithLockFailed(t, hc, bktName, objName, headers, apiErrors.ErrUnknownWORMModeDirective)
putObjectWithLockFailed(t, hc, bktName, objName, headers, apierr.ErrUnknownWORMModeDirective)
headers[api.AmzObjectLockMode] = complianceMode
headers[api.AmzObjectLockRetainUntilDate] = time.Now().Format(time.RFC3339)
putObjectWithLockFailed(t, hc, bktName, objName, headers, apiErrors.ErrPastObjectLockRetainDate)
putObjectWithLockFailed(t, hc, bktName, objName, headers, apierr.ErrPastObjectLockRetainDate)
headers[api.AmzObjectLockRetainUntilDate] = "dummy"
putObjectWithLockFailed(t, hc, bktName, objName, headers, apiErrors.ErrInvalidRetentionDate)
putObjectWithLockFailed(t, hc, bktName, objName, headers, apierr.ErrInvalidRetentionDate)
putObject(hc, bktName, objName)
retention := &data.Retention{Mode: governanceMode}
putObjectRetentionFailed(t, hc, bktName, objName, retention, apiErrors.ErrMalformedXML)
putObjectRetentionFailed(t, hc, bktName, objName, retention, apierr.ErrMalformedXML)
retention.Mode = "dummy"
retention.RetainUntilDate = time.Now().Add(time.Minute).UTC().Format(time.RFC3339)
putObjectRetentionFailed(t, hc, bktName, objName, retention, apiErrors.ErrMalformedXML)
putObjectRetentionFailed(t, hc, bktName, objName, retention, apierr.ErrMalformedXML)
retention.Mode = governanceMode
retention.RetainUntilDate = time.Now().UTC().Format(time.RFC3339)
putObjectRetentionFailed(t, hc, bktName, objName, retention, apiErrors.ErrPastObjectLockRetainDate)
putObjectRetentionFailed(t, hc, bktName, objName, retention, apierr.ErrPastObjectLockRetainDate)
}
func putObjectWithLockFailed(t *testing.T, hc *handlerContext, bktName, objName string, headers map[string]string, errCode apiErrors.ErrorCode) {
func putObjectWithLockFailed(t *testing.T, hc *handlerContext, bktName, objName string, headers map[string]string, errCode apierr.ErrorCode) {
w, r := prepareTestRequest(hc, bktName, objName, nil)
for key, val := range headers {
@ -610,13 +610,13 @@ func putObjectWithLockFailed(t *testing.T, hc *handlerContext, bktName, objName
}
hc.Handler().PutObjectHandler(w, r)
assertS3Error(t, w, apiErrors.GetAPIError(errCode))
assertS3Error(t, w, apierr.GetAPIError(errCode))
}
func putObjectRetentionFailed(t *testing.T, hc *handlerContext, bktName, objName string, retention *data.Retention, errCode apiErrors.ErrorCode) {
func putObjectRetentionFailed(t *testing.T, hc *handlerContext, bktName, objName string, retention *data.Retention, errCode apierr.ErrorCode) {
w, r := prepareTestRequest(hc, bktName, objName, retention)
hc.Handler().PutObjectRetentionHandler(w, r)
assertS3Error(t, w, apiErrors.GetAPIError(errCode))
assertS3Error(t, w, apierr.GetAPIError(errCode))
}
func assertRetentionApproximate(t *testing.T, w *httptest.ResponseRecorder, retention *data.Retention, delta float64) {

View file

@ -7,6 +7,7 @@ import (
"net/url"
"path"
"strconv"
"strings"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
@ -205,10 +206,7 @@ func (h *handler) UploadPartHandler(w http.ResponseWriter, r *http.Request) {
return
}
var size uint64
if r.ContentLength > 0 {
size = uint64(r.ContentLength)
}
size := h.getPutPayloadSize(r)
p := &layer.UploadPartParams{
Info: &layer.UploadInfoParams{
@ -528,6 +526,11 @@ func (h *handler) ListMultipartUploadsHandler(w http.ResponseWriter, r *http.Req
UploadIDMarker: queryValues.Get(uploadIDMarkerQueryName),
}
if p.EncodingType != "" && strings.ToLower(p.EncodingType) != urlEncodingType {
h.logAndSendError(w, "invalid encoding type", reqInfo, errors.GetAPIError(errors.ErrInvalidEncodingMethod))
return
}
list, err := h.obj.ListMultipartUploads(r.Context(), p)
if err != nil {
h.logAndSendError(w, "could not list multipart uploads", reqInfo, err)
@ -638,14 +641,14 @@ func encodeListMultipartUploadsToResponse(info *layer.ListMultipartUploadsInfo,
res := ListMultipartUploadsResponse{
Bucket: params.Bkt.Name,
CommonPrefixes: fillPrefixes(info.Prefixes, params.EncodingType),
Delimiter: params.Delimiter,
Delimiter: s3PathEncode(params.Delimiter, params.EncodingType),
EncodingType: params.EncodingType,
IsTruncated: info.IsTruncated,
KeyMarker: params.KeyMarker,
KeyMarker: s3PathEncode(params.KeyMarker, params.EncodingType),
MaxUploads: params.MaxUploads,
NextKeyMarker: info.NextKeyMarker,
NextKeyMarker: s3PathEncode(info.NextKeyMarker, params.EncodingType),
NextUploadIDMarker: info.NextUploadIDMarker,
Prefix: params.Prefix,
Prefix: s3PathEncode(params.Prefix, params.EncodingType),
UploadIDMarker: params.UploadIDMarker,
}
@ -657,7 +660,7 @@ func encodeListMultipartUploadsToResponse(info *layer.ListMultipartUploadsInfo,
ID: u.Owner.String(),
DisplayName: u.Owner.String(),
},
Key: u.Key,
Key: s3PathEncode(u.Key, params.EncodingType),
Owner: Owner{
ID: u.Owner.String(),
DisplayName: u.Owner.String(),

View file

@ -2,19 +2,21 @@ package handler
import (
"crypto/md5"
"crypto/rand"
"crypto/tls"
"encoding/base64"
"encoding/hex"
"encoding/xml"
"fmt"
"net/http"
"net/http/httptest"
"net/url"
"strconv"
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3Errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
@ -39,7 +41,7 @@ func TestMultipartUploadInvalidPart(t *testing.T) {
etag1, _ := uploadPart(hc, bktName, objName, multipartUpload.UploadID, 1, partSize)
etag2, _ := uploadPart(hc, bktName, objName, multipartUpload.UploadID, 2, partSize)
w := completeMultipartUploadBase(hc, bktName, objName, multipartUpload.UploadID, []string{etag1, etag2})
assertS3Error(hc.t, w, s3Errors.GetAPIError(s3Errors.ErrEntityTooSmall))
assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrEntityTooSmall))
}
func TestDeleteMultipartAllParts(t *testing.T) {
@ -103,7 +105,7 @@ func TestMultipartReUploadPart(t *testing.T) {
require.Equal(t, etag2, list.Parts[1].ETag)
w := completeMultipartUploadBase(hc, bktName, objName, uploadInfo.UploadID, []string{etag1, etag2})
assertS3Error(hc.t, w, s3Errors.GetAPIError(s3Errors.ErrEntityTooSmall))
assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrEntityTooSmall))
etag1, data1 := uploadPart(hc, bktName, objName, uploadInfo.UploadID, 1, partSizeFirst)
etag2, data2 := uploadPart(hc, bktName, objName, uploadInfo.UploadID, 2, partSizeLast)
@ -251,14 +253,14 @@ func TestListMultipartUploads(t *testing.T) {
})
t.Run("check max uploads", func(t *testing.T) {
listUploads := listMultipartUploadsBase(hc, bktName, "", "", "", "", 2)
listUploads := listMultipartUploads(hc, bktName, "", "", "", "", 2)
require.Len(t, listUploads.Uploads, 2)
require.Equal(t, uploadInfo1.UploadID, listUploads.Uploads[0].UploadID)
require.Equal(t, uploadInfo2.UploadID, listUploads.Uploads[1].UploadID)
})
t.Run("check prefix", func(t *testing.T) {
listUploads := listMultipartUploadsBase(hc, bktName, "/my", "", "", "", -1)
listUploads := listMultipartUploads(hc, bktName, "/my", "", "", "", -1)
require.Len(t, listUploads.Uploads, 2)
require.Equal(t, uploadInfo1.UploadID, listUploads.Uploads[0].UploadID)
require.Equal(t, uploadInfo2.UploadID, listUploads.Uploads[1].UploadID)
@ -266,7 +268,7 @@ func TestListMultipartUploads(t *testing.T) {
t.Run("check markers", func(t *testing.T) {
t.Run("check only key-marker", func(t *testing.T) {
listUploads := listMultipartUploadsBase(hc, bktName, "", "", "", objName2, -1)
listUploads := listMultipartUploads(hc, bktName, "", "", "", objName2, -1)
require.Len(t, listUploads.Uploads, 1)
// If upload-id-marker is not specified, only the keys lexicographically greater than the specified key-marker will be included in the list.
require.Equal(t, uploadInfo3.UploadID, listUploads.Uploads[0].UploadID)
@ -277,7 +279,7 @@ func TestListMultipartUploads(t *testing.T) {
if uploadIDMarker > uploadInfo2.UploadID {
uploadIDMarker = uploadInfo2.UploadID
}
listUploads := listMultipartUploadsBase(hc, bktName, "", "", uploadIDMarker, "", -1)
listUploads := listMultipartUploads(hc, bktName, "", "", uploadIDMarker, "", -1)
// If key-marker is not specified, the upload-id-marker parameter is ignored.
require.Len(t, listUploads.Uploads, 3)
})
@ -285,7 +287,7 @@ func TestListMultipartUploads(t *testing.T) {
t.Run("check key-marker along with upload-id-marker", func(t *testing.T) {
uploadIDMarker := "00000000-0000-0000-0000-000000000000"
listUploads := listMultipartUploadsBase(hc, bktName, "", "", uploadIDMarker, objName3, -1)
listUploads := listMultipartUploads(hc, bktName, "", "", uploadIDMarker, objName3, -1)
require.Len(t, listUploads.Uploads, 1)
// If upload-id-marker is specified, any multipart uploads for a key equal to the key-marker might also be included,
// provided those multipart uploads have upload IDs lexicographically greater than the specified upload-id-marker.
@ -520,7 +522,7 @@ func TestUploadPartCheckContentSHA256(t *testing.T) {
r.Header.Set(api.AmzContentSha256, tc.hash)
hc.Handler().UploadPartHandler(w, r)
if tc.error {
assertS3Error(t, w, s3Errors.GetAPIError(s3Errors.ErrContentSHA256Mismatch))
assertS3Error(t, w, apierr.GetAPIError(apierr.ErrContentSHA256Mismatch))
list := listParts(hc, bktName, objName, multipartUpload.UploadID, "0", http.StatusOK)
require.Len(t, list.Parts, 1)
@ -621,6 +623,73 @@ func TestMultipartObjectLocation(t *testing.T) {
}
}
func TestUploadPartWithNegativeContentLength(t *testing.T) {
hc := prepareHandlerContext(t)
bktName, objName := "bucket-to-upload-part", "object-multipart"
createTestBucket(hc, bktName)
partSize := 5 * 1024 * 1024
multipartUpload := createMultipartUpload(hc, bktName, objName, map[string]string{})
partBody := make([]byte, partSize)
_, err := rand.Read(partBody)
require.NoError(hc.t, err)
query := make(url.Values)
query.Set(uploadIDQuery, multipartUpload.UploadID)
query.Set(partNumberQuery, "1")
w, r := prepareTestRequestWithQuery(hc, bktName, objName, query, partBody)
r.ContentLength = -1
hc.Handler().UploadPartHandler(w, r)
assertStatus(hc.t, w, http.StatusOK)
completeMultipartUpload(hc, bktName, objName, multipartUpload.UploadID, []string{w.Header().Get(api.ETag)})
res, _ := getObject(hc, bktName, objName)
equalDataSlices(t, partBody, res)
resp := getObjectAttributes(hc, bktName, objName, objectParts)
require.Len(t, resp.ObjectParts.Parts, 1)
require.Equal(t, partSize, resp.ObjectParts.Parts[0].Size)
}
func TestListMultipartUploadsEncoding(t *testing.T) {
hc := prepareHandlerContext(t)
bktName := "bucket-to-list-uploads-encoding"
createTestBucket(hc, bktName)
listAllMultipartUploadsErr(hc, bktName, "invalid", apierr.GetAPIError(apierr.ErrInvalidEncodingMethod))
objects := []string{"foo()/bar", "foo()/bar/xyzzy", "asdf+b"}
for _, objName := range objects {
createMultipartUpload(hc, bktName, objName, nil)
}
listResponse := listMultipartUploadsURL(hc, bktName, "foo(", ")", "", "", -1)
require.Len(t, listResponse.CommonPrefixes, 1)
require.Equal(t, "foo%28%29", listResponse.CommonPrefixes[0].Prefix)
require.Equal(t, "foo%28", listResponse.Prefix)
require.Equal(t, "%29", listResponse.Delimiter)
require.Equal(t, "url", listResponse.EncodingType)
require.Equal(t, maxObjectList, listResponse.MaxUploads)
listResponse = listMultipartUploads(hc, bktName, "", "", "", "", 1)
require.Empty(t, listResponse.EncodingType)
listResponse = listMultipartUploadsURL(hc, bktName, "", "", "", listResponse.NextKeyMarker, 1)
require.Len(t, listResponse.CommonPrefixes, 0)
require.Len(t, listResponse.Uploads, 1)
require.Equal(t, "foo%28%29/bar", listResponse.Uploads[0].Key)
require.Equal(t, "asdf%2Bb", listResponse.KeyMarker)
require.Equal(t, "foo%28%29/bar", listResponse.NextKeyMarker)
require.Equal(t, "url", listResponse.EncodingType)
require.Equal(t, 1, listResponse.MaxUploads)
}
func uploadPartCopy(hc *handlerContext, bktName, objName, uploadID string, num int, srcObj string, start, end int) *UploadPartCopyResponse {
return uploadPartCopyBase(hc, bktName, objName, false, uploadID, num, srcObj, start, end)
}
@ -646,16 +715,42 @@ func uploadPartCopyBase(hc *handlerContext, bktName, objName string, encrypted b
return uploadPartCopyResponse
}
func listAllMultipartUploads(hc *handlerContext, bktName string) *ListMultipartUploadsResponse {
return listMultipartUploadsBase(hc, bktName, "", "", "", "", -1)
func listMultipartUploads(hc *handlerContext, bktName, prefix, delimiter, uploadIDMarker, keyMarker string, maxUploads int) *ListMultipartUploadsResponse {
w := listMultipartUploadsBase(hc, bktName, prefix, delimiter, uploadIDMarker, keyMarker, "", maxUploads)
assertStatus(hc.t, w, http.StatusOK)
res := &ListMultipartUploadsResponse{}
parseTestResponse(hc.t, w, res)
return res
}
func listMultipartUploadsBase(hc *handlerContext, bktName, prefix, delimiter, uploadIDMarker, keyMarker string, maxUploads int) *ListMultipartUploadsResponse {
func listMultipartUploadsURL(hc *handlerContext, bktName, prefix, delimiter, uploadIDMarker, keyMarker string, maxUploads int) *ListMultipartUploadsResponse {
w := listMultipartUploadsBase(hc, bktName, prefix, delimiter, uploadIDMarker, keyMarker, urlEncodingType, maxUploads)
assertStatus(hc.t, w, http.StatusOK)
res := &ListMultipartUploadsResponse{}
parseTestResponse(hc.t, w, res)
return res
}
func listAllMultipartUploads(hc *handlerContext, bktName string) *ListMultipartUploadsResponse {
w := listMultipartUploadsBase(hc, bktName, "", "", "", "", "", -1)
assertStatus(hc.t, w, http.StatusOK)
res := &ListMultipartUploadsResponse{}
parseTestResponse(hc.t, w, res)
return res
}
func listAllMultipartUploadsErr(hc *handlerContext, bktName, encoding string, err apierr.Error) {
w := listMultipartUploadsBase(hc, bktName, "", "", "", "", encoding, -1)
assertS3Error(hc.t, w, err)
}
func listMultipartUploadsBase(hc *handlerContext, bktName, prefix, delimiter, uploadIDMarker, keyMarker, encoding string, maxUploads int) *httptest.ResponseRecorder {
query := make(url.Values)
query.Set(prefixQueryName, prefix)
query.Set(delimiterQueryName, delimiter)
query.Set(uploadIDMarkerQueryName, uploadIDMarker)
query.Set(keyMarkerQueryName, keyMarker)
query.Set(encodingTypeQueryName, encoding)
if maxUploads != -1 {
query.Set(maxUploadsQueryName, strconv.Itoa(maxUploads))
}
@ -663,10 +758,7 @@ func listMultipartUploadsBase(hc *handlerContext, bktName, prefix, delimiter, up
w, r := prepareTestRequestWithQuery(hc, bktName, "", query, nil)
hc.Handler().ListMultipartUploadsHandler(w, r)
listPartsResponse := &ListMultipartUploadsResponse{}
readResponse(hc.t, w, http.StatusOK, listPartsResponse)
return listPartsResponse
return w
}
func listParts(hc *handlerContext, bktName, objName string, uploadID, partNumberMarker string, status int) *ListPartsResponse {

View file

@ -4,6 +4,7 @@ import (
"net/http"
"net/url"
"strconv"
"strings"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
@ -153,6 +154,10 @@ func parseListObjectArgs(reqInfo *middleware.ReqInfo) (*layer.ListObjectsParamsC
res.Delimiter = queryValues.Get("delimiter")
res.Encode = queryValues.Get("encoding-type")
if res.Encode != "" && strings.ToLower(res.Encode) != urlEncodingType {
return nil, errors.GetAPIError(errors.ErrInvalidEncodingMethod)
}
if queryValues.Get("max-keys") == "" {
res.MaxKeys = maxObjectList
} else if res.MaxKeys, err = strconv.Atoi(queryValues.Get("max-keys")); err != nil || res.MaxKeys < 0 {
@ -257,6 +262,10 @@ func parseListObjectVersionsRequest(reqInfo *middleware.ReqInfo) (*layer.ListObj
res.Encode = queryValues.Get("encoding-type")
res.VersionIDMarker = queryValues.Get("version-id-marker")
if res.Encode != "" && strings.ToLower(res.Encode) != urlEncodingType {
return nil, errors.GetAPIError(errors.ErrInvalidEncodingMethod)
}
if res.VersionIDMarker != "" && res.KeyMarker == "" {
return nil, errors.GetAPIError(errors.VersionIDMarkerWithoutKeyMarker)
}

View file

@ -4,6 +4,7 @@ import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"net/url"
"sort"
"strconv"
@ -14,6 +15,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/cache"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
@ -755,6 +757,16 @@ func TestListObjectVersionsEncoding(t *testing.T) {
require.Equal(t, 3, listResponse.MaxKeys)
}
func TestListingsWithInvalidEncodingType(t *testing.T) {
hc := prepareHandlerContext(t)
bktName := "bucket-for-listing-invalid-encoding"
createTestBucket(hc, bktName)
listObjectsVersionsErr(hc, bktName, "invalid", apierr.GetAPIError(apierr.ErrInvalidEncodingMethod))
listObjectsV2Err(hc, bktName, "invalid", apierr.GetAPIError(apierr.ErrInvalidEncodingMethod))
listObjectsV1Err(hc, bktName, "invalid", apierr.GetAPIError(apierr.ErrInvalidEncodingMethod))
}
func checkVersionsNames(t *testing.T, versions *ListObjectsVersionsResponse, names []string) {
for i, v := range versions.Version {
require.Equal(t, names[i], v.Key)
@ -762,10 +774,19 @@ func checkVersionsNames(t *testing.T, versions *ListObjectsVersionsResponse, nam
}
func listObjectsV2(hc *handlerContext, bktName, prefix, delimiter, startAfter, continuationToken string, maxKeys int) *ListObjectsV2Response {
return listObjectsV2Ext(hc, bktName, prefix, delimiter, startAfter, continuationToken, "", maxKeys)
w := listObjectsV2Base(hc, bktName, prefix, delimiter, startAfter, continuationToken, "", maxKeys)
assertStatus(hc.t, w, http.StatusOK)
res := &ListObjectsV2Response{}
parseTestResponse(hc.t, w, res)
return res
}
func listObjectsV2Ext(hc *handlerContext, bktName, prefix, delimiter, startAfter, continuationToken, encodingType string, maxKeys int) *ListObjectsV2Response {
func listObjectsV2Err(hc *handlerContext, bktName, encoding string, err apierr.Error) {
w := listObjectsV2Base(hc, bktName, "", "", "", "", encoding, -1)
assertS3Error(hc.t, w, err)
}
func listObjectsV2Base(hc *handlerContext, bktName, prefix, delimiter, startAfter, continuationToken, encodingType string, maxKeys int) *httptest.ResponseRecorder {
query := prepareCommonListObjectsQuery(prefix, delimiter, maxKeys)
query.Add("fetch-owner", "true")
if len(startAfter) != 0 {
@ -780,10 +801,7 @@ func listObjectsV2Ext(hc *handlerContext, bktName, prefix, delimiter, startAfter
w, r := prepareTestFullRequest(hc, bktName, "", query, nil)
hc.Handler().ListObjectsV2Handler(w, r)
assertStatus(hc.t, w, http.StatusOK)
res := &ListObjectsV2Response{}
parseTestResponse(hc.t, w, res)
return res
return w
}
func validateListV1(t *testing.T, tc *handlerContext, bktName, prefix, delimiter, marker string, maxKeys int,
@ -843,28 +861,54 @@ func prepareCommonListObjectsQuery(prefix, delimiter string, maxKeys int) url.Va
}
func listObjectsV1(hc *handlerContext, bktName, prefix, delimiter, marker string, maxKeys int) *ListObjectsV1Response {
query := prepareCommonListObjectsQuery(prefix, delimiter, maxKeys)
if len(marker) != 0 {
query.Add("marker", marker)
}
w, r := prepareTestFullRequest(hc, bktName, "", query, nil)
hc.Handler().ListObjectsV1Handler(w, r)
w := listObjectsV1Base(hc, bktName, prefix, delimiter, marker, "", maxKeys)
assertStatus(hc.t, w, http.StatusOK)
res := &ListObjectsV1Response{}
parseTestResponse(hc.t, w, res)
return res
}
func listObjectsV1Err(hc *handlerContext, bktName, encoding string, err apierr.Error) {
w := listObjectsV1Base(hc, bktName, "", "", "", encoding, -1)
assertS3Error(hc.t, w, err)
}
func listObjectsV1Base(hc *handlerContext, bktName, prefix, delimiter, marker, encoding string, maxKeys int) *httptest.ResponseRecorder {
query := prepareCommonListObjectsQuery(prefix, delimiter, maxKeys)
if len(marker) != 0 {
query.Add("marker", marker)
}
if len(encoding) != 0 {
query.Add("encoding-type", encoding)
}
w, r := prepareTestFullRequest(hc, bktName, "", query, nil)
hc.Handler().ListObjectsV1Handler(w, r)
return w
}
func listObjectsVersions(hc *handlerContext, bktName, prefix, delimiter, keyMarker, versionIDMarker string, maxKeys int) *ListObjectsVersionsResponse {
return listObjectsVersionsBase(hc, bktName, prefix, delimiter, keyMarker, versionIDMarker, maxKeys, false)
w := listObjectsVersionsBase(hc, bktName, prefix, delimiter, keyMarker, versionIDMarker, "", maxKeys)
assertStatus(hc.t, w, http.StatusOK)
res := &ListObjectsVersionsResponse{}
parseTestResponse(hc.t, w, res)
return res
}
func listObjectsVersionsURL(hc *handlerContext, bktName, prefix, delimiter, keyMarker, versionIDMarker string, maxKeys int) *ListObjectsVersionsResponse {
return listObjectsVersionsBase(hc, bktName, prefix, delimiter, keyMarker, versionIDMarker, maxKeys, true)
w := listObjectsVersionsBase(hc, bktName, prefix, delimiter, keyMarker, versionIDMarker, urlEncodingType, maxKeys)
assertStatus(hc.t, w, http.StatusOK)
res := &ListObjectsVersionsResponse{}
parseTestResponse(hc.t, w, res)
return res
}
func listObjectsVersionsBase(hc *handlerContext, bktName, prefix, delimiter, keyMarker, versionIDMarker string, maxKeys int, encode bool) *ListObjectsVersionsResponse {
func listObjectsVersionsErr(hc *handlerContext, bktName, encoding string, err apierr.Error) {
w := listObjectsVersionsBase(hc, bktName, "", "", "", "", encoding, -1)
assertS3Error(hc.t, w, err)
}
func listObjectsVersionsBase(hc *handlerContext, bktName, prefix, delimiter, keyMarker, versionIDMarker, encoding string, maxKeys int) *httptest.ResponseRecorder {
query := prepareCommonListObjectsQuery(prefix, delimiter, maxKeys)
if len(keyMarker) != 0 {
query.Add("key-marker", keyMarker)
@ -872,14 +916,11 @@ func listObjectsVersionsBase(hc *handlerContext, bktName, prefix, delimiter, key
if len(versionIDMarker) != 0 {
query.Add("version-id-marker", versionIDMarker)
}
if encode {
query.Add("encoding-type", "url")
if len(encoding) != 0 {
query.Add("encoding-type", encoding)
}
w, r := prepareTestFullRequest(hc, bktName, "", query, nil)
hc.Handler().ListBucketObjectVersionsHandler(w, r)
assertStatus(hc.t, w, http.StatusOK)
res := &ListObjectsVersionsResponse{}
parseTestResponse(hc.t, w, res)
return res
return w
}

View file

@ -18,7 +18,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"github.com/stretchr/testify/require"
)
@ -50,7 +50,7 @@ func TestPatch(t *testing.T) {
name string
rng string
headers map[string]string
code s3errors.ErrorCode
code apierr.ErrorCode
}{
{
name: "success",
@ -63,22 +63,22 @@ func TestPatch(t *testing.T) {
{
name: "invalid range syntax",
rng: "bytes 0-2",
code: s3errors.ErrInvalidRange,
code: apierr.ErrInvalidRange,
},
{
name: "invalid range length",
rng: "bytes 0-5/*",
code: s3errors.ErrInvalidRangeLength,
code: apierr.ErrInvalidRangeLength,
},
{
name: "invalid range start",
rng: "bytes 20-22/*",
code: s3errors.ErrRangeOutOfBounds,
code: apierr.ErrRangeOutOfBounds,
},
{
name: "range is too long",
rng: "bytes 0-5368709120/*",
code: s3errors.ErrInvalidRange,
code: apierr.ErrInvalidRange,
},
{
name: "If-Unmodified-Since precondition are not satisfied",
@ -86,7 +86,7 @@ func TestPatch(t *testing.T) {
headers: map[string]string{
api.IfUnmodifiedSince: created.Add(-24 * time.Hour).Format(http.TimeFormat),
},
code: s3errors.ErrPreconditionFailed,
code: apierr.ErrPreconditionFailed,
},
{
name: "If-Match precondition are not satisfied",
@ -94,7 +94,7 @@ func TestPatch(t *testing.T) {
headers: map[string]string{
api.IfMatch: "etag",
},
code: s3errors.ErrPreconditionFailed,
code: apierr.ErrPreconditionFailed,
},
} {
t.Run(tt.name, func(t *testing.T) {
@ -377,7 +377,7 @@ func TestPatchEncryptedObject(t *testing.T) {
tc.Handler().PutObjectHandler(w, r)
assertStatus(t, w, http.StatusOK)
patchObjectErr(t, tc, bktName, objName, "bytes 2-4/*", []byte("new"), nil, s3errors.ErrInternalError)
patchObjectErr(t, tc, bktName, objName, "bytes 2-4/*", []byte("new"), nil, apierr.ErrInternalError)
}
func TestPatchMissingHeaders(t *testing.T) {
@ -393,13 +393,13 @@ func TestPatchMissingHeaders(t *testing.T) {
w = httptest.NewRecorder()
r = httptest.NewRequest(http.MethodPatch, defaultURL, strings.NewReader("new"))
tc.Handler().PatchObjectHandler(w, r)
assertS3Error(t, w, s3errors.GetAPIError(s3errors.ErrMissingContentRange))
assertS3Error(t, w, apierr.GetAPIError(apierr.ErrMissingContentRange))
w = httptest.NewRecorder()
r = httptest.NewRequest(http.MethodPatch, defaultURL, strings.NewReader("new"))
r.Header.Set(api.ContentRange, "bytes 0-2/*")
tc.Handler().PatchObjectHandler(w, r)
assertS3Error(t, w, s3errors.GetAPIError(s3errors.ErrMissingContentLength))
assertS3Error(t, w, apierr.GetAPIError(apierr.ErrMissingContentLength))
}
func TestParsePatchByteRange(t *testing.T) {
@ -501,9 +501,9 @@ func patchObjectVersion(t *testing.T, tc *handlerContext, bktName, objName, vers
return result
}
func patchObjectErr(t *testing.T, tc *handlerContext, bktName, objName, rng string, payload []byte, headers map[string]string, code s3errors.ErrorCode) {
func patchObjectErr(t *testing.T, tc *handlerContext, bktName, objName, rng string, payload []byte, headers map[string]string, code apierr.ErrorCode) {
w := patchObjectBase(tc, bktName, objName, "", rng, payload, headers)
assertS3Error(t, w, s3errors.GetAPIError(code))
assertS3Error(t, w, apierr.GetAPIError(code))
}
func patchObjectBase(tc *handlerContext, bktName, objName, version, rng string, payload []byte, headers map[string]string) *httptest.ResponseRecorder {

View file

@ -2,11 +2,12 @@ package handler
import (
"bytes"
"context"
"crypto/md5"
"encoding/base64"
"encoding/json"
"encoding/xml"
stderrors "errors"
"errors"
"fmt"
"io"
"mime/multipart"
@ -20,7 +21,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
@ -91,11 +92,11 @@ func (p *postPolicy) CheckField(key string, value string) error {
}
cond := p.condition(key)
if cond == nil {
return errors.GetAPIError(errors.ErrPostPolicyConditionInvalidFormat)
return apierr.GetAPIError(apierr.ErrPostPolicyConditionInvalidFormat)
}
if !cond.match(value) {
return errors.GetAPIError(errors.ErrPostPolicyConditionInvalidFormat)
return apierr.GetAPIError(apierr.ErrPostPolicyConditionInvalidFormat)
}
return nil
@ -203,7 +204,7 @@ func (h *handler) PutObjectHandler(w http.ResponseWriter, r *http.Request) {
}
if cannedACLStatus == aclStatusYes {
h.logAndSendError(w, "acl not supported for this bucket", reqInfo, errors.GetAPIError(errors.ErrAccessControlListNotSupported))
h.logAndSendError(w, "acl not supported for this bucket", reqInfo, apierr.GetAPIError(apierr.ErrAccessControlListNotSupported))
return
}
@ -242,22 +243,22 @@ func (h *handler) PutObjectHandler(w http.ResponseWriter, r *http.Request) {
metadata[api.ContentEncoding] = encodings
}
var size uint64
if r.ContentLength > 0 {
size = uint64(r.ContentLength)
}
size := h.getPutPayloadSize(r)
params := &layer.PutObjectParams{
BktInfo: bktInfo,
Object: reqInfo.ObjectName,
Reader: body,
Size: size,
Header: metadata,
Encryption: encryptionParams,
ContentMD5: r.Header.Get(api.ContentMD5),
ContentSHA256Hash: r.Header.Get(api.AmzContentSha256),
}
if size > 0 {
params.Size = &size
}
params.CopiesNumbers, err = h.pickCopiesNumbers(metadata, reqInfo.Namespace, bktInfo.LocationConstraint)
if err != nil {
h.logAndSendError(w, "invalid copies number", reqInfo, err)
@ -332,16 +333,16 @@ func (h *handler) getBodyReader(r *http.Request) (io.ReadCloser, error) {
if !chunkedEncoding && !h.cfg.BypassContentEncodingInChunks() {
return nil, fmt.Errorf("%w: request is not chunk encoded, encodings '%s'",
errors.GetAPIError(errors.ErrInvalidEncodingMethod), strings.Join(encodings, ","))
apierr.GetAPIError(apierr.ErrInvalidEncodingMethod), strings.Join(encodings, ","))
}
decodeContentSize := r.Header.Get(api.AmzDecodedContentLength)
if len(decodeContentSize) == 0 {
return nil, errors.GetAPIError(errors.ErrMissingContentLength)
return nil, apierr.GetAPIError(apierr.ErrMissingContentLength)
}
if _, err := strconv.Atoi(decodeContentSize); err != nil {
return nil, fmt.Errorf("%w: parse decoded content length: %s", errors.GetAPIError(errors.ErrMissingContentLength), err.Error())
return nil, fmt.Errorf("%w: parse decoded content length: %s", apierr.GetAPIError(apierr.ErrMissingContentLength), err.Error())
}
chunkReader, err := newSignV4ChunkedReader(r)
@ -377,43 +378,43 @@ func formEncryptionParamsBase(r *http.Request, isCopySource bool) (enc encryptio
}
if r.TLS == nil {
return enc, errors.GetAPIError(errors.ErrInsecureSSECustomerRequest)
return enc, apierr.GetAPIError(apierr.ErrInsecureSSECustomerRequest)
}
if len(sseCustomerKey) > 0 && len(sseCustomerAlgorithm) == 0 {
return enc, errors.GetAPIError(errors.ErrMissingSSECustomerAlgorithm)
return enc, apierr.GetAPIError(apierr.ErrMissingSSECustomerAlgorithm)
}
if len(sseCustomerAlgorithm) > 0 && len(sseCustomerKey) == 0 {
return enc, errors.GetAPIError(errors.ErrMissingSSECustomerKey)
return enc, apierr.GetAPIError(apierr.ErrMissingSSECustomerKey)
}
if sseCustomerAlgorithm != layer.AESEncryptionAlgorithm {
return enc, errors.GetAPIError(errors.ErrInvalidEncryptionAlgorithm)
return enc, apierr.GetAPIError(apierr.ErrInvalidEncryptionAlgorithm)
}
key, err := base64.StdEncoding.DecodeString(sseCustomerKey)
if err != nil {
if isCopySource {
return enc, errors.GetAPIError(errors.ErrInvalidSSECustomerParameters)
return enc, apierr.GetAPIError(apierr.ErrInvalidSSECustomerParameters)
}
return enc, errors.GetAPIError(errors.ErrInvalidSSECustomerKey)
return enc, apierr.GetAPIError(apierr.ErrInvalidSSECustomerKey)
}
if len(key) != layer.AESKeySize {
if isCopySource {
return enc, errors.GetAPIError(errors.ErrInvalidSSECustomerParameters)
return enc, apierr.GetAPIError(apierr.ErrInvalidSSECustomerParameters)
}
return enc, errors.GetAPIError(errors.ErrInvalidSSECustomerKey)
return enc, apierr.GetAPIError(apierr.ErrInvalidSSECustomerKey)
}
keyMD5, err := base64.StdEncoding.DecodeString(sseCustomerKeyMD5)
if err != nil {
return enc, errors.GetAPIError(errors.ErrSSECustomerKeyMD5Mismatch)
return enc, apierr.GetAPIError(apierr.ErrSSECustomerKeyMD5Mismatch)
}
md5Sum := md5.Sum(key)
if !bytes.Equal(md5Sum[:], keyMD5) {
return enc, errors.GetAPIError(errors.ErrSSECustomerKeyMD5Mismatch)
return enc, apierr.GetAPIError(apierr.ErrSSECustomerKeyMD5Mismatch)
}
params, err := encryption.NewParams(key)
@ -443,7 +444,7 @@ func (h *handler) PostObject(w http.ResponseWriter, r *http.Request) {
tags := new(data.Tagging)
if err = h.cfg.NewXMLDecoder(buffer).Decode(tags); err != nil {
h.logAndSendError(w, "could not decode tag set", reqInfo,
fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrMalformedXML), err.Error()))
fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrMalformedXML), err.Error()))
return
}
tagSet, err = h.readTagSet(tags)
@ -466,7 +467,7 @@ func (h *handler) PostObject(w http.ResponseWriter, r *http.Request) {
}
if acl := auth.MultipartFormValue(r, "acl"); acl != "" && acl != basicACLPrivate {
h.logAndSendError(w, "acl not supported for this bucket", reqInfo, errors.GetAPIError(errors.ErrAccessControlListNotSupported))
h.logAndSendError(w, "acl not supported for this bucket", reqInfo, apierr.GetAPIError(apierr.ErrAccessControlListNotSupported))
return
}
@ -507,12 +508,12 @@ func (h *handler) PostObject(w http.ResponseWriter, r *http.Request) {
}
if reqInfo.ObjectName == "" {
h.logAndSendError(w, "missing object name", reqInfo, errors.GetAPIError(errors.ErrInvalidArgument))
h.logAndSendError(w, "missing object name", reqInfo, apierr.GetAPIError(apierr.ErrInvalidArgument))
return
}
if !policy.CheckContentLength(size) {
h.logAndSendError(w, "invalid content-length", reqInfo, errors.GetAPIError(errors.ErrInvalidArgument))
h.logAndSendError(w, "invalid content-length", reqInfo, apierr.GetAPIError(apierr.ErrInvalidArgument))
return
}
@ -520,7 +521,7 @@ func (h *handler) PostObject(w http.ResponseWriter, r *http.Request) {
BktInfo: bktInfo,
Object: reqInfo.ObjectName,
Reader: contentReader,
Size: size,
Size: &size,
Header: metadata,
}
@ -594,13 +595,13 @@ func checkPostPolicy(r *http.Request, reqInfo *middleware.ReqInfo, metadata map[
return nil, fmt.Errorf("could not unmarshal policy: %w", err)
}
if policy.Expiration.Before(time.Now()) {
return nil, fmt.Errorf("policy is expired: %w", errors.GetAPIError(errors.ErrInvalidArgument))
return nil, fmt.Errorf("policy is expired: %w", apierr.GetAPIError(apierr.ErrInvalidArgument))
}
policy.empty = false
}
if r.MultipartForm == nil {
return nil, stderrors.New("empty multipart form")
return nil, errors.New("empty multipart form")
}
for key, v := range r.MultipartForm.Value {
@ -631,7 +632,7 @@ func checkPostPolicy(r *http.Request, reqInfo *middleware.ReqInfo, metadata map[
for _, cond := range policy.Conditions {
if cond.Key == "bucket" {
if !cond.match(reqInfo.BucketName) {
return nil, errors.GetAPIError(errors.ErrPostPolicyConditionInvalidFormat)
return nil, apierr.GetAPIError(apierr.ErrPostPolicyConditionInvalidFormat)
}
}
}
@ -673,10 +674,10 @@ func parseTaggingHeader(header http.Header) (map[string]string, error) {
if tagging := header.Get(api.AmzTagging); len(tagging) > 0 {
queries, err := url.ParseQuery(tagging)
if err != nil {
return nil, errors.GetAPIError(errors.ErrInvalidArgument)
return nil, apierr.GetAPIError(apierr.ErrInvalidArgument)
}
if len(queries) > maxTags {
return nil, errors.GetAPIError(errors.ErrInvalidTagsSizeExceed)
return nil, apierr.GetAPIError(apierr.ErrInvalidTagsSizeExceed)
}
tagSet = make(map[string]string, len(queries))
for k, v := range queries {
@ -726,7 +727,7 @@ func (h *handler) parseCommonCreateBucketParams(reqInfo *middleware.ReqInfo, box
}
if p.SessionContainerCreation == nil {
return nil, nil, fmt.Errorf("%w: couldn't find session token for put", errors.GetAPIError(errors.ErrAccessDenied))
return nil, nil, fmt.Errorf("%w: couldn't find session token for put", apierr.GetAPIError(apierr.ErrAccessDenied))
}
if err := checkBucketName(reqInfo.BucketName); err != nil {
@ -783,7 +784,8 @@ func (h *handler) createBucketHandlerPolicy(w http.ResponseWriter, r *http.Reque
chains := bucketCannedACLToAPERules(cannedACL, reqInfo, bktInfo.CID)
if err = h.ape.SaveACLChains(bktInfo.CID.EncodeToString(), chains); err != nil {
h.logAndSendError(w, "failed to add morph rule chain", reqInfo, err)
cleanErr := h.cleanupBucketCreation(ctx, reqInfo, bktInfo, boxData, chains)
h.logAndSendError(w, "failed to add morph rule chain", reqInfo, err, zap.NamedError("cleanup_error", cleanErr))
return
}
@ -804,8 +806,9 @@ func (h *handler) createBucketHandlerPolicy(w http.ResponseWriter, r *http.Reque
return h.obj.PutBucketSettings(ctx, sp)
}, h.putBucketSettingsRetryer())
if err != nil {
cleanErr := h.cleanupBucketCreation(ctx, reqInfo, bktInfo, boxData, chains)
h.logAndSendError(w, "couldn't save bucket settings", reqInfo, err,
zap.String("container_id", bktInfo.CID.EncodeToString()))
zap.String("container_id", bktInfo.CID.EncodeToString()), zap.NamedError("cleanup_error", cleanErr))
return
}
@ -815,6 +818,28 @@ func (h *handler) createBucketHandlerPolicy(w http.ResponseWriter, r *http.Reque
}
}
func (h *handler) cleanupBucketCreation(ctx context.Context, reqInfo *middleware.ReqInfo, bktInfo *data.BucketInfo, boxData *accessbox.Box, chains []*chain.Chain) error {
prm := &layer.DeleteBucketParams{
BktInfo: bktInfo,
SessionToken: boxData.Gate.SessionTokenForDelete(),
}
if err := h.obj.DeleteContainer(ctx, prm); err != nil {
return err
}
chainIDs := make([]chain.ID, len(chains))
for i, c := range chains {
chainIDs[i] = c.ID
}
if err := h.ape.DeleteBucketPolicy(reqInfo.Namespace, bktInfo.CID, chainIDs); err != nil {
return fmt.Errorf("delete bucket acl policy: %w", err)
}
return nil
}
func (h *handler) putBucketSettingsRetryer() aws.RetryerV2 {
return retry.NewStandard(func(options *retry.StandardOptions) {
options.MaxAttempts = h.cfg.RetryMaxAttempts()
@ -828,7 +853,7 @@ func (h *handler) putBucketSettingsRetryer() aws.RetryerV2 {
}
options.Retryables = []retry.IsErrorRetryable{retry.IsErrorRetryableFunc(func(err error) aws.Ternary {
if stderrors.Is(err, tree.ErrNodeAccessDenied) {
if errors.Is(err, tree.ErrNodeAccessDenied) {
return aws.TrueTernary
}
return aws.FalseTernary
@ -957,7 +982,7 @@ func (h handler) setPlacementPolicy(prm *layer.CreateBucketParams, namespace, lo
return nil
}
return errors.GetAPIError(errors.ErrInvalidLocationConstraint)
return apierr.GetAPIError(apierr.ErrInvalidLocationConstraint)
}
func isLockEnabled(log *zap.Logger, header http.Header) bool {
@ -976,28 +1001,22 @@ func isLockEnabled(log *zap.Logger, header http.Header) bool {
func checkBucketName(bucketName string) error {
if len(bucketName) < 3 || len(bucketName) > 63 {
return errors.GetAPIError(errors.ErrInvalidBucketName)
return apierr.GetAPIError(apierr.ErrInvalidBucketName)
}
if strings.HasPrefix(bucketName, "xn--") || strings.HasSuffix(bucketName, "-s3alias") {
return errors.GetAPIError(errors.ErrInvalidBucketName)
return apierr.GetAPIError(apierr.ErrInvalidBucketName)
}
if net.ParseIP(bucketName) != nil {
return errors.GetAPIError(errors.ErrInvalidBucketName)
return apierr.GetAPIError(apierr.ErrInvalidBucketName)
}
labels := strings.Split(bucketName, ".")
for _, label := range labels {
if len(label) == 0 {
return errors.GetAPIError(errors.ErrInvalidBucketName)
for i, r := range bucketName {
if r == '.' || (!isAlphaNum(r) && r != '-') {
return apierr.GetAPIError(apierr.ErrInvalidBucketName)
}
for i, r := range label {
if !isAlphaNum(r) && r != '-' {
return errors.GetAPIError(errors.ErrInvalidBucketName)
}
if (i == 0 || i == len(label)-1) && r == '-' {
return errors.GetAPIError(errors.ErrInvalidBucketName)
}
if (i == 0 || i == len(bucketName)-1) && r == '-' {
return apierr.GetAPIError(apierr.ErrInvalidBucketName)
}
}
@ -1015,7 +1034,7 @@ func (h *handler) parseLocationConstraint(r *http.Request) (*createBucketParams,
params := new(createBucketParams)
if err := h.cfg.NewXMLDecoder(r.Body).Decode(params); err != nil {
return nil, fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrMalformedXML), err.Error())
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrMalformedXML), err.Error())
}
return params, nil
}

View file

@ -7,6 +7,7 @@ import (
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"io"
"mime/multipart"
"net/http"
@ -20,7 +21,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth"
v4 "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth/signer/v4"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
@ -29,6 +30,11 @@ import (
"github.com/stretchr/testify/require"
)
const (
awsChunkedRequestExampleDecodedContentLength = 66560
awsChunkedRequestExampleContentLength = 66824
)
func TestCheckBucketName(t *testing.T) {
for _, tc := range []struct {
name string
@ -36,10 +42,10 @@ func TestCheckBucketName(t *testing.T) {
}{
{name: "bucket"},
{name: "2bucket"},
{name: "buc.ket"},
{name: "buc-ket"},
{name: "abc"},
{name: "63aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"},
{name: "buc.ket", err: true},
{name: "buc.-ket", err: true},
{name: "bucket.", err: true},
{name: ".bucket", err: true},
@ -199,7 +205,7 @@ func TestPostObject(t *testing.T) {
t.Run(tc.key+";"+tc.filename, func(t *testing.T) {
w := postObjectBase(hc, ns, bktName, tc.key, tc.filename, tc.content)
if tc.err {
assertS3Error(hc.t, w, s3errors.GetAPIError(s3errors.ErrInternalError))
assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrInternalError))
return
}
assertStatus(hc.t, w, http.StatusNoContent)
@ -245,6 +251,10 @@ func TestPutObjectWithNegativeContentLength(t *testing.T) {
tc.Handler().HeadObjectHandler(w, r)
assertStatus(t, w, http.StatusOK)
require.Equal(t, strconv.Itoa(len(content)), w.Header().Get(api.ContentLength))
result := listVersions(t, tc, bktName)
require.Len(t, result.Version, 1)
require.EqualValues(t, len(content), result.Version[0].Size)
}
func TestPutObjectWithStreamBodyError(t *testing.T) {
@ -258,7 +268,7 @@ func TestPutObjectWithStreamBodyError(t *testing.T) {
r.Header.Set(api.AmzContentSha256, api.StreamingContentSHA256)
r.Header.Set(api.ContentEncoding, api.AwsChunked)
tc.Handler().PutObjectHandler(w, r)
assertS3Error(t, w, s3errors.GetAPIError(s3errors.ErrMissingContentLength))
assertS3Error(t, w, apierr.GetAPIError(apierr.ErrMissingContentLength))
checkNotFound(t, tc, bktName, objName, emptyVersion)
}
@ -274,7 +284,7 @@ func TestPutObjectWithInvalidContentMD5(t *testing.T) {
w, r := prepareTestPayloadRequest(tc, bktName, objName, bytes.NewReader(content))
r.Header.Set(api.ContentMD5, base64.StdEncoding.EncodeToString([]byte("invalid")))
tc.Handler().PutObjectHandler(w, r)
assertS3Error(t, w, s3errors.GetAPIError(s3errors.ErrInvalidDigest))
assertS3Error(t, w, apierr.GetAPIError(apierr.ErrInvalidDigest))
checkNotFound(t, tc, bktName, objName, emptyVersion)
}
@ -337,7 +347,7 @@ func TestPutObjectCheckContentSHA256(t *testing.T) {
hc.Handler().PutObjectHandler(w, r)
if tc.error {
assertS3Error(t, w, s3errors.GetAPIError(s3errors.ErrContentSHA256Mismatch))
assertS3Error(t, w, apierr.GetAPIError(apierr.ErrContentSHA256Mismatch))
w, r := prepareTestRequest(hc, bktName, objName, nil)
hc.Handler().GetObjectHandler(w, r)
@ -361,12 +371,37 @@ func TestPutObjectWithStreamBodyAWSExample(t *testing.T) {
hc.Handler().PutObjectHandler(w, req)
assertStatus(t, w, http.StatusOK)
data := getObjectRange(t, hc, bktName, objName, 0, 66824)
w, req = prepareTestRequest(hc, bktName, objName, nil)
hc.Handler().HeadObjectHandler(w, req)
assertStatus(t, w, http.StatusOK)
require.Equal(t, strconv.Itoa(awsChunkedRequestExampleDecodedContentLength), w.Header().Get(api.ContentLength))
data := getObjectRange(t, hc, bktName, objName, 0, awsChunkedRequestExampleDecodedContentLength)
for i := range chunk {
require.Equal(t, chunk[i], data[i])
}
}
func TestPutObjectWithStreamEmptyBodyAWSExample(t *testing.T) {
hc := prepareHandlerContext(t)
bktName, objName := "dkirillov", "tmp"
createTestBucket(hc, bktName)
w, req := getEmptyChunkedRequest(hc.context, t, bktName, objName)
hc.Handler().PutObjectHandler(w, req)
assertStatus(t, w, http.StatusOK)
w, req = prepareTestRequest(hc, bktName, objName, nil)
hc.Handler().HeadObjectHandler(w, req)
assertStatus(t, w, http.StatusOK)
require.Equal(t, "0", w.Header().Get(api.ContentLength))
res := listObjectsV1(hc, bktName, "", "", "", -1)
require.Len(t, res.Contents, 1)
require.Empty(t, res.Contents[0].Size)
}
func TestPutChunkedTestContentEncoding(t *testing.T) {
hc := prepareHandlerContext(t)
@ -385,7 +420,7 @@ func TestPutChunkedTestContentEncoding(t *testing.T) {
w, req, _ = getChunkedRequest(hc.context, t, bktName, objName)
req.Header.Set(api.ContentEncoding, "gzip")
hc.Handler().PutObjectHandler(w, req)
assertS3Error(t, w, s3errors.GetAPIError(s3errors.ErrInvalidEncodingMethod))
assertS3Error(t, w, apierr.GetAPIError(apierr.ErrInvalidEncodingMethod))
hc.config.bypassContentEncodingInChunks = true
w, req, _ = getChunkedRequest(hc.context, t, bktName, objName)
@ -397,6 +432,8 @@ func TestPutChunkedTestContentEncoding(t *testing.T) {
require.Equal(t, "gzip", resp.Header().Get(api.ContentEncoding))
}
// getChunkedRequest implements request example from
// https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
func getChunkedRequest(ctx context.Context, t *testing.T, bktName, objName string) (*httptest.ResponseRecorder, *http.Request, []byte) {
chunk := make([]byte, 65*1024)
for i := range chunk {
@ -424,9 +461,9 @@ func getChunkedRequest(ctx context.Context, t *testing.T, bktName, objName strin
req, err := http.NewRequest("PUT", "https://s3.amazonaws.com/"+bktName+"/"+objName, nil)
require.NoError(t, err)
req.Header.Set("content-encoding", "aws-chunked")
req.Header.Set("content-length", "66824")
req.Header.Set("content-length", strconv.Itoa(awsChunkedRequestExampleContentLength))
req.Header.Set("x-amz-content-sha256", "STREAMING-AWS4-HMAC-SHA256-PAYLOAD")
req.Header.Set("x-amz-decoded-content-length", "66560")
req.Header.Set("x-amz-decoded-content-length", strconv.Itoa(awsChunkedRequestExampleDecodedContentLength))
req.Header.Set("x-amz-storage-class", "REDUCED_REDUNDANCY")
signTime, err := time.Parse("20060102T150405Z", "20130524T000000Z")
@ -457,15 +494,72 @@ func getChunkedRequest(ctx context.Context, t *testing.T, bktName, objName strin
return w, req, chunk
}
func getEmptyChunkedRequest(ctx context.Context, t *testing.T, bktName, objName string) (*httptest.ResponseRecorder, *http.Request) {
AWSAccessKeyID := "48c1K4PLVb7SvmV3PjDKEuXaMh8yZMXZ8Wx9msrkKcYw06dZeaxeiPe8vyFm2WsoeVaNt7UWEjNsVkagDs8oX4XXh"
AWSSecretAccessKey := "09260955b4eb0279dc017ba20a1ddac909cbd226c86cbb2d868e55534c8e64b0"
//awsCreds := credentials.NewStaticCredentials(AWSAccessKeyID, AWSSecretAccessKey, "")
//signer := v4.NewSigner(awsCreds)
reqBody := bytes.NewBufferString("0;chunk-signature=311a7142c8f3a07972c3aca65c36484b513a8fee48ab7178c7225388f2ae9894\r\n\r\n")
req, err := http.NewRequest("PUT", "http://localhost:8084/"+bktName+"/"+objName, reqBody)
require.NoError(t, err)
req.Header.Set("Amz-Sdk-Invocation-Id", "8a8cd4be-aef8-8034-f08d-a6144ade41f9")
req.Header.Set("Amz-Sdk-Request", "attempt=1; max=2")
req.Header.Set(api.Authorization, "AWS4-HMAC-SHA256 Credential=48c1K4PLVb7SvmV3PjDKEuXaMh8yZMXZ8Wx9msrkKcYw06dZeaxeiPe8vyFm2WsoeVaNt7UWEjNsVkagDs8oX4XXh/20241003/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=4b530ab4af2381f214941af591266b209968264a2c94337fa1efc048c7dff352")
req.Header.Set(api.ContentEncoding, "aws-chunked")
req.Header.Set(api.ContentLength, "86")
req.Header.Set(api.ContentType, "text/plain; charset=UTF-8")
req.Header.Set(api.AmzDate, "20241003T100055Z")
req.Header.Set(api.AmzContentSha256, "STREAMING-AWS4-HMAC-SHA256-PAYLOAD")
req.Header.Set(api.AmzDecodedContentLength, "0")
signTime, err := time.Parse("20060102T150405Z", "20241003T100055Z")
require.NoError(t, err)
w := httptest.NewRecorder()
reqInfo := middleware.NewReqInfo(w, req, middleware.ObjectRequest{Bucket: bktName, Object: objName}, "")
req = req.WithContext(middleware.SetReqInfo(ctx, reqInfo))
req = req.WithContext(middleware.SetBox(req.Context(), &middleware.Box{
ClientTime: signTime,
AuthHeaders: &middleware.AuthHeader{
AccessKeyID: AWSAccessKeyID,
SignatureV4: "4b530ab4af2381f214941af591266b209968264a2c94337fa1efc048c7dff352",
Region: "us-east-1",
},
AccessBox: &accessbox.Box{
Gate: &accessbox.GateData{
SecretKey: AWSSecretAccessKey,
},
},
}))
return w, req
}
func TestCreateBucket(t *testing.T) {
hc := prepareHandlerContext(t)
bktName := "bkt-name"
info := createBucket(hc, bktName)
createBucketAssertS3Error(hc, bktName, info.Box, s3errors.ErrBucketAlreadyOwnedByYou)
createBucketAssertS3Error(hc, bktName, info.Box, apierr.ErrBucketAlreadyOwnedByYou)
box2, _ := createAccessBox(t)
createBucketAssertS3Error(hc, bktName, box2, s3errors.ErrBucketAlreadyExists)
createBucketAssertS3Error(hc, bktName, box2, apierr.ErrBucketAlreadyExists)
}
func TestCreateBucketWithoutPermissions(t *testing.T) {
hc := prepareHandlerContext(t)
bktName := "bkt-name"
hc.h.ape.(*apeMock).err = errors.New("no permissions")
box, _ := createAccessBox(t)
createBucketAssertS3Error(hc, bktName, box, apierr.ErrInternalError)
_, err := hc.tp.ContainerID(bktName)
require.Errorf(t, err, "container exists after failed creation, but shouldn't")
}
func TestCreateNamespacedBucket(t *testing.T) {

View file

@ -6,7 +6,7 @@ import (
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"github.com/stretchr/testify/require"
)
@ -98,7 +98,7 @@ func TestPutObjectTaggingCheckUniqueness(t *testing.T) {
middleware.GetReqInfo(r.Context()).Tagging = tc.body
hc.Handler().PutObjectTaggingHandler(w, r)
if tc.error {
assertS3Error(t, w, apiErrors.GetAPIError(apiErrors.ErrInvalidTagKeyUniqueness))
assertS3Error(t, w, apierr.GetAPIError(apierr.ErrInvalidTagKeyUniqueness))
return
}
assertStatus(t, w, http.StatusOK)

View file

@ -10,10 +10,9 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
frosterrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"go.opentelemetry.io/otel/trace"
@ -30,7 +29,7 @@ func (h *handler) reqLogger(ctx context.Context) *zap.Logger {
func (h *handler) logAndSendError(w http.ResponseWriter, logText string, reqInfo *middleware.ReqInfo, err error, additional ...zap.Field) {
err = handleDeleteMarker(w, err)
if code, wrErr := middleware.WriteErrorResponse(w, reqInfo, transformToS3Error(err)); wrErr != nil {
if code, wrErr := middleware.WriteErrorResponse(w, reqInfo, apierr.TransformToS3Error(err)); wrErr != nil {
additional = append(additional, zap.NamedError("write_response_error", wrErr))
} else {
additional = append(additional, zap.Int("status", code))
@ -57,25 +56,7 @@ func handleDeleteMarker(w http.ResponseWriter, err error) error {
}
w.Header().Set(api.AmzDeleteMarker, "true")
return fmt.Errorf("%w: %s", s3errors.GetAPIError(target.ErrorCode), err)
}
func transformToS3Error(err error) error {
err = frosterrors.UnwrapErr(err) // this wouldn't work with errors.Join
if _, ok := err.(s3errors.Error); ok {
return err
}
if errors.Is(err, layer.ErrAccessDenied) ||
errors.Is(err, layer.ErrNodeAccessDenied) {
return s3errors.GetAPIError(s3errors.ErrAccessDenied)
}
if errors.Is(err, layer.ErrGatewayTimeout) {
return s3errors.GetAPIError(s3errors.ErrGatewayTimeout)
}
return s3errors.GetAPIError(s3errors.ErrInternalError)
return fmt.Errorf("%w: %s", apierr.GetAPIError(target.ErrorCode), err)
}
func (h *handler) ResolveBucket(ctx context.Context, bucket string) (*data.BucketInfo, error) {
@ -106,6 +87,19 @@ func (h *handler) getBucketAndCheckOwner(r *http.Request, bucket string, header
return bktInfo, checkOwner(bktInfo, expected)
}
func (h *handler) getPutPayloadSize(r *http.Request) uint64 {
decodeContentSize := r.Header.Get(api.AmzDecodedContentLength)
decodedSize, err := strconv.Atoi(decodeContentSize)
if err != nil {
if r.ContentLength >= 0 {
return uint64(r.ContentLength)
}
return 0
}
return uint64(decodedSize)
}
func parseRange(s string) (*layer.RangeParams, error) {
if s == "" {
return nil, nil
@ -114,26 +108,26 @@ func parseRange(s string) (*layer.RangeParams, error) {
prefix := "bytes="
if !strings.HasPrefix(s, prefix) {
return nil, s3errors.GetAPIError(s3errors.ErrInvalidRange)
return nil, apierr.GetAPIError(apierr.ErrInvalidRange)
}
s = strings.TrimPrefix(s, prefix)
valuesStr := strings.Split(s, "-")
if len(valuesStr) != 2 {
return nil, s3errors.GetAPIError(s3errors.ErrInvalidRange)
return nil, apierr.GetAPIError(apierr.ErrInvalidRange)
}
values := make([]uint64, 0, len(valuesStr))
for _, v := range valuesStr {
num, err := strconv.ParseUint(v, 10, 64)
if err != nil {
return nil, s3errors.GetAPIError(s3errors.ErrInvalidRange)
return nil, apierr.GetAPIError(apierr.ErrInvalidRange)
}
values = append(values, num)
}
if values[0] > values[1] {
return nil, s3errors.GetAPIError(s3errors.ErrInvalidRange)
return nil, apierr.GetAPIError(apierr.ErrInvalidRange)
}
return &layer.RangeParams{

View file

@ -1,64 +1 @@
package handler
import (
"errors"
"fmt"
"testing"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"github.com/stretchr/testify/require"
)
func TestTransformS3Errors(t *testing.T) {
for _, tc := range []struct {
name string
err error
expected s3errors.ErrorCode
}{
{
name: "simple std error to internal error",
err: errors.New("some error"),
expected: s3errors.ErrInternalError,
},
{
name: "layer access denied error to s3 access denied error",
err: layer.ErrAccessDenied,
expected: s3errors.ErrAccessDenied,
},
{
name: "wrapped layer access denied error to s3 access denied error",
err: fmt.Errorf("wrap: %w", layer.ErrAccessDenied),
expected: s3errors.ErrAccessDenied,
},
{
name: "layer node access denied error to s3 access denied error",
err: layer.ErrNodeAccessDenied,
expected: s3errors.ErrAccessDenied,
},
{
name: "layer gateway timeout error to s3 gateway timeout error",
err: layer.ErrGatewayTimeout,
expected: s3errors.ErrGatewayTimeout,
},
{
name: "s3 error to s3 error",
err: s3errors.GetAPIError(s3errors.ErrInvalidPart),
expected: s3errors.ErrInvalidPart,
},
{
name: "wrapped s3 error to s3 error",
err: fmt.Errorf("wrap: %w", s3errors.GetAPIError(s3errors.ErrInvalidPart)),
expected: s3errors.ErrInvalidPart,
},
} {
t.Run(tc.name, func(t *testing.T) {
err := transformToS3Error(tc.err)
s3err, ok := err.(s3errors.Error)
require.True(t, ok, "error must be s3 error")
require.Equalf(t, tc.expected, s3err.ErrCode,
"expected: '%s', got: '%s'",
s3errors.GetAPIError(tc.expected).Code, s3errors.GetAPIError(s3err.ErrCode).Code)
})
}
}

View file

@ -5,6 +5,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"go.uber.org/zap"
@ -19,6 +20,7 @@ type Cache struct {
bucketCache *cache.BucketCache
systemCache *cache.SystemCache
accessCache *cache.AccessControlCache
networkInfoCache *cache.NetworkInfoCache
}
// CachesConfig contains params for caches.
@ -31,6 +33,7 @@ type CachesConfig struct {
Buckets *cache.Config
System *cache.Config
AccessControl *cache.Config
NetworkInfo *cache.NetworkInfoCacheConfig
}
// DefaultCachesConfigs returns filled configs.
@ -44,6 +47,7 @@ func DefaultCachesConfigs(logger *zap.Logger) *CachesConfig {
Buckets: cache.DefaultBucketConfig(logger),
System: cache.DefaultSystemConfig(logger),
AccessControl: cache.DefaultAccessControlConfig(logger),
NetworkInfo: cache.DefaultNetworkInfoConfig(logger),
}
}
@ -57,6 +61,7 @@ func NewCache(cfg *CachesConfig) *Cache {
bucketCache: cache.NewBucketCache(cfg.Buckets),
systemCache: cache.NewSystemCache(cfg.System),
accessCache: cache.NewAccessControlCache(cfg.AccessControl),
networkInfoCache: cache.NewNetworkInfoCache(cfg.NetworkInfo),
}
}
@ -283,3 +288,13 @@ func (c *Cache) PutLifecycleConfiguration(owner user.ID, bkt *data.BucketInfo, c
func (c *Cache) DeleteLifecycleConfiguration(bktInfo *data.BucketInfo) {
c.systemCache.Delete(bktInfo.LifecycleConfigurationObjectName())
}
func (c *Cache) GetNetworkInfo() *netmap.NetworkInfo {
return c.networkInfoCache.Get()
}
func (c *Cache) PutNetworkInfo(info netmap.NetworkInfo) {
if err := c.networkInfoCache.Put(info); err != nil {
c.logger.Warn(logs.CouldntCacheNetworkInfo, zap.Error(err))
}
}

View file

@ -6,7 +6,8 @@ import (
"fmt"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
)
func (n *Layer) GetObjectTaggingAndLock(ctx context.Context, objVersion *data.ObjectVersion, nodeVersion *data.NodeVersion) (map[string]string, data.LockInfo, error) {
@ -29,8 +30,8 @@ func (n *Layer) GetObjectTaggingAndLock(ctx context.Context, objVersion *data.Ob
tags, lockInfo, err = n.treeService.GetObjectTaggingAndLock(ctx, objVersion.BktInfo, nodeVersion)
if err != nil {
if errors.Is(err, ErrNodeNotFound) {
return nil, data.LockInfo{}, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchKey), err.Error())
if errors.Is(err, tree.ErrNodeNotFound) {
return nil, data.LockInfo{}, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
}
return nil, data.LockInfo{}, err
}

View file

@ -7,7 +7,8 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
@ -20,7 +21,7 @@ const (
AttributeLockEnabled = "LockEnabled"
)
func (n *Layer) containerInfo(ctx context.Context, prm PrmContainer) (*data.BucketInfo, error) {
func (n *Layer) containerInfo(ctx context.Context, prm frostfs.PrmContainer) (*data.BucketInfo, error) {
var (
err error
res *container.Container
@ -37,7 +38,7 @@ func (n *Layer) containerInfo(ctx context.Context, prm PrmContainer) (*data.Buck
res, err = n.frostFS.Container(ctx, prm)
if err != nil {
if client.IsErrContainerNotFound(err) {
return nil, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchBucket), err.Error())
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchBucket), err.Error())
}
return nil, fmt.Errorf("get frostfs container: %w", err)
}
@ -52,6 +53,7 @@ func (n *Layer) containerInfo(ctx context.Context, prm PrmContainer) (*data.Buck
info.Created = container.CreatedAt(cnr)
info.LocationConstraint = cnr.Attribute(attributeLocationConstraint)
info.HomomorphicHashDisabled = container.IsHomomorphicHashingDisabled(cnr)
info.PlacementPolicy = cnr.PlacementPolicy()
attrLockEnabled := cnr.Attribute(AttributeLockEnabled)
if len(attrLockEnabled) > 0 {
@ -64,7 +66,7 @@ func (n *Layer) containerInfo(ctx context.Context, prm PrmContainer) (*data.Buck
}
}
zone, _ := n.features.FormContainerZone(reqInfo.Namespace)
zone := n.features.FormContainerZone(reqInfo.Namespace)
if zone != info.Zone {
return nil, fmt.Errorf("ns '%s' and zone '%s' are mismatched for container '%s'", zone, info.Zone, prm.ContainerID)
}
@ -77,7 +79,7 @@ func (n *Layer) containerInfo(ctx context.Context, prm PrmContainer) (*data.Buck
func (n *Layer) containerList(ctx context.Context) ([]*data.BucketInfo, error) {
stoken := n.SessionTokenForRead(ctx)
prm := PrmUserContainers{
prm := frostfs.PrmUserContainers{
UserID: n.BearerOwner(ctx),
SessionToken: stoken,
}
@ -90,7 +92,7 @@ func (n *Layer) containerList(ctx context.Context) ([]*data.BucketInfo, error) {
list := make([]*data.BucketInfo, 0, len(res))
for i := range res {
getPrm := PrmContainer{
getPrm := frostfs.PrmContainer{
ContainerID: res[i],
SessionToken: stoken,
}
@ -111,7 +113,7 @@ func (n *Layer) createContainer(ctx context.Context, p *CreateBucketParams) (*da
p.LocationConstraint = api.DefaultLocationConstraint // s3tests_boto3.functional.test_s3:test_bucket_get_location
}
zone, _ := n.features.FormContainerZone(p.Namespace)
zone := n.features.FormContainerZone(p.Namespace)
bktInfo := &data.BucketInfo{
Name: p.Name,
@ -132,7 +134,7 @@ func (n *Layer) createContainer(ctx context.Context, p *CreateBucketParams) (*da
})
}
res, err := n.frostFS.CreateContainer(ctx, PrmContainerCreate{
res, err := n.frostFS.CreateContainer(ctx, frostfs.PrmContainerCreate{
Creator: bktInfo.Owner,
Policy: p.Policy,
Name: p.Name,
@ -148,6 +150,7 @@ func (n *Layer) createContainer(ctx context.Context, p *CreateBucketParams) (*da
bktInfo.CID = res.ContainerID
bktInfo.HomomorphicHashDisabled = res.HomomorphicHashDisabled
bktInfo.PlacementPolicy = p.Policy
n.cache.PutBucket(bktInfo)

View file

@ -3,12 +3,14 @@ package layer
import (
"bytes"
"context"
errorsStd "errors"
"errors"
"fmt"
"io"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
@ -31,14 +33,14 @@ func (n *Layer) PutBucketCORS(ctx context.Context, p *PutCORSParams) error {
}
if cors.CORSRules == nil {
return errors.GetAPIError(errors.ErrMalformedXML)
return apierr.GetAPIError(apierr.ErrMalformedXML)
}
if err := checkCORS(cors); err != nil {
return err
}
prm := PrmObjectCreate{
prm := frostfs.PrmObjectCreate{
Payload: &buf,
Filepath: p.BktInfo.CORSObjectName(),
CreationTime: TimeNow(ctx),
@ -61,7 +63,7 @@ func (n *Layer) PutBucketCORS(ctx context.Context, p *PutCORSParams) error {
}
objsToDelete, err := n.treeService.PutBucketCORS(ctx, p.BktInfo, newAddress(corsBkt.CID, createdObj.ID))
objToDeleteNotFound := errorsStd.Is(err, ErrNoNodeToRemove)
objToDeleteNotFound := errors.Is(err, tree.ErrNoNodeToRemove)
if err != nil && !objToDeleteNotFound {
return err
}
@ -79,7 +81,7 @@ func (n *Layer) PutBucketCORS(ctx context.Context, p *PutCORSParams) error {
// deleteCORSObject removes object and logs in case of error.
func (n *Layer) deleteCORSObject(ctx context.Context, bktInfo *data.BucketInfo, addr oid.Address) {
var prmAuth PrmAuth
var prmAuth frostfs.PrmAuth
corsBkt := bktInfo
if !addr.Container().Equals(bktInfo.CID) && !addr.Container().Equals(cid.ID{}) {
corsBkt = &data.BucketInfo{CID: addr.Container()}
@ -104,7 +106,7 @@ func (n *Layer) GetBucketCORS(ctx context.Context, bktInfo *data.BucketInfo) (*d
func (n *Layer) DeleteBucketCORS(ctx context.Context, bktInfo *data.BucketInfo) error {
objs, err := n.treeService.DeleteBucketCORS(ctx, bktInfo)
objNotFound := errorsStd.Is(err, ErrNoNodeToRemove)
objNotFound := errors.Is(err, tree.ErrNoNodeToRemove)
if err != nil && !objNotFound {
return err
}
@ -124,12 +126,12 @@ func checkCORS(cors *data.CORSConfiguration) error {
for _, r := range cors.CORSRules {
for _, m := range r.AllowedMethods {
if _, ok := supportedMethods[m]; !ok {
return errors.GetAPIErrorWithError(errors.ErrCORSUnsupportedMethod, fmt.Errorf("unsupported method is %s", m))
return apierr.GetAPIErrorWithError(apierr.ErrCORSUnsupportedMethod, fmt.Errorf("unsupported method is %s", m))
}
}
for _, h := range r.ExposeHeaders {
if h == wildcard {
return errors.GetAPIError(errors.ErrCORSWildcardExposeHeaders)
return apierr.GetAPIError(apierr.ErrCORSWildcardExposeHeaders)
}
}
}

View file

@ -1,4 +1,4 @@
package layer
package frostfs
import (
"context"

View file

@ -12,6 +12,7 @@ import (
v2container "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
objectv2 "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/checksum"
@ -52,16 +53,16 @@ func (k *FeatureSettingsMock) SetMD5Enabled(md5Enabled bool) {
k.md5Enabled = md5Enabled
}
func (k *FeatureSettingsMock) FormContainerZone(ns string) (zone string, isDefault bool) {
func (k *FeatureSettingsMock) FormContainerZone(ns string) string {
if ns == "" {
return v2container.SysAttributeZoneDefault, true
return v2container.SysAttributeZoneDefault
}
return ns + ".ns", false
return ns + ".ns"
}
type TestFrostFS struct {
FrostFS
frostfs.FrostFS
objects map[string]*object.Object
objectErrors map[string]error
@ -139,7 +140,7 @@ func (t *TestFrostFS) SetContainer(cnrID cid.ID, cnr *container.Container) {
t.containers[cnrID.EncodeToString()] = cnr
}
func (t *TestFrostFS) CreateContainer(_ context.Context, prm PrmContainerCreate) (*ContainerCreateResult, error) {
func (t *TestFrostFS) CreateContainer(_ context.Context, prm frostfs.PrmContainerCreate) (*frostfs.ContainerCreateResult, error) {
var cnr container.Container
cnr.Init()
cnr.SetOwner(prm.Creator)
@ -174,7 +175,7 @@ func (t *TestFrostFS) CreateContainer(_ context.Context, prm PrmContainerCreate)
id.SetSHA256(sha256.Sum256(b))
t.containers[id.EncodeToString()] = &cnr
return &ContainerCreateResult{ContainerID: id}, nil
return &frostfs.ContainerCreateResult{ContainerID: id}, nil
}
func (t *TestFrostFS) DeleteContainer(_ context.Context, cnrID cid.ID, _ *session.Container) error {
@ -183,7 +184,7 @@ func (t *TestFrostFS) DeleteContainer(_ context.Context, cnrID cid.ID, _ *sessio
return nil
}
func (t *TestFrostFS) Container(_ context.Context, prm PrmContainer) (*container.Container, error) {
func (t *TestFrostFS) Container(_ context.Context, prm frostfs.PrmContainer) (*container.Container, error) {
for k, v := range t.containers {
if k == prm.ContainerID.EncodeToString() {
return v, nil
@ -193,7 +194,7 @@ func (t *TestFrostFS) Container(_ context.Context, prm PrmContainer) (*container
return nil, fmt.Errorf("container not found %s", prm.ContainerID)
}
func (t *TestFrostFS) UserContainers(context.Context, PrmUserContainers) ([]cid.ID, error) {
func (t *TestFrostFS) UserContainers(context.Context, frostfs.PrmUserContainers) ([]cid.ID, error) {
var res []cid.ID
for k := range t.containers {
var idCnr cid.ID
@ -220,7 +221,7 @@ func (t *TestFrostFS) retrieveObject(ctx context.Context, cnrID cid.ID, objID oi
if obj, ok := t.objects[sAddr]; ok {
owner := getBearerOwner(ctx)
if !t.checkAccess(cnrID, owner) {
return nil, ErrAccessDenied
return nil, frostfs.ErrAccessDenied
}
return obj, nil
@ -229,23 +230,23 @@ func (t *TestFrostFS) retrieveObject(ctx context.Context, cnrID cid.ID, objID oi
return nil, fmt.Errorf("%w: %s", &apistatus.ObjectNotFound{}, addr)
}
func (t *TestFrostFS) HeadObject(ctx context.Context, prm PrmObjectHead) (*object.Object, error) {
func (t *TestFrostFS) HeadObject(ctx context.Context, prm frostfs.PrmObjectHead) (*object.Object, error) {
return t.retrieveObject(ctx, prm.Container, prm.Object)
}
func (t *TestFrostFS) GetObject(ctx context.Context, prm PrmObjectGet) (*Object, error) {
func (t *TestFrostFS) GetObject(ctx context.Context, prm frostfs.PrmObjectGet) (*frostfs.Object, error) {
obj, err := t.retrieveObject(ctx, prm.Container, prm.Object)
if err != nil {
return nil, err
}
return &Object{
return &frostfs.Object{
Header: *obj,
Payload: io.NopCloser(bytes.NewReader(obj.Payload())),
}, nil
}
func (t *TestFrostFS) RangeObject(ctx context.Context, prm PrmObjectRange) (io.ReadCloser, error) {
func (t *TestFrostFS) RangeObject(ctx context.Context, prm frostfs.PrmObjectRange) (io.ReadCloser, error) {
obj, err := t.retrieveObject(ctx, prm.Container, prm.Object)
if err != nil {
return nil, err
@ -257,7 +258,7 @@ func (t *TestFrostFS) RangeObject(ctx context.Context, prm PrmObjectRange) (io.R
return io.NopCloser(bytes.NewReader(payload)), nil
}
func (t *TestFrostFS) CreateObject(_ context.Context, prm PrmObjectCreate) (*CreateObjectResult, error) {
func (t *TestFrostFS) CreateObject(_ context.Context, prm frostfs.PrmObjectCreate) (*frostfs.CreateObjectResult, error) {
b := make([]byte, 32)
if _, err := io.ReadFull(rand.Reader, b); err != nil {
return nil, err
@ -327,13 +328,13 @@ func (t *TestFrostFS) CreateObject(_ context.Context, prm PrmObjectCreate) (*Cre
addr := newAddress(cnrID, objID)
t.objects[addr.EncodeToString()] = obj
return &CreateObjectResult{
return &frostfs.CreateObjectResult{
ObjectID: objID,
CreationEpoch: t.currentEpoch - 1,
}, nil
}
func (t *TestFrostFS) DeleteObject(ctx context.Context, prm PrmObjectDelete) error {
func (t *TestFrostFS) DeleteObject(ctx context.Context, prm frostfs.PrmObjectDelete) error {
var addr oid.Address
addr.SetContainer(prm.Container)
addr.SetObject(prm.Object)
@ -345,7 +346,7 @@ func (t *TestFrostFS) DeleteObject(ctx context.Context, prm PrmObjectDelete) err
if _, ok := t.objects[addr.EncodeToString()]; ok {
owner := getBearerOwner(ctx)
if !t.checkAccess(prm.Container, owner) {
return ErrAccessDenied
return frostfs.ErrAccessDenied
}
delete(t.objects, addr.EncodeToString())
@ -372,7 +373,7 @@ func (t *TestFrostFS) AllObjects(cnrID cid.ID) []oid.ID {
return result
}
func (t *TestFrostFS) SearchObjects(_ context.Context, prm PrmObjectSearch) ([]oid.ID, error) {
func (t *TestFrostFS) SearchObjects(_ context.Context, prm frostfs.PrmObjectSearch) ([]oid.ID, error) {
filters := object.NewSearchFilters()
filters.AddRootFilter()
@ -412,11 +413,13 @@ func (t *TestFrostFS) SearchObjects(_ context.Context, prm PrmObjectSearch) ([]o
func (t *TestFrostFS) NetworkInfo(context.Context) (netmap.NetworkInfo, error) {
ni := netmap.NetworkInfo{}
ni.SetCurrentEpoch(t.currentEpoch)
ni.SetEpochDuration(60)
ni.SetMsPerBlock(1000)
return ni, nil
}
func (t *TestFrostFS) PatchObject(ctx context.Context, prm PrmObjectPatch) (oid.ID, error) {
func (t *TestFrostFS) PatchObject(ctx context.Context, prm frostfs.PrmObjectPatch) (oid.ID, error) {
obj, err := t.retrieveObject(ctx, prm.Container, prm.Object)
if err != nil {
return oid.ID{}, err

View file

@ -6,7 +6,7 @@ import (
"crypto/rand"
"encoding/json"
"encoding/xml"
stderrors "errors"
"errors"
"fmt"
"io"
"net/url"
@ -17,9 +17,10 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
@ -35,24 +36,24 @@ import (
type (
BucketResolver interface {
Resolve(ctx context.Context, name string) (cid.ID, error)
Resolve(ctx context.Context, zone, name string) (cid.ID, error)
}
FeatureSettings interface {
ClientCut() bool
BufferMaxSizeForPut() uint64
MD5Enabled() bool
FormContainerZone(ns string) (zone string, isDefault bool)
FormContainerZone(ns string) string
}
Layer struct {
frostFS FrostFS
frostFS frostfs.FrostFS
gateOwner user.ID
log *zap.Logger
anonKey AnonymousKey
resolver BucketResolver
cache *Cache
treeService TreeService
treeService tree.Service
features FeatureSettings
gateKey *keys.PrivateKey
corsCnrInfo *data.BucketInfo
@ -65,7 +66,7 @@ type (
Cache *Cache
AnonKey AnonymousKey
Resolver BucketResolver
TreeService TreeService
TreeService tree.Service
Features FeatureSettings
GateKey *keys.PrivateKey
CORSCnrInfo *data.BucketInfo
@ -103,7 +104,7 @@ type (
PutObjectParams struct {
BktInfo *data.BucketInfo
Object string
Size uint64
Size *uint64
Reader io.Reader
Header map[string]string
Lock *data.ObjectLock
@ -235,7 +236,7 @@ func (p HeadObjectParams) Versioned() bool {
// NewLayer creates an instance of a Layer. It checks credentials
// and establishes gRPC connection with the node.
func NewLayer(log *zap.Logger, frostFS FrostFS, config *Config) *Layer {
func NewLayer(log *zap.Logger, frostFS frostfs.FrostFS, config *Config) *Layer {
return &Layer{
frostFS: frostFS,
log: log,
@ -299,7 +300,7 @@ func (n *Layer) reqLogger(ctx context.Context) *zap.Logger {
return n.log
}
func (n *Layer) prepareAuthParameters(ctx context.Context, prm *PrmAuth, bktOwner user.ID) {
func (n *Layer) prepareAuthParameters(ctx context.Context, prm *frostfs.PrmAuth, bktOwner user.ID) {
if prm.BearerToken != nil || prm.PrivateKey != nil {
return
}
@ -322,21 +323,21 @@ func (n *Layer) GetBucketInfo(ctx context.Context, name string) (*data.BucketInf
}
reqInfo := middleware.GetReqInfo(ctx)
zone, _ := n.features.FormContainerZone(reqInfo.Namespace)
zone := n.features.FormContainerZone(reqInfo.Namespace)
if bktInfo := n.cache.GetBucket(zone, name); bktInfo != nil {
return bktInfo, nil
}
containerID, err := n.ResolveBucket(ctx, name)
containerID, err := n.ResolveBucket(ctx, zone, name)
if err != nil {
if strings.Contains(err.Error(), "not found") {
return nil, fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrNoSuchBucket), err.Error())
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchBucket), err.Error())
}
return nil, err
}
prm := PrmContainer{
prm := frostfs.PrmContainer{
ContainerID: containerID,
SessionToken: n.SessionTokenForRead(ctx),
}
@ -352,13 +353,13 @@ func (n *Layer) ResolveCID(ctx context.Context, name string) (cid.ID, error) {
}
reqInfo := middleware.GetReqInfo(ctx)
zone, _ := n.features.FormContainerZone(reqInfo.Namespace)
zone := n.features.FormContainerZone(reqInfo.Namespace)
if bktInfo := n.cache.GetBucket(zone, name); bktInfo != nil {
return bktInfo.CID, nil
}
return n.ResolveBucket(ctx, name)
return n.ResolveBucket(ctx, zone, name)
}
// ListBuckets returns all user containers. The name of the bucket is a container
@ -397,9 +398,9 @@ func (n *Layer) GetObject(ctx context.Context, p *GetObjectParams) (*ObjectPaylo
if err != nil {
if client.IsErrObjectNotFound(err) {
if p.Versioned {
err = fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrNoSuchVersion), err.Error())
err = fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchVersion), err.Error())
} else {
err = fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrNoSuchKey), err.Error())
err = fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
}
}
@ -528,7 +529,7 @@ func (n *Layer) CopyObject(ctx context.Context, p *CopyObjectParams) (*data.Exte
return n.PutObject(ctx, &PutObjectParams{
BktInfo: p.DstBktInfo,
Object: p.DstObject,
Size: p.DstSize,
Size: &p.DstSize,
Reader: objPayload,
Header: p.Header,
Encryption: p.DstEncryption,
@ -655,22 +656,22 @@ func (n *Layer) handleNotFoundError(bkt *data.BucketInfo, obj *VersionedObject)
}
func isNotFoundError(err error) bool {
return errors.IsS3Error(err, errors.ErrNoSuchKey) ||
errors.IsS3Error(err, errors.ErrNoSuchVersion)
return apierr.IsS3Error(err, apierr.ErrNoSuchKey) ||
apierr.IsS3Error(err, apierr.ErrNoSuchVersion)
}
func (n *Layer) getNodeVersionsToDelete(ctx context.Context, bkt *data.BucketInfo, obj *VersionedObject) ([]*data.NodeVersion, error) {
var versionsToDelete []*data.NodeVersion
versions, err := n.treeService.GetVersions(ctx, bkt, obj.Name)
if err != nil {
if stderrors.Is(err, ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchKey), err.Error())
if errors.Is(err, tree.ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
}
return nil, err
}
if len(versions) == 0 {
return nil, fmt.Errorf("%w: there isn't tree node with requested version id", s3errors.GetAPIError(s3errors.ErrNoSuchVersion))
return nil, fmt.Errorf("%w: there isn't tree node with requested version id", apierr.GetAPIError(apierr.ErrNoSuchVersion))
}
sort.Slice(versions, func(i, j int) bool {
@ -712,7 +713,7 @@ func (n *Layer) getNodeVersionsToDelete(ctx context.Context, bkt *data.BucketInf
}
if len(versionsToDelete) == 0 {
return nil, fmt.Errorf("%w: there isn't tree node with requested version id", s3errors.GetAPIError(s3errors.ErrNoSuchVersion))
return nil, fmt.Errorf("%w: there isn't tree node with requested version id", apierr.GetAPIError(apierr.ErrNoSuchVersion))
}
n.reqLogger(ctx).Debug(logs.GetTreeNodeToDelete, zap.Stringer("cid", bkt.CID), zap.Strings("oids", oids))
@ -785,23 +786,23 @@ func (n *Layer) DeleteObjects(ctx context.Context, p *DeleteObjectParams) []*Ver
func (n *Layer) CreateBucket(ctx context.Context, p *CreateBucketParams) (*data.BucketInfo, error) {
bktInfo, err := n.GetBucketInfo(ctx, p.Name)
if err != nil {
if errors.IsS3Error(err, errors.ErrNoSuchBucket) {
if apierr.IsS3Error(err, apierr.ErrNoSuchBucket) {
return n.createContainer(ctx, p)
}
return nil, err
}
if p.SessionContainerCreation != nil && session.IssuedBy(*p.SessionContainerCreation, bktInfo.Owner) {
return nil, errors.GetAPIError(errors.ErrBucketAlreadyOwnedByYou)
return nil, apierr.GetAPIError(apierr.ErrBucketAlreadyOwnedByYou)
}
return nil, errors.GetAPIError(errors.ErrBucketAlreadyExists)
return nil, apierr.GetAPIError(apierr.ErrBucketAlreadyExists)
}
func (n *Layer) ResolveBucket(ctx context.Context, name string) (cid.ID, error) {
func (n *Layer) ResolveBucket(ctx context.Context, zone, name string) (cid.ID, error) {
var cnrID cid.ID
if err := cnrID.DecodeString(name); err != nil {
if cnrID, err = n.resolver.Resolve(ctx, name); err != nil {
if cnrID, err = n.resolver.Resolve(ctx, zone, name); err != nil {
return cid.ID{}, err
}
@ -822,7 +823,7 @@ func (n *Layer) DeleteBucket(ctx context.Context, p *DeleteBucketParams) error {
return err
}
if len(res) != 0 {
return errors.GetAPIError(errors.ErrBucketNotEmpty)
return apierr.GetAPIError(apierr.ErrBucketNotEmpty)
}
}
@ -854,11 +855,26 @@ func (n *Layer) DeleteBucket(ctx context.Context, p *DeleteBucketParams) error {
return nil
}
func (n *Layer) DeleteContainer(ctx context.Context, p *DeleteBucketParams) error {
n.cache.DeleteBucket(p.BktInfo)
if err := n.frostFS.DeleteContainer(ctx, p.BktInfo.CID, p.SessionToken); err != nil {
return fmt.Errorf("delete container: %w", err)
}
return nil
}
func (n *Layer) GetNetworkInfo(ctx context.Context) (netmap.NetworkInfo, error) {
cachedInfo := n.cache.GetNetworkInfo()
if cachedInfo != nil {
return *cachedInfo, nil
}
networkInfo, err := n.frostFS.NetworkInfo(ctx)
if err != nil {
return networkInfo, fmt.Errorf("get network info: %w", err)
return netmap.NetworkInfo{}, fmt.Errorf("get network info: %w", err)
}
n.cache.PutNetworkInfo(networkInfo)
return networkInfo, nil
}

View file

@ -3,30 +3,33 @@ package layer
import (
"bytes"
"context"
"encoding/base64"
"encoding/xml"
"errors"
"fmt"
"io"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"go.uber.org/zap"
)
type PutBucketLifecycleParams struct {
BktInfo *data.BucketInfo
LifecycleCfg *data.LifecycleConfiguration
LifecycleReader io.Reader
CopiesNumbers []uint32
MD5Hash string
BktInfo *data.BucketInfo
LifecycleCfg *data.LifecycleConfiguration
CopiesNumbers []uint32
}
func (n *Layer) PutBucketLifecycleConfiguration(ctx context.Context, p *PutBucketLifecycleParams) error {
prm := PrmObjectCreate{
Payload: p.LifecycleReader,
cfgBytes, err := xml.Marshal(p.LifecycleCfg)
if err != nil {
return fmt.Errorf("marshal lifecycle configuration: %w", err)
}
prm := frostfs.PrmObjectCreate{
Payload: bytes.NewReader(cfgBytes),
Filepath: p.BktInfo.LifecycleConfigurationObjectName(),
CreationTime: TimeNow(ctx),
}
@ -47,19 +50,8 @@ func (n *Layer) PutBucketLifecycleConfiguration(ctx context.Context, p *PutBucke
return fmt.Errorf("put lifecycle object: %w", err)
}
hashBytes, err := base64.StdEncoding.DecodeString(p.MD5Hash)
if err != nil {
return apiErr.GetAPIError(apiErr.ErrInvalidDigest)
}
if !bytes.Equal(hashBytes, createdObj.MD5Sum) {
n.deleteLifecycleObject(ctx, p.BktInfo, newAddress(lifecycleBkt.CID, createdObj.ID))
return apiErr.GetAPIError(apiErr.ErrInvalidDigest)
}
objsToDelete, err := n.treeService.PutBucketLifecycleConfiguration(ctx, p.BktInfo, newAddress(lifecycleBkt.CID, createdObj.ID))
objsToDeleteNotFound := errors.Is(err, ErrNoNodeToRemove)
objsToDeleteNotFound := errors.Is(err, tree.ErrNoNodeToRemove)
if err != nil && !objsToDeleteNotFound {
return err
}
@ -77,7 +69,7 @@ func (n *Layer) PutBucketLifecycleConfiguration(ctx context.Context, p *PutBucke
// deleteLifecycleObject removes object and logs in case of error.
func (n *Layer) deleteLifecycleObject(ctx context.Context, bktInfo *data.BucketInfo, addr oid.Address) {
var prmAuth PrmAuth
var prmAuth frostfs.PrmAuth
lifecycleBkt := bktInfo
if !addr.Container().Equals(bktInfo.CID) {
lifecycleBkt = &data.BucketInfo{CID: addr.Container()}
@ -98,16 +90,16 @@ func (n *Layer) GetBucketLifecycleConfiguration(ctx context.Context, bktInfo *da
}
addr, err := n.treeService.GetBucketLifecycleConfiguration(ctx, bktInfo)
objNotFound := errors.Is(err, ErrNodeNotFound)
objNotFound := errors.Is(err, tree.ErrNodeNotFound)
if err != nil && !objNotFound {
return nil, err
}
if objNotFound {
return nil, fmt.Errorf("%w: %s", apiErr.GetAPIError(apiErr.ErrNoSuchLifecycleConfiguration), err.Error())
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchLifecycleConfiguration), err.Error())
}
var prmAuth PrmAuth
var prmAuth frostfs.PrmAuth
lifecycleBkt := bktInfo
if !addr.Container().Equals(bktInfo.CID) {
lifecycleBkt = &data.BucketInfo{CID: addr.Container()}
@ -127,12 +119,18 @@ func (n *Layer) GetBucketLifecycleConfiguration(ctx context.Context, bktInfo *da
n.cache.PutLifecycleConfiguration(owner, bktInfo, lifecycleCfg)
for i := range lifecycleCfg.Rules {
if lifecycleCfg.Rules[i].Expiration != nil {
lifecycleCfg.Rules[i].Expiration.Epoch = nil
}
}
return lifecycleCfg, nil
}
func (n *Layer) DeleteBucketLifecycleConfiguration(ctx context.Context, bktInfo *data.BucketInfo) error {
objs, err := n.treeService.DeleteBucketLifecycleConfiguration(ctx, bktInfo)
objsNotFound := errors.Is(err, ErrNoNodeToRemove)
objsNotFound := errors.Is(err, tree.ErrNoNodeToRemove)
if err != nil && !objsNotFound {
return err
}

View file

@ -1,15 +1,13 @@
package layer
import (
"bytes"
"crypto/md5"
"encoding/base64"
"encoding/xml"
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
frostfsErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
frosterr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
"github.com/stretchr/testify/require"
)
@ -36,16 +34,14 @@ func TestBucketLifecycle(t *testing.T) {
hash.Write(lifecycleBytes)
_, err = tc.layer.GetBucketLifecycleConfiguration(tc.ctx, tc.bktInfo)
require.Equal(t, apiErr.GetAPIError(apiErr.ErrNoSuchLifecycleConfiguration), frostfsErrors.UnwrapErr(err))
require.Equal(t, apierr.GetAPIError(apierr.ErrNoSuchLifecycleConfiguration), frosterr.UnwrapErr(err))
err = tc.layer.DeleteBucketLifecycleConfiguration(tc.ctx, tc.bktInfo)
require.NoError(t, err)
err = tc.layer.PutBucketLifecycleConfiguration(tc.ctx, &PutBucketLifecycleParams{
BktInfo: tc.bktInfo,
LifecycleCfg: lifecycle,
LifecycleReader: bytes.NewReader(lifecycleBytes),
MD5Hash: base64.StdEncoding.EncodeToString(hash.Sum(nil)),
BktInfo: tc.bktInfo,
LifecycleCfg: lifecycle,
})
require.NoError(t, err)
@ -57,7 +53,7 @@ func TestBucketLifecycle(t *testing.T) {
require.NoError(t, err)
_, err = tc.layer.GetBucketLifecycleConfiguration(tc.ctx, tc.bktInfo)
require.Equal(t, apiErr.GetAPIError(apiErr.ErrNoSuchLifecycleConfiguration), frostfsErrors.UnwrapErr(err))
require.Equal(t, apierr.GetAPIError(apierr.ErrNoSuchLifecycleConfiguration), frosterr.UnwrapErr(err))
}
func ptr[T any](t T) *T {

View file

@ -11,7 +11,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/cache"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"github.com/panjf2000/ants/v2"
@ -691,10 +691,10 @@ func filterVersionsByMarker(objects []*data.ExtendedNodeVersion, p *ListObjectVe
return objects[j+1:], nil
}
}
return nil, s3errors.GetAPIError(s3errors.ErrInvalidVersion)
return nil, apierr.GetAPIError(apierr.ErrInvalidVersion)
} else if obj.NodeVersion.FilePath > p.KeyMarker {
if p.VersionIDMarker != "" {
return nil, s3errors.GetAPIError(s3errors.ErrInvalidVersion)
return nil, apierr.GetAPIError(apierr.ErrInvalidVersion)
}
return objects[i:], nil
}

View file

@ -17,8 +17,10 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
@ -150,9 +152,9 @@ func (n *Layer) CreateMultipartUpload(ctx context.Context, p *CreateMultipartPar
metaSize += len(p.Data.TagSet)
}
networkInfo, err := n.frostFS.NetworkInfo(ctx)
networkInfo, err := n.GetNetworkInfo(ctx)
if err != nil {
return fmt.Errorf("get network info: %w", err)
return err
}
info := &data.MultipartInfo{
@ -187,14 +189,14 @@ func (n *Layer) CreateMultipartUpload(ctx context.Context, p *CreateMultipartPar
func (n *Layer) UploadPart(ctx context.Context, p *UploadPartParams) (string, error) {
multipartInfo, err := n.treeService.GetMultipartUpload(ctx, p.Info.Bkt, p.Info.Key, p.Info.UploadID)
if err != nil {
if errors.Is(err, ErrNodeNotFound) {
return "", fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchUpload), err.Error())
if errors.Is(err, tree.ErrNodeNotFound) {
return "", fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchUpload), err.Error())
}
return "", err
}
if p.Size > UploadMaxSize {
return "", fmt.Errorf("%w: %d/%d", s3errors.GetAPIError(s3errors.ErrEntityTooLarge), p.Size, UploadMaxSize)
return "", fmt.Errorf("%w: %d/%d", apierr.GetAPIError(apierr.ErrEntityTooLarge), p.Size, UploadMaxSize)
}
objInfo, err := n.uploadPart(ctx, multipartInfo, p)
@ -209,11 +211,11 @@ func (n *Layer) uploadPart(ctx context.Context, multipartInfo *data.MultipartInf
encInfo := FormEncryptionInfo(multipartInfo.Meta)
if err := p.Info.Encryption.MatchObjectEncryption(encInfo); err != nil {
n.reqLogger(ctx).Warn(logs.MismatchedObjEncryptionInfo, zap.Error(err))
return nil, s3errors.GetAPIError(s3errors.ErrInvalidEncryptionParameters)
return nil, apierr.GetAPIError(apierr.ErrInvalidEncryptionParameters)
}
bktInfo := p.Info.Bkt
prm := PrmObjectCreate{
prm := frostfs.PrmObjectCreate{
Container: bktInfo.CID,
Attributes: make([][2]string, 2),
Payload: p.Reader,
@ -242,10 +244,10 @@ func (n *Layer) uploadPart(ctx context.Context, multipartInfo *data.MultipartInf
if len(p.ContentMD5) > 0 {
hashBytes, err := base64.StdEncoding.DecodeString(p.ContentMD5)
if err != nil {
return nil, s3errors.GetAPIError(s3errors.ErrInvalidDigest)
return nil, apierr.GetAPIError(apierr.ErrInvalidDigest)
}
if hex.EncodeToString(hashBytes) != hex.EncodeToString(createdObj.MD5Sum) {
prm := PrmObjectDelete{
prm := frostfs.PrmObjectDelete{
Object: createdObj.ID,
Container: bktInfo.CID,
}
@ -254,7 +256,7 @@ func (n *Layer) uploadPart(ctx context.Context, multipartInfo *data.MultipartInf
if err != nil {
n.reqLogger(ctx).Debug(logs.FailedToDeleteObject, zap.Stringer("cid", bktInfo.CID), zap.Stringer("oid", createdObj.ID))
}
return nil, s3errors.GetAPIError(s3errors.ErrInvalidDigest)
return nil, apierr.GetAPIError(apierr.ErrInvalidDigest)
}
}
if p.Info.Encryption.Enabled() {
@ -264,14 +266,14 @@ func (n *Layer) uploadPart(ctx context.Context, multipartInfo *data.MultipartInf
if !p.Info.Encryption.Enabled() && len(p.ContentSHA256Hash) > 0 && !auth.IsStandardContentSHA256(p.ContentSHA256Hash) {
contentHashBytes, err := hex.DecodeString(p.ContentSHA256Hash)
if err != nil {
return nil, s3errors.GetAPIError(s3errors.ErrContentSHA256Mismatch)
return nil, apierr.GetAPIError(apierr.ErrContentSHA256Mismatch)
}
if !bytes.Equal(contentHashBytes, createdObj.HashSum) {
err = n.objectDelete(ctx, bktInfo, createdObj.ID)
if err != nil {
n.reqLogger(ctx).Debug(logs.FailedToDeleteObject, zap.Stringer("cid", bktInfo.CID), zap.Stringer("oid", createdObj.ID))
}
return nil, s3errors.GetAPIError(s3errors.ErrContentSHA256Mismatch)
return nil, apierr.GetAPIError(apierr.ErrContentSHA256Mismatch)
}
}
@ -291,7 +293,7 @@ func (n *Layer) uploadPart(ctx context.Context, multipartInfo *data.MultipartInf
}
oldPartIDs, err := n.treeService.AddPart(ctx, bktInfo, multipartInfo.ID, partInfo)
oldPartIDNotFound := errors.Is(err, ErrNoNodeToRemove)
oldPartIDNotFound := errors.Is(err, tree.ErrNoNodeToRemove)
if err != nil && !oldPartIDNotFound {
return nil, err
}
@ -323,8 +325,8 @@ func (n *Layer) uploadPart(ctx context.Context, multipartInfo *data.MultipartInf
func (n *Layer) UploadPartCopy(ctx context.Context, p *UploadCopyParams) (*data.ObjectInfo, error) {
multipartInfo, err := n.treeService.GetMultipartUpload(ctx, p.Info.Bkt, p.Info.Key, p.Info.UploadID)
if err != nil {
if errors.Is(err, ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchUpload), err.Error())
if errors.Is(err, tree.ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchUpload), err.Error())
}
return nil, err
}
@ -340,11 +342,11 @@ func (n *Layer) UploadPartCopy(ctx context.Context, p *UploadCopyParams) (*data.
if p.Range != nil {
size = p.Range.End - p.Range.Start + 1
if p.Range.End > srcObjectSize {
return nil, fmt.Errorf("%w: %d-%d/%d", s3errors.GetAPIError(s3errors.ErrInvalidCopyPartRangeSource), p.Range.Start, p.Range.End, srcObjectSize)
return nil, fmt.Errorf("%w: %d-%d/%d", apierr.GetAPIError(apierr.ErrInvalidCopyPartRangeSource), p.Range.Start, p.Range.End, srcObjectSize)
}
}
if size > UploadMaxSize {
return nil, fmt.Errorf("%w: %d/%d", s3errors.GetAPIError(s3errors.ErrEntityTooLarge), size, UploadMaxSize)
return nil, fmt.Errorf("%w: %d/%d", apierr.GetAPIError(apierr.ErrEntityTooLarge), size, UploadMaxSize)
}
objPayload, err := n.GetObject(ctx, &GetObjectParams{
@ -371,7 +373,7 @@ func (n *Layer) UploadPartCopy(ctx context.Context, p *UploadCopyParams) (*data.
func (n *Layer) CompleteMultipartUpload(ctx context.Context, p *CompleteMultipartParams) (*UploadData, *data.ExtendedObjectInfo, error) {
for i := 1; i < len(p.Parts); i++ {
if p.Parts[i].PartNumber <= p.Parts[i-1].PartNumber {
return nil, nil, s3errors.GetAPIError(s3errors.ErrInvalidPartOrder)
return nil, nil, apierr.GetAPIError(apierr.ErrInvalidPartOrder)
}
}
@ -382,7 +384,7 @@ func (n *Layer) CompleteMultipartUpload(ctx context.Context, p *CompleteMultipar
encInfo := FormEncryptionInfo(multipartInfo.Meta)
if len(partsInfo) < len(p.Parts) {
return nil, nil, fmt.Errorf("%w: found %d parts, need %d", s3errors.GetAPIError(s3errors.ErrInvalidPart), len(partsInfo), len(p.Parts))
return nil, nil, fmt.Errorf("%w: found %d parts, need %d", apierr.GetAPIError(apierr.ErrInvalidPart), len(partsInfo), len(p.Parts))
}
var multipartObjetSize uint64
@ -394,12 +396,12 @@ func (n *Layer) CompleteMultipartUpload(ctx context.Context, p *CompleteMultipar
for i, part := range p.Parts {
partInfo := partsInfo.Extract(part.PartNumber, data.UnQuote(part.ETag), n.features.MD5Enabled())
if partInfo == nil {
return nil, nil, fmt.Errorf("%w: unknown part %d or etag mismatched", s3errors.GetAPIError(s3errors.ErrInvalidPart), part.PartNumber)
return nil, nil, fmt.Errorf("%w: unknown part %d or etag mismatched", apierr.GetAPIError(apierr.ErrInvalidPart), part.PartNumber)
}
// for the last part we have no minimum size limit
if i != len(p.Parts)-1 && partInfo.Size < UploadMinSize {
return nil, nil, fmt.Errorf("%w: %d/%d", s3errors.GetAPIError(s3errors.ErrEntityTooSmall), partInfo.Size, UploadMinSize)
return nil, nil, fmt.Errorf("%w: %d/%d", apierr.GetAPIError(apierr.ErrEntityTooSmall), partInfo.Size, UploadMinSize)
}
parts = append(parts, partInfo)
multipartObjetSize += partInfo.Size // even if encryption is enabled size is actual (decrypted)
@ -460,7 +462,7 @@ func (n *Layer) CompleteMultipartUpload(ctx context.Context, p *CompleteMultipar
Object: p.Info.Key,
Reader: bytes.NewReader(partsData),
Header: initMetadata,
Size: multipartObjetSize,
Size: &multipartObjetSize,
Encryption: p.Info.Encryption,
CopiesNumbers: multipartInfo.CopiesNumbers,
CompleteMD5Hash: hex.EncodeToString(md5Hash.Sum(nil)) + "-" + strconv.Itoa(len(p.Parts)),
@ -471,7 +473,7 @@ func (n *Layer) CompleteMultipartUpload(ctx context.Context, p *CompleteMultipar
zap.String("uploadKey", p.Info.Key),
zap.Error(err))
return nil, nil, s3errors.GetAPIError(s3errors.ErrInternalError)
return nil, nil, apierr.GetAPIError(apierr.ErrInternalError)
}
var addr oid.Address
@ -579,7 +581,7 @@ func (n *Layer) ListParts(ctx context.Context, p *ListPartsParams) (*ListPartsIn
encInfo := FormEncryptionInfo(multipartInfo.Meta)
if err = p.Info.Encryption.MatchObjectEncryption(encInfo); err != nil {
n.reqLogger(ctx).Warn(logs.MismatchedObjEncryptionInfo, zap.Error(err))
return nil, s3errors.GetAPIError(s3errors.ErrInvalidEncryptionParameters)
return nil, apierr.GetAPIError(apierr.ErrInvalidEncryptionParameters)
}
res.Owner = multipartInfo.Owner
@ -646,8 +648,8 @@ func (p PartsInfo) Extract(part int, etag string, md5Enabled bool) *data.PartInf
func (n *Layer) getUploadParts(ctx context.Context, p *UploadInfoParams) (*data.MultipartInfo, PartsInfo, error) {
multipartInfo, err := n.treeService.GetMultipartUpload(ctx, p.Bkt, p.Key, p.UploadID)
if err != nil {
if errors.Is(err, ErrNodeNotFound) {
return nil, nil, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchUpload), err.Error())
if errors.Is(err, tree.ErrNodeNotFound) {
return nil, nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchUpload), err.Error())
}
return nil, nil, err
}

View file

@ -12,6 +12,7 @@ import (
"fmt"
"io"
"mime"
"net/http"
"path/filepath"
"strconv"
"strings"
@ -19,8 +20,11 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/detector"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
@ -47,7 +51,7 @@ type (
}
DeleteMarkerError struct {
ErrorCode apiErrors.ErrorCode
ErrorCode apierr.ErrorCode
}
)
@ -68,7 +72,7 @@ func newAddress(cnr cid.ID, obj oid.ID) oid.Address {
// objectHead returns all object's headers.
func (n *Layer) objectHead(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID) (*object.Object, error) {
prm := PrmObjectHead{
prm := frostfs.PrmObjectHead{
Container: bktInfo.CID,
Object: idObj,
}
@ -126,11 +130,11 @@ func (n *Layer) initObjectPayloadReader(ctx context.Context, p getParams) (io.Re
// initializes payload reader of the FrostFS object.
// Zero range corresponds to full payload (panics if only offset is set).
func (n *Layer) initFrostFSObjectPayloadReader(ctx context.Context, p getFrostFSParams) (io.Reader, error) {
var prmAuth PrmAuth
var prmAuth frostfs.PrmAuth
n.prepareAuthParameters(ctx, &prmAuth, p.bktInfo.Owner)
if p.off+p.ln != 0 {
prm := PrmObjectRange{
prm := frostfs.PrmObjectRange{
PrmAuth: prmAuth,
Container: p.bktInfo.CID,
Object: p.oid,
@ -140,7 +144,7 @@ func (n *Layer) initFrostFSObjectPayloadReader(ctx context.Context, p getFrostFS
return n.frostFS.RangeObject(ctx, prm)
}
prm := PrmObjectGet{
prm := frostfs.PrmObjectGet{
PrmAuth: prmAuth,
Container: p.bktInfo.CID,
Object: p.oid,
@ -155,17 +159,17 @@ func (n *Layer) initFrostFSObjectPayloadReader(ctx context.Context, p getFrostFS
}
// objectGet returns an object with payload in the object.
func (n *Layer) objectGet(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID) (*Object, error) {
return n.objectGetBase(ctx, bktInfo, objID, PrmAuth{})
func (n *Layer) objectGet(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID) (*frostfs.Object, error) {
return n.objectGetBase(ctx, bktInfo, objID, frostfs.PrmAuth{})
}
// objectGetWithAuth returns an object with payload in the object. Uses provided PrmAuth.
func (n *Layer) objectGetWithAuth(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID, auth PrmAuth) (*Object, error) {
func (n *Layer) objectGetWithAuth(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID, auth frostfs.PrmAuth) (*frostfs.Object, error) {
return n.objectGetBase(ctx, bktInfo, objID, auth)
}
func (n *Layer) objectGetBase(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID, auth PrmAuth) (*Object, error) {
prm := PrmObjectGet{
func (n *Layer) objectGetBase(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID, auth frostfs.PrmAuth) (*frostfs.Object, error) {
prm := frostfs.PrmObjectGet{
PrmAuth: auth,
Container: bktInfo.CID,
Object: objID,
@ -230,40 +234,46 @@ func (n *Layer) PutObject(ctx context.Context, p *PutObjectParams) (*data.Extend
r := p.Reader
if p.Encryption.Enabled() {
p.Header[AttributeDecryptedSize] = strconv.FormatUint(p.Size, 10)
var size uint64
if p.Size != nil {
size = *p.Size
}
p.Header[AttributeDecryptedSize] = strconv.FormatUint(size, 10)
if err = addEncryptionHeaders(p.Header, p.Encryption); err != nil {
return nil, fmt.Errorf("add encryption header: %w", err)
}
var encSize uint64
if r, encSize, err = encryptionReader(p.Reader, p.Size, p.Encryption.Key()); err != nil {
if r, encSize, err = encryptionReader(p.Reader, size, p.Encryption.Key()); err != nil {
return nil, fmt.Errorf("create encrypter: %w", err)
}
p.Size = encSize
p.Size = &encSize
}
if r != nil {
if len(p.Header[api.ContentType]) == 0 {
if contentType := MimeByFilePath(p.Object); len(contentType) == 0 {
d := newDetector(r)
d := detector.NewDetector(r, http.DetectContentType)
if contentType, err := d.Detect(); err == nil {
p.Header[api.ContentType] = contentType
}
r = d.MultiReader()
r = d.RestoredReader()
} else {
p.Header[api.ContentType] = contentType
}
}
}
prm := PrmObjectCreate{
prm := frostfs.PrmObjectCreate{
Container: p.BktInfo.CID,
PayloadSize: p.Size,
Filepath: p.Object,
Payload: r,
CreationTime: TimeNow(ctx),
CopiesNumber: p.CopiesNumbers,
}
if p.Size != nil {
prm.PayloadSize = *p.Size
}
prm.Attributes = make([][2]string, 0, len(p.Header))
@ -275,31 +285,32 @@ func (n *Layer) PutObject(ctx context.Context, p *PutObjectParams) (*data.Extend
if err != nil {
return nil, err
}
if len(p.ContentMD5) > 0 {
if !p.Encryption.Enabled() && len(p.ContentMD5) > 0 {
headerMd5Hash, err := base64.StdEncoding.DecodeString(p.ContentMD5)
if err != nil {
return nil, apiErrors.GetAPIError(apiErrors.ErrInvalidDigest)
return nil, apierr.GetAPIError(apierr.ErrInvalidDigest)
}
if !bytes.Equal(headerMd5Hash, createdObj.MD5Sum) {
err = n.objectDelete(ctx, p.BktInfo, createdObj.ID)
if err != nil {
n.reqLogger(ctx).Debug(logs.FailedToDeleteObject, zap.Stringer("cid", p.BktInfo.CID), zap.Stringer("oid", createdObj.ID))
}
return nil, apiErrors.GetAPIError(apiErrors.ErrInvalidDigest)
return nil, apierr.GetAPIError(apierr.ErrInvalidDigest)
}
}
if !p.Encryption.Enabled() && len(p.ContentSHA256Hash) > 0 && !auth.IsStandardContentSHA256(p.ContentSHA256Hash) {
contentHashBytes, err := hex.DecodeString(p.ContentSHA256Hash)
if err != nil {
return nil, apiErrors.GetAPIError(apiErrors.ErrContentSHA256Mismatch)
return nil, apierr.GetAPIError(apierr.ErrContentSHA256Mismatch)
}
if !bytes.Equal(contentHashBytes, createdObj.HashSum) {
err = n.objectDelete(ctx, p.BktInfo, createdObj.ID)
if err != nil {
n.reqLogger(ctx).Debug(logs.FailedToDeleteObject, zap.Stringer("cid", p.BktInfo.CID), zap.Stringer("oid", createdObj.ID))
}
return nil, apiErrors.GetAPIError(apiErrors.ErrContentSHA256Mismatch)
return nil, apierr.GetAPIError(apierr.ErrContentSHA256Mismatch)
}
}
@ -310,7 +321,6 @@ func (n *Layer) PutObject(ctx context.Context, p *PutObjectParams) (*data.Extend
OID: createdObj.ID,
ETag: hex.EncodeToString(createdObj.HashSum),
FilePath: p.Object,
Size: p.Size,
Created: &now,
Owner: &n.gateOwner,
CreationEpoch: createdObj.CreationEpoch,
@ -318,12 +328,19 @@ func (n *Layer) PutObject(ctx context.Context, p *PutObjectParams) (*data.Extend
IsUnversioned: !bktSettings.VersioningEnabled(),
IsCombined: p.Header[MultipartObjectSize] != "",
}
if len(p.CompleteMD5Hash) > 0 {
newVersion.MD5 = p.CompleteMD5Hash
} else {
newVersion.MD5 = hex.EncodeToString(createdObj.MD5Sum)
}
if p.Size != nil {
newVersion.Size = *p.Size
} else {
newVersion.Size = createdObj.Size
}
if newVersion.ID, err = n.treeService.AddVersion(ctx, p.BktInfo, newVersion); err != nil {
return nil, fmt.Errorf("couldn't add new verion to tree service: %w", err)
}
@ -380,20 +397,20 @@ func (n *Layer) headLastVersionIfNotDeleted(ctx context.Context, bkt *data.Bucke
node, err := n.treeService.GetLatestVersion(ctx, bkt, objectName)
if err != nil {
if errors.Is(err, ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrNoSuchKey), err.Error())
if errors.Is(err, tree.ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
}
return nil, err
}
if node.IsDeleteMarker {
return nil, DeleteMarkerError{ErrorCode: apiErrors.ErrNoSuchKey}
return nil, DeleteMarkerError{ErrorCode: apierr.ErrNoSuchKey}
}
meta, err := n.objectHead(ctx, bkt, node.OID)
if err != nil {
if client.IsErrObjectNotFound(err) {
return nil, fmt.Errorf("%w: %s; %s", apiErrors.GetAPIError(apiErrors.ErrNoSuchKey), err.Error(), node.OID.EncodeToString())
return nil, fmt.Errorf("%w: %s; %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error(), node.OID.EncodeToString())
}
return nil, err
}
@ -416,8 +433,8 @@ func (n *Layer) headVersion(ctx context.Context, bkt *data.BucketInfo, p *HeadOb
if p.VersionID == data.UnversionedObjectVersionID {
foundVersion, err = n.treeService.GetUnversioned(ctx, bkt, p.Object)
if err != nil {
if errors.Is(err, ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrNoSuchVersion), err.Error())
if errors.Is(err, tree.ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchVersion), err.Error())
}
return nil, err
}
@ -434,7 +451,7 @@ func (n *Layer) headVersion(ctx context.Context, bkt *data.BucketInfo, p *HeadOb
}
}
if foundVersion == nil {
return nil, fmt.Errorf("%w: there isn't tree node with requested version id", apiErrors.GetAPIError(apiErrors.ErrNoSuchVersion))
return nil, fmt.Errorf("%w: there isn't tree node with requested version id", apierr.GetAPIError(apierr.ErrNoSuchVersion))
}
}
@ -444,13 +461,13 @@ func (n *Layer) headVersion(ctx context.Context, bkt *data.BucketInfo, p *HeadOb
}
if foundVersion.IsDeleteMarker {
return nil, DeleteMarkerError{ErrorCode: apiErrors.ErrMethodNotAllowed}
return nil, DeleteMarkerError{ErrorCode: apierr.ErrMethodNotAllowed}
}
meta, err := n.objectHead(ctx, bkt, foundVersion.OID)
if err != nil {
if client.IsErrObjectNotFound(err) {
return nil, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrNoSuchVersion), err.Error())
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchVersion), err.Error())
}
return nil, err
}
@ -469,16 +486,16 @@ func (n *Layer) headVersion(ctx context.Context, bkt *data.BucketInfo, p *HeadOb
// objectDelete puts tombstone object into frostfs.
func (n *Layer) objectDelete(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID) error {
return n.objectDeleteBase(ctx, bktInfo, idObj, PrmAuth{})
return n.objectDeleteBase(ctx, bktInfo, idObj, frostfs.PrmAuth{})
}
// objectDeleteWithAuth puts tombstone object into frostfs. Uses provided PrmAuth.
func (n *Layer) objectDeleteWithAuth(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID, auth PrmAuth) error {
func (n *Layer) objectDeleteWithAuth(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID, auth frostfs.PrmAuth) error {
return n.objectDeleteBase(ctx, bktInfo, idObj, auth)
}
func (n *Layer) objectDeleteBase(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID, auth PrmAuth) error {
prm := PrmObjectDelete{
func (n *Layer) objectDeleteBase(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID, auth frostfs.PrmAuth) error {
prm := frostfs.PrmObjectDelete{
PrmAuth: auth,
Container: bktInfo.CID,
Object: idObj,
@ -492,7 +509,7 @@ func (n *Layer) objectDeleteBase(ctx context.Context, bktInfo *data.BucketInfo,
}
// objectPutAndHash prepare auth parameters and invoke frostfs.CreateObject.
func (n *Layer) objectPutAndHash(ctx context.Context, prm PrmObjectCreate, bktInfo *data.BucketInfo) (*data.CreatedObjectInfo, error) {
func (n *Layer) objectPutAndHash(ctx context.Context, prm frostfs.PrmObjectCreate, bktInfo *data.BucketInfo) (*data.CreatedObjectInfo, error) {
n.prepareAuthParameters(ctx, &prm.PrmAuth, bktInfo.Owner)
prm.ClientCut = n.features.ClientCut()
prm.BufferMaxSize = n.features.BufferMaxSizeForPut()

View file

@ -8,6 +8,7 @@ import (
"io"
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"github.com/stretchr/testify/require"
)
@ -37,7 +38,7 @@ func TestGoroutinesDontLeakInPutAndHash(t *testing.T) {
require.NoError(t, err)
payload := bytes.NewReader(content)
prm := PrmObjectCreate{
prm := frostfs.PrmObjectCreate{
Filepath: tc.obj,
Payload: payload,
}

View file

@ -12,6 +12,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
)
type PatchObjectParams struct {
@ -32,7 +33,7 @@ func (n *Layer) PatchObject(ctx context.Context, p *PatchObjectParams) (*data.Ex
return n.patchMultipartObject(ctx, p)
}
prmPatch := PrmObjectPatch{
prmPatch := frostfs.PrmObjectPatch{
Container: p.BktInfo.CID,
Object: p.Object.ObjectInfo.ID,
Payload: p.NewBytes,
@ -74,13 +75,13 @@ func (n *Layer) PatchObject(ctx context.Context, p *PatchObjectParams) (*data.Ex
return p.Object, nil
}
func (n *Layer) patchObject(ctx context.Context, p PrmObjectPatch) (*data.CreatedObjectInfo, error) {
func (n *Layer) patchObject(ctx context.Context, p frostfs.PrmObjectPatch) (*data.CreatedObjectInfo, error) {
objID, err := n.frostFS.PatchObject(ctx, p)
if err != nil {
return nil, fmt.Errorf("patch object: %w", err)
}
prmHead := PrmObjectHead{
prmHead := frostfs.PrmObjectHead{
PrmAuth: p.PrmAuth,
Container: p.Container,
Object: objID,
@ -110,7 +111,7 @@ func (n *Layer) patchMultipartObject(ctx context.Context, p *PatchObjectParams)
return nil, fmt.Errorf("unmarshal combined object parts: %w", err)
}
prmPatch := PrmObjectPatch{
prmPatch := frostfs.PrmObjectPatch{
Container: p.BktInfo.CID,
}
n.prepareAuthParameters(ctx, &prmPatch.PrmAuth, p.BktInfo.Owner)
@ -144,13 +145,13 @@ func (n *Layer) patchMultipartObject(ctx context.Context, p *PatchObjectParams)
}
// Returns patched part info, updated offset and length.
func (n *Layer) patchPart(ctx context.Context, part *data.PartInfo, p *PatchObjectParams, prmPatch *PrmObjectPatch, off, ln uint64, lastPart bool) (*data.CreatedObjectInfo, uint64, uint64, error) {
func (n *Layer) patchPart(ctx context.Context, part *data.PartInfo, p *PatchObjectParams, prmPatch *frostfs.PrmObjectPatch, off, ln uint64, lastPart bool) (*data.CreatedObjectInfo, uint64, uint64, error) {
if off == 0 && ln >= part.Size {
curLen := part.Size
if lastPart {
curLen = ln
}
prm := PrmObjectCreate{
prm := frostfs.PrmObjectCreate{
Container: p.BktInfo.CID,
Payload: io.LimitReader(p.NewBytes, int64(curLen)),
CreationTime: part.Created,
@ -204,7 +205,7 @@ func (n *Layer) updateCombinedObject(ctx context.Context, parts []*data.PartInfo
headerParts.WriteString(headerPart)
}
prm := PrmObjectCreate{
prm := frostfs.PrmObjectCreate{
Container: p.BktInfo.CID,
PayloadSize: fullObjSize,
Filepath: p.Object.ObjectInfo.Name,

View file

@ -3,7 +3,7 @@ package layer
import (
"context"
"encoding/xml"
errorsStd "errors"
"errors"
"fmt"
"math"
"strconv"
@ -11,7 +11,9 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
)
@ -40,7 +42,7 @@ func (n *Layer) PutLockInfo(ctx context.Context, p *PutLockInfoParams) (err erro
}
lockInfo, err := n.treeService.GetLock(ctx, p.ObjVersion.BktInfo, versionNode.ID)
if err != nil && !errorsStd.Is(err, ErrNodeNotFound) {
if err != nil && !errors.Is(err, tree.ErrNodeNotFound) {
return err
}
@ -113,7 +115,7 @@ func (n *Layer) getNodeVersionFromCacheOrFrostfs(ctx context.Context, objVersion
}
func (n *Layer) putLockObject(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID, lock *data.ObjectLock, copiesNumber []uint32) (oid.ID, error) {
prm := PrmObjectCreate{
prm := frostfs.PrmObjectCreate{
Container: bktInfo.CID,
Locks: []oid.ID{objID},
CreationTime: TimeNow(ctx),
@ -146,7 +148,7 @@ func (n *Layer) GetLockInfo(ctx context.Context, objVersion *data.ObjectVersion)
}
lockInfo, err := n.treeService.GetLock(ctx, objVersion.BktInfo, versionNode.ID)
if err != nil && !errorsStd.Is(err, ErrNodeNotFound) {
if err != nil && !errors.Is(err, tree.ErrNodeNotFound) {
return nil, err
}
if lockInfo == nil {
@ -165,16 +167,16 @@ func (n *Layer) getCORS(ctx context.Context, bkt *data.BucketInfo) (*data.CORSCo
}
addr, err := n.treeService.GetBucketCORS(ctx, bkt)
objNotFound := errorsStd.Is(err, ErrNodeNotFound)
objNotFound := errors.Is(err, tree.ErrNodeNotFound)
if err != nil && !objNotFound {
return nil, err
}
if objNotFound {
return nil, fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrNoSuchCORSConfiguration), err.Error())
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchCORSConfiguration), err.Error())
}
var prmAuth PrmAuth
var prmAuth frostfs.PrmAuth
corsBkt := bkt
if !addr.Container().Equals(bkt.CID) && !addr.Container().Equals(cid.ID{}) {
corsBkt = &data.BucketInfo{CID: addr.Container()}
@ -209,7 +211,7 @@ func (n *Layer) GetBucketSettings(ctx context.Context, bktInfo *data.BucketInfo)
settings, err := n.treeService.GetSettingsNode(ctx, bktInfo)
if err != nil {
if !errorsStd.Is(err, ErrNodeNotFound) {
if !errors.Is(err, tree.ErrNodeNotFound) {
return nil, err
}
settings = &data.BucketSettings{Versioning: data.VersioningUnversioned}

View file

@ -6,7 +6,8 @@ import (
"fmt"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
@ -39,8 +40,8 @@ func (n *Layer) GetObjectTagging(ctx context.Context, p *data.GetObjectTaggingPa
tags, err := n.treeService.GetObjectTagging(ctx, p.ObjectVersion.BktInfo, nodeVersion)
if err != nil {
if errors.Is(err, ErrNodeNotFound) {
return "", nil, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchKey), err.Error())
if errors.Is(err, tree.ErrNodeNotFound) {
return "", nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
}
return "", nil, err
}
@ -62,8 +63,8 @@ func (n *Layer) PutObjectTagging(ctx context.Context, p *data.PutObjectTaggingPa
err = n.treeService.PutObjectTagging(ctx, p.ObjectVersion.BktInfo, nodeVersion, p.TagSet)
if err != nil {
if errors.Is(err, ErrNodeNotFound) {
return fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchKey), err.Error())
if errors.Is(err, tree.ErrNodeNotFound) {
return fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
}
return err
}
@ -81,8 +82,8 @@ func (n *Layer) DeleteObjectTagging(ctx context.Context, p *data.ObjectVersion)
err = n.treeService.DeleteObjectTagging(ctx, p.BktInfo, version)
if err != nil {
if errors.Is(err, ErrNodeNotFound) {
return fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchKey), err.Error())
if errors.Is(err, tree.ErrNodeNotFound) {
return fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
}
return err
}
@ -102,7 +103,7 @@ func (n *Layer) GetBucketTagging(ctx context.Context, bktInfo *data.BucketInfo)
}
tags, err := n.treeService.GetBucketTagging(ctx, bktInfo)
if err != nil && !errors.Is(err, ErrNodeNotFound) {
if err != nil && !errors.Is(err, tree.ErrNodeNotFound) {
return nil, err
}
@ -155,14 +156,14 @@ func (n *Layer) getNodeVersion(ctx context.Context, objVersion *data.ObjectVersi
}
}
if version == nil {
err = fmt.Errorf("%w: there isn't tree node with requested version id", s3errors.GetAPIError(s3errors.ErrNoSuchVersion))
err = fmt.Errorf("%w: there isn't tree node with requested version id", apierr.GetAPIError(apierr.ErrNoSuchVersion))
}
}
if err == nil && version.IsDeleteMarker && !objVersion.NoErrorOnDeleteMarker {
return nil, fmt.Errorf("%w: found version is delete marker", s3errors.GetAPIError(s3errors.ErrNoSuchKey))
} else if errors.Is(err, ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchKey), err.Error())
return nil, fmt.Errorf("%w: found version is delete marker", apierr.GetAPIError(apierr.ErrNoSuchKey))
} else if errors.Is(err, tree.ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
}
if err == nil && version != nil && !version.IsDeleteMarker {

View file

@ -1,4 +1,4 @@
package layer
package tree
import (
"context"
@ -8,8 +8,8 @@ import (
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
)
// TreeService provide interface to interact with tree service using s3 data models.
type TreeService interface {
// Service provide interface to interact with tree service using s3 data models.
type Service interface {
// PutSettingsNode update or create new settings node in tree service.
PutSettingsNode(ctx context.Context, bktInfo *data.BucketInfo, settings *data.BucketSettings) error

View file

@ -9,6 +9,7 @@ import (
"time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
)
@ -105,7 +106,7 @@ func (t *TreeServiceMock) PutSettingsNode(_ context.Context, bktInfo *data.Bucke
func (t *TreeServiceMock) GetSettingsNode(_ context.Context, bktInfo *data.BucketInfo) (*data.BucketSettings, error) {
settings, ok := t.settings[bktInfo.CID.EncodeToString()]
if !ok {
return nil, ErrNodeNotFound
return nil, tree.ErrNodeNotFound
}
return settings, nil
@ -140,7 +141,7 @@ func (t *TreeServiceMock) PutBucketCORS(_ context.Context, bktInfo *data.BucketI
t.system[bktInfo.CID.EncodeToString()] = systemMap
return nil, ErrNoNodeToRemove
return nil, tree.ErrNoNodeToRemove
}
func (t *TreeServiceMock) DeleteBucketCORS(context.Context, *data.BucketInfo) ([]oid.Address, error) {
@ -150,12 +151,12 @@ func (t *TreeServiceMock) DeleteBucketCORS(context.Context, *data.BucketInfo) ([
func (t *TreeServiceMock) GetVersions(_ context.Context, bktInfo *data.BucketInfo, objectName string) ([]*data.NodeVersion, error) {
cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()]
if !ok {
return nil, ErrNodeNotFound
return nil, tree.ErrNodeNotFound
}
versions, ok := cnrVersionsMap[objectName]
if !ok {
return nil, ErrNodeNotFound
return nil, tree.ErrNodeNotFound
}
return versions, nil
@ -164,12 +165,12 @@ func (t *TreeServiceMock) GetVersions(_ context.Context, bktInfo *data.BucketInf
func (t *TreeServiceMock) GetLatestVersion(_ context.Context, bktInfo *data.BucketInfo, objectName string) (*data.NodeVersion, error) {
cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()]
if !ok {
return nil, ErrNodeNotFound
return nil, tree.ErrNodeNotFound
}
versions, ok := cnrVersionsMap[objectName]
if !ok {
return nil, ErrNodeNotFound
return nil, tree.ErrNodeNotFound
}
sort.Slice(versions, func(i, j int) bool {
@ -180,13 +181,13 @@ func (t *TreeServiceMock) GetLatestVersion(_ context.Context, bktInfo *data.Buck
return versions[len(versions)-1], nil
}
return nil, ErrNodeNotFound
return nil, tree.ErrNodeNotFound
}
func (t *TreeServiceMock) InitVersionsByPrefixStream(_ context.Context, bktInfo *data.BucketInfo, prefix string, latestOnly bool) (data.VersionsStream, error) {
cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()]
if !ok {
return nil, ErrNodeNotFound
return nil, tree.ErrNodeNotFound
}
var result []*data.NodeVersion
@ -218,12 +219,12 @@ func (t *TreeServiceMock) InitVersionsByPrefixStream(_ context.Context, bktInfo
func (t *TreeServiceMock) GetUnversioned(_ context.Context, bktInfo *data.BucketInfo, objectName string) (*data.NodeVersion, error) {
cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()]
if !ok {
return nil, ErrNodeNotFound
return nil, tree.ErrNodeNotFound
}
versions, ok := cnrVersionsMap[objectName]
if !ok {
return nil, ErrNodeNotFound
return nil, tree.ErrNodeNotFound
}
for _, version := range versions {
@ -232,7 +233,7 @@ func (t *TreeServiceMock) GetUnversioned(_ context.Context, bktInfo *data.Bucket
}
}
return nil, ErrNodeNotFound
return nil, tree.ErrNodeNotFound
}
func (t *TreeServiceMock) AddVersion(_ context.Context, bktInfo *data.BucketInfo, newVersion *data.NodeVersion) (uint64, error) {
@ -278,7 +279,7 @@ func (t *TreeServiceMock) AddVersion(_ context.Context, bktInfo *data.BucketInfo
func (t *TreeServiceMock) RemoveVersion(_ context.Context, bktInfo *data.BucketInfo, nodeID uint64) error {
cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()]
if !ok {
return ErrNodeNotFound
return tree.ErrNodeNotFound
}
for key, versions := range cnrVersionsMap {
@ -290,7 +291,7 @@ func (t *TreeServiceMock) RemoveVersion(_ context.Context, bktInfo *data.BucketI
}
}
return ErrNodeNotFound
return tree.ErrNodeNotFound
}
func (t *TreeServiceMock) GetAllVersionsByPrefix(_ context.Context, bktInfo *data.BucketInfo, prefix string) ([]*data.NodeVersion, error) {
@ -334,7 +335,7 @@ func (t *TreeServiceMock) GetMultipartUploadsByPrefix(context.Context, *data.Buc
func (t *TreeServiceMock) GetMultipartUpload(_ context.Context, bktInfo *data.BucketInfo, objectName, uploadID string) (*data.MultipartInfo, error) {
cnrMultipartsMap, ok := t.multiparts[bktInfo.CID.EncodeToString()]
if !ok {
return nil, ErrNodeNotFound
return nil, tree.ErrNodeNotFound
}
multiparts := cnrMultipartsMap[objectName]
@ -344,7 +345,7 @@ func (t *TreeServiceMock) GetMultipartUpload(_ context.Context, bktInfo *data.Bu
}
}
return nil, ErrNodeNotFound
return nil, tree.ErrNodeNotFound
}
func (t *TreeServiceMock) AddPart(ctx context.Context, bktInfo *data.BucketInfo, multipartNodeID uint64, info *data.PartInfo) (oldObjIDsToDelete []oid.ID, err error) {
@ -387,7 +388,7 @@ LOOP:
}
if foundMultipart == nil {
return nil, ErrNodeNotFound
return nil, tree.ErrNodeNotFound
}
partsMap := t.parts[foundMultipart.UploadID]
@ -411,18 +412,18 @@ func (t *TreeServiceMock) PutBucketLifecycleConfiguration(_ context.Context, bkt
t.system[bktInfo.CID.EncodeToString()] = systemMap
return nil, ErrNoNodeToRemove
return nil, tree.ErrNoNodeToRemove
}
func (t *TreeServiceMock) GetBucketLifecycleConfiguration(_ context.Context, bktInfo *data.BucketInfo) (oid.Address, error) {
systemMap, ok := t.system[bktInfo.CID.EncodeToString()]
if !ok {
return oid.Address{}, ErrNodeNotFound
return oid.Address{}, tree.ErrNodeNotFound
}
node, ok := systemMap["lifecycle"]
if !ok {
return oid.Address{}, ErrNodeNotFound
return oid.Address{}, tree.ErrNodeNotFound
}
return newAddress(bktInfo.CID, node.OID), nil
@ -431,12 +432,12 @@ func (t *TreeServiceMock) GetBucketLifecycleConfiguration(_ context.Context, bkt
func (t *TreeServiceMock) DeleteBucketLifecycleConfiguration(_ context.Context, bktInfo *data.BucketInfo) ([]oid.Address, error) {
systemMap, ok := t.system[bktInfo.CID.EncodeToString()]
if !ok {
return nil, ErrNoNodeToRemove
return nil, tree.ErrNoNodeToRemove
}
node, ok := systemMap["lifecycle"]
if !ok {
return nil, ErrNoNodeToRemove
return nil, tree.ErrNoNodeToRemove
}
delete(systemMap, "lifecycle")
@ -461,7 +462,7 @@ LOOP:
}
if uploadID == "" {
return ErrNodeNotFound
return tree.ErrNodeNotFound
}
delete(t.parts, uploadID)

View file

@ -7,6 +7,7 @@ import (
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
bearertest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer/test"
@ -23,7 +24,7 @@ func (tc *testContext) putObject(content []byte) *data.ObjectInfo {
extObjInfo, err := tc.layer.PutObject(tc.ctx, &PutObjectParams{
BktInfo: tc.bktInfo,
Object: tc.obj,
Size: uint64(len(content)),
Size: ptr(uint64(len(content))),
Reader: bytes.NewReader(content),
Header: make(map[string]string),
})
@ -154,7 +155,7 @@ func prepareContext(t *testing.T, cachesConfig ...*CachesConfig) *testContext {
tp := NewTestFrostFS(key)
bktName := "testbucket1"
res, err := tp.CreateContainer(ctx, PrmContainerCreate{
res, err := tp.CreateContainer(ctx, frostfs.PrmContainerCreate{
Name: bktName,
})
require.NoError(t, err)

View file

@ -3,14 +3,18 @@ package middleware
import (
"net/http"
"net/url"
"strconv"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"go.uber.org/zap"
)
const wildcardPlaceholder = "<wildcard>"
const (
wildcardPlaceholder = "<wildcard>"
enabledVHS = "enabled"
disabledVHS = "disabled"
)
type VHSSettings interface {
Domains() []string
@ -26,9 +30,9 @@ func PrepareAddressStyle(settings VHSSettings, log *zap.Logger) Func {
ctx := r.Context()
reqInfo := GetReqInfo(ctx)
reqLogger := reqLogOrDefault(ctx, log)
headerVHSEnabled := r.Header.Get(settings.VHSHeader())
statusVHS := r.Header.Get(settings.VHSHeader())
if isVHSAddress(headerVHSEnabled, settings.GlobalVHS(), settings.VHSNamespacesEnabled(), reqInfo.Namespace) {
if isVHSAddress(statusVHS, settings.GlobalVHS(), settings.VHSNamespacesEnabled(), reqInfo.Namespace) {
prepareVHSAddress(reqInfo, r, settings)
} else {
preparePathStyleAddress(reqInfo, r, reqLogger)
@ -39,17 +43,20 @@ func PrepareAddressStyle(settings VHSSettings, log *zap.Logger) Func {
}
}
func isVHSAddress(headerVHSEnabled string, enabledFlag bool, vhsNamespaces map[string]bool, namespace string) bool {
if result, err := strconv.ParseBool(headerVHSEnabled); err == nil {
func isVHSAddress(statusVHS string, enabledFlag bool, vhsNamespaces map[string]bool, namespace string) bool {
switch statusVHS {
case enabledVHS:
return true
case disabledVHS:
return false
default:
result := enabledFlag
if v, ok := vhsNamespaces[namespace]; ok {
result = v
}
return result
}
result := enabledFlag
if v, ok := vhsNamespaces[namespace]; ok {
result = v
}
return result
}
func prepareVHSAddress(reqInfo *ReqInfo, r *http.Request, settings VHSSettings) {

View file

@ -41,12 +41,12 @@ func (v *VHSSettingsMock) VHSNamespacesEnabled() map[string]bool {
func TestIsVHSAddress(t *testing.T) {
for _, tc := range []struct {
name string
headerVHSEnabled string
vhsEnabledFlag bool
vhsNamespaced map[string]bool
namespace string
expected bool
name string
headerStatusVHS string
vhsEnabledFlag bool
vhsNamespaced map[string]bool
namespace string
expected bool
}{
{
name: "vhs disabled",
@ -75,9 +75,9 @@ func TestIsVHSAddress(t *testing.T) {
expected: true,
},
{
name: "vhs enabled (header)",
headerVHSEnabled: "true",
vhsEnabledFlag: false,
name: "vhs enabled (header)",
headerStatusVHS: enabledVHS,
vhsEnabledFlag: false,
vhsNamespaced: map[string]bool{
"kapusta": false,
},
@ -85,9 +85,9 @@ func TestIsVHSAddress(t *testing.T) {
expected: true,
},
{
name: "vhs disabled (header)",
headerVHSEnabled: "false",
vhsEnabledFlag: true,
name: "vhs disabled (header)",
headerStatusVHS: disabledVHS,
vhsEnabledFlag: true,
vhsNamespaced: map[string]bool{
"kapusta": true,
},
@ -96,7 +96,7 @@ func TestIsVHSAddress(t *testing.T) {
},
} {
t.Run(tc.name, func(t *testing.T) {
actual := isVHSAddress(tc.headerVHSEnabled, tc.vhsEnabledFlag, tc.vhsNamespaced, tc.namespace)
actual := isVHSAddress(tc.headerStatusVHS, tc.vhsEnabledFlag, tc.vhsNamespaced, tc.namespace)
require.Equal(t, tc.expected, actual)
})
}

View file

@ -8,9 +8,8 @@ import (
"time"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/acl"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
frostfsErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
@ -57,9 +56,9 @@ func Auth(center Center, log *zap.Logger) Func {
reqLogOrDefault(ctx, log).Debug(logs.CouldntReceiveAccessBoxForGateKeyRandomKeyWillBeUsed, zap.Error(err))
} else {
reqLogOrDefault(ctx, log).Error(logs.FailedToPassAuthentication, zap.Error(err))
err = frostfsErrors.UnwrapErr(err)
if _, ok := err.(apiErrors.Error); !ok {
err = apiErrors.GetAPIError(apiErrors.ErrAccessDenied)
err = apierr.TransformToS3Error(err)
if err.(apierr.Error).ErrCode == apierr.ErrInternalError {
err = apierr.GetAPIError(apierr.ErrAccessDenied)
}
if _, wrErr := WriteErrorResponse(w, GetReqInfo(r.Context()), err); wrErr != nil {
reqLogOrDefault(ctx, log).Error(logs.FailedToWriteResponse, zap.Error(wrErr))

View file

@ -107,3 +107,9 @@ const (
PartNumberQuery = "partNumber"
LegalHoldQuery = "legal-hold"
)
const (
StdoutPath = "stdout"
StderrPath = "stderr"
SinkName = "lumberjack"
)

237
api/middleware/log_http.go Normal file
View file

@ -0,0 +1,237 @@
//go:build loghttp
package middleware
import (
"bytes"
"io"
"net/http"
"os"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/detector"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/xmlutils"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
"gopkg.in/natefinch/lumberjack.v2"
)
type (
LogHTTPSettings interface {
LogHTTPConfig() LogHTTPConfig
}
LogHTTPConfig struct {
Enabled bool
MaxBody int64
MaxLogSize int
OutputPath string
UseGzip bool
log *httpLogger
}
httpLogger struct {
*zap.Logger
logRoller *lumberjack.Logger
}
// responseReadWriter helps read http response body.
responseReadWriter struct {
http.ResponseWriter
response *bytes.Buffer
statusCode int
}
)
const (
payloadLabel = "payload"
responseLabel = "response"
)
func (lc *LogHTTPConfig) InitHTTPLogger(log *zap.Logger) {
if err := lc.initHTTPLogger(); err != nil {
log.Error(logs.FailedToInitializeHTTPLogger, zap.Error(err))
}
}
// initHTTPLogger returns registers zap sink and returns new httpLogger.
func (lc *LogHTTPConfig) initHTTPLogger() (err error) {
lc.log = &httpLogger{
Logger: zap.NewNop(),
logRoller: &lumberjack.Logger{},
}
c := newLoggerConfig()
lc.log.Logger, err = c.Build()
if err != nil {
return err
}
lc.setLogOutput()
return nil
}
// newLoggerConfig creates new zap.Config with disabled base fields.
func newLoggerConfig() zap.Config {
c := zap.NewProductionConfig()
c.DisableCaller = true
c.DisableStacktrace = true
c.EncoderConfig = newEncoderConfig()
c.Sampling = nil
return c
}
func (lc *LogHTTPConfig) setLogOutput() {
var output zapcore.WriteSyncer
switch lc.OutputPath {
case "", StdoutPath:
output = zapcore.AddSync(os.Stdout)
case StderrPath:
output = zapcore.AddSync(os.Stderr)
default:
output = zapcore.AddSync(&lumberjack.Logger{
Filename: lc.OutputPath,
MaxSize: lc.MaxLogSize,
Compress: lc.UseGzip,
})
}
// create logger with new sync
lc.log.Logger = lc.log.Logger.WithOptions(zap.WrapCore(func(core zapcore.Core) zapcore.Core {
return zapcore.NewCore(zapcore.NewJSONEncoder(newEncoderConfig()), output, zapcore.InfoLevel)
}))
}
func newEncoderConfig() zapcore.EncoderConfig {
c := zap.NewProductionEncoderConfig()
c.MessageKey = zapcore.OmitKey
c.LevelKey = zapcore.OmitKey
c.TimeKey = zapcore.OmitKey
c.FunctionKey = zapcore.OmitKey
return c
}
func (ww *responseReadWriter) Write(data []byte) (int, error) {
ww.response.Write(data)
return ww.ResponseWriter.Write(data)
}
func (ww *responseReadWriter) WriteHeader(code int) {
ww.statusCode = code
ww.ResponseWriter.WriteHeader(code)
}
func (ww *responseReadWriter) Flush() {
if f, ok := ww.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
// LogHTTP logs http parameters from s3 request.
func LogHTTP(l *zap.Logger, settings LogHTTPSettings) Func {
return func(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
config := settings.LogHTTPConfig()
if !config.Enabled || config.log == nil {
h.ServeHTTP(w, r)
return
}
httplog := config.log.getHTTPLogger(r).
withFieldIfExist("query", r.URL.Query()).
withFieldIfExist("headers", r.Header)
payload := getBody(r.Body, l)
r.Body = io.NopCloser(bytes.NewReader(payload))
payloadReader := io.LimitReader(bytes.NewReader(payload), config.MaxBody)
httplog = httplog.withProcessedBody(payloadLabel, payloadReader, l)
wr := newResponseReadWriter(w)
h.ServeHTTP(wr, r)
respReader := io.LimitReader(wr.response, config.MaxBody)
httplog = httplog.withProcessedBody(responseLabel, respReader, l)
httplog = httplog.with(zap.Int("status", wr.statusCode))
httplog.Info(logs.LogHTTP)
})
}
}
// withFieldIfExist checks whether data is not empty and attach it to log output.
func (lg *httpLogger) withFieldIfExist(label string, data map[string][]string) *httpLogger {
if len(data) != 0 {
return lg.with(zap.Any(label, data))
}
return lg
}
func (lg *httpLogger) with(fields ...zap.Field) *httpLogger {
return &httpLogger{
Logger: lg.Logger.With(fields...),
logRoller: lg.logRoller,
}
}
func (lg *httpLogger) getHTTPLogger(r *http.Request) *httpLogger {
return lg.with(
zap.String("from", r.RemoteAddr),
zap.String("URI", r.RequestURI),
zap.String("method", r.Method),
zap.String("protocol", r.Proto),
)
}
func (lg *httpLogger) withProcessedBody(label string, bodyReader io.Reader, l *zap.Logger) *httpLogger {
resp, err := processBody(bodyReader)
if err != nil {
l.Error(logs.FailedToProcessHTTPBody,
zap.Error(err),
zap.String("body type", payloadLabel))
return lg
}
return lg.with(zap.ByteString(label, resp))
}
func newResponseReadWriter(w http.ResponseWriter) *responseReadWriter {
return &responseReadWriter{
ResponseWriter: w,
response: &bytes.Buffer{},
}
}
func getBody(httpBody io.ReadCloser, l *zap.Logger) []byte {
defer func(httpBody io.ReadCloser) {
if err := httpBody.Close(); err != nil {
l.Error(logs.FailedToCloseHTTPBody, zap.Error(err))
}
}(httpBody)
body, err := io.ReadAll(httpBody)
if err != nil {
l.Error(logs.FailedToReadHTTPBody,
zap.Error(err),
zap.String("body type", payloadLabel))
return nil
}
return body
}
// processBody reads body and base64 encode it if it's not XML.
func processBody(bodyReader io.Reader) ([]byte, error) {
resultBody := &bytes.Buffer{}
detect := detector.NewDetector(bodyReader, xmlutils.DetectXML)
dataType, err := detect.Detect()
if err != nil {
return nil, err
}
writer := xmlutils.ChooseWriter(dataType, resultBody)
if _, err = io.Copy(writer, detect.RestoredReader()); err != nil {
return nil, err
}
if err = writer.Close(); err != nil {
return nil, err
}
return resultBody.Bytes(), nil
}

View file

@ -0,0 +1,36 @@
//go:build !loghttp
package middleware
import (
"net/http"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"go.uber.org/zap"
)
type (
LogHTTPSettings interface {
LogHTTPConfig() LogHTTPConfig
}
LogHTTPConfig struct {
Enabled bool
MaxBody int64
MaxLogSize int
OutputPath string
UseGzip bool
}
)
func LogHTTP(l *zap.Logger, _ LogHTTPSettings) Func {
l.Warn(logs.LogHTTPDisabledInThisBuild)
return func(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
h.ServeHTTP(w, r)
})
}
}
func (*LogHTTPConfig) InitHTTPLogger(*zap.Logger) {
// ignore
}

View file

@ -11,8 +11,7 @@ import (
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
frostfsErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain"
"git.frostfs.info/TrueCloudLab/policy-engine/pkg/engine"
@ -25,11 +24,15 @@ import (
)
const (
QueryVersionID = "versionId"
QueryPrefix = "prefix"
QueryDelimiter = "delimiter"
QueryMaxKeys = "max-keys"
amzTagging = "x-amz-tagging"
QueryVersionID = "versionId"
QueryPrefix = "prefix"
QueryDelimiter = "delimiter"
QueryMaxKeys = "max-keys"
QueryMarker = "marker"
QueryEncodingType = "encoding-type"
amzTagging = "x-amz-tagging"
unmatchedBucketOperation = "UnmatchedBucketOperation"
)
// In these operations we don't check resource tags because
@ -85,7 +88,7 @@ func PolicyCheck(cfg PolicyConfig) Func {
ctx := r.Context()
if err := policyCheck(r, cfg); err != nil {
reqLogOrDefault(ctx, cfg.Log).Error(logs.PolicyValidationFailed, zap.Error(err))
err = frostfsErrors.UnwrapErr(err)
err = apierr.TransformToS3Error(err)
if _, wrErr := WriteErrorResponse(w, GetReqInfo(ctx), err); wrErr != nil {
reqLogOrDefault(ctx, cfg.Log).Error(logs.FailedToWriteResponse, zap.Error(wrErr))
}
@ -145,11 +148,11 @@ func policyCheck(r *http.Request, cfg PolicyConfig) error {
case st == chain.Allow:
return nil
case st != chain.NoRuleFound:
return apiErr.GetAPIErrorWithError(apiErr.ErrAccessDenied, fmt.Errorf("policy check: %s", st.String()))
return apierr.GetAPIErrorWithError(apierr.ErrAccessDenied, fmt.Errorf("policy check: %s", st.String()))
}
if cfg.Settings.PolicyDenyByDefault() {
return apiErr.GetAPIErrorWithError(apiErr.ErrAccessDenied, fmt.Errorf("policy check: %s", st.String()))
return apierr.GetAPIErrorWithError(apierr.ErrAccessDenied, fmt.Errorf("policy check: %s", st.String()))
}
return nil
@ -269,8 +272,17 @@ func determineBucketOperation(r *http.Request) string {
return ListObjectsV2MOperation
case query.Get(ListTypeQuery) == "2":
return ListObjectsV2Operation
default:
case len(query) == 0 || func() bool {
for key := range query {
if key != QueryDelimiter && key != QueryMaxKeys && key != QueryPrefix && key != QueryMarker && key != QueryEncodingType {
return false
}
}
return true
}():
return ListObjectsV1Operation
default:
return unmatchedBucketOperation
}
case http.MethodPut:
switch {
@ -292,8 +304,10 @@ func determineBucketOperation(r *http.Request) string {
return PutBucketVersioningOperation
case query.Has(NotificationQuery):
return PutBucketNotificationOperation
default:
case len(query) == 0:
return CreateBucketOperation
default:
return unmatchedBucketOperation
}
case http.MethodPost:
switch {
@ -316,12 +330,14 @@ func determineBucketOperation(r *http.Request) string {
return DeleteBucketLifecycleOperation
case query.Has(EncryptionQuery):
return DeleteBucketEncryptionOperation
default:
case len(query) == 0:
return DeleteBucketOperation
default:
return unmatchedBucketOperation
}
}
return "UnmatchedBucketOperation"
return unmatchedBucketOperation
}
func determineObjectOperation(r *http.Request) string {
@ -461,7 +477,7 @@ func determineRequestTags(r *http.Request, decoder XMLDecoder, op string) (map[s
if strings.HasSuffix(op, PutObjectTaggingOperation) || strings.HasSuffix(op, PutBucketTaggingOperation) {
tagging := new(data.Tagging)
if err := decoder.NewXMLDecoder(r.Body).Decode(tagging); err != nil {
return nil, fmt.Errorf("%w: %s", apiErr.GetAPIError(apiErr.ErrMalformedXML), err.Error())
return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrMalformedXML), err.Error())
}
GetReqInfo(r.Context()).Tagging = tagging
@ -473,7 +489,7 @@ func determineRequestTags(r *http.Request, decoder XMLDecoder, op string) (map[s
if tagging := r.Header.Get(amzTagging); len(tagging) > 0 {
queries, err := url.ParseQuery(tagging)
if err != nil {
return nil, apiErr.GetAPIError(apiErr.ErrInvalidArgument)
return nil, apierr.GetAPIError(apierr.ErrInvalidArgument)
}
for key := range queries {
tags[fmt.Sprintf(s3.PropertyKeyFormatRequestTag, key)] = queries.Get(key)

View file

@ -152,6 +152,12 @@ func TestDetermineBucketOperation(t *testing.T) {
method: http.MethodGet,
expected: ListObjectsV1Operation,
},
{
name: "UnmatchedBucketOperation GET",
method: http.MethodGet,
queryParam: map[string]string{"query": ""},
expected: unmatchedBucketOperation,
},
{
name: "PutBucketCorsOperation",
method: http.MethodPut,
@ -211,6 +217,12 @@ func TestDetermineBucketOperation(t *testing.T) {
method: http.MethodPut,
expected: CreateBucketOperation,
},
{
name: "UnmatchedBucketOperation PUT",
method: http.MethodPut,
queryParam: map[string]string{"query": ""},
expected: unmatchedBucketOperation,
},
{
name: "DeleteMultipleObjectsOperation",
method: http.MethodPost,
@ -263,10 +275,16 @@ func TestDetermineBucketOperation(t *testing.T) {
method: http.MethodDelete,
expected: DeleteBucketOperation,
},
{
name: "UnmatchedBucketOperation DELETE",
method: http.MethodDelete,
queryParam: map[string]string{"query": ""},
expected: unmatchedBucketOperation,
},
{
name: "UnmatchedBucketOperation",
method: "invalid-method",
expected: "UnmatchedBucketOperation",
expected: unmatchedBucketOperation,
},
} {
t.Run(tc.name, func(t *testing.T) {

View file

@ -62,7 +62,7 @@ const (
hdrSSE = "X-Amz-Server-Side-Encryption"
// hdrSSECustomerKey is the HTTP header key referencing the
// SSE-C client-provided key..
// SSE-C client-provided key.
hdrSSECustomerKey = hdrSSE + "-Customer-Key"
// hdrSSECopyKey is the HTTP header key referencing the SSE-C
@ -74,7 +74,7 @@ var (
xmlHeader = []byte(xml.Header)
)
// Non exhaustive list of AWS S3 standard error responses -
// Non-exhaustive list of AWS S3 standard error responses -
// http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
var s3ErrorResponseMap = map[string]string{
"AccessDenied": "Access Denied.",

View file

@ -6,7 +6,7 @@ import (
"fmt"
"sync"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
v2container "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/ns"
@ -29,20 +29,14 @@ type FrostFS interface {
SystemDNS(context.Context) (string, error)
}
type Settings interface {
FormContainerZone(ns string) (zone string, isDefault bool)
}
type Config struct {
FrostFS FrostFS
RPCAddress string
Settings Settings
}
type BucketResolver struct {
rpcAddress string
frostfs FrostFS
settings Settings
mu sync.RWMutex
resolvers []*Resolver
@ -50,15 +44,15 @@ type BucketResolver struct {
type Resolver struct {
Name string
resolve func(context.Context, string) (cid.ID, error)
resolve func(context.Context, string, string) (cid.ID, error)
}
func (r *Resolver) SetResolveFunc(fn func(context.Context, string) (cid.ID, error)) {
func (r *Resolver) SetResolveFunc(fn func(context.Context, string, string) (cid.ID, error)) {
r.resolve = fn
}
func (r *Resolver) Resolve(ctx context.Context, name string) (cid.ID, error) {
return r.resolve(ctx, name)
func (r *Resolver) Resolve(ctx context.Context, zone, name string) (cid.ID, error) {
return r.resolve(ctx, zone, name)
}
func NewBucketResolver(resolverNames []string, cfg *Config) (*BucketResolver, error) {
@ -87,12 +81,12 @@ func createResolvers(resolverNames []string, cfg *Config) ([]*Resolver, error) {
return resolvers, nil
}
func (r *BucketResolver) Resolve(ctx context.Context, bktName string) (cnrID cid.ID, err error) {
func (r *BucketResolver) Resolve(ctx context.Context, zone, bktName string) (cnrID cid.ID, err error) {
r.mu.RLock()
defer r.mu.RUnlock()
for _, resolver := range r.resolvers {
cnrID, resolverErr := resolver.Resolve(ctx, bktName)
cnrID, resolverErr := resolver.Resolve(ctx, zone, bktName)
if resolverErr != nil {
resolverErr = fmt.Errorf("%s: %w", resolver.Name, resolverErr)
if err == nil {
@ -123,7 +117,6 @@ func (r *BucketResolver) UpdateResolvers(resolverNames []string) error {
cfg := &Config{
FrostFS: r.frostfs,
RPCAddress: r.rpcAddress,
Settings: r.settings,
}
resolvers, err := createResolvers(resolverNames, cfg)
@ -152,30 +145,25 @@ func (r *BucketResolver) equals(resolverNames []string) bool {
func newResolver(name string, cfg *Config) (*Resolver, error) {
switch name {
case DNSResolver:
return NewDNSResolver(cfg.FrostFS, cfg.Settings)
return NewDNSResolver(cfg.FrostFS)
case NNSResolver:
return NewNNSResolver(cfg.RPCAddress, cfg.Settings)
return NewNNSResolver(cfg.RPCAddress)
default:
return nil, fmt.Errorf("unknown resolver: %s", name)
}
}
func NewDNSResolver(frostFS FrostFS, settings Settings) (*Resolver, error) {
func NewDNSResolver(frostFS FrostFS) (*Resolver, error) {
if frostFS == nil {
return nil, fmt.Errorf("pool must not be nil for DNS resolver")
}
if settings == nil {
return nil, fmt.Errorf("resolver settings must not be nil for DNS resolver")
}
var dns ns.DNS
resolveFunc := func(ctx context.Context, name string) (cid.ID, error) {
resolveFunc := func(ctx context.Context, zone, name string) (cid.ID, error) {
var err error
reqInfo := middleware.GetReqInfo(ctx)
zone, isDefault := settings.FormContainerZone(reqInfo.Namespace)
if isDefault {
if zone == v2container.SysAttributeZoneDefault {
zone, err = frostFS.SystemDNS(ctx)
if err != nil {
return cid.ID{}, fmt.Errorf("read system DNS parameter of the FrostFS: %w", err)
@ -196,13 +184,10 @@ func NewDNSResolver(frostFS FrostFS, settings Settings) (*Resolver, error) {
}, nil
}
func NewNNSResolver(address string, settings Settings) (*Resolver, error) {
func NewNNSResolver(address string) (*Resolver, error) {
if address == "" {
return nil, fmt.Errorf("rpc address must not be empty for NNS resolver")
}
if settings == nil {
return nil, fmt.Errorf("resolver settings must not be nil for NNS resolver")
}
var nns ns.NNS
@ -210,12 +195,9 @@ func NewNNSResolver(address string, settings Settings) (*Resolver, error) {
return nil, fmt.Errorf("dial %s: %w", address, err)
}
resolveFunc := func(ctx context.Context, name string) (cid.ID, error) {
resolveFunc := func(_ context.Context, zone, name string) (cid.ID, error) {
var d container.Domain
d.SetName(name)
reqInfo := middleware.GetReqInfo(ctx)
zone, _ := settings.FormContainerZone(reqInfo.Namespace)
d.SetZone(zone)
cnrID, err := nns.ResolveContainerDomain(d)

View file

@ -99,6 +99,7 @@ type Settings interface {
s3middleware.PolicySettings
s3middleware.MetricsSettings
s3middleware.VHSSettings
s3middleware.LogHTTPSettings
}
type FrostFSID interface {
@ -127,7 +128,9 @@ type Config struct {
func NewRouter(cfg Config) *chi.Mux {
api := chi.NewRouter()
api.Use(
s3middleware.LogHTTP(cfg.Log, cfg.MiddlewareSettings),
s3middleware.Request(cfg.Log, cfg.MiddlewareSettings),
middleware.ThrottleWithOpts(cfg.Throttle),
middleware.Recoverer,
@ -223,6 +226,28 @@ func errorResponseHandler(w http.ResponseWriter, r *http.Request) {
}
}
func notSupportedHandler() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
reqInfo := s3middleware.GetReqInfo(ctx)
_, wrErr := s3middleware.WriteErrorResponse(w, reqInfo, errors.GetAPIError(errors.ErrNotSupported))
if log := s3middleware.GetReqLog(ctx); log != nil {
fields := []zap.Field{
zap.String("http method", r.Method),
zap.String("url", r.RequestURI),
}
if wrErr != nil {
fields = append(fields, zap.NamedError("write_response_error", wrErr))
}
log.Error(logs.NotSupported, fields...)
}
}
}
// attachErrorHandler set NotFoundHandler and MethodNotAllowedHandler for chi.Router.
func attachErrorHandler(api *chi.Mux) {
errorHandler := http.HandlerFunc(errorResponseHandler)
@ -310,7 +335,14 @@ func bucketRouter(h Handler) chi.Router {
Add(NewFilter().
Queries(s3middleware.VersionsQuery).
Handler(named(s3middleware.ListBucketObjectVersionsOperation, h.ListBucketObjectVersionsHandler))).
DefaultHandler(listWrapper(h)))
Add(NewFilter().
AllowedQueries(s3middleware.QueryDelimiter, s3middleware.QueryMaxKeys, s3middleware.QueryPrefix,
s3middleware.QueryMarker, s3middleware.QueryEncodingType).
Handler(named(s3middleware.ListObjectsV1Operation, h.ListObjectsV1Handler))).
Add(NewFilter().
NoQueries().
Handler(listWrapper(h))).
DefaultHandler(notSupportedHandler()))
})
// PUT method handlers
@ -343,7 +375,10 @@ func bucketRouter(h Handler) chi.Router {
Add(NewFilter().
Queries(s3middleware.NotificationQuery).
Handler(named(s3middleware.PutBucketNotificationOperation, h.PutBucketNotificationHandler))).
DefaultHandler(named(s3middleware.CreateBucketOperation, h.CreateBucketHandler)))
Add(NewFilter().
NoQueries().
Handler(named(s3middleware.CreateBucketOperation, h.CreateBucketHandler))).
DefaultHandler(notSupportedHandler()))
})
// POST method handlers
@ -377,7 +412,10 @@ func bucketRouter(h Handler) chi.Router {
Add(NewFilter().
Queries(s3middleware.EncryptionQuery).
Handler(named(s3middleware.DeleteBucketEncryptionOperation, h.DeleteBucketEncryptionHandler))).
DefaultHandler(named(s3middleware.DeleteBucketOperation, h.DeleteBucketHandler)))
Add(NewFilter().
NoQueries().
Handler(named(s3middleware.DeleteBucketOperation, h.DeleteBucketHandler))).
DefaultHandler(notSupportedHandler()))
})
attachErrorHandler(bktRouter)

View file

@ -11,9 +11,11 @@ type HandlerFilters struct {
}
type Filter struct {
queries []Pair
headers []Pair
h http.Handler
queries []Pair
headers []Pair
allowedQueries map[string]struct{}
noQueries bool
h http.Handler
}
type Pair struct {
@ -105,6 +107,22 @@ func (f *Filter) Queries(queries ...string) *Filter {
return f
}
// NoQueries sets flag indicating that request shouldn't have query parameters.
func (f *Filter) NoQueries() *Filter {
f.noQueries = true
return f
}
// AllowedQueries adds query parameter keys that may be present in request.
func (f *Filter) AllowedQueries(queries ...string) *Filter {
f.allowedQueries = make(map[string]struct{}, len(queries))
for _, query := range queries {
f.allowedQueries[query] = struct{}{}
}
return f
}
func (hf *HandlerFilters) DefaultHandler(handler http.HandlerFunc) *HandlerFilters {
hf.defaultHandler = handler
return hf
@ -122,6 +140,17 @@ func (hf *HandlerFilters) ServeHTTP(w http.ResponseWriter, r *http.Request) {
func (hf *HandlerFilters) match(r *http.Request) http.Handler {
LOOP:
for _, filter := range hf.filters {
if filter.noQueries && len(r.URL.Query()) > 0 {
continue
}
if len(filter.allowedQueries) > 0 {
queries := r.URL.Query()
for key := range queries {
if _, ok := filter.allowedQueries[key]; !ok {
continue LOOP
}
}
}
for _, header := range filter.headers {
hdrVals := r.Header.Values(header.Key)
if len(hdrVals) == 0 || header.Value != "" && header.Value != hdrVals[0] {

View file

@ -10,7 +10,7 @@ import (
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
@ -40,8 +40,9 @@ type centerMock struct {
t *testing.T
anon bool
noAuthHeader bool
isError bool
err error
attrs []object.Attribute
key *keys.PrivateKey
}
func (c *centerMock) Authenticate(*http.Request) (*middleware.Box, error) {
@ -49,8 +50,8 @@ func (c *centerMock) Authenticate(*http.Request) (*middleware.Box, error) {
return nil, middleware.ErrNoAuthorizationHeader
}
if c.isError {
return nil, fmt.Errorf("some error")
if c.err != nil {
return nil, c.err
}
var token *bearer.Token
@ -58,8 +59,12 @@ func (c *centerMock) Authenticate(*http.Request) (*middleware.Box, error) {
if !c.anon {
bt := bearertest.Token()
token = &bt
key, err := keys.NewPrivateKey()
require.NoError(c.t, err)
key := c.key
if key == nil {
var err error
key, err = keys.NewPrivateKey()
require.NoError(c.t, err)
}
require.NoError(c.t, token.Sign(key.PrivateKey))
}
@ -80,6 +85,7 @@ type middlewareSettingsMock struct {
domains []string
vhsEnabled bool
vhsNamespacesEnabled map[string]bool
logHTTP middleware.LogHTTPConfig
}
func (r *middlewareSettingsMock) SourceIPHeader() string {
@ -117,6 +123,9 @@ func (r *middlewareSettingsMock) ServernameHeader() string {
func (r *middlewareSettingsMock) VHSNamespacesEnabled() map[string]bool {
return r.vhsNamespacesEnabled
}
func (r *middlewareSettingsMock) LogHTTPConfig() middleware.LogHTTPConfig {
return r.logHTTP
}
type frostFSIDMock struct {
tags map[string]string
@ -147,22 +156,21 @@ func (m *xmlMock) NewXMLDecoder(r io.Reader) *xml.Decoder {
}
type resourceTaggingMock struct {
bucketTags map[string]string
objectTags map[string]string
noSuchObjectKey bool
noSuchBucketKey bool
bucketTags map[string]string
objectTags map[string]string
err error
}
func (m *resourceTaggingMock) GetBucketTagging(context.Context, *data.BucketInfo) (map[string]string, error) {
if m.noSuchBucketKey {
return nil, apiErrors.GetAPIError(apiErrors.ErrNoSuchKey)
if m.err != nil {
return nil, m.err
}
return m.bucketTags, nil
}
func (m *resourceTaggingMock) GetObjectTagging(context.Context, *data.GetObjectTaggingParams) (string, map[string]string, error) {
if m.noSuchObjectKey {
return "", nil, apiErrors.GetAPIError(apiErrors.ErrNoSuchKey)
if m.err != nil {
return "", nil, m.err
}
return "", m.objectTags, nil
}
@ -569,7 +577,7 @@ func (h *handlerMock) ResolveBucket(ctx context.Context, name string) (*data.Buc
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, ok := h.buckets[reqInfo.Namespace+name]
if !ok {
return nil, apiErrors.GetAPIError(apiErrors.ErrNoSuchBucket)
return nil, apierr.GetAPIError(apierr.ErrNoSuchBucket)
}
return bktInfo, nil
}

View file

@ -14,7 +14,8 @@ import (
"time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
s3middleware "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/metrics"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
@ -26,6 +27,7 @@ import (
"git.frostfs.info/TrueCloudLab/policy-engine/schema/s3"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/prometheus/client_golang/prometheus"
"github.com/stretchr/testify/require"
"go.uber.org/zap/zaptest"
@ -214,18 +216,18 @@ func TestPolicyChecker(t *testing.T) {
deleteObject(chiRouter, ns2, bktName2, objName2, nil)
// check we cannot access 'bucket' in custom namespace
putObjectErr(chiRouter, ns2, bktName1, objName2, nil, apiErrors.ErrAccessDenied)
deleteObjectErr(chiRouter, ns2, bktName1, objName2, nil, apiErrors.ErrAccessDenied)
putObjectErr(chiRouter, ns2, bktName1, objName2, nil, apierr.ErrAccessDenied)
deleteObjectErr(chiRouter, ns2, bktName1, objName2, nil, apierr.ErrAccessDenied)
}
func TestPolicyCheckerError(t *testing.T) {
chiRouter := prepareRouter(t)
ns1, bktName1, objName1 := "", "bucket", "object"
putObjectErr(chiRouter, ns1, bktName1, objName1, nil, apiErrors.ErrNoSuchBucket)
putObjectErr(chiRouter, ns1, bktName1, objName1, nil, apierr.ErrNoSuchBucket)
chiRouter = prepareRouter(t)
chiRouter.cfg.FrostfsID.(*frostFSIDMock).userGroupsError = true
putObjectErr(chiRouter, ns1, bktName1, objName1, nil, apiErrors.ErrInternalError)
putObjectErr(chiRouter, ns1, bktName1, objName1, nil, apierr.ErrInternalError)
}
func TestPolicyCheckerReqTypeDetermination(t *testing.T) {
@ -274,6 +276,36 @@ func TestPolicyCheckerReqTypeDetermination(t *testing.T) {
})
}
func TestPolicyCheckFrostfsErrors(t *testing.T) {
chiRouter := prepareRouter(t)
ns1, bktName1, objName1 := "", "bucket", "object"
createBucket(chiRouter, ns1, bktName1)
key, err := keys.NewPrivateKey()
require.NoError(t, err)
chiRouter.cfg.Center.(*centerMock).key = key
chiRouter.cfg.MiddlewareSettings.(*middlewareSettingsMock).denyByDefault = true
ruleChain := &chain.Chain{
ID: chain.ID("id"),
Rules: []chain.Rule{{
Status: chain.Allow,
Actions: chain.Actions{Names: []string{"*"}},
Resources: chain.Resources{Names: []string{fmt.Sprintf(s3.ResourceFormatS3BucketObjects, bktName1)}},
}},
}
_, _, err = chiRouter.policyChecker.MorphRuleChainStorage().AddMorphRuleChain(chain.S3, engine.UserTarget(ns1+":"+key.Address()), ruleChain)
require.NoError(t, err)
// check we can access 'bucket' in default namespace
putObject(chiRouter, ns1, bktName1, objName1, nil)
chiRouter.cfg.Center.(*centerMock).anon = true
chiRouter.cfg.Tagging.(*resourceTaggingMock).err = frostfs.ErrAccessDenied
getObjectErr(chiRouter, ns1, bktName1, objName1, apierr.ErrAccessDenied)
}
func TestDefaultBehaviorPolicyChecker(t *testing.T) {
chiRouter := prepareRouter(t)
ns, bktName := "", "bucket"
@ -283,7 +315,7 @@ func TestDefaultBehaviorPolicyChecker(t *testing.T) {
// check we cannot access if rules not found when settings is enabled
chiRouter.middlewareSettings.denyByDefault = true
createBucketErr(chiRouter, ns, bktName, nil, apiErrors.ErrAccessDenied)
createBucketErr(chiRouter, ns, bktName, nil, apierr.ErrAccessDenied)
}
func TestDefaultPolicyCheckerWithUserTags(t *testing.T) {
@ -294,7 +326,7 @@ func TestDefaultPolicyCheckerWithUserTags(t *testing.T) {
allowOperations(router, ns, []string{"s3:CreateBucket"}, engineiam.Conditions{
engineiam.CondStringEquals: engineiam.Condition{fmt.Sprintf(common.PropertyKeyFormatFrostFSIDUserClaim, "tag-test"): []string{"test"}},
})
createBucketErr(router, ns, bktName, nil, apiErrors.ErrAccessDenied)
createBucketErr(router, ns, bktName, nil, apierr.ErrAccessDenied)
tags := make(map[string]string)
tags["tag-test"] = "test"
@ -321,8 +353,8 @@ func TestRequestParametersCheck(t *testing.T) {
})
listObjectsV1(router, ns, bktName, prefix, "", "")
listObjectsV1Err(router, ns, bktName, "", "", "", apiErrors.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "invalid", "", "", apiErrors.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "", "", "", apierr.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "invalid", "", "", apierr.ErrAccessDenied)
})
t.Run("delimiter parameter, prohibit specific value", func(t *testing.T) {
@ -344,7 +376,7 @@ func TestRequestParametersCheck(t *testing.T) {
listObjectsV1(router, ns, bktName, "", "", "")
listObjectsV1(router, ns, bktName, "", "some-delimiter", "")
listObjectsV1Err(router, ns, bktName, "", delimiter, "", apiErrors.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "", delimiter, "", apierr.ErrAccessDenied)
})
t.Run("max-keys parameter, allow specific value", func(t *testing.T) {
@ -365,9 +397,9 @@ func TestRequestParametersCheck(t *testing.T) {
})
listObjectsV1(router, ns, bktName, "", "", strconv.Itoa(maxKeys))
listObjectsV1Err(router, ns, bktName, "", "", "", apiErrors.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "", "", strconv.Itoa(maxKeys-1), apiErrors.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "", "", "invalid", apiErrors.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "", "", "", apierr.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "", "", strconv.Itoa(maxKeys-1), apierr.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "", "", "invalid", apierr.ErrAccessDenied)
})
t.Run("max-keys parameter, allow range of values", func(t *testing.T) {
@ -389,7 +421,7 @@ func TestRequestParametersCheck(t *testing.T) {
listObjectsV1(router, ns, bktName, "", "", strconv.Itoa(maxKeys))
listObjectsV1(router, ns, bktName, "", "", strconv.Itoa(maxKeys-1))
listObjectsV1Err(router, ns, bktName, "", "", strconv.Itoa(maxKeys+1), apiErrors.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "", "", strconv.Itoa(maxKeys+1), apierr.ErrAccessDenied)
})
t.Run("max-keys parameter, prohibit specific value", func(t *testing.T) {
@ -411,7 +443,7 @@ func TestRequestParametersCheck(t *testing.T) {
listObjectsV1(router, ns, bktName, "", "", "")
listObjectsV1(router, ns, bktName, "", "", strconv.Itoa(maxKeys-1))
listObjectsV1Err(router, ns, bktName, "", "", strconv.Itoa(maxKeys), apiErrors.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "", "", strconv.Itoa(maxKeys), apierr.ErrAccessDenied)
})
}
@ -439,10 +471,10 @@ func TestRequestTagsCheck(t *testing.T) {
tagging, err = xml.Marshal(data.Tagging{TagSet: []data.Tag{{Key: "key", Value: tagValue}}})
require.NoError(t, err)
putBucketTaggingErr(router, ns, bktName, tagging, apiErrors.ErrAccessDenied)
putBucketTaggingErr(router, ns, bktName, tagging, apierr.ErrAccessDenied)
tagging = nil
putBucketTaggingErr(router, ns, bktName, tagging, apiErrors.ErrMalformedXML)
putBucketTaggingErr(router, ns, bktName, tagging, apierr.ErrMalformedXML)
})
t.Run("put object with tag", func(t *testing.T) {
@ -464,7 +496,7 @@ func TestRequestTagsCheck(t *testing.T) {
putObject(router, ns, bktName, objName, &data.Tag{Key: tagKey, Value: tagValue})
putObjectErr(router, ns, bktName, objName, &data.Tag{Key: "key", Value: tagValue}, apiErrors.ErrAccessDenied)
putObjectErr(router, ns, bktName, objName, &data.Tag{Key: "key", Value: tagValue}, apierr.ErrAccessDenied)
})
}
@ -490,7 +522,7 @@ func TestResourceTagsCheck(t *testing.T) {
listObjectsV1(router, ns, bktName, "", "", "")
router.cfg.Tagging.(*resourceTaggingMock).bucketTags = map[string]string{}
listObjectsV1Err(router, ns, bktName, "", "", "", apiErrors.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "", "", "", apierr.ErrAccessDenied)
})
t.Run("object tagging", func(t *testing.T) {
@ -515,22 +547,21 @@ func TestResourceTagsCheck(t *testing.T) {
getObject(router, ns, bktName, objName)
router.cfg.Tagging.(*resourceTaggingMock).objectTags = map[string]string{}
getObjectErr(router, ns, bktName, objName, apiErrors.ErrAccessDenied)
getObjectErr(router, ns, bktName, objName, apierr.ErrAccessDenied)
})
t.Run("non-existent resources", func(t *testing.T) {
router := prepareRouter(t)
ns, bktName, objName := "", "bucket", "object"
listObjectsV1Err(router, ns, bktName, "", "", "", apiErrors.ErrNoSuchBucket)
listObjectsV1Err(router, ns, bktName, "", "", "", apierr.ErrNoSuchBucket)
router.cfg.Tagging.(*resourceTaggingMock).noSuchBucketKey = true
router.cfg.Tagging.(*resourceTaggingMock).err = apierr.GetAPIError(apierr.ErrNoSuchKey)
createBucket(router, ns, bktName)
getBucketErr(router, ns, bktName, apiErrors.ErrNoSuchKey)
getBucketErr(router, ns, bktName, apierr.ErrNoSuchKey)
router.cfg.Tagging.(*resourceTaggingMock).noSuchObjectKey = true
createBucket(router, ns, bktName)
getObjectErr(router, ns, bktName, objName, apiErrors.ErrNoSuchKey)
getObjectErr(router, ns, bktName, objName, apierr.ErrNoSuchKey)
})
}
@ -548,7 +579,7 @@ func TestAccessBoxAttributesCheck(t *testing.T) {
engineiam.CondBool: engineiam.Condition{fmt.Sprintf(s3.PropertyKeyFormatAccessBoxAttr, attrKey): []string{attrValue}},
})
listObjectsV1Err(router, ns, bktName, "", "", "", apiErrors.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "", "", "", apierr.ErrAccessDenied)
var attr object.Attribute
attr.SetKey(attrKey)
@ -570,7 +601,7 @@ func TestSourceIPCheck(t *testing.T) {
router.middlewareSettings.sourceIPHeader = hdr
header := map[string][]string{hdr: {"192.0.3.0"}}
createBucketErr(router, ns, bktName, header, apiErrors.ErrAccessDenied)
createBucketErr(router, ns, bktName, header, apierr.ErrAccessDenied)
router.middlewareSettings.sourceIPHeader = ""
createBucket(router, ns, bktName)
@ -586,7 +617,7 @@ func TestMFAPolicy(t *testing.T) {
denyOperations(router, ns, []string{"s3:CreateBucket"}, engineiam.Conditions{
engineiam.CondBool: engineiam.Condition{s3.PropertyKeyAccessBoxAttrMFA: []string{"false"}},
})
createBucketErr(router, ns, bktName, nil, apiErrors.ErrAccessDenied)
createBucketErr(router, ns, bktName, nil, apierr.ErrAccessDenied)
var attr object.Attribute
attr.SetKey("IAM-MFA")
@ -630,7 +661,7 @@ func createBucket(router *routerMock, namespace, bktName string) {
require.Equal(router.t, s3middleware.CreateBucketOperation, resp.Method)
}
func createBucketErr(router *routerMock, namespace, bktName string, header http.Header, errCode apiErrors.ErrorCode) {
func createBucketErr(router *routerMock, namespace, bktName string, header http.Header, errCode apierr.ErrorCode) {
w := createBucketBase(router, namespace, bktName, header)
assertAPIError(router.t, w, errCode)
}
@ -646,7 +677,7 @@ func createBucketBase(router *routerMock, namespace, bktName string, header http
return w
}
func getBucketErr(router *routerMock, namespace, bktName string, errCode apiErrors.ErrorCode) {
func getBucketErr(router *routerMock, namespace, bktName string, errCode apierr.ErrorCode) {
w := getBucketBase(router, namespace, bktName)
assertAPIError(router.t, w, errCode)
}
@ -665,7 +696,7 @@ func putObject(router *routerMock, namespace, bktName, objName string, tag *data
return resp
}
func putObjectErr(router *routerMock, namespace, bktName, objName string, tag *data.Tag, errCode apiErrors.ErrorCode) {
func putObjectErr(router *routerMock, namespace, bktName, objName string, tag *data.Tag, errCode apierr.ErrorCode) {
w := putObjectBase(router, namespace, bktName, objName, tag)
assertAPIError(router.t, w, errCode)
}
@ -690,7 +721,7 @@ func deleteObject(router *routerMock, namespace, bktName, objName string, tag *d
return resp
}
func deleteObjectErr(router *routerMock, namespace, bktName, objName string, tag *data.Tag, errCode apiErrors.ErrorCode) {
func deleteObjectErr(router *routerMock, namespace, bktName, objName string, tag *data.Tag, errCode apierr.ErrorCode) {
w := deleteObjectBase(router, namespace, bktName, objName, tag)
assertAPIError(router.t, w, errCode)
}
@ -715,7 +746,7 @@ func putBucketTagging(router *routerMock, namespace, bktName string, tagging []b
return resp
}
func putBucketTaggingErr(router *routerMock, namespace, bktName string, tagging []byte, errCode apiErrors.ErrorCode) {
func putBucketTaggingErr(router *routerMock, namespace, bktName string, tagging []byte, errCode apierr.ErrorCode) {
w := putBucketTaggingBase(router, namespace, bktName, tagging)
assertAPIError(router.t, w, errCode)
}
@ -738,7 +769,7 @@ func getObject(router *routerMock, namespace, bktName, objName string) handlerRe
return resp
}
func getObjectErr(router *routerMock, namespace, bktName, objName string, errCode apiErrors.ErrorCode) {
func getObjectErr(router *routerMock, namespace, bktName, objName string, errCode apierr.ErrorCode) {
w := getObjectBase(router, namespace, bktName, objName)
assertAPIError(router.t, w, errCode)
}
@ -757,7 +788,7 @@ func listObjectsV1(router *routerMock, namespace, bktName, prefix, delimiter, ma
return resp
}
func listObjectsV1Err(router *routerMock, namespace, bktName, prefix, delimiter, maxKeys string, errCode apiErrors.ErrorCode) {
func listObjectsV1Err(router *routerMock, namespace, bktName, prefix, delimiter, maxKeys string, errCode apierr.ErrorCode) {
w := listObjectsV1Base(router, namespace, bktName, prefix, delimiter, maxKeys)
assertAPIError(router.t, w, errCode)
}
@ -826,8 +857,17 @@ func TestAuthenticate(t *testing.T) {
createBucket(chiRouter, "", "bkt-2")
chiRouter = prepareRouter(t)
chiRouter.cfg.Center.(*centerMock).isError = true
createBucketErr(chiRouter, "", "bkt-3", nil, apiErrors.ErrAccessDenied)
chiRouter.cfg.Center.(*centerMock).err = apierr.GetAPIError(apierr.ErrAccessDenied)
createBucketErr(chiRouter, "", "bkt-3", nil, apierr.ErrAccessDenied)
chiRouter.cfg.Center.(*centerMock).err = frostfs.ErrGatewayTimeout
createBucketErr(chiRouter, "", "bkt-3", nil, apierr.ErrGatewayTimeout)
chiRouter.cfg.Center.(*centerMock).err = apierr.GetAPIError(apierr.ErrInternalError)
createBucketErr(chiRouter, "", "bkt-3", nil, apierr.ErrAccessDenied)
chiRouter.cfg.Center.(*centerMock).err = apierr.GetAPIError(apierr.ErrBadRequest)
createBucketErr(chiRouter, "", "bkt-3", nil, apierr.ErrBadRequest)
}
func TestFrostFSIDValidation(t *testing.T) {
@ -843,7 +883,7 @@ func TestFrostFSIDValidation(t *testing.T) {
// frostFSID validation failed
chiRouter = prepareRouter(t, frostFSIDValidation(true))
chiRouter.cfg.FrostfsID.(*frostFSIDMock).validateError = true
createBucketErr(chiRouter, "", "bkt-3", nil, apiErrors.ErrInternalError)
createBucketErr(chiRouter, "", "bkt-3", nil, apierr.ErrInternalError)
}
func TestRouterListObjectsV2Domains(t *testing.T) {
@ -882,17 +922,17 @@ func readResponse(t *testing.T, w *httptest.ResponseRecorder) handlerResult {
return res
}
func assertAPIError(t *testing.T, w *httptest.ResponseRecorder, expectedErrorCode apiErrors.ErrorCode) {
func assertAPIError(t *testing.T, w *httptest.ResponseRecorder, expectedErrorCode apierr.ErrorCode) {
actualErrorResponse := &s3middleware.ErrorResponse{}
err := xml.NewDecoder(w.Result().Body).Decode(actualErrorResponse)
require.NoError(t, err)
expectedError := apiErrors.GetAPIError(expectedErrorCode)
expectedError := apierr.GetAPIError(expectedErrorCode)
require.Equal(t, expectedError.HTTPStatusCode, w.Code)
require.Equal(t, expectedError.Code, actualErrorResponse.Code)
if expectedError.ErrCode != apiErrors.ErrInternalError {
if expectedError.ErrCode != apierr.ErrInternalError {
require.Contains(t, actualErrorResponse.Message, expectedError.Description)
}
}

View file

@ -98,7 +98,9 @@ type (
// IssueSecretOptions contains options for passing to Agent.IssueSecret method.
IssueSecretOptions struct {
Container ContainerOptions
Container cid.ID
AccessKeyID string
SecretAccessKey string
FrostFSKey *keys.PrivateKey
GatesPublicKeys []*keys.PublicKey
Impersonate bool
@ -114,7 +116,9 @@ type (
UpdateSecretOptions struct {
FrostFSKey *keys.PrivateKey
GatesPublicKeys []*keys.PublicKey
Address oid.Address
IsCustom bool
AccessKeyID string
ContainerID cid.ID
GatePrivateKey *keys.PrivateKey
CustomAttributes []object.Attribute
}
@ -141,7 +145,8 @@ type (
// ObtainSecretOptions contains options for passing to Agent.ObtainSecret method.
ObtainSecretOptions struct {
SecretAddress string
Container cid.ID
AccessKeyID string
GatePrivateKey *keys.PrivateKey
}
)
@ -168,32 +173,9 @@ type (
}
)
func (a *Agent) checkContainer(ctx context.Context, opts ContainerOptions, idOwner user.ID) (cid.ID, error) {
if !opts.ID.Equals(cid.ID{}) {
a.log.Info(logs.CheckContainer, zap.Stringer("cid", opts.ID))
return opts.ID, a.frostFS.ContainerExists(ctx, opts.ID)
}
a.log.Info(logs.CreateContainer,
zap.String("friendly_name", opts.FriendlyName),
zap.String("placement_policy", opts.PlacementPolicy))
var prm PrmContainerCreate
err := prm.Policy.DecodeString(opts.PlacementPolicy)
if err != nil {
return cid.ID{}, fmt.Errorf("failed to build placement policy: %w", err)
}
prm.Owner = idOwner
prm.FriendlyName = opts.FriendlyName
cnrID, err := a.frostFS.CreateContainer(ctx, prm)
if err != nil {
return cid.ID{}, fmt.Errorf("create container in FrostFS: %w", err)
}
return cnrID, nil
func (a *Agent) checkContainer(ctx context.Context, cnrID cid.ID) error {
a.log.Info(logs.CheckContainer, zap.Stringer("cid", cnrID))
return a.frostFS.ContainerExists(ctx, cnrID)
}
func checkPolicy(policyString string) (*netmap.PlacementPolicy, error) {
@ -255,20 +237,24 @@ func (a *Agent) IssueSecret(ctx context.Context, w io.Writer, options *IssueSecr
return fmt.Errorf("create tokens: %w", err)
}
box, secrets, err := accessbox.PackTokens(gatesData, nil)
var secret []byte
isCustom := options.AccessKeyID != ""
if isCustom {
secret = []byte(options.SecretAccessKey)
}
box, secrets, err := accessbox.PackTokens(gatesData, secret, isCustom)
if err != nil {
return fmt.Errorf("pack tokens: %w", err)
}
box.ContainerPolicy = policies
var idOwner user.ID
user.IDFromKey(&idOwner, options.FrostFSKey.PrivateKey.PublicKey)
id, err := a.checkContainer(ctx, options.Container, idOwner)
if err != nil {
if err = a.checkContainer(ctx, options.Container); err != nil {
return fmt.Errorf("check container: %w", err)
}
var idOwner user.ID
user.IDFromKey(&idOwner, options.FrostFSKey.PrivateKey.PublicKey)
a.log.Info(logs.StoreBearerTokenIntoFrostFS,
zap.Stringer("owner_tkn", idOwner))
@ -281,26 +267,31 @@ func (a *Agent) IssueSecret(ctx context.Context, w io.Writer, options *IssueSecr
creds := tokens.New(cfg)
prm := tokens.CredentialsParam{
OwnerID: idOwner,
Container: options.Container,
AccessKeyID: options.AccessKeyID,
AccessBox: box,
Expiration: lifetime.Exp,
Keys: options.GatesPublicKeys,
CustomAttributes: options.CustomAttributes,
}
addr, err := creds.Put(ctx, id, prm)
addr, err := creds.Put(ctx, prm)
if err != nil {
return fmt.Errorf("failed to put creds: %w", err)
}
accessKeyID := accessKeyIDFromAddr(addr)
accessKeyID := options.AccessKeyID
if accessKeyID == "" {
accessKeyID = accessKeyIDFromAddr(addr)
}
ir := &issuingResult{
InitialAccessKeyID: accessKeyID,
AccessKeyID: accessKeyID,
SecretAccessKey: secrets.SecretKey,
OwnerPrivateKey: hex.EncodeToString(secrets.EphemeralKey.Bytes()),
WalletPublicKey: hex.EncodeToString(options.FrostFSKey.PublicKey().Bytes()),
ContainerID: id.EncodeToString(),
ContainerID: options.Container.EncodeToString(),
}
enc := json.NewEncoder(w)
@ -337,13 +328,15 @@ func (a *Agent) UpdateSecret(ctx context.Context, w io.Writer, options *UpdateSe
creds := tokens.New(cfg)
box, _, err := creds.GetBox(ctx, options.Address)
box, _, err := creds.GetBox(ctx, options.ContainerID, options.AccessKeyID)
if err != nil {
return fmt.Errorf("get accessbox: %w", err)
}
secret, err := hex.DecodeString(box.Gate.SecretKey)
if err != nil {
var secret []byte
if options.IsCustom {
secret = []byte(box.Gate.SecretKey)
} else if secret, err = hex.DecodeString(box.Gate.SecretKey); err != nil {
return fmt.Errorf("failed to decode secret key access box: %w", err)
}
@ -360,7 +353,7 @@ func (a *Agent) UpdateSecret(ctx context.Context, w io.Writer, options *UpdateSe
return fmt.Errorf("create tokens: %w", err)
}
updatedBox, secrets, err := accessbox.PackTokens(gatesData, secret)
updatedBox, secrets, err := accessbox.PackTokens(gatesData, secret, options.IsCustom)
if err != nil {
return fmt.Errorf("pack tokens: %w", err)
}
@ -371,22 +364,26 @@ func (a *Agent) UpdateSecret(ctx context.Context, w io.Writer, options *UpdateSe
zap.Stringer("owner_tkn", idOwner))
prm := tokens.CredentialsParam{
OwnerID: idOwner,
Container: options.ContainerID,
AccessBox: updatedBox,
Expiration: lifetime.Exp,
Keys: options.GatesPublicKeys,
CustomAttributes: options.CustomAttributes,
}
oldAddr := options.Address
addr, err := creds.Update(ctx, oldAddr, prm)
addr, err := creds.Update(ctx, prm)
if err != nil {
return fmt.Errorf("failed to update creds: %w", err)
}
accessKeyID := options.AccessKeyID
if !options.IsCustom {
accessKeyID = accessKeyIDFromAddr(addr)
}
ir := &issuingResult{
AccessKeyID: accessKeyIDFromAddr(addr),
InitialAccessKeyID: accessKeyIDFromAddr(oldAddr),
AccessKeyID: accessKeyID,
InitialAccessKeyID: options.AccessKeyID,
SecretAccessKey: secrets.SecretKey,
OwnerPrivateKey: hex.EncodeToString(secrets.EphemeralKey.Bytes()),
WalletPublicKey: hex.EncodeToString(options.FrostFSKey.PublicKey().Bytes()),
@ -419,12 +416,7 @@ func (a *Agent) ObtainSecret(ctx context.Context, w io.Writer, options *ObtainSe
bearerCreds := tokens.New(cfg)
var addr oid.Address
if err := addr.DecodeString(options.SecretAddress); err != nil {
return fmt.Errorf("failed to parse secret address: %w", err)
}
box, _, err := bearerCreds.GetBox(ctx, addr)
box, _, err := bearerCreds.GetBox(ctx, options.Container, options.AccessKeyID)
if err != nil {
return fmt.Errorf("failed to get tokens: %w", err)
}

View file

@ -4,22 +4,34 @@ import (
"context"
"fmt"
"os"
"strings"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/authmate"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/util"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/wallet"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"go.uber.org/zap"
)
var issueSecretCmd = &cobra.Command{
Use: "issue-secret",
Short: "Issue a secret in FrostFS network",
Long: "Creates new s3 credentials to use with frostfs-s3-gw",
Example: `frostfs-s3-authmate issue-secret --wallet wallet.json --peer s01.frostfs.devenv:8080 --gate-public-key 031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a
frostfs-s3-authmate issue-secret --wallet wallet.json --peer s01.frostfs.devenv:8080 --gate-public-key 031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a --attributes LOGIN=NUUb82KR2JrVByHs2YSKgtK29gKnF5q6Vt`,
Example: `To create new s3 credentials use:
frostfs-s3-authmate issue-secret --wallet wallet.json --peer s01.frostfs.devenv:8080 --gate-public-key 031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a
frostfs-s3-authmate issue-secret --wallet wallet.json --peer s01.frostfs.devenv:8080 --gate-public-key 031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a --attributes LOGIN=NUUb82KR2JrVByHs2YSKgtK29gKnF5q6Vt
To create new s3 credentials using specific access key id and secret access key use:
frostfs-s3-authmate issue-secret --wallet wallet.json --peer s01.frostfs.devenv:8080 --gate-public-key 031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a --access-key-id my-access-key-id --secret-access-key my-secret-key --container-id BpExV76416Vo7GrkJsGwXGoLM35xsBwup8voedDZR3c6
`,
RunE: runIssueSecretCmd,
}
@ -54,6 +66,9 @@ const (
poolHealthcheckTimeoutFlag = "pool-healthcheck-timeout"
poolRebalanceIntervalFlag = "pool-rebalance-interval"
poolStreamTimeoutFlag = "pool-stream-timeout"
accessKeyIDFlag = "access-key-id"
secretAccessKeyFlag = "secret-access-key"
)
func initIssueSecretCmd() {
@ -73,6 +88,9 @@ func initIssueSecretCmd() {
issueSecretCmd.Flags().Duration(poolRebalanceIntervalFlag, defaultPoolRebalanceInterval, "Interval for updating nodes health status")
issueSecretCmd.Flags().Duration(poolStreamTimeoutFlag, defaultPoolStreamTimeout, "Timeout for individual operation in streaming RPC")
issueSecretCmd.Flags().String(attributesFlag, "", "User attributes in form of Key1=Value1,Key2=Value2 (note: you cannot override system attributes)")
issueSecretCmd.Flags().String(accessKeyIDFlag, "", "Access key id of s3 credential that must be created")
issueSecretCmd.Flags().String(secretAccessKeyFlag, "", "Secret access key of s3 credential that must be used")
issueSecretCmd.Flags().String(rpcEndpointFlag, "", "NEO node RPC address (must be provided if container-id is nns name)")
_ = issueSecretCmd.MarkFlagRequired(walletFlag)
_ = issueSecretCmd.MarkFlagRequired(peerFlag)
@ -91,14 +109,6 @@ func runIssueSecretCmd(cmd *cobra.Command, _ []string) error {
return wrapPreparationError(fmt.Errorf("failed to load frostfs private key: %s", err))
}
var cnrID cid.ID
containerID := viper.GetString(containerIDFlag)
if len(containerID) > 0 {
if err = cnrID.DecodeString(containerID); err != nil {
return wrapPreparationError(fmt.Errorf("failed to parse auth container id: %s", err))
}
}
var gatesPublicKeys []*keys.PublicKey
for _, keyStr := range viper.GetStringSlice(gatePublicKeyFlag) {
gpk, err := keys.NewPublicKeyFromString(keyStr)
@ -137,17 +147,29 @@ func runIssueSecretCmd(cmd *cobra.Command, _ []string) error {
return wrapFrostFSInitError(fmt.Errorf("failed to create FrostFS component: %s", err))
}
var accessBox cid.ID
if viper.IsSet(containerIDFlag) {
if accessBox, err = util.ResolveContainerID(viper.GetString(containerIDFlag), viper.GetString(rpcEndpointFlag)); err != nil {
return wrapPreparationError(fmt.Errorf("resolve accessbox container id (make sure you provided %s): %w", rpcEndpointFlag, err))
}
} else if accessBox, err = createAccessBox(ctx, frostFS, key, log); err != nil {
return wrapPreparationError(err)
}
accessKeyID, secretAccessKey, err := parseAccessKeys()
if err != nil {
return wrapPreparationError(err)
}
customAttrs, err := parseObjectAttrs(viper.GetString(attributesFlag))
if err != nil {
return wrapPreparationError(fmt.Errorf("failed to parse attributes: %s", err))
}
issueSecretOptions := &authmate.IssueSecretOptions{
Container: authmate.ContainerOptions{
ID: cnrID,
FriendlyName: viper.GetString(containerFriendlyNameFlag),
PlacementPolicy: viper.GetString(containerPlacementPolicyFlag),
},
Container: accessBox,
AccessKeyID: accessKeyID,
SecretAccessKey: secretAccessKey,
FrostFSKey: key,
GatesPublicKeys: gatesPublicKeys,
Impersonate: true,
@ -164,3 +186,59 @@ func runIssueSecretCmd(cmd *cobra.Command, _ []string) error {
}
return nil
}
func parseAccessKeys() (accessKeyID, secretAccessKey string, err error) {
accessKeyID = viper.GetString(accessKeyIDFlag)
secretAccessKey = viper.GetString(secretAccessKeyFlag)
if accessKeyID == "" && secretAccessKey != "" || accessKeyID != "" && secretAccessKey == "" {
return "", "", fmt.Errorf("flags %s and %s must be both provided or not", accessKeyIDFlag, secretAccessKeyFlag)
}
if accessKeyID != "" {
if !isCustomCreds(accessKeyID) {
return "", "", fmt.Errorf("invalid custom AccessKeyID format: %s", accessKeyID)
}
if !checkAccessKeyLength(accessKeyID) {
return "", "", fmt.Errorf("invalid custom AccessKeyID length: %s", accessKeyID)
}
if !checkAccessKeyLength(secretAccessKey) {
return "", "", fmt.Errorf("invalid custom SecretAccessKey length: %s", secretAccessKey)
}
}
return accessKeyID, secretAccessKey, nil
}
func isCustomCreds(accessKeyID string) bool {
var addr oid.Address
return addr.DecodeString(strings.ReplaceAll(accessKeyID, "0", "/")) != nil
}
func checkAccessKeyLength(key string) bool {
return 4 <= len(key) && len(key) <= 128
}
func createAccessBox(ctx context.Context, frostFS *frostfs.AuthmateFrostFS, key *keys.PrivateKey, log *zap.Logger) (cid.ID, error) {
friendlyName := viper.GetString(containerFriendlyNameFlag)
placementPolicy := viper.GetString(containerPlacementPolicyFlag)
log.Info(logs.CreateContainer, zap.String("friendly_name", friendlyName), zap.String("placement_policy", placementPolicy))
prm := authmate.PrmContainerCreate{
FriendlyName: friendlyName,
}
user.IDFromKey(&prm.Owner, key.PrivateKey.PublicKey)
if err := prm.Policy.DecodeString(placementPolicy); err != nil {
return cid.ID{}, fmt.Errorf("failed to build placement policy: %w", err)
}
accessBox, err := frostFS.CreateContainer(ctx, prm)
if err != nil {
return cid.ID{}, fmt.Errorf("create container in FrostFS: %w", err)
}
return accessBox, nil
}

View file

@ -4,7 +4,6 @@ import (
"context"
"fmt"
"os"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/authmate"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/wallet"
@ -24,7 +23,6 @@ var obtainSecretCmd = &cobra.Command{
const (
gateWalletFlag = "gate-wallet"
gateAddressFlag = "gate-address"
accessKeyIDFlag = "access-key-id"
)
const (
@ -38,10 +36,12 @@ func initObtainSecretCmd() {
obtainSecretCmd.Flags().String(gateWalletFlag, "", "Path to the s3 gateway wallet to decrypt accessbox")
obtainSecretCmd.Flags().String(gateAddressFlag, "", "Address of the s3 gateway wallet account")
obtainSecretCmd.Flags().String(accessKeyIDFlag, "", "Access key id of s3 credential for which secret must be obtained")
obtainSecretCmd.Flags().String(containerIDFlag, "", "CID or NNS name of auth container that contains provided credential (must be provided if custom access key id is used)")
obtainSecretCmd.Flags().Duration(poolDialTimeoutFlag, defaultPoolDialTimeout, "Timeout for connection to the node in pool to be established")
obtainSecretCmd.Flags().Duration(poolHealthcheckTimeoutFlag, defaultPoolHealthcheckTimeout, "Timeout for request to node to decide if it is alive")
obtainSecretCmd.Flags().Duration(poolRebalanceIntervalFlag, defaultPoolRebalanceInterval, "Interval for updating nodes health status")
obtainSecretCmd.Flags().Duration(poolStreamTimeoutFlag, defaultPoolStreamTimeout, "Timeout for individual operation in streaming RPC")
obtainSecretCmd.Flags().String(rpcEndpointFlag, "", "NEO node RPC address (must be provided if container-id is nns name)")
_ = obtainSecretCmd.MarkFlagRequired(walletFlag)
_ = obtainSecretCmd.MarkFlagRequired(peerFlag)
@ -81,8 +81,14 @@ func runObtainSecretCmd(cmd *cobra.Command, _ []string) error {
return wrapFrostFSInitError(cli.Exit(fmt.Sprintf("failed to create FrostFS component: %s", err), 2))
}
accessBox, accessKeyID, _, err := getAccessBoxID()
if err != nil {
return wrapPreparationError(err)
}
obtainSecretOptions := &authmate.ObtainSecretOptions{
SecretAddress: strings.Replace(viper.GetString(accessKeyIDFlag), "0", "/", 1),
Container: accessBox,
AccessKeyID: accessKeyID,
GatePrivateKey: gateKey,
}

View file

@ -7,6 +7,7 @@ import (
"os"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-contract/commonclient"
"git.frostfs.info/TrueCloudLab/frostfs-contract/policy"
ffsidContract "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/frostfsid/contract"
policyContact "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/policy/contract"
@ -116,6 +117,10 @@ func initFrostFSIDContract(ctx context.Context, log *zap.Logger, key *keys.Priva
Contract: viper.GetString(frostfsIDContractFlag),
ProxyContract: viper.GetString(proxyContractFlag),
Key: key,
Waiter: commonclient.WaiterOptions{
IgnoreAlreadyExistsError: false,
VerifyExecResults: true,
},
}
cli, err := ffsidContract.New(ctx, cfg)

View file

@ -68,4 +68,7 @@ GoVersion: {{ runtimeVersion }}
rootCmd.AddCommand(registerUserCmd)
initRegisterUserCmd()
rootCmd.AddCommand(signCmd)
initSignCmd()
}

View file

@ -0,0 +1,115 @@
package modules
import (
"errors"
"fmt"
"os"
"strings"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
var signCmd = &cobra.Command{
Use: "sign",
Short: "Sign arbitrary data using AWS Signature Version 4",
Long: `Generate signature for provided data using AWS credentials. Credentials must be placed in ~/.aws/credentials.
You can provide profile to load using --profile flag or explicitly provide credentials and region using
--aws-access-key-id, --aws-secret-access-key, --region.
Note to override credentials you must provide both access key and secret key.`,
Example: `frostfs-s3-authmate sign --data some-data
frostfs-s3-authmate sign --data file://data.txt
frostfs-s3-authmate sign --data file://data.txt --profile my-profile --time 2024-09-27
frostfs-s3-authmate sign --data some-data --region ru --service s3 --time 2024-09-27 --aws-access-key-id ETaA2CadPcA7bAkLsML2PbTudXY8uRt2PDjCCwkvRv9s0FDCxWDXYc1SA1vKv8KbyCNsLY2AmAjJ92Vz5rgvsFCy --aws-secret-access-key c2d65ef2980f03f4f495bdebedeeae760496697880d61d106bb9a4e5cd2e0607`,
RunE: runSignCmd,
}
const (
serviceFlag = "s3"
timeFlag = "time"
dataFlag = "data"
)
func initSignCmd() {
signCmd.Flags().StringP(dataFlag, "d", "", "Data to sign. Can be provided as string or as a file ('file://path-to-file')")
signCmd.Flags().String(profileFlag, "", "AWS profile to load")
signCmd.Flags().String(serviceFlag, "s3", "AWS service name to form signature")
signCmd.Flags().String(timeFlag, "", "Signing time in '2006-01-02' format (default is current UTC time)")
signCmd.Flags().String(regionFlag, "", "AWS region to use in signature (default is taken from ~/.aws/config)")
signCmd.Flags().String(awsAccessKeyIDFlag, "", "AWS access key id to sign data (default is taken from ~/.aws/credentials)")
signCmd.Flags().String(awsSecretAccessKeyFlag, "", "AWS secret access key to sign data (default is taken from ~/.aws/credentials)")
_ = signCmd.MarkFlagRequired(dataFlag)
}
func runSignCmd(cmd *cobra.Command, _ []string) error {
var cfg aws.Config
if region := viper.GetString(regionFlag); region != "" {
cfg.Region = &region
}
accessKeyID := viper.GetString(awsAccessKeyIDFlag)
secretAccessKey := viper.GetString(awsSecretAccessKeyFlag)
if accessKeyID != "" && secretAccessKey != "" {
cfg.Credentials = credentials.NewStaticCredentialsFromCreds(credentials.Value{
AccessKeyID: accessKeyID,
SecretAccessKey: secretAccessKey,
})
} else if accessKeyID != "" || secretAccessKey != "" {
return wrapPreparationError(fmt.Errorf("both flags '%s' and '%s' must be provided", accessKeyIDFlag, awsSecretAccessKeyFlag))
}
sess, err := session.NewSessionWithOptions(session.Options{
Config: cfg,
Profile: viper.GetString(profileFlag),
SharedConfigState: session.SharedConfigEnable,
})
if err != nil {
return wrapPreparationError(fmt.Errorf("couldn't get aws credentials: %w", err))
}
data := viper.GetString(dataFlag)
if strings.HasPrefix(data, "file://") {
dataToSign, err := os.ReadFile(data[7:])
if err != nil {
return wrapPreparationError(fmt.Errorf("read data file: %w", err))
}
data = string(dataToSign)
}
creds, err := sess.Config.Credentials.Get()
if err != nil {
return wrapPreparationError(fmt.Errorf("get creds: %w", err))
}
if sess.Config.Region == nil || *sess.Config.Region == "" {
return wrapPreparationError(errors.New("missing region"))
}
service := viper.GetString(serviceFlag)
if service == "" {
return wrapPreparationError(errors.New("missing service"))
}
signTime := viper.GetTime(timeFlag)
if signTime.IsZero() {
signTime = time.Now()
}
signature := auth.SignStr(creds.SecretAccessKey, service, *sess.Config.Region, signTime, data)
cmd.Println("service:", service)
cmd.Println("region:", *sess.Config.Region)
cmd.Println("time:", signTime.UTC().Format("20060102"))
cmd.Println("accessKeyId:", creds.AccessKeyID)
cmd.Printf("secretAccessKey: [****************%s]\n", creds.SecretAccessKey[max(0, len(creds.SecretAccessKey)-4):])
cmd.Println("signature:", signature)
return nil
}

View file

@ -4,11 +4,9 @@ import (
"context"
"fmt"
"os"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/authmate"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/wallet"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/spf13/cobra"
"github.com/spf13/viper"
@ -33,13 +31,15 @@ func initUpdateSecretCmd() {
updateSecretCmd.Flags().String(peerFlag, "", "Address of a frostfs peer to connect to")
updateSecretCmd.Flags().String(gateWalletFlag, "", "Path to the s3 gateway wallet to decrypt accessbox")
updateSecretCmd.Flags().String(gateAddressFlag, "", "Address of the s3 gateway wallet account")
updateSecretCmd.Flags().String(accessKeyIDFlag, "", "Access key id of s3 credential for which secret must be obtained")
updateSecretCmd.Flags().String(accessKeyIDFlag, "", "Access key id of s3 credential for which secret must be updatedd")
updateSecretCmd.Flags().String(containerIDFlag, "", "CID or NNS name of auth container that contains provided credential (must be provided if custom access key id is used)")
updateSecretCmd.Flags().StringSlice(gatePublicKeyFlag, nil, "Public 256r1 key of a gate (use flags repeatedly for multiple gates or separate them by comma)")
updateSecretCmd.Flags().Duration(poolDialTimeoutFlag, defaultPoolDialTimeout, "Timeout for connection to the node in pool to be established")
updateSecretCmd.Flags().Duration(poolHealthcheckTimeoutFlag, defaultPoolHealthcheckTimeout, "Timeout for request to node to decide if it is alive")
updateSecretCmd.Flags().Duration(poolRebalanceIntervalFlag, defaultPoolRebalanceInterval, "Interval for updating nodes health status")
updateSecretCmd.Flags().Duration(poolStreamTimeoutFlag, defaultPoolStreamTimeout, "Timeout for individual operation in streaming RPC")
updateSecretCmd.Flags().String(attributesFlag, "", "User attributes in form of Key1=Value1,Key2=Value2 (note: you cannot override system attributes)")
updateSecretCmd.Flags().String(rpcEndpointFlag, "", "NEO node RPC address (must be provided if container-id is nns name)")
_ = updateSecretCmd.MarkFlagRequired(walletFlag)
_ = updateSecretCmd.MarkFlagRequired(peerFlag)
@ -66,10 +66,9 @@ func runUpdateSecretCmd(cmd *cobra.Command, _ []string) error {
return wrapPreparationError(fmt.Errorf("failed to load s3 gate private key: %s", err))
}
var accessBoxAddress oid.Address
credAddr := strings.Replace(viper.GetString(accessKeyIDFlag), "0", "/", 1)
if err = accessBoxAddress.DecodeString(credAddr); err != nil {
return wrapPreparationError(fmt.Errorf("failed to parse creds address: %w", err))
accessBox, accessKeyID, isCustom, err := getAccessBoxID()
if err != nil {
return wrapPreparationError(err)
}
var gatesPublicKeys []*keys.PublicKey
@ -101,7 +100,9 @@ func runUpdateSecretCmd(cmd *cobra.Command, _ []string) error {
}
updateSecretOptions := &authmate.UpdateSecretOptions{
Address: accessBoxAddress,
ContainerID: accessBox,
AccessKeyID: accessKeyID,
IsCustom: isCustom,
FrostFSKey: key,
GatesPublicKeys: gatesPublicKeys,
GatePrivateKey: gateKey,

View file

@ -3,6 +3,7 @@ package modules
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"strings"
@ -11,8 +12,11 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/authmate"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/util"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/pool"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/spf13/viper"
@ -163,3 +167,23 @@ func parseObjectAttrs(attributes string) ([]object.Attribute, error) {
return attrs, nil
}
func getAccessBoxID() (cid.ID, string, bool, error) {
accessKeyID := viper.GetString(accessKeyIDFlag)
var accessBoxAddress oid.Address
if err := accessBoxAddress.DecodeString(strings.Replace(accessKeyID, "0", "/", 1)); err == nil {
return accessBoxAddress.Container(), accessKeyID, false, nil
}
if !viper.IsSet(containerIDFlag) {
return cid.ID{}, "", false, errors.New("accessbox parameter must be set when custom access key id is used")
}
accessBox, err := util.ResolveContainerID(viper.GetString(containerIDFlag), viper.GetString(rpcEndpointFlag))
if err != nil {
return cid.ID{}, "", false, fmt.Errorf("resolve accessbox container id (make sure you provided %s): %w", rpcEndpointFlag, err)
}
return accessBox, accessKeyID, true, nil
}

View file

@ -2,6 +2,7 @@ package main
import (
"context"
"crypto/x509"
"encoding/hex"
"encoding/xml"
"errors"
@ -11,11 +12,13 @@ import (
"os"
"os/signal"
"runtime/debug"
"strings"
"sync"
"syscall"
"time"
v2container "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
"git.frostfs.info/TrueCloudLab/frostfs-contract/commonclient"
"git.frostfs.info/TrueCloudLab/frostfs-observability/tracing"
grpctracing "git.frostfs.info/TrueCloudLab/frostfs-observability/tracing/grpc"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
@ -86,12 +89,14 @@ type (
appSettings struct {
logLevel zap.AtomicLevel
httpLogging s3middleware.LogHTTPConfig
maxClient maxClientsConfig
defaultMaxAge int
reconnectInterval time.Duration
resolveZoneList []string
isResolveListAllow bool // True if ResolveZoneList contains allowed zones
frostfsidValidation bool
accessbox *cid.ID
mu sync.RWMutex
namespaces Namespaces
@ -128,18 +133,7 @@ type (
func newApp(ctx context.Context, log *Logger, v *viper.Viper) *App {
objPool, treePool, key := getPools(ctx, log.logger, v)
cfg := tokens.Config{
FrostFS: frostfs.NewAuthmateFrostFS(frostfs.NewFrostFS(objPool, key), log.logger),
Key: key,
CacheConfig: getAccessBoxCacheConfig(v, log.logger),
RemovingCheckAfterDurations: fetchRemovingCheckInterval(v, log.logger),
}
// prepare auth center
ctr := auth.New(tokens.New(cfg), v.GetStringSlice(cfgAllowedAccessKeyIDPrefixes))
app := &App{
ctr: ctr,
log: log.logger,
cfg: v,
pool: objPool,
@ -158,6 +152,8 @@ func newApp(ctx context.Context, log *Logger, v *viper.Viper) *App {
}
func (a *App) init(ctx context.Context) {
a.initResolver()
a.initAuthCenter(ctx)
a.setRuntimeParameters()
a.initFrostfsID(ctx)
a.initPolicyStorage(ctx)
@ -167,9 +163,26 @@ func (a *App) init(ctx context.Context) {
a.initTracing(ctx)
}
func (a *App) initLayer(ctx context.Context) {
a.initResolver()
func (a *App) initAuthCenter(ctx context.Context) {
if a.cfg.IsSet(cfgContainersAccessBox) {
cnrID, err := a.resolveContainerID(ctx, cfgContainersAccessBox)
if err != nil {
a.log.Fatal(logs.CouldNotFetchAccessBoxContainerInfo, zap.Error(err))
}
a.settings.accessbox = &cnrID
}
cfg := tokens.Config{
FrostFS: frostfs.NewAuthmateFrostFS(frostfs.NewFrostFS(a.pool, a.key), a.log),
Key: a.key,
CacheConfig: getAccessBoxCacheConfig(a.cfg, a.log),
RemovingCheckAfterDurations: fetchRemovingCheckInterval(a.cfg, a.log),
}
a.ctr = auth.New(tokens.New(cfg), a.cfg.GetStringSlice(cfgAllowedAccessKeyIDPrefixes), a.settings)
}
func (a *App) initLayer(ctx context.Context) {
// prepare random key for anonymous requests
randomKey, err := keys.NewPrivateKey()
if err != nil {
@ -216,6 +229,7 @@ func (a *App) initLayer(ctx context.Context) {
func newAppSettings(log *Logger, v *viper.Viper) *appSettings {
settings := &appSettings{
logLevel: log.lvl,
httpLogging: s3middleware.LogHTTPConfig{},
maxClient: newMaxClients(v),
defaultMaxAge: fetchDefaultMaxAge(v, log.logger),
reconnectInterval: fetchReconnectInterval(v),
@ -236,6 +250,7 @@ func newAppSettings(log *Logger, v *viper.Viper) *appSettings {
func (s *appSettings) update(v *viper.Viper, log *zap.Logger) {
namespaceHeader := v.GetString(cfgResolveNamespaceHeader)
nsConfig, defaultNamespaces := fetchNamespacesConfig(log, v)
vhsNamespacesEnabled := s.prepareVHSNamespaces(v, log, defaultNamespaces)
defaultXMLNS := v.GetBool(cfgKludgeUseDefaultXMLNS)
bypassContentEncodingInChunks := v.GetBool(cfgKludgeBypassContentEncodingCheckInChunks)
clientCut := v.GetBool(cfgClientCut)
@ -250,11 +265,22 @@ func (s *appSettings) update(v *viper.Viper, log *zap.Logger) {
vhsEnabled := v.GetBool(cfgVHSEnabled)
vhsHeader := v.GetString(cfgVHSHeader)
servernameHeader := v.GetString(cfgServernameHeader)
vhsNamespacesEnabled := s.prepareVHSNamespaces(v, log)
httpLoggingEnabled := v.GetBool(cfgHTTPLoggingEnabled)
httpLoggingMaxBody := v.GetInt64(cfgHTTPLoggingMaxBody)
httpLoggingMaxLogSize := v.GetInt(cfgHTTPLoggingMaxLogSize)
httpLoggingOutputPath := v.GetString(cfgHTTPLoggingDestination)
httpLoggingUseGzip := v.GetBool(cfgHTTPLoggingGzip)
s.mu.Lock()
defer s.mu.Unlock()
s.httpLogging.Enabled = httpLoggingEnabled
s.httpLogging.MaxBody = httpLoggingMaxBody
s.httpLogging.MaxLogSize = httpLoggingMaxLogSize
s.httpLogging.OutputPath = httpLoggingOutputPath
s.httpLogging.UseGzip = httpLoggingUseGzip
s.httpLogging.InitHTTPLogger(log)
s.namespaceHeader = namespaceHeader
s.defaultNamespaces = defaultNamespaces
s.namespaces = nsConfig.Namespaces
@ -275,11 +301,14 @@ func (s *appSettings) update(v *viper.Viper, log *zap.Logger) {
s.vhsNamespacesEnabled = vhsNamespacesEnabled
}
func (s *appSettings) prepareVHSNamespaces(v *viper.Viper, log *zap.Logger) map[string]bool {
func (s *appSettings) prepareVHSNamespaces(v *viper.Viper, log *zap.Logger, defaultNamespaces []string) map[string]bool {
nsMap := fetchVHSNamespaces(v, log)
vhsNamespaces := make(map[string]bool, len(nsMap))
for ns, flag := range nsMap {
vhsNamespaces[s.ResolveNamespaceAlias(ns)] = flag
if slices.Contains(defaultNamespaces, ns) {
ns = defaultNamespace
}
vhsNamespaces[ns] = flag
}
return vhsNamespaces
@ -361,6 +390,13 @@ func (s *appSettings) DefaultCopiesNumbers(namespace string) []uint32 {
return s.namespaces[namespace].CopiesNumbers[defaultConstraintName]
}
func (s *appSettings) LogHTTPConfig() s3middleware.LogHTTPConfig {
s.mu.RLock()
defer s.mu.RUnlock()
return s.httpLogging
}
func (s *appSettings) NewXMLDecoder(r io.Reader) *xml.Decoder {
dec := xml.NewDecoder(r)
dec.CharsetReader = func(charset string, reader io.Reader) (io.Reader, error) {
@ -404,12 +440,12 @@ func (s *appSettings) NamespaceHeader() string {
return s.namespaceHeader
}
func (s *appSettings) FormContainerZone(ns string) (zone string, isDefault bool) {
func (s *appSettings) FormContainerZone(ns string) string {
if len(ns) == 0 {
return v2container.SysAttributeZoneDefault, true
return v2container.SysAttributeZoneDefault
}
return ns + ".ns", false
return ns + ".ns"
}
func (s *appSettings) isDefaultNamespace(ns string) bool {
@ -457,6 +493,14 @@ func (s *appSettings) RetryStrategy() handler.RetryStrategy {
return s.retryStrategy
}
func (s *appSettings) AccessBoxContainer() (cid.ID, bool) {
if s.accessbox != nil {
return *s.accessbox, true
}
return cid.ID{}, false
}
func (a *App) initAPI(ctx context.Context) {
a.initLayer(ctx)
a.initHandler()
@ -480,6 +524,10 @@ func (a *App) initFrostfsID(ctx context.Context) {
Contract: a.cfg.GetString(cfgFrostfsIDContract),
ProxyContract: a.cfg.GetString(cfgProxyContract),
Key: a.key,
Waiter: commonclient.WaiterOptions{
IgnoreAlreadyExistsError: false,
VerifyExecResults: true,
},
})
if err != nil {
a.log.Fatal(logs.InitFrostfsIDContractFailed, zap.Error(err))
@ -501,6 +549,10 @@ func (a *App) initPolicyStorage(ctx context.Context) {
Contract: a.cfg.GetString(cfgPolicyContract),
ProxyContract: a.cfg.GetString(cfgProxyContract),
Key: a.key,
Waiter: commonclient.WaiterOptions{
IgnoreAlreadyExistsError: false,
VerifyExecResults: true,
},
})
if err != nil {
a.log.Fatal(logs.InitPolicyContractFailed, zap.Error(err))
@ -525,7 +577,6 @@ func (a *App) getResolverConfig() *resolver.Config {
return &resolver.Config{
FrostFS: frostfs.NewResolverFrostFS(a.pool),
RPCAddress: a.cfg.GetString(cfgRPCEndpoint),
Settings: a.settings,
}
}
@ -556,6 +607,22 @@ func (a *App) initTracing(ctx context.Context) {
InstanceID: instanceID,
Version: version.Version,
}
if trustedCa := a.cfg.GetString(cfgTracingTrustedCa); trustedCa != "" {
caBytes, err := os.ReadFile(trustedCa)
if err != nil {
a.log.Warn(logs.FailedToInitializeTracing, zap.Error(err))
return
}
certPool := x509.NewCertPool()
ok := certPool.AppendCertsFromPEM(caBytes)
if !ok {
a.log.Warn(logs.FailedToInitializeTracing, zap.String("error", "can't fill cert pool by ca cert"))
return
}
cfg.ServerCaCertPool = certPool
}
updated, err := tracing.Setup(ctx, cfg)
if err != nil {
a.log.Warn(logs.FailedToInitializeTracing, zap.Error(err))
@ -599,7 +666,8 @@ func getPools(ctx context.Context, logger *zap.Logger, cfg *viper.Viper) (*pool.
prmTree.SetKey(key)
logger.Info(logs.UsingCredentials, zap.String("FrostFS", hex.EncodeToString(key.PublicKey().Bytes())))
for _, peer := range fetchPeers(logger, cfg) {
peers := fetchPeers(logger, cfg)
for _, peer := range peers {
prm.AddNode(peer)
prmTree.AddNode(peer)
}
@ -630,18 +698,12 @@ func getPools(ctx context.Context, logger *zap.Logger, cfg *viper.Viper) (*pool.
prmTree.SetMaxRequestAttempts(cfg.GetInt(cfgTreePoolMaxAttempts))
var apiGRPCDialOpts []grpc.DialOption
var treeGRPCDialOpts []grpc.DialOption
if cfg.GetBool(cfgTracingEnabled) {
interceptors := []grpc.DialOption{
grpc.WithUnaryInterceptor(grpctracing.NewUnaryClientInteceptor()),
grpc.WithStreamInterceptor(grpctracing.NewStreamClientInterceptor()),
}
treeGRPCDialOpts = append(treeGRPCDialOpts, interceptors...)
apiGRPCDialOpts = append(apiGRPCDialOpts, interceptors...)
interceptors := []grpc.DialOption{
grpc.WithUnaryInterceptor(grpctracing.NewUnaryClientInteceptor()),
grpc.WithStreamInterceptor(grpctracing.NewStreamClientInterceptor()),
}
prm.SetGRPCDialOptions(apiGRPCDialOpts...)
prmTree.SetGRPCDialOptions(treeGRPCDialOpts...)
prm.SetGRPCDialOptions(interceptors...)
prmTree.SetGRPCDialOptions(interceptors...)
p, err := pool.NewPool(prm)
if err != nil {
@ -652,6 +714,8 @@ func getPools(ctx context.Context, logger *zap.Logger, cfg *viper.Viper) (*pool.
logger.Fatal(logs.FailedToDialConnectionPool, zap.Error(err))
}
prmTree.SetBootstrapAddress(peers[0].Address())
treePool, err := treepool.NewPool(prmTree)
if err != nil {
logger.Fatal(logs.FailedToCreateTreePool, zap.Error(err))
@ -920,6 +984,8 @@ func getCacheOptions(v *viper.Viper, l *zap.Logger) *layer.CachesConfig {
cacheCfg.AccessControl.Lifetime = fetchCacheLifetime(v, l, cfgAccessControlCacheLifetime, cacheCfg.AccessControl.Lifetime)
cacheCfg.AccessControl.Size = fetchCacheSize(v, l, cfgAccessControlCacheSize, cacheCfg.AccessControl.Size)
cacheCfg.NetworkInfo.Lifetime = fetchCacheLifetime(v, l, cfgNetworkInfoCacheLifetime, cacheCfg.NetworkInfo.Lifetime)
return cacheCfg
}
@ -1058,16 +1124,30 @@ func (a *App) tryReconnect(ctx context.Context, sr *http.Server) bool {
}
func (a *App) fetchContainerInfo(ctx context.Context, cfgKey string) (info *data.BucketInfo, err error) {
cnrID, err := a.resolveContainerID(ctx, cfgKey)
if err != nil {
return nil, err
}
return getContainerInfo(ctx, cnrID, a.pool)
}
func (a *App) resolveContainerID(ctx context.Context, cfgKey string) (cid.ID, error) {
containerString := a.cfg.GetString(cfgKey)
var id cid.ID
if err = id.DecodeString(containerString); err != nil {
if id, err = a.bucketResolver.Resolve(ctx, containerString); err != nil {
return nil, fmt.Errorf("resolve container name %s: %w", containerString, err)
if err := id.DecodeString(containerString); err != nil {
i := strings.Index(containerString, ".")
if i < 0 {
return cid.ID{}, fmt.Errorf("invalid container address: %s", containerString)
}
if id, err = a.bucketResolver.Resolve(ctx, containerString[i+1:], containerString[:i]); err != nil {
return cid.ID{}, fmt.Errorf("resolve container address %s: %w", containerString, err)
}
}
return getContainerInfo(ctx, id, a.pool)
return id, nil
}
func getContainerInfo(ctx context.Context, id cid.ID, frostFSPool *pool.Pool) (*data.BucketInfo, error) {

View file

@ -41,6 +41,8 @@ const (
defaultStreamTimeout = 10 * time.Second
defaultShutdownTimeout = 15 * time.Second
defaultLoggerSamplerInterval = 1 * time.Second
defaultGracefulCloseOnSwitchTimeout = 10 * time.Second
defaultPoolErrorThreshold uint32 = 100
@ -81,6 +83,19 @@ const ( // Settings.
cfgLoggerLevel = "logger.level"
cfgLoggerDestination = "logger.destination"
cfgLoggerSamplingEnabled = "logger.sampling.enabled"
cfgLoggerSamplingInitial = "logger.sampling.initial"
cfgLoggerSamplingThereafter = "logger.sampling.thereafter"
cfgLoggerSamplingInterval = "logger.sampling.interval"
// HttpLogging.
cfgHTTPLoggingEnabled = "http_logging.enabled"
cfgHTTPLoggingMaxBody = "http_logging.max_body"
cfgHTTPLoggingMaxLogSize = "http_logging.max_log_size"
cfgHTTPLoggingDestination = "http_logging.destination"
cfgHTTPLoggingGzip = "http_logging.gzip"
cfgHTTPLoggingLogResponse = "http_logging.log_response"
// Wallet.
cfgWalletPath = "wallet.path"
cfgWalletAddress = "wallet.address"
@ -122,6 +137,7 @@ const ( // Settings.
cfgMorphPolicyCacheSize = "cache.morph_policy.size"
cfgFrostfsIDCacheLifetime = "cache.frostfsid.lifetime"
cfgFrostfsIDCacheSize = "cache.frostfsid.size"
cfgNetworkInfoCacheLifetime = "cache.network_info.lifetime"
cfgAccessBoxCacheRemovingCheckInterval = "cache.accessbox.removing_check_interval"
@ -144,9 +160,10 @@ const ( // Settings.
cfgPProfAddress = "pprof.address"
// Tracing.
cfgTracingEnabled = "tracing.enabled"
cfgTracingExporter = "tracing.exporter"
cfgTracingEndpoint = "tracing.endpoint"
cfgTracingEnabled = "tracing.enabled"
cfgTracingExporter = "tracing.exporter"
cfgTracingEndpoint = "tracing.endpoint"
cfgTracingTrustedCa = "tracing.trusted_ca"
cfgListenDomains = "listen_domains"
@ -191,6 +208,7 @@ const ( // Settings.
// Containers.
cfgContainersCORS = "containers.cors"
cfgContainersLifecycle = "containers.lifecycle"
cfgContainersAccessBox = "containers.accessbox"
// Command line args.
cmdHelp = "help"
@ -778,6 +796,18 @@ func newSettings() *viper.Viper {
// logger:
v.SetDefault(cfgLoggerLevel, "debug")
v.SetDefault(cfgLoggerDestination, "stdout")
v.SetDefault(cfgLoggerSamplingEnabled, false)
v.SetDefault(cfgLoggerSamplingThereafter, 100)
v.SetDefault(cfgLoggerSamplingInitial, 100)
v.SetDefault(cfgLoggerSamplingInterval, defaultLoggerSamplerInterval)
// http logger
v.SetDefault(cfgHTTPLoggingEnabled, false)
v.SetDefault(cfgHTTPLoggingMaxBody, 1024)
v.SetDefault(cfgHTTPLoggingMaxLogSize, 50)
v.SetDefault(cfgHTTPLoggingDestination, "stdout")
v.SetDefault(cfgHTTPLoggingGzip, false)
v.SetDefault(cfgHTTPLoggingLogResponse, true)
// pool:
v.SetDefault(cfgPoolErrorThreshold, defaultPoolErrorThreshold)
@ -1013,9 +1043,9 @@ func pickLogger(v *viper.Viper) *Logger {
switch dest {
case destinationStdout:
return newStdoutLogger(lvl)
return newStdoutLogger(v, lvl)
case destinationJournald:
return newJournaldLogger(lvl)
return newJournaldLogger(v, lvl)
default:
panic(fmt.Sprintf("wrong destination for logger: %s", dest))
}
@ -1028,53 +1058,74 @@ func pickLogger(v *viper.Viper) *Logger {
// - parameterized level (debug by default)
// - console encoding
// - ISO8601 time encoding
// - sampling intervals
//
// and atomic log level to dynamically change it.
//
// Logger records a stack trace for all messages at or above fatal level.
//
// See also zapcore.Level, zap.NewProductionConfig, zap.AddStacktrace.
func newStdoutLogger(lvl zapcore.Level) *Logger {
c := zap.NewProductionConfig()
c.Level = zap.NewAtomicLevelAt(lvl)
c.Encoding = "console"
c.EncoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder
func newStdoutLogger(v *viper.Viper, lvl zapcore.Level) *Logger {
stdout := zapcore.AddSync(os.Stderr)
level := zap.NewAtomicLevelAt(lvl)
l, err := c.Build(
zap.AddStacktrace(zap.NewAtomicLevelAt(zap.FatalLevel)),
)
if err != nil {
panic(fmt.Sprintf("build zap logger instance: %v", err))
}
consoleOutCore := zapcore.NewCore(newLogEncoder(), stdout, level)
consoleOutCore = samplingEnabling(v, consoleOutCore)
return &Logger{
logger: l,
lvl: c.Level,
logger: zap.New(consoleOutCore, zap.AddStacktrace(zap.NewAtomicLevelAt(zap.FatalLevel))),
lvl: level,
}
}
func newJournaldLogger(lvl zapcore.Level) *Logger {
c := zap.NewProductionConfig()
c.EncoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder
c.Level = zap.NewAtomicLevelAt(lvl)
func newJournaldLogger(v *viper.Viper, lvl zapcore.Level) *Logger {
level := zap.NewAtomicLevelAt(lvl)
encoder := zapjournald.NewPartialEncoder(zapcore.NewConsoleEncoder(c.EncoderConfig), zapjournald.SyslogFields)
encoder := zapjournald.NewPartialEncoder(newLogEncoder(), zapjournald.SyslogFields)
core := zapjournald.NewCore(c.Level, encoder, &journald.Journal{}, zapjournald.SyslogFields)
core := zapjournald.NewCore(level, encoder, &journald.Journal{}, zapjournald.SyslogFields)
coreWithContext := core.With([]zapcore.Field{
zapjournald.SyslogFacility(zapjournald.LogDaemon),
zapjournald.SyslogIdentifier(),
zapjournald.SyslogPid(),
})
coreWithContext = samplingEnabling(v, coreWithContext)
l := zap.New(coreWithContext, zap.AddStacktrace(zap.NewAtomicLevelAt(zap.FatalLevel)))
return &Logger{
logger: l,
lvl: c.Level,
lvl: level,
}
}
func newLogEncoder() zapcore.Encoder {
c := zap.NewProductionEncoderConfig()
c.EncodeTime = zapcore.ISO8601TimeEncoder
return zapcore.NewConsoleEncoder(c)
}
func samplingEnabling(v *viper.Viper, core zapcore.Core) zapcore.Core {
// Zap samples by logging the first cgfLoggerSamplingInitial entries with a given level
// and message within the specified time interval.
// In the above config, only the first cgfLoggerSamplingInitial log entries with the same level and message
// are recorded in cfgLoggerSamplingInterval interval. Every other log entry will be dropped within the interval since
// cfgLoggerSamplingThereafter is specified here.
if v.GetBool(cfgLoggerSamplingEnabled) {
core = zapcore.NewSamplerWithOptions(
core,
v.GetDuration(cfgLoggerSamplingInterval),
v.GetInt(cfgLoggerSamplingInitial),
v.GetInt(cfgLoggerSamplingThereafter),
)
}
return core
}
func getLogLevel(v *viper.Viper) (zapcore.Level, error) {
var lvl zapcore.Level
lvlStr := v.GetString(cfgLoggerLevel)

View file

@ -0,0 +1,50 @@
package playback
import (
"encoding/xml"
"fmt"
"net/http"
)
type MultipartUpload struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ InitiateMultipartUploadResult" json:"-"`
Bucket string `json:"bucket" xml:"Bucket"`
Key string `json:"key" xml:"Key"`
UploadID string `json:"uploadId" xml:"UploadId"`
}
func HandleResponse(r *http.Request, mparts map[string]MultipartUpload, resp []byte, logResponse []byte) error {
var mpart, mpartOld MultipartUpload
if r.Method != "POST" || !r.URL.Query().Has("uploads") {
return nil
}
// get new uploadId from response
err := xml.Unmarshal(resp, &mpart)
if err != nil {
return fmt.Errorf("xml unmarshal error: %w", err)
}
// get old uploadId from logs
err = xml.Unmarshal(logResponse, &mpartOld)
if err != nil {
return fmt.Errorf("xml unmarshal error: %w", err)
}
if mpartOld.UploadID != "" {
mparts[mpartOld.UploadID] = mpart
}
return nil
}
func SwapUploadID(r *http.Request, settings *Settings) error {
var uploadID string
query := r.URL.Query()
uploadID = query.Get("uploadId")
mpart, ok := settings.Multiparts[uploadID]
if !ok {
return fmt.Errorf("no multipart upload with specified uploadId")
}
query.Set("uploadId", mpart.UploadID)
r.URL.RawQuery = query.Encode()
return nil
}

View file

@ -0,0 +1,97 @@
package playback
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"strings"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/detector"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/xmlutils"
v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
"github.com/aws/aws-sdk-go-v2/credentials"
)
type (
httpBody []byte
LoggedRequest struct {
From string `json:"from"`
URI string `json:"URI"`
Method string `json:"method"`
Payload httpBody `json:"payload,omitempty"`
Response httpBody `json:"response,omitempty"`
Query url.Values `json:"query"`
Header http.Header `json:"headers"`
}
Credentials struct {
AccessKey string
SecretKey string
}
Settings struct {
Endpoint string
Creds Credentials
Multiparts map[string]MultipartUpload
Client *http.Client
}
)
func (h *httpBody) UnmarshalJSON(data []byte) error {
unquoted, err := strconv.Unquote(string(data))
if err != nil {
return fmt.Errorf("failed to unquote data: %w", err)
}
detect := detector.NewDetector(strings.NewReader(unquoted), xmlutils.DetectXML)
dataType, err := detect.Detect()
if err != nil {
return fmt.Errorf("failed to detect data: %w", err)
}
reader := xmlutils.ChooseReader(dataType, detect.RestoredReader())
*h, err = io.ReadAll(reader)
if err != nil {
return fmt.Errorf("failed to unmarshal httpbody: %w", err)
}
return nil
}
// Sign replace Authorization header with new Access key id and Signature values.
func Sign(ctx context.Context, r *http.Request, creds Credentials) error {
credProvider := credentials.NewStaticCredentialsProvider(creds.AccessKey, creds.SecretKey, "")
awsCred, err := credProvider.Retrieve(ctx)
if err != nil {
return err
}
authHdr := r.Header.Get(auth.AuthorizationHdr)
authInfo, err := parseAuthHeader(authHdr)
if err != nil {
return err
}
newHeader := strings.Replace(authHdr, authInfo["access_key_id"], creds.AccessKey, 1)
r.Header.Set(auth.AuthorizationHdr, newHeader)
signer := v4.NewSigner()
signatureDateTimeStr := r.Header.Get(api.AmzDate)
signatureDateTime, err := time.Parse("20060102T150405Z", signatureDateTimeStr)
if err != nil {
return err
}
return signer.SignHTTP(ctx, awsCred, r, r.Header.Get(api.AmzContentSha256), authInfo["service"], authInfo["region"], signatureDateTime)
}
func parseAuthHeader(authHeader string) (map[string]string, error) {
authInfo := auth.NewRegexpMatcher(auth.AuthorizationFieldRegexp).GetSubmatches(authHeader)
if len(authInfo) == 0 {
return nil, errors.New("no matches found")
}
return authInfo, nil
}

View file

@ -0,0 +1,98 @@
package playback
import (
"errors"
"strings"
"testing"
"github.com/stretchr/testify/require"
)
var errNoMatches = errors.New("no matches found")
func withoutValue(data map[string]string, field string) map[string]string {
result := make(map[string]string)
for k, v := range data {
result[k] = v
}
result[field] = ""
return result
}
func TestParseAuthHeader(t *testing.T) {
defaultHeader := "AWS4-HMAC-SHA256 Credential=oid0cid/20210809/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2811ccb9e242f41426738fb1f"
defaultAuthInfo := map[string]string{
"access_key_id": "oid0cid",
"service": "s3",
"region": "us-east-1",
"v4_signature": "2811ccb9e242f41426738fb1f",
"signed_header_fields": "host;x-amz-content-sha256;x-amz-date",
"date": "20210809",
}
for _, tc := range []struct {
title string
header string
err error
expected map[string]string
}{
{
title: "correct full header",
header: defaultHeader,
err: nil,
expected: defaultAuthInfo,
},
{
title: "correct with empty region",
header: strings.Replace(defaultHeader, "/us-east-1/", "//", 1),
err: nil,
expected: withoutValue(defaultAuthInfo, "region"),
},
{
title: "empty access key",
header: strings.Replace(defaultHeader, "oid0cid", "", 1),
err: errNoMatches,
expected: nil,
},
{
title: "empty service",
header: strings.Replace(defaultHeader, "/s3/", "//", 1),
err: errNoMatches,
expected: nil,
},
{
title: "empty date",
header: strings.Replace(defaultHeader, "/20210809/", "//", 1),
err: errNoMatches,
expected: nil,
},
{
title: "empty v4_signature",
header: strings.Replace(defaultHeader, "Signature=2811ccb9e242f41426738fb1f",
"Signature=", 1),
err: errNoMatches,
expected: nil,
},
{
title: "empty signed_fields",
header: strings.Replace(defaultHeader, "SignedHeaders=host;x-amz-content-sha256;x-amz-date",
"SignedHeaders=", 1),
err: errNoMatches,
expected: nil,
},
{
title: "empty signed_fields",
header: strings.Replace(defaultHeader, "SignedHeaders=host;x-amz-content-sha256;x-amz-date",
"SignedHeaders=", 1),
err: errNoMatches,
expected: nil,
},
} {
t.Run(tc.title, func(t *testing.T) {
authInfo, err := parseAuthHeader(tc.header)
require.Equal(t, err, tc.err, tc.header)
require.Equal(t, tc.expected, authInfo, tc.header)
})
}
}

20
cmd/s3-playback/main.go Normal file
View file

@ -0,0 +1,20 @@
package main
import (
"context"
"os"
"os/signal"
"syscall"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/cmd/s3-playback/modules"
)
func main() {
ctx, _ := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM, syscall.SIGHUP)
if cmd, err := modules.Execute(ctx); err != nil {
cmd.PrintErrln("Error:", err.Error())
cmd.PrintErrf("Run '%v --help' for usage.\n", cmd.CommandPath())
os.Exit(1)
}
}

View file

@ -0,0 +1,67 @@
package modules
import (
"context"
"os"
"strings"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/version"
"github.com/spf13/cobra"
"github.com/spf13/pflag"
"github.com/spf13/viper"
)
const (
defaultPrintResponseLimit = 1024
cfgConfigPath = "config"
cfgHTTPTimeoutFlag = "http-timeout"
cfgSkipVerifyTLS = "skip-verify-tls"
)
var (
cfgFile string
rootCmd = &cobra.Command{
Use: "frostfs-s3-playback",
Version: version.Version,
Short: "FrostFS S3 Traffic Playback",
Long: "Helps to reproduce s3 commands from log files",
Example: "frostfs-s3-playback [--skip-verify-tls] [--http-timeout <timeout>] " +
"[--version] --config <config_path> <command>",
SilenceUsage: true,
SilenceErrors: true,
PersistentPreRunE: func(cmd *cobra.Command, _ []string) error {
return viper.BindPFlags(cmd.Flags())
},
RunE: func(cmd *cobra.Command, _ []string) error {
return cmd.Help()
},
}
)
func Execute(ctx context.Context) (*cobra.Command, error) {
return rootCmd.ExecuteContextC(ctx)
}
func initConfig() {
viper.SetConfigFile(cfgFile)
_ = viper.ReadInConfig()
}
func init() {
cobra.OnInitialize(initConfig)
cobra.EnableTraverseRunHooks = true
rootCmd.SetGlobalNormalizationFunc(func(_ *pflag.FlagSet, name string) pflag.NormalizedName {
return pflag.NormalizedName(strings.ReplaceAll(name, "_", "-"))
})
rootCmd.PersistentFlags().StringVar(&cfgFile, cfgConfigPath, "", "configuration filepath")
_ = rootCmd.MarkPersistentFlagRequired(cfgConfigPath)
_ = rootCmd.MarkPersistentFlagFilename(cfgConfigPath)
rootCmd.PersistentFlags().Duration(cfgHTTPTimeoutFlag, time.Minute, "http request timeout")
rootCmd.PersistentFlags().Bool(cfgSkipVerifyTLS, false, "skip TLS certificate verification")
rootCmd.SetOut(os.Stdout)
initRunCmd()
rootCmd.AddCommand(runCmd)
}

View file

@ -0,0 +1,208 @@
package modules
import (
"bufio"
"bytes"
"context"
"crypto/md5"
"crypto/sha256"
"crypto/tls"
"encoding/base64"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/cmd/s3-playback/internal/playback"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/detector"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/xmlutils"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
const (
cfgPrintResponseLimit = "print-response-limit"
cfgLogPath = "log"
cfgEndpoint = "endpoint"
awsAccessKey = "credentials.access_key"
awsSecretKey = "credentials.secret_key"
)
var runCmd = &cobra.Command{
Use: "run",
Short: "Send requests from log file",
Long: "Reads the network log file and sends each request to the specified URL",
Example: "frostfs-s3-playback --config <config_path> run [--endpoint=<endpoint>] [--log=<log_path>]",
PersistentPreRunE: func(cmd *cobra.Command, _ []string) (err error) {
viper.SetDefault(cfgPrintResponseLimit, defaultPrintResponseLimit)
return viper.BindPFlags(cmd.Flags())
},
RunE: run,
}
func initRunCmd() {
runCmd.Flags().String(cfgLogPath, "./request.log", "log file path")
runCmd.Flags().String(cfgEndpoint, "", "endpoint URL")
runCmd.Flags().Int(cfgPrintResponseLimit, defaultPrintResponseLimit, "print limit for http response body")
}
func logResponse(cmd *cobra.Command, r *http.Request, resp *http.Response) {
cmd.Println(r.Method, r.URL.RequestURI())
cmd.Println(resp.Status)
if resp.ContentLength == 0 {
return
}
detect := detector.NewDetector(resp.Body, xmlutils.DetectXML)
dataType, err := detect.Detect()
if err != nil {
cmd.PrintErrln("type detection error:", err.Error())
return
}
body := &bytes.Buffer{}
resultWriter := xmlutils.ChooseWriter(dataType, body)
_, err = io.Copy(resultWriter, io.LimitReader(detect.RestoredReader(), viper.GetInt64(cfgPrintResponseLimit)))
if err != nil {
cmd.Println(err)
return
}
if err = resultWriter.Close(); err != nil {
cmd.Printf("could not close response body: %s\n", err)
return
}
cmd.Println(body.String())
cmd.Println()
}
func run(cmd *cobra.Command, _ []string) error {
ctx := cmd.Context()
settings := &playback.Settings{
Endpoint: viper.GetString(cfgEndpoint),
Creds: playback.Credentials{
AccessKey: viper.GetString(awsAccessKey),
SecretKey: viper.GetString(awsSecretKey),
},
Multiparts: make(map[string]playback.MultipartUpload),
Client: &http.Client{
Transport: http.DefaultTransport,
Timeout: viper.GetDuration(cfgHTTPTimeoutFlag),
},
}
file, err := os.Open(viper.GetString(cfgLogPath))
if err != nil {
return err
}
defer file.Close()
reader := bufio.NewReader(file)
if viper.GetBool(cfgSkipVerifyTLS) {
settings.Client.Transport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
}
id := 1
for {
logReq, err := getRequestFromLog(reader)
if err != nil {
if err == io.EOF {
break
}
cmd.PrintErrln(strconv.Itoa(id)+")", "failed to parse request", err)
id++
continue
}
select {
case <-ctx.Done():
return fmt.Errorf("interrupted: %w", ctx.Err())
default:
r, resp, err := playbackRequest(ctx, logReq, settings)
if err != nil {
cmd.PrintErrln(strconv.Itoa(id)+")", "failed to playback request:", err)
id++
continue
}
cmd.Print(strconv.Itoa(id) + ") ")
logResponse(cmd, r, resp)
id++
}
}
return nil
}
func getRequestFromLog(reader *bufio.Reader) (playback.LoggedRequest, error) {
var logReq playback.LoggedRequest
req, err := reader.ReadString('\n')
if err != nil {
return logReq, err
}
err = json.Unmarshal([]byte(req), &logReq)
if err != nil {
return logReq, err
}
return logReq, nil
}
// playbackRequest creates http.Request from LoggedRequest and sends it to specified endpoint.
func playbackRequest(ctx context.Context, logReq playback.LoggedRequest, settings *playback.Settings) (*http.Request, *http.Response, error) {
r, err := prepareRequest(ctx, logReq, settings)
if err != nil {
return nil, nil, fmt.Errorf("failed to prepare request: %w", err)
}
resp, err := settings.Client.Do(r)
if err != nil {
return nil, nil, fmt.Errorf("failed to send request: %w", err)
}
respBody, err := io.ReadAll(resp.Body)
if err != nil {
return nil, nil, fmt.Errorf("failed to read response body: %w", err)
}
defer resp.Body.Close()
if err = playback.HandleResponse(r, settings.Multiparts, respBody, logReq.Response); err != nil {
return nil, nil, fmt.Errorf("failed to register multipart upload: %w", err)
}
resp.Body = io.NopCloser(bytes.NewBuffer(respBody))
return r, resp, nil
}
// prepareRequest creates request from logs and modifies its signature and uploadId (if presents).
func prepareRequest(ctx context.Context, logReq playback.LoggedRequest, settings *playback.Settings) (*http.Request, error) {
r, err := http.NewRequestWithContext(ctx, logReq.Method, settings.Endpoint+logReq.URI, bytes.NewReader(logReq.Payload))
if err != nil {
return nil, err
}
r.Header = logReq.Header
sha256hash := sha256.New()
sha256hash.Write(logReq.Payload)
r.Header.Set(auth.AmzContentSHA256, hex.EncodeToString(sha256hash.Sum(nil)))
if r.Header.Get(api.ContentMD5) != "" {
sha256hash.Reset()
md5hash := md5.New()
md5hash.Write(logReq.Payload)
r.Header.Set(api.ContentMD5, base64.StdEncoding.EncodeToString(md5hash.Sum(nil)))
}
if r.URL.Query().Has("uploadId") {
if err = playback.SwapUploadID(r, settings); err != nil {
return nil, err
}
}
if r.Header.Get(auth.AuthorizationHdr) != "" {
if err = playback.Sign(ctx, r, settings.Creds); err != nil {
return nil, err
}
}
return r, nil
}

View file

@ -51,6 +51,21 @@ S3_GW_CONFIG=/path/to/config/yaml
# Logger
S3_GW_LOGGER_LEVEL=debug
S3_GW_LOGGER_SAMPLING_ENABLED=false
S3_GW_LOGGER_SAMPLING_INITIAL=100
S3_GW_LOGGER_SAMPLING_THEREAFTER=100
S3_GW_LOGGER_SAMPLING_INTERVAL=1s
# HTTP logger
S3_GW_HTTP_LOGGING_ENABLED=false
# max body size to log
S3_GW_HTTP_LOGGING_MAX_BODY=1024
# max log size in Mb
S3_GW_HTTP_LOGGING_MAX_LOG_SIZE=20
# use log compression
S3_GW_HTTP_LOGGING_GZIP=true
# possible destination output values: filesystem path, url, "stdout", "stderr"
S3_GW_HTTP_LOGGING_DESTINATION=stdout
# RPC endpoint and order of resolving of bucket names
S3_GW_RPC_ENDPOINT=http://morph-chain.frostfs.devenv:30333/
@ -111,6 +126,8 @@ S3_GW_CACHE_MORPH_POLICY_SIZE=10000
# Cache which stores frostfsid subject info
S3_GW_CACHE_FROSTFSID_LIFETIME=1m
S3_GW_CACHE_FROSTFSID_SIZE=10000
# Cache which stores network info
S3_GW_CACHE_NETWORK_INFO_LIFETIME=1m
# Default policy of placing containers in FrostFS
# If a user sends a request `CreateBucket` and doesn't define policy for placing of a container in FrostFS, the S3 Gateway
@ -167,6 +184,7 @@ S3_GW_KLUDGE_DEFAULT_NAMESPACES="" "root"
S3_GW_TRACING_ENABLED=false
S3_GW_TRACING_ENDPOINT="localhost:4318"
S3_GW_TRACING_EXPORTER="otlp_grpc"
S3_GW_TRACING_TRUSTED_CA=""
S3_GW_RUNTIME_SOFT_MEMORY_LIMIT=1073741824

View file

@ -55,7 +55,24 @@ vhs:
logger:
level: debug
destination: stdout
sampling:
enabled: false
initial: 100
thereafter: 100
interval: 1s
# log http request data (URI, headers, query, etc)
http_logging:
enabled: false
# max body size to log
max_body: 1024
# max log size in Mb
max_log_size: 20
# use log compression
gzip: true
# possible output values: filesystem path, url, "stdout", "stderr"
destination: stdout
# RPC endpoint and order of resolving of bucket names
rpc_endpoint: http://morph-chain.frostfs.devenv:30333
resolve_order:
@ -74,6 +91,7 @@ tracing:
enabled: false
exporter: "otlp_grpc"
endpoint: "localhost:4318"
trusted_ca: ""
# Timeout to connect to a node
connect_timeout: 10s
@ -135,6 +153,9 @@ cache:
frostfsid:
lifetime: 1m
size: 10000
# Cache which stores network info
network_info:
lifetime: 1m
# Parameters of FrostFS container placement policy
placement_policy:

View file

@ -0,0 +1,8 @@
endpoint: http://localhost:8084
log: ./log/request.log
credentials:
access_key: CAtUxDSSFtuVyVCjHTMhwx3eP3YSPo5ffwbPcnKfcdrD06WwUSn72T5EBNe3jcgjL54rmxFM6u3nUAoNBps8qJ1PD
secret_key: 560027d81c277de7378f71cbf12a32e4f7f541de724be59bcfdbfdc925425f30
http_timeout: 60s
skip_verify_tls: false
print_response_limit: 1024

View file

@ -99,13 +99,14 @@ func (x *AccessBox) Unmarshal(data []byte) error {
// PackTokens adds bearer and session tokens to BearerTokens and SessionToken lists respectively.
// Session token can be nil.
// Secret can be nil. In such case secret will be generated.
func PackTokens(gatesData []*GateData, secret []byte) (*AccessBox, *Secrets, error) {
func PackTokens(gatesData []*GateData, secret []byte, isCustomSecret bool) (*AccessBox, *Secrets, error) {
box := &AccessBox{}
ephemeralKey, err := keys.NewPrivateKey()
if err != nil {
return nil, nil, fmt.Errorf("create ephemeral key: %w", err)
}
box.SeedKey = ephemeralKey.PublicKey().Bytes()
box.IsCustom = isCustomSecret
if secret == nil {
secret, err = generateSecret()
@ -118,7 +119,12 @@ func PackTokens(gatesData []*GateData, secret []byte) (*AccessBox, *Secrets, err
return nil, nil, fmt.Errorf("failed to add tokens to accessbox: %w", err)
}
return box, &Secrets{hex.EncodeToString(secret), ephemeralKey}, err
secretKey := string(secret)
if !isCustomSecret {
secretKey = hex.EncodeToString(secret)
}
return box, &Secrets{SecretKey: secretKey, EphemeralKey: ephemeralKey}, err
}
// GetTokens returns gate tokens from AccessBox.
@ -133,7 +139,7 @@ func (x *AccessBox) GetTokens(owner *keys.PrivateKey) (*GateData, error) {
continue
}
gateData, err := decodeGate(gate, owner, seedKey)
gateData, err := x.decodeGate(gate, owner, seedKey)
if err != nil {
return nil, fmt.Errorf("failed to decode gate: %w", err)
}
@ -217,7 +223,7 @@ func encodeGate(ephemeralKey *keys.PrivateKey, seedKey *keys.PublicKey, tokens *
return gate, nil
}
func decodeGate(gate *AccessBox_Gate, owner *keys.PrivateKey, seedKey *keys.PublicKey) (*GateData, error) {
func (x *AccessBox) decodeGate(gate *AccessBox_Gate, owner *keys.PrivateKey, seedKey *keys.PublicKey) (*GateData, error) {
data, err := decrypt(owner, seedKey, gate.Tokens)
if err != nil {
return nil, fmt.Errorf("decrypt tokens: %w", err)
@ -243,7 +249,11 @@ func decodeGate(gate *AccessBox_Gate, owner *keys.PrivateKey, seedKey *keys.Publ
gateData := NewGateData(owner.PublicKey(), &bearerTkn)
gateData.SessionTokens = sessionTkns
gateData.SecretKey = hex.EncodeToString(tokens.SecretKey)
if x.IsCustom {
gateData.SecretKey = string(tokens.SecretKey)
} else {
gateData.SecretKey = hex.EncodeToString(tokens.SecretKey)
}
return gateData, nil
}

View file

@ -1,7 +1,7 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.30.0
// protoc v3.12.4
// protoc-gen-go v1.34.2
// protoc v3.21.9
// source: creds/accessbox/accessbox.proto
package accessbox
@ -28,6 +28,7 @@ type AccessBox struct {
SeedKey []byte `protobuf:"bytes,1,opt,name=seedKey,proto3" json:"seedKey,omitempty"`
Gates []*AccessBox_Gate `protobuf:"bytes,2,rep,name=gates,proto3" json:"gates,omitempty"`
ContainerPolicy []*AccessBox_ContainerPolicy `protobuf:"bytes,3,rep,name=containerPolicy,proto3" json:"containerPolicy,omitempty"`
IsCustom bool `protobuf:"varint,4,opt,name=isCustom,proto3" json:"isCustom,omitempty"`
}
func (x *AccessBox) Reset() {
@ -83,6 +84,13 @@ func (x *AccessBox) GetContainerPolicy() []*AccessBox_ContainerPolicy {
return nil
}
func (x *AccessBox) GetIsCustom() bool {
if x != nil {
return x.IsCustom
}
return false
}
type Tokens struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
@ -261,7 +269,7 @@ var File_creds_accessbox_accessbox_proto protoreflect.FileDescriptor
var file_creds_accessbox_accessbox_proto_rawDesc = []byte{
0x0a, 0x1f, 0x63, 0x72, 0x65, 0x64, 0x73, 0x2f, 0x61, 0x63, 0x63, 0x65, 0x73, 0x73, 0x62, 0x6f,
0x78, 0x2f, 0x61, 0x63, 0x63, 0x65, 0x73, 0x73, 0x62, 0x6f, 0x78, 0x2e, 0x70, 0x72, 0x6f, 0x74,
0x6f, 0x12, 0x09, 0x61, 0x63, 0x63, 0x65, 0x73, 0x73, 0x62, 0x6f, 0x78, 0x22, 0xc7, 0x02, 0x0a,
0x6f, 0x12, 0x09, 0x61, 0x63, 0x63, 0x65, 0x73, 0x73, 0x62, 0x6f, 0x78, 0x22, 0xe3, 0x02, 0x0a,
0x09, 0x41, 0x63, 0x63, 0x65, 0x73, 0x73, 0x42, 0x6f, 0x78, 0x12, 0x18, 0x0a, 0x07, 0x73, 0x65,
0x65, 0x64, 0x4b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x07, 0x73, 0x65, 0x65,
0x64, 0x4b, 0x65, 0x79, 0x12, 0x2f, 0x0a, 0x05, 0x67, 0x61, 0x74, 0x65, 0x73, 0x18, 0x02, 0x20,
@ -272,29 +280,31 @@ var file_creds_accessbox_accessbox_proto_rawDesc = []byte{
0x2e, 0x61, 0x63, 0x63, 0x65, 0x73, 0x73, 0x62, 0x6f, 0x78, 0x2e, 0x41, 0x63, 0x63, 0x65, 0x73,
0x73, 0x42, 0x6f, 0x78, 0x2e, 0x43, 0x6f, 0x6e, 0x74, 0x61, 0x69, 0x6e, 0x65, 0x72, 0x50, 0x6f,
0x6c, 0x69, 0x63, 0x79, 0x52, 0x0f, 0x63, 0x6f, 0x6e, 0x74, 0x61, 0x69, 0x6e, 0x65, 0x72, 0x50,
0x6f, 0x6c, 0x69, 0x63, 0x79, 0x1a, 0x44, 0x0a, 0x04, 0x47, 0x61, 0x74, 0x65, 0x12, 0x16, 0x0a,
0x06, 0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x73, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x06, 0x74,
0x6f, 0x6b, 0x65, 0x6e, 0x73, 0x12, 0x24, 0x0a, 0x0d, 0x67, 0x61, 0x74, 0x65, 0x50, 0x75, 0x62,
0x6c, 0x69, 0x63, 0x4b, 0x65, 0x79, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x0d, 0x67, 0x61,
0x74, 0x65, 0x50, 0x75, 0x62, 0x6c, 0x69, 0x63, 0x4b, 0x65, 0x79, 0x1a, 0x59, 0x0a, 0x0f, 0x43,
0x6f, 0x6e, 0x74, 0x61, 0x69, 0x6e, 0x65, 0x72, 0x50, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x12, 0x2e,
0x0a, 0x12, 0x6c, 0x6f, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x43, 0x6f, 0x6e, 0x73, 0x74, 0x72,
0x61, 0x69, 0x6e, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x12, 0x6c, 0x6f, 0x63, 0x61,
0x74, 0x69, 0x6f, 0x6e, 0x43, 0x6f, 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, 0x74, 0x12, 0x16,
0x0a, 0x06, 0x70, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x06,
0x70, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x22, 0x6e, 0x0a, 0x06, 0x54, 0x6f, 0x6b, 0x65, 0x6e, 0x73,
0x12, 0x1c, 0x0a, 0x09, 0x73, 0x65, 0x63, 0x72, 0x65, 0x74, 0x4b, 0x65, 0x79, 0x18, 0x01, 0x20,
0x01, 0x28, 0x0c, 0x52, 0x09, 0x73, 0x65, 0x63, 0x72, 0x65, 0x74, 0x4b, 0x65, 0x79, 0x12, 0x20,
0x0a, 0x0b, 0x62, 0x65, 0x61, 0x72, 0x65, 0x72, 0x54, 0x6f, 0x6b, 0x65, 0x6e, 0x18, 0x02, 0x20,
0x01, 0x28, 0x0c, 0x52, 0x0b, 0x62, 0x65, 0x61, 0x72, 0x65, 0x72, 0x54, 0x6f, 0x6b, 0x65, 0x6e,
0x12, 0x24, 0x0a, 0x0d, 0x73, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e, 0x54, 0x6f, 0x6b, 0x65, 0x6e,
0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x0c, 0x52, 0x0d, 0x73, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e,
0x54, 0x6f, 0x6b, 0x65, 0x6e, 0x73, 0x42, 0x46, 0x5a, 0x44, 0x67, 0x69, 0x74, 0x2e, 0x66, 0x72,
0x6f, 0x73, 0x74, 0x66, 0x73, 0x2e, 0x69, 0x6e, 0x66, 0x6f, 0x2f, 0x54, 0x72, 0x75, 0x65, 0x43,
0x6c, 0x6f, 0x75, 0x64, 0x4c, 0x61, 0x62, 0x2f, 0x66, 0x72, 0x6f, 0x73, 0x74, 0x66, 0x73, 0x2d,
0x73, 0x33, 0x2d, 0x67, 0x77, 0x2f, 0x63, 0x72, 0x65, 0x64, 0x73, 0x2f, 0x74, 0x6f, 0x6b, 0x65,
0x6e, 0x62, 0x6f, 0x78, 0x3b, 0x61, 0x63, 0x63, 0x65, 0x73, 0x73, 0x62, 0x6f, 0x78, 0x62, 0x06,
0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
0x6f, 0x6c, 0x69, 0x63, 0x79, 0x12, 0x1a, 0x0a, 0x08, 0x69, 0x73, 0x43, 0x75, 0x73, 0x74, 0x6f,
0x6d, 0x18, 0x04, 0x20, 0x01, 0x28, 0x08, 0x52, 0x08, 0x69, 0x73, 0x43, 0x75, 0x73, 0x74, 0x6f,
0x6d, 0x1a, 0x44, 0x0a, 0x04, 0x47, 0x61, 0x74, 0x65, 0x12, 0x16, 0x0a, 0x06, 0x74, 0x6f, 0x6b,
0x65, 0x6e, 0x73, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x06, 0x74, 0x6f, 0x6b, 0x65, 0x6e,
0x73, 0x12, 0x24, 0x0a, 0x0d, 0x67, 0x61, 0x74, 0x65, 0x50, 0x75, 0x62, 0x6c, 0x69, 0x63, 0x4b,
0x65, 0x79, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x0d, 0x67, 0x61, 0x74, 0x65, 0x50, 0x75,
0x62, 0x6c, 0x69, 0x63, 0x4b, 0x65, 0x79, 0x1a, 0x59, 0x0a, 0x0f, 0x43, 0x6f, 0x6e, 0x74, 0x61,
0x69, 0x6e, 0x65, 0x72, 0x50, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x12, 0x2e, 0x0a, 0x12, 0x6c, 0x6f,
0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x43, 0x6f, 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, 0x74,
0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x12, 0x6c, 0x6f, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e,
0x43, 0x6f, 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, 0x74, 0x12, 0x16, 0x0a, 0x06, 0x70, 0x6f,
0x6c, 0x69, 0x63, 0x79, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x06, 0x70, 0x6f, 0x6c, 0x69,
0x63, 0x79, 0x22, 0x6e, 0x0a, 0x06, 0x54, 0x6f, 0x6b, 0x65, 0x6e, 0x73, 0x12, 0x1c, 0x0a, 0x09,
0x73, 0x65, 0x63, 0x72, 0x65, 0x74, 0x4b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52,
0x09, 0x73, 0x65, 0x63, 0x72, 0x65, 0x74, 0x4b, 0x65, 0x79, 0x12, 0x20, 0x0a, 0x0b, 0x62, 0x65,
0x61, 0x72, 0x65, 0x72, 0x54, 0x6f, 0x6b, 0x65, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52,
0x0b, 0x62, 0x65, 0x61, 0x72, 0x65, 0x72, 0x54, 0x6f, 0x6b, 0x65, 0x6e, 0x12, 0x24, 0x0a, 0x0d,
0x73, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e, 0x54, 0x6f, 0x6b, 0x65, 0x6e, 0x73, 0x18, 0x03, 0x20,
0x03, 0x28, 0x0c, 0x52, 0x0d, 0x73, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e, 0x54, 0x6f, 0x6b, 0x65,
0x6e, 0x73, 0x42, 0x46, 0x5a, 0x44, 0x67, 0x69, 0x74, 0x2e, 0x66, 0x72, 0x6f, 0x73, 0x74, 0x66,
0x73, 0x2e, 0x69, 0x6e, 0x66, 0x6f, 0x2f, 0x54, 0x72, 0x75, 0x65, 0x43, 0x6c, 0x6f, 0x75, 0x64,
0x4c, 0x61, 0x62, 0x2f, 0x66, 0x72, 0x6f, 0x73, 0x74, 0x66, 0x73, 0x2d, 0x73, 0x33, 0x2d, 0x67,
0x77, 0x2f, 0x63, 0x72, 0x65, 0x64, 0x73, 0x2f, 0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x62, 0x6f, 0x78,
0x3b, 0x61, 0x63, 0x63, 0x65, 0x73, 0x73, 0x62, 0x6f, 0x78, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74,
0x6f, 0x33,
}
var (
@ -310,7 +320,7 @@ func file_creds_accessbox_accessbox_proto_rawDescGZIP() []byte {
}
var file_creds_accessbox_accessbox_proto_msgTypes = make([]protoimpl.MessageInfo, 4)
var file_creds_accessbox_accessbox_proto_goTypes = []interface{}{
var file_creds_accessbox_accessbox_proto_goTypes = []any{
(*AccessBox)(nil), // 0: accessbox.AccessBox
(*Tokens)(nil), // 1: accessbox.Tokens
(*AccessBox_Gate)(nil), // 2: accessbox.AccessBox.Gate
@ -332,7 +342,7 @@ func file_creds_accessbox_accessbox_proto_init() {
return
}
if !protoimpl.UnsafeEnabled {
file_creds_accessbox_accessbox_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
file_creds_accessbox_accessbox_proto_msgTypes[0].Exporter = func(v any, i int) any {
switch v := v.(*AccessBox); i {
case 0:
return &v.state
@ -344,7 +354,7 @@ func file_creds_accessbox_accessbox_proto_init() {
return nil
}
}
file_creds_accessbox_accessbox_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} {
file_creds_accessbox_accessbox_proto_msgTypes[1].Exporter = func(v any, i int) any {
switch v := v.(*Tokens); i {
case 0:
return &v.state
@ -356,7 +366,7 @@ func file_creds_accessbox_accessbox_proto_init() {
return nil
}
}
file_creds_accessbox_accessbox_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} {
file_creds_accessbox_accessbox_proto_msgTypes[2].Exporter = func(v any, i int) any {
switch v := v.(*AccessBox_Gate); i {
case 0:
return &v.state
@ -368,7 +378,7 @@ func file_creds_accessbox_accessbox_proto_init() {
return nil
}
}
file_creds_accessbox_accessbox_proto_msgTypes[3].Exporter = func(v interface{}, i int) interface{} {
file_creds_accessbox_accessbox_proto_msgTypes[3].Exporter = func(v any, i int) any {
switch v := v.(*AccessBox_ContainerPolicy); i {
case 0:
return &v.state

View file

@ -20,6 +20,7 @@ message AccessBox {
bytes seedKey = 1 [json_name = "seedKey"];
repeated Gate gates = 2 [json_name = "gates"];
repeated ContainerPolicy containerPolicy = 3 [json_name = "containerPolicy"];
bool isCustom = 4 [json_name = "isCustom"];
}
message Tokens {

View file

@ -61,7 +61,7 @@ func TestBearerTokenInAccessBox(t *testing.T) {
require.NoError(t, tkn.Sign(sec.PrivateKey))
gate := NewGateData(cred.PublicKey(), &tkn)
box, _, err = PackTokens([]*GateData{gate}, nil)
box, _, err = PackTokens([]*GateData{gate}, nil, false)
require.NoError(t, err)
data, err := box.Marshal()
@ -96,7 +96,7 @@ func TestSessionTokenInAccessBox(t *testing.T) {
var newTkn bearer.Token
gate := NewGateData(cred.PublicKey(), &newTkn)
gate.SessionTokens = []*session.Container{tkn}
box, _, err = PackTokens([]*GateData{gate}, nil)
box, _, err = PackTokens([]*GateData{gate}, nil, false)
require.NoError(t, err)
data, err := box.Marshal()
@ -136,7 +136,7 @@ func TestAccessboxMultipleKeys(t *testing.T) {
}
}
box, _, err = PackTokens(gates, nil)
box, _, err = PackTokens(gates, nil, false)
require.NoError(t, err)
for i, k := range privateKeys {
@ -165,7 +165,7 @@ func TestUnknownKey(t *testing.T) {
require.NoError(t, tkn.Sign(sec.PrivateKey))
gate := NewGateData(cred.PublicKey(), &tkn)
box, _, err = PackTokens([]*GateData{gate}, nil)
box, _, err = PackTokens([]*GateData{gate}, nil, false)
require.NoError(t, err)
_, err = box.GetTokens(wrongCred)
@ -224,14 +224,27 @@ func TestGetBox(t *testing.T) {
var tkn bearer.Token
gate := NewGateData(cred.PublicKey(), &tkn)
secret := []byte("secret")
accessBox, _, err := PackTokens([]*GateData{gate}, secret)
require.NoError(t, err)
box, err := accessBox.GetBox(cred)
require.NoError(t, err)
require.Equal(t, hex.EncodeToString(secret), box.Gate.SecretKey)
t.Run("regular secret", func(t *testing.T) {
accessBox, secrets, err := PackTokens([]*GateData{gate}, secret, false)
require.NoError(t, err)
require.Equal(t, hex.EncodeToString(secret), secrets.SecretKey)
box, err := accessBox.GetBox(cred)
require.NoError(t, err)
require.Equal(t, hex.EncodeToString(secret), box.Gate.SecretKey)
})
t.Run("custom secret", func(t *testing.T) {
accessBox, secrets, err := PackTokens([]*GateData{gate}, secret, true)
require.NoError(t, err)
require.Equal(t, string(secret), secrets.SecretKey)
box, err := accessBox.GetBox(cred)
require.NoError(t, err)
require.Equal(t, string(secret), box.Gate.SecretKey)
})
}
func TestAccessBox(t *testing.T) {
@ -241,7 +254,7 @@ func TestAccessBox(t *testing.T) {
var tkn bearer.Token
gate := NewGateData(cred.PublicKey(), &tkn)
accessBox, _, err := PackTokens([]*GateData{gate}, nil)
accessBox, _, err := PackTokens([]*GateData{gate}, nil, false)
require.NoError(t, err)
t.Run("invalid owner", func(t *testing.T) {
@ -300,7 +313,7 @@ func TestAccessBox(t *testing.T) {
BearerToken: &tkn,
GateKey: &keys.PublicKey{},
}
_, _, err = PackTokens([]*GateData{gate}, nil)
_, _, err = PackTokens([]*GateData{gate}, nil, false)
require.Error(t, err)
})
}

View file

@ -14,7 +14,6 @@ import (
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"go.uber.org/zap"
)
@ -22,13 +21,14 @@ import (
type (
// Credentials is a bearer token get/put interface.
Credentials interface {
GetBox(context.Context, oid.Address) (*accessbox.Box, []object.Attribute, error)
Put(context.Context, cid.ID, CredentialsParam) (oid.Address, error)
Update(context.Context, oid.Address, CredentialsParam) (oid.Address, error)
GetBox(context.Context, cid.ID, string) (*accessbox.Box, []object.Attribute, error)
Put(context.Context, CredentialsParam) (oid.Address, error)
Update(context.Context, CredentialsParam) (oid.Address, error)
}
CredentialsParam struct {
OwnerID user.ID
Container cid.ID
AccessKeyID string
AccessBox *accessbox.AccessBox
Expiration uint64
Keys keys.PublicKeys
@ -49,13 +49,16 @@ type (
CacheConfig *cache.Config
RemovingCheckAfterDurations time.Duration
}
Box struct {
AccessBox *accessbox.AccessBox
Attributes []object.Attribute
Address *oid.Address
}
)
// PrmObjectCreate groups parameters of objects created by credential tool.
type PrmObjectCreate struct {
// FrostFS identifier of the object creator.
Creator user.ID
// FrostFS container to store the object.
Container cid.ID
@ -64,7 +67,12 @@ type PrmObjectCreate struct {
// Optional.
// If provided cred object will be created using crdt approach.
NewVersionFor *oid.ID
NewVersionForAccessKeyID string
// Optional.
// If provided cred object will contain specific crdt name attribute for first accessbox object version.
// If NewVersionForAccessKeyID is provided this field isn't used.
CustomAccessKey string
// Last FrostFS epoch of the object lifetime.
ExpirationEpoch uint64
@ -76,6 +84,21 @@ type PrmObjectCreate struct {
CustomAttributes []object.Attribute
}
// PrmGetCredsObject groups parameters of getting credential object.
type PrmGetCredsObject struct {
// FrostFS container to get the object.
Container cid.ID
// S3 access key id.
AccessKeyID string
// FallbackAddress is an address that should be used to get creds if we couldn't find it by AccessKeyID.
// Optional.
FallbackAddress *oid.Address
}
var ErrCustomAccessKeyIDNotFound = errors.New("custom AccessKeyId not found")
// FrostFS represents virtual connection to FrostFS network.
type FrostFS interface {
// CreateObject creates and saves a parameterized object in the specified
@ -92,8 +115,9 @@ type FrostFS interface {
//
// It returns exactly one non-nil value. It returns any error encountered which
// prevented the object payload from being read.
// Returns ErrCustomAccessKeyIDNotFound if provided AccessKey is custom, and it was not found.
// Object must contain full payload.
GetCredsObject(context.Context, oid.Address) (*object.Object, error)
GetCredsObject(context.Context, PrmGetCredsObject) (*object.Object, error)
}
var (
@ -116,84 +140,128 @@ func New(cfg Config) Credentials {
}
}
func (c *cred) GetBox(ctx context.Context, addr oid.Address) (*accessbox.Box, []object.Attribute, error) {
cachedBoxValue := c.cache.Get(addr)
func (c *cred) GetBox(ctx context.Context, cnrID cid.ID, accessKeyID string) (*accessbox.Box, []object.Attribute, error) {
cachedBoxValue := c.cache.Get(accessKeyID)
if cachedBoxValue != nil {
return c.checkIfCredentialsAreRemoved(ctx, addr, cachedBoxValue)
return c.checkIfCredentialsAreRemoved(ctx, cnrID, accessKeyID, cachedBoxValue)
}
box, attrs, err := c.getAccessBox(ctx, addr)
box, err := c.getAccessBox(ctx, cnrID, accessKeyID, nil)
if err != nil {
return nil, nil, fmt.Errorf("get access box: %w", err)
}
cachedBox, err := box.GetBox(c.key)
cachedBox, err := box.AccessBox.GetBox(c.key)
if err != nil {
return nil, nil, fmt.Errorf("get gate box: %w", err)
}
c.putBoxToCache(addr, cachedBox, attrs)
val := &cache.AccessBoxCacheValue{
Box: cachedBox,
Attributes: box.Attributes,
PutTime: time.Now(),
Address: box.Address,
}
return cachedBox, attrs, nil
c.putBoxToCache(accessKeyID, val)
return cachedBox, box.Attributes, nil
}
func (c *cred) checkIfCredentialsAreRemoved(ctx context.Context, addr oid.Address, cachedBoxValue *cache.AccessBoxCacheValue) (*accessbox.Box, []object.Attribute, error) {
func (c *cred) checkIfCredentialsAreRemoved(ctx context.Context, cnrID cid.ID, accessKeyID string, cachedBoxValue *cache.AccessBoxCacheValue) (*accessbox.Box, []object.Attribute, error) {
if time.Since(cachedBoxValue.PutTime) < c.removingCheckDuration {
return cachedBoxValue.Box, cachedBoxValue.Attributes, nil
}
box, attrs, err := c.getAccessBox(ctx, addr)
box, err := c.getAccessBox(ctx, cnrID, accessKeyID, cachedBoxValue.Address)
if err != nil {
if client.IsErrObjectAlreadyRemoved(err) {
c.cache.Delete(addr)
c.cache.Delete(accessKeyID)
return nil, nil, fmt.Errorf("get access box: %w", err)
}
return cachedBoxValue.Box, cachedBoxValue.Attributes, nil
}
cachedBox, err := box.GetBox(c.key)
cachedBox, err := box.AccessBox.GetBox(c.key)
if err != nil {
c.cache.Delete(addr)
c.cache.Delete(accessKeyID)
return nil, nil, fmt.Errorf("get gate box: %w", err)
}
// we need this to reset PutTime
// to don't check for removing each time after removingCheckDuration interval
c.putBoxToCache(addr, cachedBox, attrs)
val := &cache.AccessBoxCacheValue{
Box: cachedBox,
Attributes: box.Attributes,
PutTime: time.Now(),
Address: box.Address,
}
c.putBoxToCache(accessKeyID, val)
return cachedBoxValue.Box, attrs, nil
return cachedBoxValue.Box, box.Attributes, nil
}
func (c *cred) putBoxToCache(addr oid.Address, box *accessbox.Box, attrs []object.Attribute) {
if err := c.cache.Put(addr, box, attrs); err != nil {
c.log.Warn(logs.CouldntPutAccessBoxIntoCache, zap.String("address", addr.EncodeToString()))
func (c *cred) putBoxToCache(accessKeyID string, val *cache.AccessBoxCacheValue) {
if err := c.cache.Put(accessKeyID, val); err != nil {
c.log.Warn(logs.CouldntPutAccessBoxIntoCache, zap.String("accessKeyID", accessKeyID))
}
}
func (c *cred) getAccessBox(ctx context.Context, addr oid.Address) (*accessbox.AccessBox, []object.Attribute, error) {
obj, err := c.frostFS.GetCredsObject(ctx, addr)
func (c *cred) getAccessBox(ctx context.Context, cnrID cid.ID, accessKeyID string, fallbackAddr *oid.Address) (*Box, error) {
prm := PrmGetCredsObject{
Container: cnrID,
AccessKeyID: accessKeyID,
FallbackAddress: fallbackAddr,
}
obj, err := c.frostFS.GetCredsObject(ctx, prm)
if err != nil {
return nil, nil, fmt.Errorf("read payload and attributes: %w", err)
return nil, fmt.Errorf("read payload and attributes: %w", err)
}
// decode access box
var box accessbox.AccessBox
if err = box.Unmarshal(obj.Payload()); err != nil {
return nil, nil, fmt.Errorf("unmarhal access box: %w", err)
return nil, fmt.Errorf("unmarhal access box: %w", err)
}
return &box, obj.Attributes(), nil
addr := &oid.Address{}
boxCnrID, cnrIDOk := obj.ContainerID()
boxObjID, objIDOk := obj.ID()
addr.SetContainer(boxCnrID)
addr.SetObject(boxObjID)
if !cnrIDOk || !objIDOk {
addr = nil
}
return &Box{
AccessBox: &box,
Attributes: obj.Attributes(),
Address: addr,
}, nil
}
func (c *cred) Put(ctx context.Context, idCnr cid.ID, prm CredentialsParam) (oid.Address, error) {
return c.createObject(ctx, idCnr, nil, prm)
func (c *cred) Put(ctx context.Context, prm CredentialsParam) (oid.Address, error) {
if prm.AccessKeyID != "" {
c.log.Info(logs.CheckCustomAccessKeyIDUniqueness, zap.String("access_key_id", prm.AccessKeyID))
credsPrm := PrmGetCredsObject{
Container: prm.Container,
AccessKeyID: prm.AccessKeyID,
}
if _, err := c.frostFS.GetCredsObject(ctx, credsPrm); err == nil {
return oid.Address{}, fmt.Errorf("access key id '%s' already exists", prm.AccessKeyID)
} else if !errors.Is(err, ErrCustomAccessKeyIDNotFound) {
return oid.Address{}, fmt.Errorf("check AccessKeyID uniqueness: %w", err)
}
}
return c.createObject(ctx, prm, false)
}
func (c *cred) Update(ctx context.Context, addr oid.Address, prm CredentialsParam) (oid.Address, error) {
objID := addr.Object()
return c.createObject(ctx, addr.Container(), &objID, prm)
func (c *cred) Update(ctx context.Context, prm CredentialsParam) (oid.Address, error) {
return c.createObject(ctx, prm, true)
}
func (c *cred) createObject(ctx context.Context, cnrID cid.ID, newVersionFor *oid.ID, prm CredentialsParam) (oid.Address, error) {
func (c *cred) createObject(ctx context.Context, prm CredentialsParam, update bool) (oid.Address, error) {
if len(prm.Keys) == 0 {
return oid.Address{}, ErrEmptyPublicKeys
} else if prm.AccessBox == nil {
@ -204,14 +272,19 @@ func (c *cred) createObject(ctx context.Context, cnrID cid.ID, newVersionFor *oi
return oid.Address{}, fmt.Errorf("marshall box: %w", err)
}
var newVersionFor string
if update {
newVersionFor = prm.AccessKeyID
}
idObj, err := c.frostFS.CreateObject(ctx, PrmObjectCreate{
Creator: prm.OwnerID,
Container: cnrID,
Filepath: strconv.FormatInt(time.Now().Unix(), 10) + "_access.box",
ExpirationEpoch: prm.Expiration,
NewVersionFor: newVersionFor,
Payload: data,
CustomAttributes: prm.CustomAttributes,
Container: prm.Container,
Filepath: strconv.FormatInt(time.Now().Unix(), 10) + "_access.box",
ExpirationEpoch: prm.Expiration,
CustomAccessKey: prm.AccessKeyID,
NewVersionForAccessKeyID: newVersionFor,
Payload: data,
CustomAttributes: prm.CustomAttributes,
})
if err != nil {
return oid.Address{}, fmt.Errorf("create object: %w", err)
@ -219,7 +292,7 @@ func (c *cred) createObject(ctx context.Context, cnrID cid.ID, newVersionFor *oi
var addr oid.Address
addr.SetObject(idObj)
addr.SetContainer(cnrID)
addr.SetContainer(prm.Container)
return addr, nil
}

View file

@ -4,6 +4,7 @@ import (
"context"
"encoding/hex"
"errors"
"strings"
"testing"
"time"
@ -15,27 +16,40 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
oidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id/test"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/stretchr/testify/require"
"go.uber.org/zap/zaptest"
)
type frostfsMock struct {
objects map[oid.Address][]*object.Object
errors map[oid.Address]error
key *keys.PrivateKey
objects map[string][]*object.Object
errors map[string]error
}
func newFrostfsMock() *frostfsMock {
func newFrostfsMock(key *keys.PrivateKey) *frostfsMock {
return &frostfsMock{
objects: map[oid.Address][]*object.Object{},
errors: map[oid.Address]error{},
objects: map[string][]*object.Object{},
errors: map[string]error{},
key: key,
}
}
func (f *frostfsMock) ownerID() user.ID {
if f.key == nil {
return user.ID{}
}
var ownerID user.ID
user.IDFromKey(&ownerID, f.key.PrivateKey.PublicKey)
return ownerID
}
func (f *frostfsMock) CreateObject(_ context.Context, prm PrmObjectCreate) (oid.ID, error) {
var obj object.Object
obj.SetPayload(prm.Payload)
obj.SetOwnerID(prm.Creator)
obj.SetOwnerID(f.ownerID())
obj.SetContainerID(prm.Container)
a := object.NewAttribute()
@ -44,19 +58,15 @@ func (f *frostfsMock) CreateObject(_ context.Context, prm PrmObjectCreate) (oid.
prm.CustomAttributes = append(prm.CustomAttributes, *a)
obj.SetAttributes(prm.CustomAttributes...)
if prm.NewVersionFor != nil {
var addr oid.Address
addr.SetObject(*prm.NewVersionFor)
addr.SetContainer(prm.Container)
_, ok := f.objects[addr]
if prm.NewVersionForAccessKeyID != "" {
_, ok := f.objects[prm.NewVersionForAccessKeyID]
if !ok {
return oid.ID{}, errors.New("not found")
}
objID := oidtest.ID()
obj.SetID(objID)
f.objects[addr] = append(f.objects[addr], &obj)
f.objects[prm.NewVersionForAccessKeyID] = append(f.objects[prm.NewVersionForAccessKeyID], &obj)
return objID, nil
}
@ -64,22 +74,27 @@ func (f *frostfsMock) CreateObject(_ context.Context, prm PrmObjectCreate) (oid.
objID := oidtest.ID()
obj.SetID(objID)
accessKeyID := prm.CustomAccessKey
if accessKeyID == "" {
accessKeyID = prm.Container.EncodeToString() + "0" + objID.EncodeToString()
}
var addr oid.Address
addr.SetObject(objID)
addr.SetContainer(prm.Container)
f.objects[addr] = []*object.Object{&obj}
f.objects[accessKeyID] = []*object.Object{&obj}
return objID, nil
}
func (f *frostfsMock) GetCredsObject(_ context.Context, address oid.Address) (*object.Object, error) {
if err := f.errors[address]; err != nil {
func (f *frostfsMock) GetCredsObject(_ context.Context, prm PrmGetCredsObject) (*object.Object, error) {
if err := f.errors[prm.AccessKeyID]; err != nil {
return nil, err
}
objects, ok := f.objects[address]
objects, ok := f.objects[prm.AccessKeyID]
if !ok {
return nil, errors.New("not found")
return nil, ErrCustomAccessKeyIDNotFound
}
return objects[len(objects)-1], nil
@ -100,7 +115,7 @@ func TestRemovingAccessBox(t *testing.T) {
sk, err := hex.DecodeString(secretKey)
require.NoError(t, err)
accessBox, _, err := accessbox.PackTokens(gateData, sk)
accessBox, _, err := accessbox.PackTokens(gateData, sk, false)
require.NoError(t, err)
data, err := accessBox.Marshal()
require.NoError(t, err)
@ -111,9 +126,24 @@ func TestRemovingAccessBox(t *testing.T) {
obj.SetID(addr.Object())
obj.SetContainerID(addr.Container())
accessKeyID := getAccessKeyID(addr)
accessBoxCustom, _, err := accessbox.PackTokens(gateData, []byte("secret"), true)
require.NoError(t, err)
dataCustom, err := accessBoxCustom.Marshal()
require.NoError(t, err)
var objCustom object.Object
objCustom.SetPayload(dataCustom)
addrCustom := oidtest.Address()
objCustom.SetID(addrCustom.Object())
objCustom.SetContainerID(addrCustom.Container())
accessKeyIDCustom := "accessKeyID"
frostfs := &frostfsMock{
objects: map[oid.Address][]*object.Object{addr: {&obj}},
errors: map[oid.Address]error{},
objects: map[string][]*object.Object{accessKeyID: {&obj}, accessKeyIDCustom: {&objCustom}},
errors: map[string]error{},
}
cfg := Config{
@ -129,15 +159,30 @@ func TestRemovingAccessBox(t *testing.T) {
creds := New(cfg)
_, _, err = creds.GetBox(ctx, addr)
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyID)
require.NoError(t, err)
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyIDCustom)
require.NoError(t, err)
frostfs.errors[addr] = errors.New("network error")
_, _, err = creds.GetBox(ctx, addr)
frostfs.errors[accessKeyID] = errors.New("network error")
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyID)
require.NoError(t, err)
frostfs.errors[accessKeyIDCustom] = errors.New("network error")
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyIDCustom)
require.NoError(t, err)
frostfs.errors[addr] = &apistatus.ObjectAlreadyRemoved{}
_, _, err = creds.GetBox(ctx, addr)
frostfs.errors[accessKeyID] = &apistatus.ObjectAlreadyRemoved{}
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyID)
require.Error(t, err)
frostfs.errors[accessKeyIDCustom] = &apistatus.ObjectAlreadyRemoved{}
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyIDCustom)
require.Error(t, err)
frostfs.errors[accessKeyID] = &apistatus.ObjectAlreadyRemoved{}
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyID)
require.Error(t, err)
frostfs.errors[accessKeyIDCustom] = &apistatus.ObjectAlreadyRemoved{}
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyIDCustom)
require.Error(t, err)
}
@ -153,8 +198,9 @@ func TestGetBox(t *testing.T) {
}}
secret := []byte("secret")
accessBox, _, err := accessbox.PackTokens(gateData, secret)
accessBox, secrets, err := accessbox.PackTokens(gateData, secret, false)
require.NoError(t, err)
require.Equal(t, hex.EncodeToString(secret), secrets.SecretKey)
data, err := accessBox.Marshal()
require.NoError(t, err)
@ -172,108 +218,107 @@ func TestGetBox(t *testing.T) {
}
t.Run("no removing check, accessbox from cache", func(t *testing.T) {
frostfs := newFrostfsMock()
cfg.FrostFS = frostfs
cfg.RemovingCheckAfterDurations = time.Hour
cfg.Key = key
creds := New(cfg)
creds := newCreds(key, cfg, time.Hour)
cnrID := cidtest.ID()
addr, err := creds.Put(ctx, cnrID, CredentialsParam{Keys: keys.PublicKeys{key.PublicKey()}, AccessBox: accessBox})
addr, err := creds.Put(ctx, CredentialsParam{Container: cnrID, Keys: keys.PublicKeys{key.PublicKey()}, AccessBox: accessBox})
require.NoError(t, err)
_, _, err = creds.GetBox(ctx, addr)
accessKeyID := getAccessKeyID(addr)
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyID)
require.NoError(t, err)
frostfs.errors[addr] = &apistatus.ObjectAlreadyRemoved{}
_, _, err = creds.GetBox(ctx, addr)
creds.(*cred).frostFS.(*frostfsMock).errors[accessKeyID] = &apistatus.ObjectAlreadyRemoved{}
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyID)
require.NoError(t, err)
})
t.Run("error while getting box from frostfs", func(t *testing.T) {
frostfs := newFrostfsMock()
cfg.FrostFS = frostfs
cfg.RemovingCheckAfterDurations = 0
cfg.Key = key
creds := New(cfg)
creds := newCreds(key, cfg, 0)
cnrID := cidtest.ID()
addr, err := creds.Put(ctx, cnrID, CredentialsParam{Keys: keys.PublicKeys{key.PublicKey()}, AccessBox: accessBox})
addr, err := creds.Put(ctx, CredentialsParam{Container: cnrID, Keys: keys.PublicKeys{key.PublicKey()}, AccessBox: accessBox})
require.NoError(t, err)
frostfs.errors[addr] = errors.New("network error")
_, _, err = creds.GetBox(ctx, addr)
accessKeyID := getAccessKeyID(addr)
creds.(*cred).frostFS.(*frostfsMock).errors[accessKeyID] = errors.New("network error")
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyID)
require.Error(t, err)
})
t.Run("invalid key", func(t *testing.T) {
frostfs := newFrostfsMock()
frostfs := newFrostfsMock(key)
var obj object.Object
obj.SetPayload(data)
addr := oidtest.Address()
frostfs.objects[addr] = []*object.Object{&obj}
accessKeyID := getAccessKeyID(addr)
frostfs.objects[accessKeyID] = []*object.Object{&obj}
cfg.FrostFS = frostfs
cfg.RemovingCheckAfterDurations = 0
cfg.Key = &keys.PrivateKey{}
creds := New(cfg)
_, _, err = creds.GetBox(ctx, addr)
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyID)
require.Error(t, err)
})
t.Run("invalid payload", func(t *testing.T) {
frostfs := newFrostfsMock()
frostfs := newFrostfsMock(key)
var obj object.Object
obj.SetPayload([]byte("invalid"))
addr := oidtest.Address()
frostfs.objects[addr] = []*object.Object{&obj}
accessKeyID := getAccessKeyID(addr)
frostfs.objects[accessKeyID] = []*object.Object{&obj}
cfg.FrostFS = frostfs
cfg.RemovingCheckAfterDurations = 0
cfg.Key = key
creds := New(cfg)
_, _, err = creds.GetBox(ctx, addr)
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyID)
require.Error(t, err)
})
t.Run("check attributes update", func(t *testing.T) {
frostfs := newFrostfsMock()
cfg.FrostFS = frostfs
cfg.RemovingCheckAfterDurations = 0
cfg.Key = key
creds := New(cfg)
creds := newCreds(key, cfg, 0)
cnrID := cidtest.ID()
addr, err := creds.Put(ctx, cnrID, CredentialsParam{Keys: keys.PublicKeys{key.PublicKey()}, AccessBox: accessBox})
addr, err := creds.Put(ctx, CredentialsParam{Container: cnrID, Keys: keys.PublicKeys{key.PublicKey()}, AccessBox: accessBox})
require.NoError(t, err)
_, boxAttrs, err := creds.GetBox(ctx, addr)
accessKeyID := getAccessKeyID(addr)
_, boxAttrs, err := creds.GetBox(ctx, addr.Container(), accessKeyID)
require.NoError(t, err)
_, err = creds.Update(ctx, addr, CredentialsParam{Keys: keys.PublicKeys{key.PublicKey()}, AccessBox: accessBox, CustomAttributes: attrs})
prm := CredentialsParam{
Container: addr.Container(),
AccessKeyID: accessKeyID,
Keys: keys.PublicKeys{key.PublicKey()},
AccessBox: accessBox,
CustomAttributes: attrs,
}
_, err = creds.Update(ctx, prm)
require.NoError(t, err)
_, newBoxAttrs, err := creds.GetBox(ctx, addr)
_, newBoxAttrs, err := creds.GetBox(ctx, addr.Container(), accessKeyID)
require.NoError(t, err)
require.Equal(t, len(boxAttrs)+1, len(newBoxAttrs))
})
t.Run("check accessbox update", func(t *testing.T) {
frostfs := newFrostfsMock()
cfg.FrostFS = frostfs
cfg.RemovingCheckAfterDurations = 0
cfg.Key = key
creds := New(cfg)
creds := newCreds(key, cfg, 0)
cnrID := cidtest.ID()
addr, err := creds.Put(ctx, cnrID, CredentialsParam{Keys: keys.PublicKeys{key.PublicKey()}, AccessBox: accessBox})
addr, err := creds.Put(ctx, CredentialsParam{Container: cnrID, Keys: keys.PublicKeys{key.PublicKey()}, AccessBox: accessBox})
require.NoError(t, err)
box, _, err := creds.GetBox(ctx, addr)
accessKeyID := getAccessKeyID(addr)
box, _, err := creds.GetBox(ctx, addr.Container(), accessKeyID)
require.NoError(t, err)
require.Equal(t, hex.EncodeToString(secret), box.Gate.SecretKey)
@ -286,44 +331,134 @@ func TestGetBox(t *testing.T) {
}}
newSecret := []byte("new-secret")
newAccessBox, _, err := accessbox.PackTokens(newGateData, newSecret)
newAccessBox, _, err := accessbox.PackTokens(newGateData, newSecret, false)
require.NoError(t, err)
_, err = creds.Update(ctx, addr, CredentialsParam{Keys: keys.PublicKeys{newKey.PublicKey()}, AccessBox: newAccessBox})
prm := CredentialsParam{
Container: addr.Container(),
AccessKeyID: accessKeyID,
Keys: keys.PublicKeys{newKey.PublicKey()},
AccessBox: newAccessBox,
}
_, err = creds.Update(ctx, prm)
require.NoError(t, err)
_, _, err = creds.GetBox(ctx, addr)
_, _, err = creds.GetBox(ctx, addr.Container(), accessKeyID)
require.Error(t, err)
cfg.Key = newKey
newCreds := New(cfg)
newCfg := Config{
FrostFS: creds.(*cred).frostFS,
Key: newKey,
CacheConfig: cfg.CacheConfig,
}
newCreds := New(newCfg)
box, _, err = newCreds.GetBox(ctx, addr)
box, _, err = newCreds.GetBox(ctx, addr.Container(), accessKeyID)
require.NoError(t, err)
require.Equal(t, hex.EncodeToString(newSecret), box.Gate.SecretKey)
})
t.Run("check access key id uniqueness", func(t *testing.T) {
creds := newCreds(key, cfg, 0)
prm := CredentialsParam{
Container: cidtest.ID(),
AccessBox: accessBox,
Keys: keys.PublicKeys{key.PublicKey()},
}
_, err = creds.Put(ctx, prm)
require.NoError(t, err)
_, err = creds.Put(ctx, prm)
require.NoError(t, err)
})
t.Run("empty keys", func(t *testing.T) {
frostfs := newFrostfsMock()
cfg.FrostFS = frostfs
cfg.RemovingCheckAfterDurations = 0
cfg.Key = key
creds := New(cfg)
creds := newCreds(key, cfg, 0)
cnrID := cidtest.ID()
_, err = creds.Put(ctx, cnrID, CredentialsParam{AccessBox: accessBox})
_, err = creds.Put(ctx, CredentialsParam{Container: cnrID, AccessBox: accessBox})
require.ErrorIs(t, err, ErrEmptyPublicKeys)
})
t.Run("empty accessbox", func(t *testing.T) {
frostfs := newFrostfsMock()
cfg.FrostFS = frostfs
cfg.RemovingCheckAfterDurations = 0
cfg.Key = key
creds := New(cfg)
creds := newCreds(key, cfg, 0)
cnrID := cidtest.ID()
_, err = creds.Put(ctx, cnrID, CredentialsParam{Keys: keys.PublicKeys{key.PublicKey()}})
_, err = creds.Put(ctx, CredentialsParam{Container: cnrID, Keys: keys.PublicKeys{key.PublicKey()}})
require.ErrorIs(t, err, ErrEmptyBearerToken)
})
}
func TestBoxWithCustomAccessKeyID(t *testing.T) {
ctx := context.Background()
key, err := keys.NewPrivateKey()
require.NoError(t, err)
gateData := []*accessbox.GateData{{
BearerToken: &bearer.Token{},
GateKey: key.PublicKey(),
}}
secret := []byte("secret")
accessBox, secrets, err := accessbox.PackTokens(gateData, secret, true)
require.NoError(t, err)
require.Equal(t, string(secret), secrets.SecretKey)
cfg := Config{
CacheConfig: &cache.Config{
Size: 10,
Lifetime: 24 * time.Hour,
Logger: zaptest.NewLogger(t),
},
}
t.Run("check secret format", func(t *testing.T) {
creds := newCreds(key, cfg, 0)
prm := CredentialsParam{
Container: cidtest.ID(),
AccessKeyID: "custom-access-key-id",
AccessBox: accessBox,
Keys: keys.PublicKeys{key.PublicKey()},
}
_, err = creds.Put(ctx, prm)
require.NoError(t, err)
box, _, err := creds.GetBox(ctx, prm.Container, prm.AccessKeyID)
require.NoError(t, err)
require.Equal(t, string(secret), box.Gate.SecretKey)
})
t.Run("check custom access key id uniqueness", func(t *testing.T) {
creds := newCreds(key, cfg, 0)
prm := CredentialsParam{
Container: cidtest.ID(),
AccessKeyID: "custom-access-key-id",
AccessBox: accessBox,
Keys: keys.PublicKeys{key.PublicKey()},
}
_, err = creds.Put(ctx, prm)
require.NoError(t, err)
_, err = creds.Put(ctx, prm)
require.Error(t, err)
})
}
func newCreds(key *keys.PrivateKey, cfg Config, removingCheckDuration time.Duration) Credentials {
frostfs := newFrostfsMock(key)
cfg.FrostFS = frostfs
cfg.RemovingCheckAfterDurations = removingCheckDuration
cfg.Key = key
return New(cfg)
}
func getAccessKeyID(addr oid.Address) string {
return strings.ReplaceAll(addr.EncodeToString(), "/", "0")
}

View file

@ -159,8 +159,10 @@ storage node.
Object s3 credentials are formed based on:
* `AccessKeyId` - is concatenated container id and object id (`<cid>0<oid>`) of `AccessBox` (
e.g. `2XGRML5EW3LMHdf64W2DkBy1Nkuu4y4wGhUj44QjbXBi05ZNvs8WVwy1XTmSEkcVkydPKzCgtmR7U3zyLYTj3Snxf`)
* `SecretAccessKey` - hex-encoded random generated 32 bytes (that is encrypted and stored in object payload)
e.g. `2XGRML5EW3LMHdf64W2DkBy1Nkuu4y4wGhUj44QjbXBi05ZNvs8WVwy1XTmSEkcVkydPKzCgtmR7U3zyLYTj3Snxf`).
Or it can be arbitrary user-provided unique string with min length 4 and max length 128.
* `SecretAccessKey` - hex-encoded random generated 32 bytes (that is encrypted and stored in object payload).
Or it can be arbitrary user-provided unique string with min length 4 and max length 128.
> **Note**: sensitive info in `AccessBox` is [encrypted](#encryption), so only someone who posses specific private key
> can decrypt such info.
@ -192,7 +194,7 @@ It contains:
* List of gate data:
* Gate public key (so that gate (when it will decrypt data later) know which item from the list it should process)
* Encrypted tokens:
* `SecretAccessKey` - hex-encoded random generated 32 bytes
* `SecretAccessKey` - hex-encoded random generated 32 bytes (or arbitrary user-provided string)
* Marshaled bearer token - more detail
in [spec](https://git.frostfs.info/TrueCloudLab/frostfs-api/src/commit/4c68d92468503b10282c8a92af83a56f170c8a3a/acl/types.proto#L189)
* Marshaled session token - more detail
@ -229,10 +231,12 @@ relevant data) the following sequence is used:
</a>
* Search all object whose attribute `S3-Access-Box-CRDT-Name` is equal to `AccessKeyId` (extract container id
from `AccessKeyId` that has format: `<cid>0<oid>`).
from `AccessKeyId` that has format: `<cid>0<oid>` if `AccessBox` was created with default parameters, or it can also
be arbitrary user-defined string).
* Get metadata for these object using `HEAD` requests (not `Get` to reduce network traffic)
* Sort all these objects by creation epoch and object id
* Pick last object id (If no object is found then extract object id from `AccessKeyId` that has format: `<cid>0<oid>`.
* Pick last object id (If no object is found then extract object id from `AccessKeyId` that has format: `<cid>0<oid>`
(if `AccessBox` was created with default parameters, or it can also be arbitrary user-defined string).
We need to do this because versions of `AccessBox` can miss the `S3-Access-Box-CRDT-Name` attribute.)
* Get appropriate object from FrostFS storage
* Decrypt `AccessBox` (see [encryption](#encryption))
@ -253,7 +257,7 @@ secp256r1 or prime256v1) is used (unless otherwise stated).
* Create ephemeral key (`SeedKey`), it's need to generate shared secret
* Generate random 32-byte (that after hex-encoded be `SecretAccessKey`) or use existing secret access key
(if `AccessBox` is being updated rather than creating brand new)
(if `AccessBox` is being updated rather than creating brand new) or use arbitrary user-provided string
* Generate shared secret as [ECDH](https://en.wikipedia.org/wiki/Elliptic-curve_Diffie%E2%80%93Hellman)
* Derive 32-byte key using shared secret from previous step with key derivation function based on
HMAC with SHA256 [HKDF](https://en.wikipedia.org/wiki/HKDF)

View file

@ -146,6 +146,32 @@ the secret. Format of `access_key_id`: `%cid0%oid`, where 0(zero) is a delimiter
24h). Default value is `720h` (30 days). It will be ceil rounded to the nearest amount of epoch
* `--aws-cli-credentials` - path to the aws cli credentials file, where authmate will write `access_key_id` and
`secret_access_key` to
* `--rpc-endpoint` - NEO node RPC address (must be provided if `--container-id` is NNS name)
* `--access-key-id` - access key id of s3 credential that must be created (must be unique)
* `--secret-access-key` - secret access key of s3 credential that must be used
You also can specify `AccessKeyID`/`SecretAccessKey` pair that should be created:
```shell
$ frostfs-s3-authmate issue-secret --wallet wallet.json \
--peer 192.168.130.71:8080 \
--gate-public-key 0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf \
--gate-public-key 0317585fa8274f7afdf1fc5f2a2e7bece549d5175c4e5182e37924f30229aef967 \
--access-key-id my-access-key \
--secret-access-key my-secret-key \
--container-id BpExV76416Vo7GrkJsGwXGoLM35xsBwup8voedDZR3c6
Enter password for wallet.json >
{
"initial_access_key_id": "my-access-key-3",
"access_key_id": "my-access-key",
"secret_access_key": "my-secret-key",
"owner_private_key": "d9972cc4f21b07a90f4b347c72c33c1d1611c2b9a2cfd0cc28cee8cb221e8e55",
"wallet_public_key": "031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a",
"container_id": "BpExV76416Vo7GrkJsGwXGoLM35xsBwup8voedDZR3c6"
}
```
### Bearer tokens

View file

@ -176,6 +176,7 @@ There are some custom types used for brevity:
| `placement_policy` | [Placement policy configuration](#placement_policy-section) |
| `server` | [Server configuration](#server-section) |
| `logger` | [Logger configuration](#logger-section) |
| `http_logging` | [HTTP Request logger configuration](#http_logging-section) |
| `cache` | [Cache configuration](#cache-section) |
| `cors` | [CORS configuration](#cors-section) |
| `pprof` | [Pprof configuration](#pprof-section) |
@ -220,7 +221,7 @@ max_clients_deadline: 30s
allowed_access_key_id_prefixes:
- Ck9BHsgKcnwfCTUSFm6pxhoNS4cBqgN2NQ8zVgPjqZDX
- 3stjWenX15YwYzczMr88gy3CQr4NYFBQ8P7keGzH5QFn
reconnect_interval: 1m
source_ip_header: "Source-Ip"
@ -369,12 +370,47 @@ server:
logger:
level: debug
destination: stdout
sampling:
enabled: false
initial: 100
thereafter: 100
interval: 1s
```
| Parameter | Type | SIGHUP reload | Default value | Description |
|---------------|----------|---------------|---------------|----------------------------------------------------------------------------------------------------|
| `level` | `string` | yes | `debug` | Logging level.<br/>Possible values: `debug`, `info`, `warn`, `error`, `dpanic`, `panic`, `fatal`. |
| `destination` | `string` | no | `stdout` | Destination for logger: `stdout` or `journald` |
| Parameter | Type | SIGHUP reload | Default value | Description |
|-----------------------|------------|---------------|---------------|----------------------------------------------------------------------------------------------------|
| `level` | `string` | yes | `debug` | Logging level.<br/>Possible values: `debug`, `info`, `warn`, `error`, `dpanic`, `panic`, `fatal`. |
| `destination` | `string` | no | `stdout` | Destination for logger: `stdout` or `journald` |
| `sampling.enabled` | `bool` | no | false | Sampling enabling flag. |
| `sampling.initial` | `int` | no | '100' | Sampling count of first log entries. |
| `sampling.thereafter` | `int` | no | '100' | Sampling count of entries after an `interval`. |
| `sampling.interval` | `duration` | no | '1s' | Sampling interval of messaging similar entries. |
### `http_logging` section
Could be enabled only in builds with `loghttp` build tag. To build with `loghttp` tag, pass `GOFLAGS` var to `make`:
```bash
make GOFLAGS="-tags=loghttp" [target]
```
```yaml
http_logging:
enabled: false
max_body: 1024
max_log_size: 20
gzip: true
destination: stdout
```
| Parameter | Type | SIGHUP reload | Default value | Description |
|----------------|----------|---------------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `enabled` | `bool` | yes | `false` | Flag to enable the logger. |
| `max_body` | `int` | yes | `1024` | Max body size for log output in bytes. |
| `max_log_size` | `int` | yes | `50` | Log file size threshold (in megabytes) to be moved in backup file. After reaching threshold, initial filename is appended with timestamp. And new empty file with initial name is created. |
| `gzip` | `bool` | yes | `false` | Whether to enable Gzip compression to backup log files. |
| `destination` | `string` | yes | `stdout` | Specify path for log output. Accepts log file path, or "stdout" and "stderr" reserved words to print in output streams. File and folders are created if necessary. |
### `cache` section
@ -411,6 +447,8 @@ cache:
frostfsid:
lifetime: 1m
size: 10000
network_info:
lifetime: 1m
```
| Parameter | Type | Default value | Description |
@ -425,6 +463,7 @@ cache:
| `accesscontrol` | [Cache config](#cache-subsection) | `lifetime: 1m`<br>`size: 100000` | Cache which stores owner to cache operation mapping. |
| `morph_policy` | [Cache config](#cache-subsection) | `lifetime: 1m`<br>`size: 10000` | Cache which stores list of policy chains. |
| `frostfsid` | [Cache config](#cache-subsection) | `lifetime: 1m`<br>`size: 10000` | Cache which stores FrostfsID subject info. |
| `network_info` | [Cache config](#cache-subsection) | `lifetime: 1m` | Cache which stores network info. |
#### `cache` subsection
@ -503,14 +542,15 @@ tracing:
enabled: false
exporter: "otlp_grpc"
endpoint: "localhost:4318"
trusted_ca: "/etc/ssl/telemetry-trusted-ca.pem"
```
| Parameter | Type | SIGHUP reload | Default value | Description |
|-------------|----------|---------------|---------------|-----------------------------------------|
| `enabled` | `bool` | yes | `false` | Flag to enable the service. |
| `exporter` | `string` | yes | `` | Type of tracing exporter. |
| `endpoint` | `string` | yes | `` | Address that service listener binds to. |
| Parameter | Type | SIGHUP reload | Default value | Description |
|--------------|----------|---------------|---------------|---------------------------------------------------------------------------------------------------------------------------------|
| `enabled` | `bool` | yes | `false` | Flag to enable the service. |
| `exporter` | `string` | yes | `` | Type of tracing exporter. |
| `endpoint` | `string` | yes | `` | Address that service listener binds to. |
| `trusted_ca` | `string` | yes | | Path to certificate of a certification authority in pem format, that issued the TLS certificate of the telemetry remote server. |
# `frostfs` section
@ -721,12 +761,14 @@ Section for well-known containers to store s3-related data and settings.
containers:
cors: AZjLTXfK4vs4ovxMic2xEJKSymMNLqdwq9JT64ASFCRj
lifecycle: AZjLTXfK4vs4ovxMic2xEJKSymMNLqdwq9JT64ASFCRj
accessbox: ExnA1gSY3kzgomi2wJxNyWo1ytWv9VAKXRE55fNXEPL2
```
| Parameter | Type | SIGHUP reload | Default value | Description |
|-------------|----------|---------------|---------------|-------------------------------------------------------------------------------------------|
| `cors` | `string` | no | | Container name for CORS configurations. If not set, container of the bucket is used. |
| `lifecycle` | `string` | no | | Container name for lifecycle configurations. If not set, container of the bucket is used. |
| Parameter | Type | SIGHUP reload | Default value | Description |
|-------------|----------|---------------|---------------|-------------------------------------------------------------------------------------------------------------------------|
| `cors` | `string` | no | | Container name for CORS configurations. If not set, container of the bucket is used. |
| `lifecycle` | `string` | no | | Container name for lifecycle configurations. If not set, container of the bucket is used. |
| `accessbox` | `string` | no | | Container name to lookup accessbox if custom aws credentials is used. If not set, custom credentials are not supported. |
# `vhs` section
@ -743,7 +785,7 @@ vhs:
```
| Parameter | Type | SIGHUP reload | Default value | Description |
| ------------------- | ----------------- | ------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|---------------------|-------------------|---------------|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `enabled` | `bool` | yes | `false` | Enables the use of virtual host addressing for buckets at the application level. |
| `vhs_header` | `string` | yes | `X-Frostfs-S3-VHS` | Header for determining whether VHS is enabled for the request. |
| `servername_header` | `string` | yes | `X-Frostfs-Servername` | Header for determining servername. |

View file

@ -21,6 +21,7 @@ package AccessBox {
SeedKey => Encoded public seed key
List of Gates *--> Gate
List of container policies *--> ContainerPolicy
IsCustom => True if SecretKey was imported and must be treated as it is
}

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 13 KiB

After

Width:  |  Height:  |  Size: 13 KiB

48
docs/playback.md Normal file
View file

@ -0,0 +1,48 @@
# FrostFS S3 Playback
Playback is a tool to reproduce queries to `frostfs-s3-gw` in dev environment. Network logs could be
gathered from `s3-gw` via HTTP Logger which could be enabled on build with `loghttp` build tag
and `http_logging.enabled` option set to `true` in `s3-gw` configuration.
## Commands
`run` - reads log file and reproduces send requests from it to specified endpoint
#### Example
```bash
frostfs-s3-playback --config <config_path> run [--endpoint=<endpoint>] [--log=<log_path>]
```
## Configuration
Playback accepts configuration file path in yaml with corresponding options:
```yaml
endpoint: http://localhost:8084
log: ./request.log
env: .env
credentials:
access_key: CAtUxDSSFtuVyVCjHTMhwx3eP3YSPo5ffwbPcnKfcdrD06WwUSn72T5EBNe3jcgjL54rmxFM6u3nUAoNBps8qJ1PD
secret_key: 560027d81c277de7378f71cbf12a32e4f7f541de724be59bcfdbfdc925425f30
http_timeout: 60s
skip_verify_tls: true
```
Configuration path is passed via required `--config` flag.
If corresponding flag is set, it overrides parameter from config.
### Configuration parameters
#### Global parameters
| Config parameter name | Flag name | Type | Default value | Description |
|-------------------------|-------------------|------------|---------------|-------------------------------------------------------------------------------|
| - | `config` | `string` | - | config file path (e.g. `./config/playback.yaml`) |
| `http_timeout` | `http-timeout` | `duration` | `60s` | http request timeout |
| `skip_verify_tls` | `skip-verify-tls` | `bool` | `false` | skips tls certificate verification for https endpoints |
| `credentials.accessKey` | - | `string` | - | AWS access key id |
| `credentials.secretKey` | - | `string` | - | AWS secret key |
| `print_response_limit` | - | `int` | `1024` | max response length to be printed in stdout, the rest of body will be omitted |
#### `run` command parameters
| Config parameter name | Flag name | Type | Default value | Description |
|-----------------------|-----------|--------|---------------|--------------------------------------------------------|
| `endpoint` | endpoint | string | - | s3-gw endpoint URL |
| `log` | log | string | ./request.log | path to log file, could be either absolute or relative |

91
go.mod
View file

@ -3,99 +3,114 @@ module git.frostfs.info/TrueCloudLab/frostfs-s3-gw
go 1.22
require (
git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.16.1-0.20240819074700-a43110e36326
git.frostfs.info/TrueCloudLab/frostfs-contract v0.19.3-0.20240621131249-49e5270f673e
git.frostfs.info/TrueCloudLab/frostfs-observability v0.0.0-20230531082742-c97d21411eb6
git.frostfs.info/TrueCloudLab/frostfs-sdk-go v0.0.0-20240822080251-28f140bf06c1
git.frostfs.info/TrueCloudLab/frostfs-api-go/v2 v2.16.1-0.20241011114054-f0fc40e116d1
git.frostfs.info/TrueCloudLab/frostfs-contract v0.20.1-0.20241022094040-5f956751d48b
git.frostfs.info/TrueCloudLab/frostfs-observability v0.0.0-20240909114314-666d326cc573
git.frostfs.info/TrueCloudLab/frostfs-sdk-go v0.0.0-20241022124111-5361f0ecebd3
git.frostfs.info/TrueCloudLab/policy-engine v0.0.0-20240822104152-a3bc3099bd5b
git.frostfs.info/TrueCloudLab/zapjournald v0.0.0-20240124114243-cb2e66427d02
github.com/aws/aws-sdk-go v1.44.6
github.com/aws/aws-sdk-go-v2 v1.18.1
github.com/aws/aws-sdk-go-v2 v1.30.3
github.com/aws/aws-sdk-go-v2/credentials v1.17.27
github.com/bluele/gcache v0.0.2
github.com/go-chi/chi/v5 v5.0.8
github.com/google/uuid v1.6.0
github.com/minio/sio v0.3.0
github.com/mr-tron/base58 v1.2.0
github.com/nspcc-dev/neo-go v0.106.2
github.com/nspcc-dev/neo-go v0.106.3
github.com/panjf2000/ants/v2 v2.5.0
github.com/prometheus/client_golang v1.19.0
github.com/prometheus/client_model v0.5.0
github.com/spf13/cobra v1.7.0
github.com/spf13/cobra v1.8.1
github.com/spf13/pflag v1.0.5
github.com/spf13/viper v1.15.0
github.com/ssgreg/journald v1.0.0
github.com/stretchr/testify v1.9.0
github.com/trailofbits/go-fuzz-utils v0.0.0-20230413173806-58c38daa3cb4
github.com/urfave/cli/v2 v2.3.0
go.opentelemetry.io/otel v1.16.0
go.opentelemetry.io/otel/trace v1.16.0
github.com/urfave/cli/v2 v2.27.2
go.opentelemetry.io/otel v1.28.0
go.opentelemetry.io/otel/trace v1.28.0
go.uber.org/zap v1.27.0
golang.org/x/crypto v0.21.0
golang.org/x/exp v0.0.0-20240222234643-814bf88cf225
golang.org/x/net v0.23.0
golang.org/x/text v0.14.0
google.golang.org/grpc v1.63.2
google.golang.org/protobuf v1.33.0
golang.org/x/crypto v0.24.0
golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842
golang.org/x/net v0.26.0
golang.org/x/text v0.16.0
google.golang.org/grpc v1.66.2
google.golang.org/protobuf v1.34.2
gopkg.in/natefinch/lumberjack.v2 v2.2.1
)
replace git.frostfs.info/TrueCloudLab/frostfs-sdk-go v0.0.0-20241022124111-5361f0ecebd3 => git.frostfs.info/mbiryukova/frostfs-sdk-go v0.0.0-20241028144246-b11bc616b491
require (
git.frostfs.info/TrueCloudLab/frostfs-crypto v0.6.0 // indirect
git.frostfs.info/TrueCloudLab/hrw v1.2.1 // indirect
git.frostfs.info/TrueCloudLab/rfc6979 v0.4.0 // indirect
git.frostfs.info/TrueCloudLab/tzhash v1.8.0 // indirect
github.com/VictoriaMetrics/easyproto v0.1.4 // indirect
github.com/antlr4-go/antlr/v4 v4.13.0 // indirect
github.com/aws/smithy-go v1.13.5 // indirect
github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
github.com/aws/smithy-go v1.20.3 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.4 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect
github.com/fsnotify/fsnotify v1.6.0 // indirect
github.com/go-logr/logr v1.2.4 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang/snappy v0.0.1 // indirect
github.com/gorilla/websocket v1.5.1 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.11.3 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/holiman/uint256 v1.2.4 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/ipfs/go-cid v0.0.7 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.6 // indirect
github.com/magiconair/properties v1.8.7 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 // indirect
github.com/minio/sha256-simd v1.0.1 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/multiformats/go-base32 v0.0.3 // indirect
github.com/multiformats/go-base36 v0.1.0 // indirect
github.com/multiformats/go-multiaddr v0.12.1 // indirect
github.com/multiformats/go-multibase v0.0.3 // indirect
github.com/multiformats/go-multihash v0.0.14 // indirect
github.com/multiformats/go-varint v0.0.6 // indirect
github.com/nspcc-dev/go-ordered-json v0.0.0-20240301084351-0246b013f8b2 // indirect
github.com/nspcc-dev/neo-go/pkg/interop v0.0.0-20240521091047-78685785716d // indirect
github.com/nspcc-dev/neo-go/pkg/interop v0.0.0-20240727093519-1a48f1ce43ec // indirect
github.com/nspcc-dev/rfc6979 v0.2.1 // indirect
github.com/pelletier/go-toml/v2 v2.0.6 // indirect
github.com/pierrec/lz4 v2.6.1+incompatible // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/common v0.48.0 // indirect
github.com/prometheus/procfs v0.12.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/spf13/afero v1.9.3 // indirect
github.com/spf13/cast v1.5.0 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/subosito/gotenv v1.4.2 // indirect
github.com/syndtr/goleveldb v1.0.1-0.20210305035536-64b5b1c73954 // indirect
github.com/twmb/murmur3 v1.1.8 // indirect
github.com/urfave/cli v1.22.5 // indirect
github.com/xrash/smetrics v0.0.0-20240312152122-5f08fbb34913 // indirect
go.etcd.io/bbolt v1.3.9 // indirect
go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.16.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.16.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.16.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.16.0 // indirect
go.opentelemetry.io/otel/metric v1.16.0 // indirect
go.opentelemetry.io/otel/sdk v1.16.0 // indirect
go.opentelemetry.io/proto/otlp v0.19.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.28.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.28.0 // indirect
go.opentelemetry.io/otel/metric v1.28.0 // indirect
go.opentelemetry.io/otel/sdk v1.28.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/sync v0.6.0 // indirect
golang.org/x/sys v0.18.0 // indirect
golang.org/x/term v0.18.0 // indirect
google.golang.org/genproto v0.0.0-20240227224415-6ceb2ff114de // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240227224415-6ceb2ff114de // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240227224415-6ceb2ff114de // indirect
golang.org/x/sync v0.7.0 // indirect
golang.org/x/sys v0.22.0 // indirect
golang.org/x/term v0.21.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240701130421-f6361c86f094 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240701130421-f6361c86f094 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

Some files were not shown because too many files have changed in this diff Show more