Compare commits

..

1 commit

Author SHA1 Message Date
Denis Kirillov
c987076071 [#681] acl: Prepare owner ID before processing
Some checks failed
CodeQL / Analyze (go) (pull_request) Failing after 2s
Builds / Build CLI (pull_request) Failing after 2s
Builds / Build Docker image (pull_request) Has been skipped
DCO check / Commits Check (pull_request) Failing after 2s
Tests / Coverage (pull_request) Failing after 2s
Tests / Tests (1.17) (pull_request) Failing after 2s
Tests / Lint (pull_request) Failing after 4s
Tests / Tests (1.18.x) (pull_request) Failing after 1s
Tests / Tests (1.19.x) (pull_request) Failing after 2s
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2023-01-12 16:34:40 +03:00
124 changed files with 1215 additions and 2395 deletions

View file

@ -1,7 +1,7 @@
FROM golang:1.19 as builder FROM golang:1.19 as builder
ARG BUILD=now ARG BUILD=now
ARG REPO=git.frostfs.info/TrueCloudLab/frostfs-s3-gw ARG REPO=github.com/TrueCloudLab/frostfs-s3-gw
ARG VERSION=dev ARG VERSION=dev
WORKDIR /src WORKDIR /src

View file

@ -1,45 +0,0 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: community, triage, bug
assignees: ''
---
<!--- Provide a general summary of the issue in the Title above -->
## Expected Behavior
<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->
## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->
## Possible Solution
<!--- Not obligatory -->
<!--- If no reason/fix/additions for the bug can be suggested, -->
<!--- uncomment the following phrase: -->
<!--- No fix can be suggested by a QA engineer. Further solutions shall be up to developers. -->
## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. -->
1.
## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->
## Regression
<!-- Is this issue a regression? (Yes / No) -->
<!-- If Yes, optionally please include version or commit id or PR# that caused this regression, if you have these details. -->
## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used:
* Server setup and configuration:
* Operating System and version (`uname -a`):

View file

@ -1 +0,0 @@
blank_issues_enabled: false

View file

@ -1,20 +0,0 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: community, triage
assignees: ''
---
## Is your feature request related to a problem? Please describe.
<!--- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
## Describe the solution you'd like
<!--- A clear and concise description of what you want to happen. -->
## Describe alternatives you've considered
<!--- A clear and concise description of any alternative solutions or features you've considered. -->
## Additional context
<!--- Add any other context or screenshots about the feature request here. -->

View file

@ -68,7 +68,7 @@ jobs:
runs-on: ubuntu-20.04 runs-on: ubuntu-20.04
strategy: strategy:
matrix: matrix:
go_versions: [ '1.18.x', '1.19.x' ] go_versions: [ '1.17', '1.18.x', '1.19.x' ]
fail-fast: false fail-fast: false
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v2

9
.gitignore vendored
View file

@ -21,11 +21,4 @@ coverage.txt
coverage.html coverage.html
# debhelpers # debhelpers
**/*debhelper* **/.debhelper
# debian package build files
debian/files
debian/*.log
debian/*.substvars
debian/frostfs-s3-gw/

View file

@ -4,30 +4,6 @@ This document outlines major changes between releases.
## [Unreleased] ## [Unreleased]
### Fixed
- Get empty bucket CORS from frostfs (TrueCloudLab#36)
- Don't count pool error on client abort (#35)
### Added
- Return container name in `head-bucket` response (TrueCloudLab#18)
- Billing metrics (TrueCloudLab#5)
- Multiple configs support (TrueCloudLab#21)
- Bucket name resolving policy (TrueCloudLab#25)
- Support string `Action` and `Resource` fields in `bucketPolicy.Statement` (TrueCloudLab#32)
- Add new `kludge.use_default_xmlns_for_complete_multipart` config param (TrueCloudLab#40)
### Changed
- Update neo-go to v0.101.0 (#14)
- Update viper to v1.15.0 (#14)
- Using multiple servers require only one healthy (TrueCloudLab#12)
- Update go version to go1.18 (TrueCloudLab#16)
- Return error on invalid LocationConstraint (TrueCloudLab#23)
- Place billing metrics to separate url path (TrueCloudLab#26)
- Add generated deb builder files to .gitignore, and fix typo (TrueCloudLab#28)
- Limit number of objects to delete at one time (TrueCloudLab#37)
- CompleteMultipartUpload handler now sends whitespace characters to keep alive client's connection (#60)
- Support new system attributes (#64)
## [0.26.0] - 2022-12-28 ## [0.26.0] - 2022-12-28
### Added ### Added

View file

@ -6,7 +6,9 @@
</p> </p>
--- ---
[![Report](https://goreportcard.com/badge/git.frostfs.info/TrueCloudLab/frostfs-s3-gw)](https://goreportcard.com/report/git.frostfs.info/TrueCloudLab/frostfs-s3-gw) [![Report](https://goreportcard.com/badge/github.com/TrueCloudLab/frostfs-s3-gw)](https://goreportcard.com/report/github.com/TrueCloudLab/frostfs-s3-gw)
![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/TrueCloudLab/frostfs-s3-gw?sort=semver)
![License](https://img.shields.io/github/license/TrueCloudLab/frostfs-s3-gw.svg?style=popout)
# FrostFS S3 Gateway # FrostFS S3 Gateway
@ -14,7 +16,7 @@ FrostFS S3 gateway provides API compatible with Amazon S3 cloud storage service.
## Installation ## Installation
```go get -u git.frostfs.info/TrueCloudLab/frostfs-s3-gw``` ```go get -u github.com/TrueCloudLab/frostfs-s3-gw```
Or you can call `make` to build it from the cloned repository (the binary will Or you can call `make` to build it from the cloned repository (the binary will
end up in `bin/frostfs-s3-gw` with authmate helper in `bin/frostfs-s3-authmate`). end up in `bin/frostfs-s3-gw` with authmate helper in `bin/frostfs-s3-authmate`).

View file

@ -14,12 +14,12 @@ import (
"strings" "strings"
"time" "time"
v4 "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth/signer/v4" v4 "github.com/TrueCloudLab/frostfs-s3-gw/api/auth/signer/v4"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/cache" "github.com/TrueCloudLab/frostfs-s3-gw/api/cache"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apiErrors "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "github.com/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/tokens" "github.com/TrueCloudLab/frostfs-s3-gw/creds/tokens"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/aws/aws-sdk-go/aws/credentials" "github.com/aws/aws-sdk-go/aws/credentials"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
) )

View file

@ -5,7 +5,7 @@ import (
"testing" "testing"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )

View file

@ -4,7 +4,7 @@ import (
"fmt" "fmt"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
"github.com/bluele/gcache" "github.com/bluele/gcache"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -4,8 +4,8 @@ import (
"fmt" "fmt"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "github.com/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/bluele/gcache" "github.com/bluele/gcache"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -4,7 +4,7 @@ import (
"fmt" "fmt"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"github.com/bluele/gcache" "github.com/bluele/gcache"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -3,10 +3,10 @@ package cache
import ( import (
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "github.com/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
cidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id/test" cidtest "github.com/TrueCloudLab/frostfs-sdk-go/container/id/test"
oidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id/test" oidtest "github.com/TrueCloudLab/frostfs-sdk-go/object/id/test"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"go.uber.org/zap" "go.uber.org/zap"
"go.uber.org/zap/zaptest/observer" "go.uber.org/zap/zaptest/observer"

2
api/cache/names.go vendored
View file

@ -4,7 +4,7 @@ import (
"fmt" "fmt"
"time" "time"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/bluele/gcache" "github.com/bluele/gcache"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -4,8 +4,8 @@ import (
"fmt" "fmt"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/bluele/gcache" "github.com/bluele/gcache"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -4,9 +4,9 @@ import (
"testing" "testing"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
objecttest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/test" objecttest "github.com/TrueCloudLab/frostfs-sdk-go/object/test"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -6,8 +6,8 @@ import (
"strings" "strings"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"github.com/bluele/gcache" "github.com/bluele/gcache"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -4,9 +4,9 @@ import (
"testing" "testing"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
cidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id/test" cidtest "github.com/TrueCloudLab/frostfs-sdk-go/container/id/test"
oidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id/test" oidtest "github.com/TrueCloudLab/frostfs-sdk-go/object/id/test"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"go.uber.org/zap" "go.uber.org/zap"
) )

2
api/cache/system.go vendored
View file

@ -4,7 +4,7 @@ import (
"fmt" "fmt"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"github.com/bluele/gcache" "github.com/bluele/gcache"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -4,9 +4,9 @@ import (
"encoding/xml" "encoding/xml"
"time" "time"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
) )
const ( const (
@ -22,8 +22,7 @@ const (
type ( type (
// BucketInfo stores basic bucket data. // BucketInfo stores basic bucket data.
BucketInfo struct { BucketInfo struct {
Name string // container name from system attribute Name string
Zone string // container zone from system attribute
CID cid.ID CID cid.ID
Owner user.ID Owner user.ID
Created time.Time Created time.Time

View file

@ -4,9 +4,9 @@ import (
"strconv" "strconv"
"time" "time"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
) )
const ( const (

View file

@ -90,7 +90,6 @@ const (
ErrMissingFields ErrMissingFields
ErrMissingCredTag ErrMissingCredTag
ErrCredMalformed ErrCredMalformed
ErrInvalidLocationConstraint
ErrInvalidRegion ErrInvalidRegion
ErrInvalidServiceS3 ErrInvalidServiceS3
ErrInvalidServiceSTS ErrInvalidServiceSTS
@ -681,12 +680,6 @@ var errorCodes = errorCodeMap{
Description: "Error parsing the X-Amz-Credential parameter; the region is wrong;", Description: "Error parsing the X-Amz-Credential parameter; the region is wrong;",
HTTPStatusCode: http.StatusBadRequest, HTTPStatusCode: http.StatusBadRequest,
}, },
ErrInvalidLocationConstraint: {
ErrCode: ErrInvalidLocationConstraint,
Code: "InvalidLocationConstraint",
Description: "The specified location (Region) constraint is not valid.",
HTTPStatusCode: http.StatusBadRequest,
},
ErrInvalidRegion: { ErrInvalidRegion: {
ErrCode: ErrInvalidRegion, ErrCode: ErrInvalidRegion,
Code: "InvalidRegion", Code: "InvalidRegion",

View file

@ -14,15 +14,15 @@ import (
"strconv" "strconv"
"strings" "strings"
v2acl "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/acl" v2acl "github.com/TrueCloudLab/frostfs-api-go/v2/acl"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl" "github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "github.com/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "github.com/TrueCloudLab/frostfs-sdk-go/session"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"go.uber.org/zap" "go.uber.org/zap"
) )
@ -65,13 +65,10 @@ const (
aclRead AWSACL = "READ" aclRead AWSACL = "READ"
) )
// GranteeType is aws grantee permission type constants.
type GranteeType string
const ( const (
acpCanonicalUser GranteeType = "CanonicalUser" acpCanonicalUser = "CanonicalUser"
acpAmazonCustomerByEmail GranteeType = "AmazonCustomerByEmail" acpAmazonCustomerByEmail = "AmazonCustomerByEmail"
acpGroup GranteeType = "Group" acpGroup = "Group"
) )
type bucketPolicy struct { type bucketPolicy struct {
@ -158,90 +155,6 @@ func (s ServiceRecord) ToEACLRecord() *eacl.Record {
return serviceRecord return serviceRecord
} }
var (
errInvalidStatement = stderrors.New("invalid statement")
errInvalidPrincipal = stderrors.New("invalid principal")
)
func (s *statement) UnmarshalJSON(data []byte) error {
var statementMap map[string]interface{}
if err := json.Unmarshal(data, &statementMap); err != nil {
return err
}
sidField, ok := statementMap["Sid"]
if ok {
if s.Sid, ok = sidField.(string); !ok {
return errInvalidStatement
}
}
effectField, ok := statementMap["Effect"]
if ok {
if s.Effect, ok = effectField.(string); !ok {
return errInvalidStatement
}
}
principalField, ok := statementMap["Principal"]
if ok {
principalMap, ok := principalField.(map[string]interface{})
if !ok {
return errInvalidPrincipal
}
awsField, ok := principalMap["AWS"]
if ok {
if s.Principal.AWS, ok = awsField.(string); !ok {
return fmt.Errorf("%w: 'AWS' field must be string", errInvalidPrincipal)
}
}
canonicalUserField, ok := principalMap["CanonicalUser"]
if ok {
if s.Principal.CanonicalUser, ok = canonicalUserField.(string); !ok {
return errInvalidPrincipal
}
}
}
actionField, ok := statementMap["Action"]
if ok {
switch actionField := actionField.(type) {
case []interface{}:
s.Action = make([]string, len(actionField))
for i, action := range actionField {
if s.Action[i], ok = action.(string); !ok {
return errInvalidStatement
}
}
case string:
s.Action = []string{actionField}
default:
return errInvalidStatement
}
}
resourceField, ok := statementMap["Resource"]
if ok {
switch resourceField := resourceField.(type) {
case []interface{}:
s.Resource = make([]string, len(resourceField))
for i, action := range resourceField {
if s.Resource[i], ok = action.(string); !ok {
return errInvalidStatement
}
}
case string:
s.Resource = []string{resourceField}
default:
return errInvalidStatement
}
}
return nil
}
func (h *handler) GetBucketACLHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketACLHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := api.GetReqInfo(r.Context()) reqInfo := api.GetReqInfo(r.Context())
@ -304,6 +217,9 @@ func (h *handler) PutBucketACLHandler(w http.ResponseWriter, r *http.Request) {
} else if err = xml.NewDecoder(r.Body).Decode(list); err != nil { } else if err = xml.NewDecoder(r.Body).Decode(list); err != nil {
h.logAndSendError(w, "could not parse bucket acl", reqInfo, errors.GetAPIError(errors.ErrMalformedXML)) h.logAndSendError(w, "could not parse bucket acl", reqInfo, errors.GetAPIError(errors.ErrMalformedXML))
return return
} else {
// workaround to decode owner ID
list.Owner.ID = hex.EncodeToString(key.Bytes())
} }
resInfo := &resourceInfo{Bucket: reqInfo.BucketName} resInfo := &resourceInfo{Bucket: reqInfo.BucketName}
@ -439,6 +355,9 @@ func (h *handler) PutObjectACLHandler(w http.ResponseWriter, r *http.Request) {
} else if err = xml.NewDecoder(r.Body).Decode(list); err != nil { } else if err = xml.NewDecoder(r.Body).Decode(list); err != nil {
h.logAndSendError(w, "could not parse bucket acl", reqInfo, errors.GetAPIError(errors.ErrMalformedXML)) h.logAndSendError(w, "could not parse bucket acl", reqInfo, errors.GetAPIError(errors.ErrMalformedXML))
return return
} else {
// workaround to decode owner ID
list.Owner.ID = hex.EncodeToString(key.Bytes())
} }
resInfo := &resourceInfo{ resInfo := &resourceInfo{
@ -553,7 +472,7 @@ func parseACLHeaders(header http.Header, key *keys.PublicKey) (*AccessControlPol
Grantee: &Grantee{ Grantee: &Grantee{
ID: hex.EncodeToString(key.Bytes()), ID: hex.EncodeToString(key.Bytes()),
DisplayName: key.Address(), DisplayName: key.Address(),
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclFullControl, Permission: aclFullControl,
}} }}
@ -593,7 +512,7 @@ func addGrantees(list []*Grant, headers http.Header, hdr string) ([]*Grant, erro
} }
for _, grantee := range grantees { for _, grantee := range grantees {
if grantee.Type == acpAmazonCustomerByEmail || (grantee.Type == acpGroup && grantee.URI != allUsersGroup) { if grantee.matchType(acpAmazonCustomerByEmail) || (grantee.matchType(acpGroup) && grantee.URI != allUsersGroup) {
return nil, stderrors.New("unsupported grantee type") return nil, stderrors.New("unsupported grantee type")
} }
@ -643,17 +562,17 @@ func formGrantee(granteeType, value string) (*Grantee, error) {
case "id": case "id":
return &Grantee{ return &Grantee{
ID: value, ID: value,
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, nil }, nil
case "uri": case "uri":
return &Grantee{ return &Grantee{
URI: value, URI: value,
Type: acpGroup, Type: formGranteeType(acpGroup),
}, nil }, nil
case "emailAddress": case "emailAddress":
return &Grantee{ return &Grantee{
EmailAddress: value, EmailAddress: value,
Type: acpAmazonCustomerByEmail, Type: formGranteeType(acpAmazonCustomerByEmail),
}, nil }, nil
} }
// do not return grantee type to avoid sensitive data logging (#489) // do not return grantee type to avoid sensitive data logging (#489)
@ -667,7 +586,7 @@ func addPredefinedACP(acp *AccessControlPolicy, cannedACL string) (*AccessContro
acp.AccessControlList = append(acp.AccessControlList, &Grant{ acp.AccessControlList = append(acp.AccessControlList, &Grant{
Grantee: &Grantee{ Grantee: &Grantee{
URI: allUsersGroup, URI: allUsersGroup,
Type: acpGroup, Type: formGranteeType(acpGroup),
}, },
Permission: aclFullControl, Permission: aclFullControl,
}) })
@ -677,7 +596,7 @@ func addPredefinedACP(acp *AccessControlPolicy, cannedACL string) (*AccessContro
acp.AccessControlList = append(acp.AccessControlList, &Grant{ acp.AccessControlList = append(acp.AccessControlList, &Grant{
Grantee: &Grantee{ Grantee: &Grantee{
URI: allUsersGroup, URI: allUsersGroup,
Type: acpGroup, Type: formGranteeType(acpGroup),
}, },
Permission: aclRead, Permission: aclRead,
}) })
@ -1257,12 +1176,12 @@ func aclToAst(acl *AccessControlPolicy, resInfo *resourceInfo) (*ast, error) {
} }
for _, grant := range acl.AccessControlList { for _, grant := range acl.AccessControlList {
if grant.Grantee.Type == acpAmazonCustomerByEmail || (grant.Grantee.Type == acpGroup && grant.Grantee.URI != allUsersGroup) { if grant.Grantee.matchType(acpAmazonCustomerByEmail) || (grant.Grantee.matchType(acpGroup) && grant.Grantee.URI != allUsersGroup) {
return nil, stderrors.New("unsupported grantee type") return nil, stderrors.New("unsupported grantee type")
} }
var groupGrantee bool var groupGrantee bool
if grant.Grantee.Type == acpGroup { if grant.Grantee.matchType(acpGroup) {
groupGrantee = true groupGrantee = true
} else if grant.Grantee.ID == acl.Owner.ID { } else if grant.Grantee.ID == acl.Owner.ID {
continue continue
@ -1296,12 +1215,12 @@ func aclToPolicy(acl *AccessControlPolicy, resInfo *resourceInfo) (*bucketPolicy
} }
for _, grant := range acl.AccessControlList { for _, grant := range acl.AccessControlList {
if grant.Grantee.Type == acpAmazonCustomerByEmail || (grant.Grantee.Type == acpGroup && grant.Grantee.URI != allUsersGroup) { if grant.Grantee.matchType(acpAmazonCustomerByEmail) || (grant.Grantee.matchType(acpGroup) && grant.Grantee.URI != allUsersGroup) {
return nil, stderrors.New("unsupported grantee type") return nil, stderrors.New("unsupported grantee type")
} }
user := grant.Grantee.ID user := grant.Grantee.ID
if grant.Grantee.Type == acpGroup { if grant.Grantee.matchType(acpGroup) {
user = allUsersWildcard user = allUsersWildcard
} else if user == acl.Owner.ID { } else if user == acl.Owner.ID {
continue continue
@ -1461,10 +1380,10 @@ func (h *handler) encodeObjectACL(bucketACL *layer.BucketACL, bucketName, object
var grantee *Grantee var grantee *Grantee
if key == allUsersGroup { if key == allUsersGroup {
grantee = NewGrantee(acpGroup) grantee = NewGrantee(formGranteeType(acpGroup))
grantee.URI = allUsersGroup grantee.URI = allUsersGroup
} else { } else {
grantee = NewGrantee(acpCanonicalUser) grantee = NewGrantee(formGranteeType(acpCanonicalUser))
grantee.ID = key grantee.ID = key
} }
@ -1537,7 +1456,7 @@ func bucketACLToTable(acp *AccessControlPolicy, resInfo *resourceInfo) (*eacl.Ta
} }
func getRecordFunction(grantee *Grantee) (getRecordFunc, error) { func getRecordFunction(grantee *Grantee) (getRecordFunc, error) {
switch grantee.Type { switch grantee.TypeString() {
case acpAmazonCustomerByEmail: case acpAmazonCustomerByEmail:
case acpCanonicalUser: case acpCanonicalUser:
pk, err := keys.NewPublicKeyFromString(grantee.ID) pk, err := keys.NewPublicKeyFromString(grantee.ID)
@ -1557,7 +1476,7 @@ func getRecordFunction(grantee *Grantee) (getRecordFunc, error) {
func isValidGrant(grant *Grant) bool { func isValidGrant(grant *Grant) bool {
return (grant.Permission == aclFullControl || grant.Permission == aclRead || grant.Permission == aclWrite) && return (grant.Permission == aclFullControl || grant.Permission == aclRead || grant.Permission == aclWrite) &&
(grant.Grantee.Type == acpCanonicalUser || (grant.Grantee.Type == acpGroup && grant.Grantee.URI == allUsersGroup)) (grant.Grantee.matchType(acpCanonicalUser) || (grant.Grantee.matchType(acpGroup) && grant.Grantee.URI == allUsersGroup))
} }
func getAllowRecord(op eacl.Operation, pk *keys.PublicKey) *eacl.Record { func getAllowRecord(op eacl.Operation, pk *keys.PublicKey) *eacl.Record {

View file

@ -8,19 +8,20 @@ import (
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"encoding/json" "encoding/json"
"encoding/xml"
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "github.com/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "github.com/TrueCloudLab/frostfs-sdk-go/bearer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl" "github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "github.com/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "github.com/TrueCloudLab/frostfs-sdk-go/session"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@ -724,13 +725,13 @@ func TestBucketAclToPolicy(t *testing.T) {
AccessControlList: []*Grant{{ AccessControlList: []*Grant{{
Grantee: &Grantee{ Grantee: &Grantee{
URI: allUsersGroup, URI: allUsersGroup,
Type: acpGroup, Type: formGranteeType(acpGroup),
}, },
Permission: aclRead, Permission: aclRead,
}, { }, {
Grantee: &Grantee{ Grantee: &Grantee{
ID: id2, ID: id2,
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclWrite, Permission: aclWrite,
}}, }},
@ -790,19 +791,19 @@ func TestObjectAclToPolicy(t *testing.T) {
AccessControlList: []*Grant{{ AccessControlList: []*Grant{{
Grantee: &Grantee{ Grantee: &Grantee{
ID: id, ID: id,
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclFullControl, Permission: aclFullControl,
}, { }, {
Grantee: &Grantee{ Grantee: &Grantee{
ID: id2, ID: id2,
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclFullControl, Permission: aclFullControl,
}, { }, {
Grantee: &Grantee{ Grantee: &Grantee{
URI: allUsersGroup, URI: allUsersGroup,
Type: acpGroup, Type: formGranteeType(acpGroup),
}, },
Permission: aclRead, Permission: aclRead,
}}, }},
@ -859,7 +860,7 @@ func TestObjectWithVersionAclToTable(t *testing.T) {
AccessControlList: []*Grant{{ AccessControlList: []*Grant{{
Grantee: &Grantee{ Grantee: &Grantee{
ID: id, ID: id,
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclFullControl, Permission: aclFullControl,
}}, }},
@ -980,13 +981,13 @@ func TestParseCannedACLHeaders(t *testing.T) {
Grantee: &Grantee{ Grantee: &Grantee{
ID: id, ID: id,
DisplayName: address, DisplayName: address,
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclFullControl, Permission: aclFullControl,
}, { }, {
Grantee: &Grantee{ Grantee: &Grantee{
URI: allUsersGroup, URI: allUsersGroup,
Type: acpGroup, Type: formGranteeType(acpGroup),
}, },
Permission: aclRead, Permission: aclRead,
}}, }},
@ -1021,37 +1022,37 @@ func TestParseACLHeaders(t *testing.T) {
Grantee: &Grantee{ Grantee: &Grantee{
ID: id, ID: id,
DisplayName: address, DisplayName: address,
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclFullControl, Permission: aclFullControl,
}, { }, {
Grantee: &Grantee{ Grantee: &Grantee{
ID: "user1", ID: "user1",
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclFullControl, Permission: aclFullControl,
}, { }, {
Grantee: &Grantee{ Grantee: &Grantee{
URI: allUsersGroup, URI: allUsersGroup,
Type: acpGroup, Type: formGranteeType(acpGroup),
}, },
Permission: aclRead, Permission: aclRead,
}, { }, {
Grantee: &Grantee{ Grantee: &Grantee{
ID: "user2", ID: "user2",
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclRead, Permission: aclRead,
}, { }, {
Grantee: &Grantee{ Grantee: &Grantee{
ID: "user2", ID: "user2",
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclWrite, Permission: aclWrite,
}, { }, {
Grantee: &Grantee{ Grantee: &Grantee{
ID: "user3", ID: "user3",
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclWrite, Permission: aclWrite,
}}, }},
@ -1112,13 +1113,13 @@ func TestBucketAclToTable(t *testing.T) {
AccessControlList: []*Grant{{ AccessControlList: []*Grant{{
Grantee: &Grantee{ Grantee: &Grantee{
URI: allUsersGroup, URI: allUsersGroup,
Type: acpGroup, Type: formGranteeType(acpGroup),
}, },
Permission: aclRead, Permission: aclRead,
}, { }, {
Grantee: &Grantee{ Grantee: &Grantee{
ID: id2, ID: id2,
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclWrite, Permission: aclWrite,
}}, }},
@ -1169,13 +1170,13 @@ func TestObjectAclToAst(t *testing.T) {
AccessControlList: []*Grant{{ AccessControlList: []*Grant{{
Grantee: &Grantee{ Grantee: &Grantee{
ID: id, ID: id,
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclFullControl, Permission: aclFullControl,
}, { }, {
Grantee: &Grantee{ Grantee: &Grantee{
ID: id2, ID: id2,
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclRead, Permission: aclRead,
}, },
@ -1238,13 +1239,13 @@ func TestBucketAclToAst(t *testing.T) {
{ {
Grantee: &Grantee{ Grantee: &Grantee{
ID: id2, ID: id2,
Type: acpCanonicalUser, Type: formGranteeType(acpCanonicalUser),
}, },
Permission: aclWrite, Permission: aclWrite,
}, { }, {
Grantee: &Grantee{ Grantee: &Grantee{
URI: allUsersGroup, URI: allUsersGroup,
Type: acpGroup, Type: formGranteeType(acpGroup),
}, },
Permission: aclRead, Permission: aclRead,
}, },
@ -1352,85 +1353,6 @@ func TestBucketPolicy(t *testing.T) {
} }
} }
func TestBucketPolicyUnmarshal(t *testing.T) {
for _, tc := range []struct {
name string
policy string
}{
{
name: "action/resource array",
policy: `
{
"Version": "2012-10-17",
"Statement": [{
"Principal": {
"AWS": "arn:aws:iam::111122223333:role/JohnDoe"
},
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion"
],
"Resource": [
"arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",
"arn:aws:s3:::DOC-EXAMPLE-BUCKET2/*"
]
}]
}
`,
},
{
name: "action/resource string",
policy: `
{
"Version": "2012-10-17",
"Statement": [{
"Principal": {
"AWS": "arn:aws:iam::111122223333:role/JohnDoe"
},
"Effect": "Deny",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"
}]
}
`,
},
} {
t.Run(tc.name, func(t *testing.T) {
bktPolicy := &bucketPolicy{}
err := json.Unmarshal([]byte(tc.policy), bktPolicy)
require.NoError(t, err)
})
}
}
func TestPutBucketPolicy(t *testing.T) {
bktPolicy := `
{
"Version": "2012-10-17",
"Statement": [{
"Principal": {
"AWS": "*"
},
"Effect": "Deny",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::bucket-for-policy/*"
}]
}
`
hc := prepareHandlerContext(t)
bktName := "bucket-for-policy"
box, _ := createAccessBox(t)
createBucket(t, hc, bktName, box)
w, r := prepareTestPayloadRequest(hc, bktName, "", bytes.NewReader([]byte(bktPolicy)))
ctx := context.WithValue(r.Context(), api.BoxData, box)
r = r.WithContext(ctx)
hc.Handler().PutBucketPolicyHandler(w, r)
assertStatus(hc.t, w, http.StatusOK)
}
func getBucketPolicy(hc *handlerContext, bktName string) *bucketPolicy { func getBucketPolicy(hc *handlerContext, bktName string) *bucketPolicy {
w, r := prepareTestRequest(hc, bktName, "", nil) w, r := prepareTestRequest(hc, bktName, "", nil)
hc.Handler().GetBucketPolicyHandler(w, r) hc.Handler().GetBucketPolicyHandler(w, r)
@ -1515,3 +1437,47 @@ func putBucketACL(t *testing.T, tc *handlerContext, bktName string, box *accessb
tc.Handler().PutBucketACLHandler(w, r) tc.Handler().PutBucketACLHandler(w, r)
assertStatus(t, w, http.StatusOK) assertStatus(t, w, http.StatusOK)
} }
func TestACLGranteeParse(t *testing.T) {
body := []byte(`
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Owner>
<DisplayName>NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM</DisplayName>
<ID>NbUgTSFvPmsRxmGeWpuuGeJUoRoi6PErcM</ID>
</Owner>
<AccessControlList>
<Grant>
<Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
<ID>031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a</ID>
</Grantee>
<Permission>FULL_CONTROL</Permission>
</Grant>
<Grant>
<Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group">
<URI>http://acs.amazonaws.com/groups/global/AllUsers</URI>
</Grantee>
<Permission>FULL_CONTROL</Permission>
</Grant>
</AccessControlList>
</AccessControlPolicy>
`)
acl := &AccessControlPolicy{}
err := xml.Unmarshal(body, acl)
require.NoError(t, err)
require.True(t, acl.AccessControlList[0].Grantee.matchType(acpCanonicalUser))
require.True(t, acl.AccessControlList[1].Grantee.matchType(acpGroup))
grantee := NewGrantee(formGranteeType(acpGroup))
grantee.URI = allUsersGroup
raw, err := xml.MarshalIndent(grantee, "", " ")
require.NoError(t, err)
grantee2 := &Grantee{}
err = xml.Unmarshal(raw, grantee2)
require.NoError(t, err)
grantee2.XMLNS.Value = granteeXMLNS
require.Equal(t, grantee, grantee2)
}

View file

@ -1,14 +1,12 @@
package handler package handler
import ( import (
"encoding/xml"
"errors" "errors"
"io"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" "github.com/TrueCloudLab/frostfs-sdk-go/netmap"
"go.uber.org/zap" "go.uber.org/zap"
) )
@ -27,24 +25,16 @@ type (
// Config contains data which handler needs to keep. // Config contains data which handler needs to keep.
Config struct { Config struct {
Policy PlacementPolicy Policy PlacementPolicy
XMLDecoder XMLDecoderProvider DefaultMaxAge int
DefaultMaxAge int NotificatorEnabled bool
NotificatorEnabled bool CopiesNumber uint32
CopiesNumber uint32
ResolveZoneList []string
IsResolveListAllow bool // True if ResolveZoneList contains allowed zones
CompleteMultipartKeepalive time.Duration
} }
PlacementPolicy interface { PlacementPolicy interface {
Default() netmap.PlacementPolicy Default() netmap.PlacementPolicy
Get(string) (netmap.PlacementPolicy, bool) Get(string) (netmap.PlacementPolicy, bool)
} }
XMLDecoderProvider interface {
NewCompleteMultipartDecoder(io.Reader) *xml.Decoder
}
) )
const ( const (

View file

@ -6,10 +6,10 @@ import (
"strconv" "strconv"
"strings" "strings"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -4,7 +4,7 @@ import (
"strings" "strings"
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )

View file

@ -6,12 +6,12 @@ import (
"regexp" "regexp"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth" "github.com/TrueCloudLab/frostfs-s3-gw/api/auth"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "github.com/TrueCloudLab/frostfs-sdk-go/session"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -6,7 +6,7 @@ import (
"net/url" "net/url"
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )

View file

@ -5,9 +5,9 @@ import (
"strconv" "strconv"
"strings" "strings"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -1,41 +0,0 @@
package handler
import (
"context"
"net/http"
"strings"
"testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
)
func TestCORSOriginWildcard(t *testing.T) {
body := `
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedMethod>GET</AllowedMethod>
<AllowedOrigin>*</AllowedOrigin>
</CORSRule>
</CORSConfiguration>
`
hc := prepareHandlerContext(t)
bktName := "bucket-for-cors"
box, _ := createAccessBox(t)
w, r := prepareTestRequest(hc, bktName, "", nil)
ctx := context.WithValue(r.Context(), api.BoxData, box)
r = r.WithContext(ctx)
r.Header.Add(api.AmzACL, "public-read")
hc.Handler().CreateBucketHandler(w, r)
assertStatus(t, w, http.StatusOK)
w, r = prepareTestPayloadRequest(hc, bktName, "", strings.NewReader(body))
ctx = context.WithValue(r.Context(), api.BoxData, box)
r = r.WithContext(ctx)
hc.Handler().PutBucketCorsHandler(w, r)
assertStatus(t, w, http.StatusOK)
w, r = prepareTestPayloadRequest(hc, bktName, "", nil)
hc.Handler().GetBucketCorsHandler(w, r)
assertStatus(t, w, http.StatusOK)
}

View file

@ -6,20 +6,17 @@ import (
"strconv" "strconv"
"strings" "strings"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
apistatus "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client/status" apistatus "github.com/TrueCloudLab/frostfs-sdk-go/client/status"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "github.com/TrueCloudLab/frostfs-sdk-go/session"
"go.uber.org/zap" "go.uber.org/zap"
"go.uber.org/zap/zapcore" "go.uber.org/zap/zapcore"
) )
// limitation of AWS https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
const maxObjectsToDelete = 1000
// DeleteObjectsRequest -- xml carrying the object key names which should be deleted. // DeleteObjectsRequest -- xml carrying the object key names which should be deleted.
type DeleteObjectsRequest struct { type DeleteObjectsRequest struct {
// Element to enable quiet mode for the request // Element to enable quiet mode for the request
@ -179,11 +176,6 @@ func (h *handler) DeleteMultipleObjectsHandler(w http.ResponseWriter, r *http.Re
return return
} }
if len(requested.Objects) == 0 || len(requested.Objects) > maxObjectsToDelete {
h.logAndSendError(w, "number of objects to delete must be greater than 0 and less or equal to 1000", reqInfo, errors.GetAPIError(errors.ErrMalformedXML))
return
}
removed := make(map[string]*layer.VersionedObject) removed := make(map[string]*layer.VersionedObject)
toRemove := make([]*layer.VersionedObject, 0, len(requested.Objects)) toRemove := make([]*layer.VersionedObject, 0, len(requested.Objects))
for _, obj := range requested.Objects { for _, obj := range requested.Objects {

View file

@ -6,8 +6,8 @@ import (
"net/url" "net/url"
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )

View file

@ -12,8 +12,8 @@ import (
"strings" "strings"
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )

View file

@ -8,10 +8,10 @@ import (
"strings" "strings"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -8,9 +8,9 @@ import (
"testing" "testing"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )

View file

@ -13,15 +13,15 @@ import (
"testing" "testing"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/resolver" "github.com/TrueCloudLab/frostfs-s3-gw/api/resolver"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" "github.com/TrueCloudLab/frostfs-sdk-go/netmap"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "github.com/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"go.uber.org/zap" "go.uber.org/zap"
@ -63,12 +63,6 @@ func (p *placementPolicyMock) Get(string) (netmap.PlacementPolicy, bool) {
return netmap.PlacementPolicy{}, false return netmap.PlacementPolicy{}, false
} }
type xmlDecoderProviderMock struct{}
func (p *xmlDecoderProviderMock) NewCompleteMultipartDecoder(r io.Reader) *xml.Decoder {
return xml.NewDecoder(r)
}
func prepareHandlerContext(t *testing.T) *handlerContext { func prepareHandlerContext(t *testing.T) *handlerContext {
key, err := keys.NewPrivateKey() key, err := keys.NewPrivateKey()
require.NoError(t, err) require.NoError(t, err)
@ -99,8 +93,7 @@ func prepareHandlerContext(t *testing.T) *handlerContext {
log: l, log: l,
obj: layer.NewLayer(l, tp, layerCfg), obj: layer.NewLayer(l, tp, layerCfg),
cfg: &Config{ cfg: &Config{
Policy: &placementPolicyMock{defaultPolicy: pp}, Policy: &placementPolicyMock{defaultPolicy: pp},
XMLDecoder: &xmlDecoderProviderMock{},
}, },
} }

View file

@ -4,10 +4,10 @@ import (
"bytes" "bytes"
"net/http" "net/http"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"go.uber.org/zap" "go.uber.org/zap"
) )
@ -124,12 +124,6 @@ func (h *handler) HeadBucketHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set(api.ContainerID, bktInfo.CID.EncodeToString()) w.Header().Set(api.ContainerID, bktInfo.CID.EncodeToString())
w.Header().Set(api.AmzBucketRegion, bktInfo.LocationConstraint) w.Header().Set(api.AmzBucketRegion, bktInfo.LocationConstraint)
if isAvailableToResolve(bktInfo.Zone, h.cfg.ResolveZoneList, h.cfg.IsResolveListAllow) {
w.Header().Set(api.ContainerName, bktInfo.Name)
w.Header().Set(api.ContainerZone, bktInfo.Zone)
}
api.WriteResponse(w, http.StatusOK, nil, api.MimeNone) api.WriteResponse(w, http.StatusOK, nil, api.MimeNone)
} }
@ -163,25 +157,3 @@ func writeLockHeaders(h http.Header, legalHold *data.LegalHold, retention *data.
h.Set(api.AmzObjectLockMode, retention.Mode) h.Set(api.AmzObjectLockMode, retention.Mode)
} }
} }
func isAvailableToResolve(zone string, list []string, isAllowList bool) bool {
// empty zone means container doesn't have proper system name,
// so we don't have to resolve it
if len(zone) == 0 {
return false
}
var zoneInList bool
for _, t := range list {
if t == zone {
zoneInList = true
break
}
}
// InList | IsAllowList | Result
// 0 0 1
// 0 1 0
// 1 0 0
// 1 1 1
return zoneInList == isAllowList
}

View file

@ -6,10 +6,10 @@ import (
"testing" "testing"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "github.com/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "github.com/TrueCloudLab/frostfs-sdk-go/bearer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl" "github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@ -86,26 +86,6 @@ func TestInvalidAccessThroughCache(t *testing.T) {
assertStatus(t, w, http.StatusForbidden) assertStatus(t, w, http.StatusForbidden)
} }
func TestIsAvailableToResolve(t *testing.T) {
list := []string{"container", "s3"}
for i, testCase := range [...]struct {
isAllowList bool
list []string
zone string
expected bool
}{
{isAllowList: true, list: list, zone: "container", expected: true},
{isAllowList: true, list: list, zone: "sftp", expected: false},
{isAllowList: false, list: list, zone: "s3", expected: false},
{isAllowList: false, list: list, zone: "system", expected: true},
{isAllowList: true, list: list, zone: "", expected: false},
} {
result := isAvailableToResolve(testCase.zone, testCase.list, testCase.isAllowList)
require.Equal(t, testCase.expected, result, "case %d", i+1)
}
}
func newTestAccessBox(t *testing.T, key *keys.PrivateKey) *accessbox.Box { func newTestAccessBox(t *testing.T, key *keys.PrivateKey) *accessbox.Box {
var err error var err error
if key == nil { if key == nil {

View file

@ -3,7 +3,7 @@ package handler
import ( import (
"net/http" "net/http"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
) )
func (h *handler) GetBucketLocationHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketLocationHandler(w http.ResponseWriter, r *http.Request) {

View file

@ -4,8 +4,8 @@ import (
"net/http" "net/http"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
) )
const maxObjectList = 1000 // Limit number of objects in a listObjectsResponse/listObjectsVersionsResponse. const maxObjectList = 1000 // Limit number of objects in a listObjectsResponse/listObjectsVersionsResponse.

View file

@ -8,10 +8,10 @@ import (
"strconv" "strconv"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apiErrors "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
) )
const ( const (

View file

@ -10,9 +10,9 @@ import (
"testing" "testing"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apiErrors "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )

View file

@ -2,17 +2,16 @@ package handler
import ( import (
"encoding/xml" "encoding/xml"
"fmt"
"io"
"net/http" "net/http"
"net/url" "net/url"
"strconv" "strconv"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"github.com/TrueCloudLab/frostfs-sdk-go/session"
"github.com/google/uuid" "github.com/google/uuid"
"go.uber.org/zap" "go.uber.org/zap"
) )
@ -373,6 +372,8 @@ func (h *handler) CompleteMultipartUploadHandler(w http.ResponseWriter, r *http.
} }
var ( var (
sessionTokenSetEACL *session.Container
uploadID = r.URL.Query().Get(uploadIDHeaderName) uploadID = r.URL.Query().Get(uploadIDHeaderName)
uploadInfo = &layer.UploadInfoParams{ uploadInfo = &layer.UploadInfoParams{
UploadID: uploadID, UploadID: uploadID,
@ -383,7 +384,7 @@ func (h *handler) CompleteMultipartUploadHandler(w http.ResponseWriter, r *http.
) )
reqBody := new(CompleteMultipartUpload) reqBody := new(CompleteMultipartUpload)
if err = h.cfg.XMLDecoder.NewCompleteMultipartDecoder(r.Body).Decode(reqBody); err != nil { if err = xml.NewDecoder(r.Body).Decode(reqBody); err != nil {
h.logAndSendError(w, "could not read complete multipart upload xml", reqInfo, h.logAndSendError(w, "could not read complete multipart upload xml", reqInfo,
errors.GetAPIError(errors.ErrMalformedXML), additional...) errors.GetAPIError(errors.ErrMalformedXML), additional...)
return return
@ -398,51 +399,10 @@ func (h *handler) CompleteMultipartUploadHandler(w http.ResponseWriter, r *http.
Parts: reqBody.Parts, Parts: reqBody.Parts,
} }
// Next operations might take some time, so we want to keep client's
// connection alive. To do so, gateway sends periodic white spaces
// back to the client the same way as Amazon S3 service does.
stopPeriodicResponseWriter := periodicXMLWriter(w, h.cfg.CompleteMultipartKeepalive)
// Start complete multipart upload which may take some time to fetch object
// and re-upload it part by part.
objInfo, err := h.completeMultipartUpload(r, c, bktInfo, reqInfo)
// Stop periodic writer as complete multipart upload is finished
// successfully or not.
headerIsWritten := stopPeriodicResponseWriter()
responseWriter := api.EncodeToResponse
errLogger := h.logAndSendError
// Do not send XML and HTTP headers if periodic writer was invoked at this point.
if headerIsWritten {
responseWriter = api.EncodeToResponseNoHeader
errLogger = h.logAndSendErrorNoHeader
}
if err != nil {
errLogger(w, "complete multipart error", reqInfo, err, additional...)
return
}
response := CompleteMultipartUploadResponse{
Bucket: objInfo.Bucket,
ETag: objInfo.HashSum,
Key: objInfo.Name,
}
// Here we previously set api.AmzVersionID header for versioned bucket.
// It is not possible after #60, because of periodic white
// space XML writer to keep connection with the client.
if err = responseWriter(w, response); err != nil {
errLogger(w, "something went wrong", reqInfo, err)
}
}
func (h *handler) completeMultipartUpload(r *http.Request, c *layer.CompleteMultipartParams, bktInfo *data.BucketInfo, reqInfo *api.ReqInfo) (*data.ObjectInfo, error) {
uploadData, extendedObjInfo, err := h.obj.CompleteMultipartUpload(r.Context(), c) uploadData, extendedObjInfo, err := h.obj.CompleteMultipartUpload(r.Context(), c)
if err != nil { if err != nil {
return nil, fmt.Errorf("could not complete multipart upload: %w", err) h.logAndSendError(w, "could not complete multipart upload", reqInfo, err, additional...)
return
} }
objInfo := extendedObjInfo.ObjectInfo objInfo := extendedObjInfo.ObjectInfo
@ -457,22 +417,21 @@ func (h *handler) completeMultipartUpload(r *http.Request, c *layer.CompleteMult
NodeVersion: extendedObjInfo.NodeVersion, NodeVersion: extendedObjInfo.NodeVersion,
} }
if _, err = h.obj.PutObjectTagging(r.Context(), tagPrm); err != nil { if _, err = h.obj.PutObjectTagging(r.Context(), tagPrm); err != nil {
return nil, fmt.Errorf("could not put tagging file of completed multipart upload: %w", err) h.logAndSendError(w, "could not put tagging file of completed multipart upload", reqInfo, err, additional...)
return
} }
} }
if len(uploadData.ACLHeaders) != 0 { if len(uploadData.ACLHeaders) != 0 {
sessionTokenSetEACL, err := getSessionTokenSetEACL(r.Context())
if err != nil {
return nil, fmt.Errorf("couldn't get eacl token: %w", err)
}
key, err := h.bearerTokenIssuerKey(r.Context()) key, err := h.bearerTokenIssuerKey(r.Context())
if err != nil { if err != nil {
return nil, fmt.Errorf("couldn't get gate key: %w", err) h.logAndSendError(w, "couldn't get gate key", reqInfo, err)
return
} }
acl, err := parseACLHeaders(r.Header, key) acl, err := parseACLHeaders(r.Header, key)
if err != nil { if err != nil {
return nil, fmt.Errorf("could not parse acl: %w", err) h.logAndSendError(w, "could not parse acl", reqInfo, err)
return
} }
resInfo := &resourceInfo{ resInfo := &resourceInfo{
@ -481,10 +440,12 @@ func (h *handler) completeMultipartUpload(r *http.Request, c *layer.CompleteMult
} }
astObject, err := aclToAst(acl, resInfo) astObject, err := aclToAst(acl, resInfo)
if err != nil { if err != nil {
return nil, fmt.Errorf("could not translate acl of completed multipart upload to ast: %w", err) h.logAndSendError(w, "could not translate acl of completed multipart upload to ast", reqInfo, err, additional...)
return
} }
if _, err = h.updateBucketACL(r, astObject, bktInfo, sessionTokenSetEACL); err != nil { if _, err = h.updateBucketACL(r, astObject, bktInfo, sessionTokenSetEACL); err != nil {
return nil, fmt.Errorf("could not update bucket acl while completing multipart upload: %w", err) h.logAndSendError(w, "could not update bucket acl while completing multipart upload", reqInfo, err, additional...)
return
} }
} }
@ -498,7 +459,24 @@ func (h *handler) completeMultipartUpload(r *http.Request, c *layer.CompleteMult
h.log.Error("couldn't send notification: %w", zap.Error(err)) h.log.Error("couldn't send notification: %w", zap.Error(err))
} }
return objInfo, nil bktSettings, err := h.obj.GetBucketSettings(r.Context(), bktInfo)
if err != nil {
h.logAndSendError(w, "could not get bucket settings", reqInfo, err)
}
response := CompleteMultipartUploadResponse{
Bucket: objInfo.Bucket,
ETag: objInfo.HashSum,
Key: objInfo.Name,
}
if bktSettings.VersioningEnabled() {
w.Header().Set(api.AmzVersionID, objInfo.VersionID())
}
if err = api.EncodeToResponse(w, response); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err)
}
} }
func (h *handler) ListMultipartUploadsHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) ListMultipartUploadsHandler(w http.ResponseWriter, r *http.Request) {
@ -703,62 +681,3 @@ func encodeListPartsToResponse(info *layer.ListPartsInfo, params *layer.ListPart
Parts: info.Parts, Parts: info.Parts,
} }
} }
// periodicXMLWriter creates go routine to write xml header and whitespaces
// over time to avoid connection drop from the client. To work properly,
// pass `http.ResponseWriter` with implemented `http.Flusher` interface.
// Returns stop function which returns boolean if writer has been used
// during goroutine execution. To disable writer, pass 0 duration value.
func periodicXMLWriter(w io.Writer, dur time.Duration) (stop func() bool) {
if dur == 0 { // 0 duration disables periodic writer
return func() bool { return false }
}
whitespaceChar := []byte(" ")
closer := make(chan struct{})
done := make(chan struct{})
headerWritten := false
go func() {
defer close(done)
tick := time.NewTicker(dur)
defer tick.Stop()
for {
select {
case <-tick.C:
if !headerWritten {
_, err := w.Write([]byte(xml.Header))
headerWritten = err == nil
}
_, err := w.Write(whitespaceChar)
if err != nil {
return // is there anything we can do better than ignore error?
}
if buffered, ok := w.(http.Flusher); ok {
buffered.Flush()
}
case <-closer:
return
}
}
}()
stop = func() bool {
close(closer)
<-done // wait for goroutine to stop
return headerWritten
}
return stop
}
// periodicWriterErrorSender returns handler function to send error. If header is
// alreay written by periodic XML writer, do not send HTTP and XML headers.
func (h *handler) periodicWriterErrorSender(headerWritten bool) func(http.ResponseWriter, string, *api.ReqInfo, error, ...zap.Field) {
if headerWritten {
return h.logAndSendErrorNoHeader
}
return h.logAndSendError
}

View file

@ -1,48 +0,0 @@
package handler
import (
"bytes"
"encoding/xml"
"testing"
"time"
"github.com/stretchr/testify/require"
)
func TestPeriodicWriter(t *testing.T) {
const dur = 100 * time.Millisecond
const whitespaces = 8
expected := []byte(xml.Header)
for i := 0; i < whitespaces; i++ {
expected = append(expected, []byte(" ")...)
}
t.Run("writes data", func(t *testing.T) {
buf := bytes.NewBuffer(nil)
stop := periodicXMLWriter(buf, dur)
// N number of whitespaces + half durations to guarantee at least N writes in buffer
time.Sleep(whitespaces*dur + dur/2)
require.True(t, stop())
require.Equal(t, expected, buf.Bytes())
t.Run("no additional data after stop", func(t *testing.T) {
time.Sleep(2 * dur)
require.Equal(t, expected, buf.Bytes())
})
})
t.Run("does not write data", func(t *testing.T) {
buf := bytes.NewBuffer(nil)
stop := periodicXMLWriter(buf, dur)
time.Sleep(dur / 2)
require.False(t, stop())
require.Empty(t, buf.Bytes())
t.Run("disabled", func(t *testing.T) {
stop = periodicXMLWriter(buf, 0)
require.False(t, stop())
require.Empty(t, buf.Bytes())
})
})
}

View file

@ -3,8 +3,8 @@ package handler
import ( import (
"net/http" "net/http"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
) )
func (h *handler) DeleteBucketPolicyHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) DeleteBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {

View file

@ -8,11 +8,11 @@ import (
"strings" "strings"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "github.com/TrueCloudLab/frostfs-sdk-go/bearer"
"github.com/google/uuid" "github.com/google/uuid"
) )

View file

@ -3,8 +3,8 @@ package handler
import ( import (
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )

View file

@ -6,11 +6,11 @@ import (
"strconv" "strconv"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
) )
// ListObjectsV1Handler handles objects listing requests for API version 1. // ListObjectsV1Handler handles objects listing requests for API version 1.

View file

@ -6,7 +6,7 @@ import (
"strconv" "strconv"
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )

View file

@ -16,15 +16,15 @@ import (
"strings" "strings"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth" "github.com/TrueCloudLab/frostfs-s3-gw/api/auth"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "github.com/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl" "github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "github.com/TrueCloudLab/frostfs-sdk-go/session"
"go.uber.org/zap" "go.uber.org/zap"
) )
@ -720,10 +720,7 @@ func (h *handler) CreateBucketHandler(w http.ResponseWriter, r *http.Request) {
return return
} }
if err = h.setPolicy(p, createParams.LocationConstraint, policies); err != nil { h.setPolicy(p, createParams.LocationConstraint, policies)
h.logAndSendError(w, "couldn't set placement policy", reqInfo, err)
return
}
p.ObjectLockEnabled = isLockEnabled(r.Header) p.ObjectLockEnabled = isLockEnabled(r.Header)
@ -751,27 +748,25 @@ func (h *handler) CreateBucketHandler(w http.ResponseWriter, r *http.Request) {
api.WriteSuccessResponseHeadersOnly(w) api.WriteSuccessResponseHeadersOnly(w)
} }
func (h handler) setPolicy(prm *layer.CreateBucketParams, locationConstraint string, userPolicies []*accessbox.ContainerPolicy) error { func (h handler) setPolicy(prm *layer.CreateBucketParams, locationConstraint string, userPolicies []*accessbox.ContainerPolicy) {
prm.Policy = h.cfg.Policy.Default() prm.Policy = h.cfg.Policy.Default()
prm.LocationConstraint = locationConstraint
if locationConstraint == "" { if locationConstraint == "" {
return nil return
}
if policy, ok := h.cfg.Policy.Get(locationConstraint); ok {
prm.Policy = policy
prm.LocationConstraint = locationConstraint
} }
for _, placementPolicy := range userPolicies { for _, placementPolicy := range userPolicies {
if placementPolicy.LocationConstraint == locationConstraint { if placementPolicy.LocationConstraint == locationConstraint {
prm.Policy = placementPolicy.Policy prm.Policy = placementPolicy.Policy
return nil prm.LocationConstraint = locationConstraint
return
} }
} }
if policy, ok := h.cfg.Policy.Get(locationConstraint); ok {
prm.Policy = policy
return nil
}
return errors.GetAPIError(errors.ErrInvalidLocationConstraint)
} }
func isLockEnabled(header http.Header) bool { func isLockEnabled(header http.Header) bool {

View file

@ -8,8 +8,8 @@ import (
"testing" "testing"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )

View file

@ -70,21 +70,50 @@ type Grant struct {
// Grantee is info about access rights of some actor. // Grantee is info about access rights of some actor.
type Grantee struct { type Grantee struct {
XMLName xml.Name `xml:"Grantee"` XMLName xml.Name `xml:"Grantee"`
XMLNS string `xml:"xmlns:xsi,attr"` XMLNS xml.Attr `xml:"xsi,attr"`
ID string `xml:"ID,omitempty"` ID string `xml:"ID,omitempty"`
DisplayName string `xml:"DisplayName,omitempty"` DisplayName string `xml:"DisplayName,omitempty"`
EmailAddress string `xml:"EmailAddress,omitempty"` EmailAddress string `xml:"EmailAddress,omitempty"`
URI string `xml:"URI,omitempty"` URI string `xml:"URI,omitempty"`
Type GranteeType `xml:"xsi:type,attr"` Type xml.Attr `xml:"type,attr"`
} }
// NewGrantee creates new grantee using workaround func (g Grantee) matchType(t string) bool {
// https://github.com/golang/go/issues/9519#issuecomment-252196382 return g.Type.Value == t
func NewGrantee(t GranteeType) *Grantee { }
func (g Grantee) TypeString() string {
return g.Type.Value
}
func formGranteeType(str string) xml.Attr {
return xml.Attr{
Name: xml.Name{
Space: "xsi",
Local: "type",
},
Value: str,
}
}
const granteeXMLNS = "http://www.w3.org/2001/XMLSchema-instance"
// NewGrantee creates new grantee.
func NewGrantee(t xml.Attr) *Grantee {
return &Grantee{ return &Grantee{
XMLNS: "http://www.w3.org/2001/XMLSchema-instance", XMLName: xml.Name{
Type: t, Local: "Grantee",
},
XMLNS: xml.Attr{
Name: xml.Name{
Space: "xmlns",
Local: "xsi",
},
Value: granteeXMLNS,
},
Type: t,
} }
} }

View file

@ -8,10 +8,10 @@ import (
"strings" "strings"
"unicode" "unicode"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -3,8 +3,8 @@ package handler
import ( import (
"net/http" "net/http"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
) )
func (h *handler) SelectObjectContentHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) SelectObjectContentHandler(w http.ResponseWriter, r *http.Request) {

View file

@ -7,11 +7,11 @@ import (
"strconv" "strconv"
"strings" "strings"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "github.com/TrueCloudLab/frostfs-sdk-go/session"
"go.uber.org/zap" "go.uber.org/zap"
) )
@ -29,19 +29,6 @@ func (h *handler) logAndSendError(w http.ResponseWriter, logText string, reqInfo
h.log.Error("call method", fields...) h.log.Error("call method", fields...)
} }
func (h *handler) logAndSendErrorNoHeader(w http.ResponseWriter, logText string, reqInfo *api.ReqInfo, err error, additional ...zap.Field) {
api.WriteErrorResponseNoHeader(w, reqInfo, transformToS3Error(err))
fields := []zap.Field{
zap.String("request_id", reqInfo.RequestID),
zap.String("method", reqInfo.API),
zap.String("bucket", reqInfo.BucketName),
zap.String("object", reqInfo.ObjectName),
zap.String("description", logText),
zap.Error(err)}
fields = append(fields, additional...)
h.log.Error("call method", fields...)
}
func transformToS3Error(err error) error { func transformToS3Error(err error) error {
if _, ok := err.(errors.Error); ok { if _, ok := err.(errors.Error); ok {
return err return err
@ -55,10 +42,6 @@ func transformToS3Error(err error) error {
return errors.GetAPIError(errors.ErrInternalError) return errors.GetAPIError(errors.ErrInternalError)
} }
func (h *handler) ResolveBucket(ctx context.Context, bucket string) (*data.BucketInfo, error) {
return h.obj.GetBucketInfo(ctx, bucket)
}
func (h *handler) getBucketAndCheckOwner(r *http.Request, bucket string, header ...string) (*data.BucketInfo, error) { func (h *handler) getBucketAndCheckOwner(r *http.Request, bucket string, header ...string) (*data.BucketInfo, error) {
bktInfo, err := h.obj.GetBucketInfo(r.Context(), bucket) bktInfo, err := h.obj.GetBucketInfo(r.Context(), bucket)
if err != nil { if err != nil {

View file

@ -4,10 +4,10 @@ import (
"encoding/xml" "encoding/xml"
"net/http" "net/http"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
) )
func (h *handler) PutBucketVersioningHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) PutBucketVersioningHandler(w http.ResponseWriter, r *http.Request) {

View file

@ -62,9 +62,7 @@ const (
AmzServerSideEncryptionCustomerKey = "x-amz-server-side-encryption-customer-key" AmzServerSideEncryptionCustomerKey = "x-amz-server-side-encryption-customer-key"
AmzServerSideEncryptionCustomerKeyMD5 = "x-amz-server-side-encryption-customer-key-MD5" AmzServerSideEncryptionCustomerKeyMD5 = "x-amz-server-side-encryption-customer-key-MD5"
ContainerID = "X-Container-Id" ContainerID = "X-Container-Id"
ContainerName = "X-Container-Name"
ContainerZone = "X-Container-Zone"
AccessControlAllowOrigin = "Access-Control-Allow-Origin" AccessControlAllowOrigin = "Access-Control-Allow-Origin"
AccessControlAllowMethods = "Access-Control-Allow-Methods" AccessControlAllowMethods = "Access-Control-Allow-Methods"

View file

@ -1,11 +1,11 @@
package layer package layer
import ( import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/cache" "github.com/TrueCloudLab/frostfs-s3-gw/api/cache"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -4,8 +4,8 @@ import (
"context" "context"
errorsStd "errors" errorsStd "errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
) )
func (n *layer) GetObjectTaggingAndLock(ctx context.Context, objVersion *ObjectVersion, nodeVersion *data.NodeVersion) (map[string]string, *data.LockInfo, error) { func (n *layer) GetObjectTaggingAndLock(ctx context.Context, objVersion *ObjectVersion, nodeVersion *data.NodeVersion) (map[string]string, *data.LockInfo, error) {

View file

@ -5,15 +5,14 @@ import (
"fmt" "fmt"
"strconv" "strconv"
v2container "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-sdk-go/client"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client" "github.com/TrueCloudLab/frostfs-sdk-go/container"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" "github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl" "github.com/TrueCloudLab/frostfs-sdk-go/session"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
"go.uber.org/zap" "go.uber.org/zap"
) )
@ -57,7 +56,6 @@ func (n *layer) containerInfo(ctx context.Context, idCnr cid.ID) (*data.BucketIn
info.Owner = cnr.Owner() info.Owner = cnr.Owner()
if domain := container.ReadDomain(cnr); domain.Name() != "" { if domain := container.ReadDomain(cnr); domain.Name() != "" {
info.Name = domain.Name() info.Name = domain.Name()
info.Zone = domain.Zone()
} }
info.Created = container.CreatedAt(cnr) info.Created = container.CreatedAt(cnr)
info.LocationConstraint = cnr.Attribute(attributeLocationConstraint) info.LocationConstraint = cnr.Attribute(attributeLocationConstraint)
@ -116,7 +114,6 @@ func (n *layer) createContainer(ctx context.Context, p *CreateBucketParams) (*da
} }
bktInfo := &data.BucketInfo{ bktInfo := &data.BucketInfo{
Name: p.Name, Name: p.Name,
Zone: v2container.SysAttributeZoneDefault,
Owner: ownerID, Owner: ownerID,
Created: TimeNow(ctx), Created: TimeNow(ctx),
LocationConstraint: p.LocationConstraint, LocationConstraint: p.LocationConstraint,

View file

@ -8,8 +8,8 @@ import (
"fmt" "fmt"
"io" "io"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"go.uber.org/zap" "go.uber.org/zap"
) )
@ -39,7 +39,7 @@ func (n *layer) PutBucketCORS(ctx context.Context, p *PutCORSParams) error {
prm := PrmObjectCreate{ prm := PrmObjectCreate{
Container: p.BktInfo.CID, Container: p.BktInfo.CID,
Creator: p.BktInfo.Owner, Creator: p.BktInfo.Owner,
Payload: &buf, Payload: p.Reader,
Filepath: p.BktInfo.CORSObjectName(), Filepath: p.BktInfo.CORSObjectName(),
CreationTime: TimeNow(ctx), CreationTime: TimeNow(ctx),
CopiesNumber: p.CopiesNumber, CopiesNumber: p.CopiesNumber,

View file

@ -7,16 +7,16 @@ import (
"io" "io"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "github.com/TrueCloudLab/frostfs-sdk-go/bearer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container" "github.com/TrueCloudLab/frostfs-sdk-go/container"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/acl" "github.com/TrueCloudLab/frostfs-sdk-go/container/acl"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl" "github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" "github.com/TrueCloudLab/frostfs-sdk-go/netmap"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "github.com/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "github.com/TrueCloudLab/frostfs-sdk-go/session"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
) )
// PrmContainerCreate groups parameters of FrostFS.CreateContainer operation. // PrmContainerCreate groups parameters of FrostFS.CreateContainer operation.

View file

@ -11,18 +11,18 @@ import (
"strings" "strings"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "github.com/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "github.com/TrueCloudLab/frostfs-sdk-go/bearer"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl" "github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" "github.com/TrueCloudLab/frostfs-sdk-go/netmap"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "github.com/TrueCloudLab/frostfs-sdk-go/session"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
"github.com/nats-io/nats.go" "github.com/nats-io/nats.go"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"go.uber.org/zap" "go.uber.org/zap"

View file

@ -4,8 +4,7 @@ import (
"testing" "testing"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@ -44,10 +43,10 @@ func TestObjectLockAttributes(t *testing.T) {
expEpoch := false expEpoch := false
for _, attr := range lockObj.Attributes() { for _, attr := range lockObj.Attributes() {
if attr.Key() == object.SysAttributeExpEpoch { if attr.Key() == AttributeExpirationEpoch {
expEpoch = true expEpoch = true
} }
} }
require.Truef(t, expEpoch, "system header __SYSTEM__EXPIRATION_EPOCH presence") require.Truef(t, expEpoch, "system header __NEOFS__EXPIRATION_EPOCH presence")
} }

View file

@ -11,12 +11,12 @@ import (
"strings" "strings"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
"github.com/minio/sio" "github.com/minio/sio"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -10,18 +10,18 @@ import (
"io" "io"
"time" "time"
objectv2 "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object" objectv2 "github.com/TrueCloudLab/frostfs-api-go/v2/object"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "github.com/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "github.com/TrueCloudLab/frostfs-sdk-go/bearer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/checksum" "github.com/TrueCloudLab/frostfs-sdk-go/checksum"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container" "github.com/TrueCloudLab/frostfs-sdk-go/container"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl" "github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "github.com/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "github.com/TrueCloudLab/frostfs-sdk-go/session"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
) )
type TestFrostFS struct { type TestFrostFS struct {
@ -144,7 +144,7 @@ func (t *TestFrostFS) ReadObject(ctx context.Context, prm PrmObjectRead) (*Objec
if obj, ok := t.objects[sAddr]; ok { if obj, ok := t.objects[sAddr]; ok {
owner := getOwner(ctx) owner := getOwner(ctx)
if !obj.OwnerID().Equals(owner) && !t.isPublicRead(prm.Container) { if !obj.OwnerID().Equals(owner) {
return nil, ErrAccessDenied return nil, ErrAccessDenied
} }
@ -282,25 +282,6 @@ func (t *TestFrostFS) ContainerEACL(_ context.Context, cnrID cid.ID) (*eacl.Tabl
return table, nil return table, nil
} }
func (t *TestFrostFS) isPublicRead(cnrID cid.ID) bool {
table, ok := t.eaclTables[cnrID.EncodeToString()]
if !ok {
return false
}
for _, rec := range table.Records() {
if rec.Operation() == eacl.OperationGet && len(rec.Filters()) == 0 {
for _, trgt := range rec.Targets() {
if trgt.Role() == eacl.RoleOthers {
return rec.Action() == eacl.ActionAllow
}
}
}
}
return false
}
func getOwner(ctx context.Context) user.ID { func getOwner(ctx context.Context) user.ID {
if bd, ok := ctx.Value(api.BoxData).(*accessbox.Box); ok && bd != nil && bd.Gate != nil && bd.Gate.BearerToken != nil { if bd, ok := ctx.Value(api.BoxData).(*accessbox.Box); ok && bd != nil && bd.Gate != nil && bd.Gate.BearerToken != nil {
return bearer.ResolveIssuer(*bd.Gate.BearerToken) return bearer.ResolveIssuer(*bd.Gate.BearerToken)

View file

@ -7,8 +7,8 @@ import (
errorsStd "errors" errorsStd "errors"
"fmt" "fmt"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -14,14 +14,14 @@ import (
"strings" "strings"
"sync" "sync"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/cache" "github.com/TrueCloudLab/frostfs-s3-gw/api/cache"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apiErrors "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client" "github.com/TrueCloudLab/frostfs-sdk-go/client"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "github.com/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/minio/sio" "github.com/minio/sio"
"github.com/panjf2000/ants/v2" "github.com/panjf2000/ants/v2"
"go.uber.org/zap" "go.uber.org/zap"

View file

@ -9,14 +9,14 @@ import (
"strconv" "strconv"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
) )
const ( const (
AttributeComplianceMode = ".s3-compliance-mode" AttributeComplianceMode = ".s3-compliance-mode"
AttributeExpirationEpoch = "__NEOFS__EXPIRATION_EPOCH"
) )
type PutLockInfoParams struct { type PutLockInfoParams struct {
@ -238,7 +238,7 @@ func (n *layer) attributesFromLock(ctx context.Context, lock *data.ObjectLock) (
} }
if lock.LegalHold != nil && lock.LegalHold.Enabled { if lock.LegalHold != nil && lock.LegalHold.Enabled {
// todo: (@KirillovDenis) reconsider this when FrostFS will support Legal Hold https://git.frostfs.info/TrueCloudLab/frostfs-contract/issues/2 // todo: (@KirillovDenis) reconsider this when FrostFS will support Legal Hold https://github.com/TrueCloudLab/frostfs-contract/issues/2
// Currently lock object must have an expiration epoch. // Currently lock object must have an expiration epoch.
// Besides we need to override retention expiration epoch since legal hold cannot be deleted yet. // Besides we need to override retention expiration epoch since legal hold cannot be deleted yet.
expEpoch = math.MaxUint64 expEpoch = math.MaxUint64
@ -246,7 +246,7 @@ func (n *layer) attributesFromLock(ctx context.Context, lock *data.ObjectLock) (
if expEpoch != 0 { if expEpoch != 0 {
result = append(result, [2]string{ result = append(result, [2]string{
object.SysAttributeExpEpoch, strconv.FormatUint(expEpoch, 10), AttributeExpirationEpoch, strconv.FormatUint(expEpoch, 10),
}) })
} }

View file

@ -4,12 +4,12 @@ import (
"context" "context"
errorsStd "errors" errorsStd "errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -6,8 +6,8 @@ import (
"sort" "sort"
"strings" "strings"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
) )
type TreeServiceMock struct { type TreeServiceMock struct {
@ -109,32 +109,11 @@ func (t *TreeServiceMock) PutNotificationConfigurationNode(ctx context.Context,
} }
func (t *TreeServiceMock) GetBucketCORS(ctx context.Context, bktInfo *data.BucketInfo) (oid.ID, error) { func (t *TreeServiceMock) GetBucketCORS(ctx context.Context, bktInfo *data.BucketInfo) (oid.ID, error) {
systemMap, ok := t.system[bktInfo.CID.EncodeToString()] panic("implement me")
if !ok {
return oid.ID{}, nil
}
node, ok := systemMap["cors"]
if !ok {
return oid.ID{}, nil
}
return node.OID, nil
} }
func (t *TreeServiceMock) PutBucketCORS(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID) (oid.ID, error) { func (t *TreeServiceMock) PutBucketCORS(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID) (oid.ID, error) {
systemMap, ok := t.system[bktInfo.CID.EncodeToString()] panic("implement me")
if !ok {
systemMap = make(map[string]*data.BaseNodeVersion)
}
systemMap["cors"] = &data.BaseNodeVersion{
OID: objID,
}
t.system[bktInfo.CID.EncodeToString()] = systemMap
return oid.ID{}, ErrNoNodeToRemove
} }
func (t *TreeServiceMock) DeleteBucketCORS(ctx context.Context, bktInfo *data.BucketInfo) (oid.ID, error) { func (t *TreeServiceMock) DeleteBucketCORS(ctx context.Context, bktInfo *data.BucketInfo) (oid.ID, error) {

View file

@ -4,8 +4,8 @@ import (
"context" "context"
"errors" "errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
) )
// TreeService provide interface to interact with tree service using s3 data models. // TreeService provide interface to interact with tree service using s3 data models.

View file

@ -9,11 +9,11 @@ import (
"strings" "strings"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "github.com/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "github.com/TrueCloudLab/frostfs-sdk-go/object"
) )
type ( type (

View file

@ -6,11 +6,11 @@ import (
"testing" "testing"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/checksum" "github.com/TrueCloudLab/frostfs-sdk-go/checksum"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )

View file

@ -4,7 +4,7 @@ import (
"context" "context"
"sort" "sort"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
) )
func (n *layer) ListObjectVersions(ctx context.Context, p *ListObjectVersionsParams) (*ListObjectVersionsInfo, error) { func (n *layer) ListObjectVersions(ctx context.Context, p *ListObjectVersionsParams) (*ListObjectVersionsInfo, error) {

View file

@ -5,13 +5,13 @@ import (
"context" "context"
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "github.com/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
bearertest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer/test" bearertest "github.com/TrueCloudLab/frostfs-sdk-go/bearer/test"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "github.com/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"go.uber.org/zap" "go.uber.org/zap"

View file

@ -4,7 +4,7 @@ import (
"net/http" "net/http"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
) )
type ( type (

View file

@ -1,7 +1,6 @@
package api package metrics
import ( import (
"context"
"io" "io"
"net/http" "net/http"
"strings" "strings"
@ -9,68 +8,9 @@ import (
"sync/atomic" "sync/atomic"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
"github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus"
) )
type RequestType int
const (
UNKNOWNRequest RequestType = iota
HEADRequest RequestType = iota
PUTRequest RequestType = iota
LISTRequest RequestType = iota
GETRequest RequestType = iota
DELETERequest RequestType = iota
)
func (t RequestType) String() string {
switch t {
case 1:
return "HEAD"
case 2:
return "PUT"
case 3:
return "LIST"
case 4:
return "GET"
case 5:
return "DELETE"
default:
return "Unknown"
}
}
func RequestTypeFromAPI(api string) RequestType {
switch api {
case "Options", "HeadObject", "HeadBucket":
return HEADRequest
case "CreateMultipartUpload", "UploadPartCopy", "UploadPart", "CompleteMultipartUpload",
"PutObjectACL", "PutObjectTagging", "CopyObject", "PutObjectRetention", "PutObjectLegalHold",
"PutObject", "PutBucketCors", "PutBucketACL", "PutBucketLifecycle", "PutBucketEncryption",
"PutBucketPolicy", "PutBucketObjectLockConfig", "PutBucketTagging", "PutBucketVersioning",
"PutBucketNotification", "CreateBucket", "PostObject":
return PUTRequest
case "ListObjectParts", "ListMultipartUploads", "ListObjectsV2M", "ListObjectsV2", "ListBucketVersions",
"ListObjectsV1", "ListBuckets":
return LISTRequest
case "GetObjectACL", "GetObjectTagging", "SelectObjectContent", "GetObjectRetention", "getobjectlegalhold",
"GetObjectAttributes", "GetObject", "GetBucketLocation", "GetBucketPolicy",
"GetBucketLifecycle", "GetBucketEncryption", "GetBucketCors", "GetBucketACL",
"GetBucketWebsite", "GetBucketAccelerate", "GetBucketRequestPayment", "GetBucketLogging",
"GetBucketReplication", "GetBucketTagging", "GetBucketObjectLockConfig",
"GetBucketVersioning", "GetBucketNotification", "ListenBucketNotification":
return GETRequest
case "AbortMultipartUpload", "DeleteObjectTagging", "DeleteObject", "DeleteBucketCors",
"DeleteBucketWebsite", "DeleteBucketTagging", "DeleteMultipleObjects", "DeleteBucketPolicy",
"DeleteBucketLifecycle", "DeleteBucketEncryption", "DeleteBucket":
return DELETERequest
default:
return UNKNOWNRequest
}
}
type ( type (
// HTTPAPIStats holds statistics information about // HTTPAPIStats holds statistics information about
// the API given in the requests. // the API given in the requests.
@ -79,10 +19,6 @@ type (
sync.RWMutex sync.RWMutex
} }
UsersStat interface {
Update(user, bucket, cnrID string, reqType RequestType, in, out uint64)
}
// HTTPStats holds statistics information about // HTTPStats holds statistics information about
// HTTP requests made by all clients. // HTTP requests made by all clients.
HTTPStats struct { HTTPStats struct {
@ -167,16 +103,11 @@ func collectHTTPMetrics(ch chan<- prometheus.Metric) {
} }
} }
// CIDResolveFunc is a func to resolve CID in Stats handler. // APIStats wraps http handler for api with basic statistics collection.
type CIDResolveFunc func(ctx context.Context, reqInfo *ReqInfo) (cnrID string) func APIStats(api string, f http.HandlerFunc) http.HandlerFunc {
// Stats is a handler that update metrics.
func Stats(f http.HandlerFunc, resolveCID CIDResolveFunc, usersStat UsersStat) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
reqInfo := GetReqInfo(r.Context()) httpStatsMetric.currentS3Requests.Inc(api)
defer httpStatsMetric.currentS3Requests.Dec(api)
httpStatsMetric.currentS3Requests.Inc(reqInfo.API)
defer httpStatsMetric.currentS3Requests.Dec(reqInfo.API)
in := &readCounter{ReadCloser: r.Body} in := &readCounter{ReadCloser: r.Body}
out := &writeCounter{ResponseWriter: w} out := &writeCounter{ResponseWriter: w}
@ -188,45 +119,20 @@ func Stats(f http.HandlerFunc, resolveCID CIDResolveFunc, usersStat UsersStat) h
startTime: time.Now(), startTime: time.Now(),
} }
f(statsWriter, r) f.ServeHTTP(statsWriter, r)
// Time duration in secs since the call started. // Time duration in secs since the call started.
// We don't need to do nanosecond precision here // We don't need to do nanosecond precision here
// simply for the fact that it is not human-readable. // simply for the fact that it is not human readable.
durationSecs := time.Since(statsWriter.startTime).Seconds() durationSecs := time.Since(statsWriter.startTime).Seconds()
user := resolveUser(r.Context()) httpStatsMetric.updateStats(api, statsWriter, r, durationSecs)
cnrID := resolveCID(r.Context(), reqInfo)
usersStat.Update(user, reqInfo.BucketName, cnrID, RequestTypeFromAPI(reqInfo.API), in.countBytes, out.countBytes)
code := statsWriter.statusCode
// A successful request has a 2xx response code
successReq := code >= http.StatusOK && code < http.StatusMultipleChoices
if !strings.HasSuffix(r.URL.Path, systemPath) {
httpStatsMetric.totalS3Requests.Inc(reqInfo.API)
if !successReq && code != 0 {
httpStatsMetric.totalS3Errors.Inc(reqInfo.API)
}
}
if r.Method == http.MethodGet {
// Increment the prometheus http request response histogram with appropriate label
httpRequestsDuration.With(prometheus.Labels{"api": reqInfo.API}).Observe(durationSecs)
}
atomic.AddUint64(&httpStatsMetric.totalInputBytes, in.countBytes) atomic.AddUint64(&httpStatsMetric.totalInputBytes, in.countBytes)
atomic.AddUint64(&httpStatsMetric.totalOutputBytes, out.countBytes) atomic.AddUint64(&httpStatsMetric.totalOutputBytes, out.countBytes)
} }
} }
func resolveUser(ctx context.Context) string {
user := "anon"
if bd, ok := ctx.Value(BoxData).(*accessbox.Box); ok && bd != nil && bd.Gate != nil && bd.Gate.BearerToken != nil {
user = bearer.ResolveIssuer(*bd.Gate.BearerToken).String()
}
return user
}
// Inc increments the api stats counter. // Inc increments the api stats counter.
func (stats *HTTPAPIStats) Inc(api string) { func (stats *HTTPAPIStats) Inc(api string) {
if stats == nil { if stats == nil {
@ -271,6 +177,30 @@ func (st *HTTPStats) getOutputBytes() uint64 {
return atomic.LoadUint64(&st.totalOutputBytes) return atomic.LoadUint64(&st.totalOutputBytes)
} }
// Update statistics from http request and response data.
func (st *HTTPStats) updateStats(api string, w http.ResponseWriter, r *http.Request, durationSecs float64) {
var code int
if res, ok := w.(*responseWrapper); ok {
code = res.statusCode
}
// A successful request has a 2xx response code
successReq := code >= http.StatusOK && code < http.StatusMultipleChoices
if !strings.HasSuffix(r.URL.Path, systemPath) {
st.totalS3Requests.Inc(api)
if !successReq && code != 0 {
st.totalS3Errors.Inc(api)
}
}
if r.Method == http.MethodGet {
// Increment the prometheus http request response histogram with appropriate label
httpRequestsDuration.With(prometheus.Labels{"api": api}).Observe(durationSecs)
}
}
// WriteHeader -- writes http status code. // WriteHeader -- writes http status code.
func (w *responseWrapper) WriteHeader(code int) { func (w *responseWrapper) WriteHeader(code int) {
w.Do(func() { w.Do(func() {
@ -286,12 +216,6 @@ func (w *responseWrapper) Flush() {
} }
} }
func (w *writeCounter) Flush() {
if f, ok := w.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
func (w *writeCounter) Write(p []byte) (int, error) { func (w *writeCounter) Write(p []byte) (int, error) {
n, err := w.ResponseWriter.Write(p) n, err := w.ResponseWriter.Write(p)
atomic.AddUint64(&w.countBytes, uint64(n)) atomic.AddUint64(&w.countBytes, uint64(n))

View file

@ -1,7 +1,7 @@
package api package metrics
import ( import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/version" "github.com/TrueCloudLab/frostfs-s3-gw/internal/version"
"github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus"
) )

View file

@ -7,8 +7,8 @@ import (
"sync" "sync"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/handler" "github.com/TrueCloudLab/frostfs-s3-gw/api/handler"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"github.com/nats-io/nats.go" "github.com/nats-io/nats.go"
"go.uber.org/zap" "go.uber.org/zap"
) )

View file

@ -6,9 +6,9 @@ import (
"fmt" "fmt"
"sync" "sync"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container" "github.com/TrueCloudLab/frostfs-sdk-go/container"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/ns" "github.com/TrueCloudLab/frostfs-sdk-go/ns"
) )
const ( const (

View file

@ -7,8 +7,8 @@ import (
"net/http" "net/http"
"strconv" "strconv"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/version" "github.com/TrueCloudLab/frostfs-s3-gw/internal/version"
"github.com/google/uuid" "github.com/google/uuid"
) )
@ -133,13 +133,6 @@ func WriteErrorResponse(w http.ResponseWriter, reqInfo *ReqInfo, err error) int
return code return code
} }
// WriteErrorResponseNoHeader writes XML encoded error to the response body.
func WriteErrorResponseNoHeader(w http.ResponseWriter, reqInfo *ReqInfo, err error) {
errorResponse := getAPIErrorResponse(reqInfo, err)
encodedErrorResponse := EncodeResponse(errorResponse)
WriteResponseBody(w, encodedErrorResponse)
}
// If none of the http routes match respond with appropriate errors. // If none of the http routes match respond with appropriate errors.
func errorResponseHandler(w http.ResponseWriter, r *http.Request) { func errorResponseHandler(w http.ResponseWriter, r *http.Request) {
desc := fmt.Sprintf("Unknown API request at %s", r.URL.Path) desc := fmt.Sprintf("Unknown API request at %s", r.URL.Path)
@ -179,11 +172,6 @@ func WriteResponse(w http.ResponseWriter, statusCode int, response []byte, mType
return return
} }
WriteResponseBody(w, response)
}
// WriteResponseBody writes response into w.
func WriteResponseBody(w http.ResponseWriter, response []byte) {
_, _ = w.Write(response) _, _ = w.Write(response)
if flusher, ok := w.(http.Flusher); ok { if flusher, ok := w.(http.Flusher); ok {
flusher.Flush() flusher.Flush()
@ -200,30 +188,13 @@ func EncodeResponse(response interface{}) []byte {
return bytesBuffer.Bytes() return bytesBuffer.Bytes()
} }
// EncodeResponseNoHeader encodes response without setting xml.Header.
// Should be used with periodicXMLWriter which sends xml.Header to the client
// with whitespaces to keep connection alive.
func EncodeResponseNoHeader(response interface{}) []byte {
var bytesBuffer bytes.Buffer
_ = xml.NewEncoder(&bytesBuffer).Encode(response)
return bytesBuffer.Bytes()
}
// EncodeToResponse encodes the response into ResponseWriter. // EncodeToResponse encodes the response into ResponseWriter.
func EncodeToResponse(w http.ResponseWriter, response interface{}) error { func EncodeToResponse(w http.ResponseWriter, response interface{}) error {
w.WriteHeader(http.StatusOK) w.WriteHeader(http.StatusOK)
if _, err := w.Write(xmlHeader); err != nil { if _, err := w.Write(xmlHeader); err != nil {
return fmt.Errorf("write headers: %w", err) return fmt.Errorf("write headers: %w", err)
} } else if err = xml.NewEncoder(w).Encode(response); err != nil {
return EncodeToResponseNoHeader(w, response)
}
// EncodeToResponseNoHeader encodes the response into ResponseWriter without
// header status.
func EncodeToResponseNoHeader(w http.ResponseWriter, response interface{}) error {
if err := xml.NewEncoder(w).Encode(response); err != nil {
return fmt.Errorf("encode xml response: %w", err) return fmt.Errorf("encode xml response: %w", err)
} }

View file

@ -5,8 +5,8 @@ import (
"net/http" "net/http"
"sync" "sync"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth" "github.com/TrueCloudLab/frostfs-s3-gw/api/auth"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "github.com/TrueCloudLab/frostfs-s3-gw/api/metrics"
"github.com/google/uuid" "github.com/google/uuid"
"github.com/gorilla/mux" "github.com/gorilla/mux"
"go.uber.org/zap" "go.uber.org/zap"
@ -82,8 +82,6 @@ type (
AbortMultipartUploadHandler(http.ResponseWriter, *http.Request) AbortMultipartUploadHandler(http.ResponseWriter, *http.Request)
ListPartsHandler(w http.ResponseWriter, r *http.Request) ListPartsHandler(w http.ResponseWriter, r *http.Request)
ListMultipartUploadsHandler(http.ResponseWriter, *http.Request) ListMultipartUploadsHandler(http.ResponseWriter, *http.Request)
ResolveBucket(ctx context.Context, bucket string) (*data.BucketInfo, error)
} }
// mimeType represents various MIME types used in API responses. // mimeType represents various MIME types used in API responses.
@ -108,7 +106,7 @@ const (
MimeXML mimeType = "application/xml" MimeXML mimeType = "application/xml"
) )
var _ = logSuccessResponse var _ = logErrorResponse
func (lrw *logResponseWriter) WriteHeader(code int) { func (lrw *logResponseWriter) WriteHeader(code int) {
lrw.Do(func() { lrw.Do(func() {
@ -117,12 +115,6 @@ func (lrw *logResponseWriter) WriteHeader(code int) {
}) })
} }
func (lrw *logResponseWriter) Flush() {
if f, ok := lrw.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
func setRequestID(h http.Handler) http.Handler { func setRequestID(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// generate random UUIDv4 // generate random UUIDv4
@ -153,37 +145,7 @@ func appendCORS(handler Handler) mux.MiddlewareFunc {
} }
} }
// BucketResolveFunc is a func to resolve bucket info by name. func logErrorResponse(l *zap.Logger) mux.MiddlewareFunc {
type BucketResolveFunc func(ctx context.Context, bucket string) (*data.BucketInfo, error)
// metricsMiddleware wraps http handler for api with basic statistics collection.
func metricsMiddleware(log *zap.Logger, resolveBucket BucketResolveFunc, usersStat UsersStat) mux.MiddlewareFunc {
return func(h http.Handler) http.Handler {
return Stats(h.ServeHTTP, resolveCID(log, resolveBucket), usersStat)
}
}
// resolveCID forms CIDResolveFunc using BucketResolveFunc.
func resolveCID(log *zap.Logger, resolveBucket BucketResolveFunc) CIDResolveFunc {
return func(ctx context.Context, reqInfo *ReqInfo) (cnrID string) {
if reqInfo.BucketName == "" || reqInfo.API == "CreateBucket" || reqInfo.API == "" {
return ""
}
bktInfo, err := resolveBucket(ctx, reqInfo.BucketName)
if err != nil {
log.Debug("failed to resolve CID",
zap.String("request_id", reqInfo.RequestID), zap.String("method", reqInfo.API),
zap.String("bucket", reqInfo.BucketName), zap.String("object", reqInfo.ObjectName),
zap.Error(err))
return ""
}
return bktInfo.CID.EncodeToString()
}
}
func logSuccessResponse(l *zap.Logger) mux.MiddlewareFunc {
return func(h http.Handler) http.Handler { return func(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
lw := &logResponseWriter{ResponseWriter: w} lw := &logResponseWriter{ResponseWriter: w}
@ -221,49 +183,21 @@ func GetRequestID(v interface{}) string {
} }
} }
func setErrorAPI(apiName string, h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ctx := SetReqInfo(r.Context(), &ReqInfo{API: apiName})
h.ServeHTTP(w, r.WithContext(ctx))
})
}
// attachErrorHandler set NotFoundHandler and MethodNotAllowedHandler for mux.Router.
func attachErrorHandler(api *mux.Router, log *zap.Logger, h Handler, center auth.Center, usersStat UsersStat) {
middlewares := []mux.MiddlewareFunc{
AuthMiddleware(log, center),
metricsMiddleware(log, h.ResolveBucket, usersStat),
}
var errorHandler http.Handler = http.HandlerFunc(errorResponseHandler)
for i := len(middlewares) - 1; i >= 0; i-- {
errorHandler = middlewares[i](errorHandler)
}
// If none of the routes match, add default error handler routes
api.NotFoundHandler = setErrorAPI("NotFound", errorHandler)
api.MethodNotAllowedHandler = setErrorAPI("MethodNotAllowed", errorHandler)
}
// Attach adds S3 API handlers from h to r for domains with m client limit using // Attach adds S3 API handlers from h to r for domains with m client limit using
// center authentication and log logger. // center authentication and log logger.
func Attach(r *mux.Router, domains []string, m MaxClients, h Handler, center auth.Center, log *zap.Logger, usersStat UsersStat) { func Attach(r *mux.Router, domains []string, m MaxClients, h Handler, center auth.Center, log *zap.Logger) {
api := r.PathPrefix(SlashSeparator).Subrouter() api := r.PathPrefix(SlashSeparator).Subrouter()
api.Use( api.Use(
// -- prepare request // -- prepare request
setRequestID, setRequestID,
// Attach user authentication for all S3 routes.
AuthMiddleware(log, center),
metricsMiddleware(log, h.ResolveBucket, usersStat),
// -- logging error requests // -- logging error requests
logSuccessResponse(log), logErrorResponse(log),
) )
attachErrorHandler(api, log, h, center, usersStat) // Attach user authentication for all S3 routes.
AttachUserAuth(api, center, log)
buckets := make([]*mux.Router, 0, len(domains)+1) buckets := make([]*mux.Router, 0, len(domains)+1)
buckets = append(buckets, api.PathPrefix("/{bucket}").Subrouter()) buckets = append(buckets, api.PathPrefix("/{bucket}").Subrouter())
@ -279,327 +213,277 @@ func Attach(r *mux.Router, domains []string, m MaxClients, h Handler, center aut
// -- append CORS headers to a response for // -- append CORS headers to a response for
appendCORS(h), appendCORS(h),
) )
bucket.Methods(http.MethodOptions).HandlerFunc( bucket.Methods(http.MethodOptions).HandlerFunc(m.Handle(metrics.APIStats("preflight", h.Preflight))).Name("Options")
m.Handle(h.Preflight)).
Name("Options")
bucket.Methods(http.MethodHead).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodHead).Path("/{object:.+}").HandlerFunc(
m.Handle(h.HeadObjectHandler)). m.Handle(metrics.APIStats("headobject", h.HeadObjectHandler))).Name("HeadObject")
Name("HeadObject")
// CopyObjectPart // CopyObjectPart
bucket.Methods(http.MethodPut).Path("/{object:.+}").Headers(hdrAmzCopySource, "").HandlerFunc( bucket.Methods(http.MethodPut).Path("/{object:.+}").Headers(hdrAmzCopySource, "").HandlerFunc(m.Handle(metrics.APIStats("uploadpartcopy", h.UploadPartCopy))).Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}").
m.Handle(h.UploadPartCopy)).
Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}").
Name("UploadPartCopy") Name("UploadPartCopy")
// PutObjectPart // PutObjectPart
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(
m.Handle(h.UploadPartHandler)). m.Handle(metrics.APIStats("uploadpart", h.UploadPartHandler))).Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}").
Queries("partNumber", "{partNumber:[0-9]+}", "uploadId", "{uploadId:.*}").
Name("UploadPart") Name("UploadPart")
// ListParts // ListParts
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(
m.Handle(h.ListPartsHandler)). m.Handle(metrics.APIStats("listobjectparts", h.ListPartsHandler))).Queries("uploadId", "{uploadId:.*}").
Queries("uploadId", "{uploadId:.*}").
Name("ListObjectParts") Name("ListObjectParts")
// CompleteMultipartUpload // CompleteMultipartUpload
bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc(
m.Handle(h.CompleteMultipartUploadHandler)). m.Handle(metrics.APIStats("completemutipartupload", h.CompleteMultipartUploadHandler))).Queries("uploadId", "{uploadId:.*}").
Queries("uploadId", "{uploadId:.*}").
Name("CompleteMultipartUpload") Name("CompleteMultipartUpload")
// CreateMultipartUpload // CreateMultipartUpload
bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc(
m.Handle(h.CreateMultipartUploadHandler)). m.Handle(metrics.APIStats("createmultipartupload", h.CreateMultipartUploadHandler))).Queries("uploads", "").
Queries("uploads", "").
Name("CreateMultipartUpload") Name("CreateMultipartUpload")
// AbortMultipartUpload // AbortMultipartUpload
bucket.Methods(http.MethodDelete).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodDelete).Path("/{object:.+}").HandlerFunc(
m.Handle(h.AbortMultipartUploadHandler)). m.Handle(metrics.APIStats("abortmultipartupload", h.AbortMultipartUploadHandler))).Queries("uploadId", "{uploadId:.*}").
Queries("uploadId", "{uploadId:.*}").
Name("AbortMultipartUpload") Name("AbortMultipartUpload")
// ListMultipartUploads // ListMultipartUploads
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.ListMultipartUploadsHandler)). m.Handle(metrics.APIStats("listmultipartuploads", h.ListMultipartUploadsHandler))).Queries("uploads", "").
Queries("uploads", "").
Name("ListMultipartUploads") Name("ListMultipartUploads")
// GetObjectACL -- this is a dummy call. // GetObjectACL -- this is a dummy call.
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(
m.Handle(h.GetObjectACLHandler)). m.Handle(metrics.APIStats("getobjectacl", h.GetObjectACLHandler))).Queries("acl", "").
Queries("acl", "").
Name("GetObjectACL") Name("GetObjectACL")
// PutObjectACL -- this is a dummy call. // PutObjectACL -- this is a dummy call.
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(
m.Handle(h.PutObjectACLHandler)). m.Handle(metrics.APIStats("putobjectacl", h.PutObjectACLHandler))).Queries("acl", "").
Queries("acl", "").
Name("PutObjectACL") Name("PutObjectACL")
// GetObjectTagging // GetObjectTagging
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(
m.Handle(h.GetObjectTaggingHandler)). m.Handle(metrics.APIStats("getobjecttagging", h.GetObjectTaggingHandler))).Queries("tagging", "").
Queries("tagging", "").
Name("GetObjectTagging") Name("GetObjectTagging")
// PutObjectTagging // PutObjectTagging
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(
m.Handle(h.PutObjectTaggingHandler)). m.Handle(metrics.APIStats("putobjecttagging", h.PutObjectTaggingHandler))).Queries("tagging", "").
Queries("tagging", "").
Name("PutObjectTagging") Name("PutObjectTagging")
// DeleteObjectTagging // DeleteObjectTagging
bucket.Methods(http.MethodDelete).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodDelete).Path("/{object:.+}").HandlerFunc(
m.Handle(h.DeleteObjectTaggingHandler)). m.Handle(metrics.APIStats("deleteobjecttagging", h.DeleteObjectTaggingHandler))).Queries("tagging", "").
Queries("tagging", "").
Name("DeleteObjectTagging") Name("DeleteObjectTagging")
// SelectObjectContent // SelectObjectContent
bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodPost).Path("/{object:.+}").HandlerFunc(
m.Handle(h.SelectObjectContentHandler)). m.Handle(metrics.APIStats("selectobjectcontent", h.SelectObjectContentHandler))).Queries("select", "").Queries("select-type", "2").
Queries("select", "").Queries("select-type", "2").
Name("SelectObjectContent") Name("SelectObjectContent")
// GetObjectRetention // GetObjectRetention
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(
m.Handle(h.GetObjectRetentionHandler)). m.Handle(metrics.APIStats("getobjectretention", h.GetObjectRetentionHandler))).Queries("retention", "").
Queries("retention", "").
Name("GetObjectRetention") Name("GetObjectRetention")
// GetObjectLegalHold // GetObjectLegalHold
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(
m.Handle(h.GetObjectLegalHoldHandler)). m.Handle(metrics.APIStats("getobjectlegalhold", h.GetObjectLegalHoldHandler))).Queries("legal-hold", "").
Queries("legal-hold", "").
Name("GetObjectLegalHold") Name("GetObjectLegalHold")
// GetObjectAttributes // GetObjectAttributes
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(
m.Handle(h.GetObjectAttributesHandler)). m.Handle(metrics.APIStats("getobjectattributes", h.GetObjectAttributesHandler))).Queries("attributes", "").
Queries("attributes", "").
Name("GetObjectAttributes") Name("GetObjectAttributes")
// GetObject // GetObject
bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodGet).Path("/{object:.+}").HandlerFunc(
m.Handle(h.GetObjectHandler)). m.Handle(metrics.APIStats("getobject", h.GetObjectHandler))).
Name("GetObject") Name("GetObject")
// CopyObject // CopyObject
bucket.Methods(http.MethodPut).Path("/{object:.+}").Headers(hdrAmzCopySource, "").HandlerFunc( bucket.Methods(http.MethodPut).Path("/{object:.+}").Headers(hdrAmzCopySource, "").HandlerFunc(m.Handle(metrics.APIStats("copyobject", h.CopyObjectHandler))).
m.Handle(h.CopyObjectHandler)).
Name("CopyObject") Name("CopyObject")
// PutObjectRetention // PutObjectRetention
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(
m.Handle(h.PutObjectRetentionHandler)). m.Handle(metrics.APIStats("putobjectretention", h.PutObjectRetentionHandler))).Queries("retention", "").
Queries("retention", "").
Name("PutObjectRetention") Name("PutObjectRetention")
// PutObjectLegalHold // PutObjectLegalHold
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(
m.Handle(h.PutObjectLegalHoldHandler)). m.Handle(metrics.APIStats("putobjectlegalhold", h.PutObjectLegalHoldHandler))).Queries("legal-hold", "").
Queries("legal-hold", "").
Name("PutObjectLegalHold") Name("PutObjectLegalHold")
// PutObject // PutObject
bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodPut).Path("/{object:.+}").HandlerFunc(
m.Handle(h.PutObjectHandler)). m.Handle(metrics.APIStats("putobject", h.PutObjectHandler))).
Name("PutObject") Name("PutObject")
// DeleteObject // DeleteObject
bucket.Methods(http.MethodDelete).Path("/{object:.+}").HandlerFunc( bucket.Methods(http.MethodDelete).Path("/{object:.+}").HandlerFunc(
m.Handle(h.DeleteObjectHandler)). m.Handle(metrics.APIStats("deleteobject", h.DeleteObjectHandler))).
Name("DeleteObject") Name("DeleteObject")
// Bucket operations // Bucket operations
// GetBucketLocation // GetBucketLocation
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketLocationHandler)). m.Handle(metrics.APIStats("getbucketlocation", h.GetBucketLocationHandler))).Queries("location", "").
Queries("location", "").
Name("GetBucketLocation") Name("GetBucketLocation")
// GetBucketPolicy // GetBucketPolicy
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketPolicyHandler)). m.Handle(metrics.APIStats("getbucketpolicy", h.GetBucketPolicyHandler))).Queries("policy", "").
Queries("policy", "").
Name("GetBucketPolicy") Name("GetBucketPolicy")
// GetBucketLifecycle // GetBucketLifecycle
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketLifecycleHandler)). m.Handle(metrics.APIStats("getbucketlifecycle", h.GetBucketLifecycleHandler))).Queries("lifecycle", "").
Queries("lifecycle", "").
Name("GetBucketLifecycle") Name("GetBucketLifecycle")
// GetBucketEncryption // GetBucketEncryption
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketEncryptionHandler)). m.Handle(metrics.APIStats("getbucketencryption", h.GetBucketEncryptionHandler))).Queries("encryption", "").
Queries("encryption", "").
Name("GetBucketEncryption") Name("GetBucketEncryption")
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketCorsHandler)). m.Handle(metrics.APIStats("getbucketcors", h.GetBucketCorsHandler))).Queries("cors", "").
Queries("cors", "").
Name("GetBucketCors") Name("GetBucketCors")
bucket.Methods(http.MethodPut).HandlerFunc( bucket.Methods(http.MethodPut).HandlerFunc(
m.Handle(h.PutBucketCorsHandler)). m.Handle(metrics.APIStats("putbucketcors", h.PutBucketCorsHandler))).Queries("cors", "").
Queries("cors", "").
Name("PutBucketCors") Name("PutBucketCors")
bucket.Methods(http.MethodDelete).HandlerFunc( bucket.Methods(http.MethodDelete).HandlerFunc(
m.Handle(h.DeleteBucketCorsHandler)). m.Handle(metrics.APIStats("deletebucketcors", h.DeleteBucketCorsHandler))).Queries("cors", "").
Queries("cors", "").
Name("DeleteBucketCors") Name("DeleteBucketCors")
// Dummy Bucket Calls // Dummy Bucket Calls
// GetBucketACL -- this is a dummy call. // GetBucketACL -- this is a dummy call.
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketACLHandler)). m.Handle(metrics.APIStats("getbucketacl", h.GetBucketACLHandler))).Queries("acl", "").
Queries("acl", "").
Name("GetBucketACL") Name("GetBucketACL")
// PutBucketACL -- this is a dummy call. // PutBucketACL -- this is a dummy call.
bucket.Methods(http.MethodPut).HandlerFunc( bucket.Methods(http.MethodPut).HandlerFunc(
m.Handle(h.PutBucketACLHandler)). m.Handle(metrics.APIStats("putbucketacl", h.PutBucketACLHandler))).Queries("acl", "").
Queries("acl", "").
Name("PutBucketACL") Name("PutBucketACL")
// GetBucketWebsiteHandler -- this is a dummy call. // GetBucketWebsiteHandler -- this is a dummy call.
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketWebsiteHandler)). m.Handle(metrics.APIStats("getbucketwebsite", h.GetBucketWebsiteHandler))).Queries("website", "").
Queries("website", "").
Name("GetBucketWebsite") Name("GetBucketWebsite")
// GetBucketAccelerateHandler -- this is a dummy call. // GetBucketAccelerateHandler -- this is a dummy call.
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketAccelerateHandler)). m.Handle(metrics.APIStats("getbucketaccelerate", h.GetBucketAccelerateHandler))).Queries("accelerate", "").
Queries("accelerate", "").
Name("GetBucketAccelerate") Name("GetBucketAccelerate")
// GetBucketRequestPaymentHandler -- this is a dummy call. // GetBucketRequestPaymentHandler -- this is a dummy call.
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketRequestPaymentHandler)). m.Handle(metrics.APIStats("getbucketrequestpayment", h.GetBucketRequestPaymentHandler))).Queries("requestPayment", "").
Queries("requestPayment", "").
Name("GetBucketRequestPayment") Name("GetBucketRequestPayment")
// GetBucketLoggingHandler -- this is a dummy call. // GetBucketLoggingHandler -- this is a dummy call.
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketLoggingHandler)). m.Handle(metrics.APIStats("getbucketlogging", h.GetBucketLoggingHandler))).Queries("logging", "").
Queries("logging", "").
Name("GetBucketLogging") Name("GetBucketLogging")
// GetBucketLifecycleHandler -- this is a dummy call.
bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(metrics.APIStats("getbucketlifecycle", h.GetBucketLifecycleHandler))).Queries("lifecycle", "").
Name("GetBucketLifecycle")
// GetBucketReplicationHandler -- this is a dummy call. // GetBucketReplicationHandler -- this is a dummy call.
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketReplicationHandler)). m.Handle(metrics.APIStats("getbucketreplication", h.GetBucketReplicationHandler))).Queries("replication", "").
Queries("replication", "").
Name("GetBucketReplication") Name("GetBucketReplication")
// GetBucketTaggingHandler // GetBucketTaggingHandler
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketTaggingHandler)). m.Handle(metrics.APIStats("getbuckettagging", h.GetBucketTaggingHandler))).Queries("tagging", "").
Queries("tagging", "").
Name("GetBucketTagging") Name("GetBucketTagging")
// DeleteBucketWebsiteHandler // DeleteBucketWebsiteHandler
bucket.Methods(http.MethodDelete).HandlerFunc( bucket.Methods(http.MethodDelete).HandlerFunc(
m.Handle(h.DeleteBucketWebsiteHandler)). m.Handle(metrics.APIStats("deletebucketwebsite", h.DeleteBucketWebsiteHandler))).Queries("website", "").
Queries("website", "").
Name("DeleteBucketWebsite") Name("DeleteBucketWebsite")
// DeleteBucketTaggingHandler // DeleteBucketTaggingHandler
bucket.Methods(http.MethodDelete).HandlerFunc( bucket.Methods(http.MethodDelete).HandlerFunc(
m.Handle(h.DeleteBucketTaggingHandler)). m.Handle(metrics.APIStats("deletebuckettagging", h.DeleteBucketTaggingHandler))).Queries("tagging", "").
Queries("tagging", "").
Name("DeleteBucketTagging") Name("DeleteBucketTagging")
// GetBucketObjectLockConfig // GetBucketObjectLockConfig
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketObjectLockConfigHandler)). m.Handle(metrics.APIStats("getbucketobjectlockconfiguration", h.GetBucketObjectLockConfigHandler))).Queries("object-lock", "").
Queries("object-lock", "").
Name("GetBucketObjectLockConfig") Name("GetBucketObjectLockConfig")
// GetBucketVersioning // GetBucketVersioning
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketVersioningHandler)). m.Handle(metrics.APIStats("getbucketversioning", h.GetBucketVersioningHandler))).Queries("versioning", "").
Queries("versioning", "").
Name("GetBucketVersioning") Name("GetBucketVersioning")
// GetBucketNotification // GetBucketNotification
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.GetBucketNotificationHandler)). m.Handle(metrics.APIStats("getbucketnotification", h.GetBucketNotificationHandler))).Queries("notification", "").
Queries("notification", "").
Name("GetBucketNotification") Name("GetBucketNotification")
// ListenBucketNotification // ListenBucketNotification
bucket.Methods(http.MethodGet).HandlerFunc(h.ListenBucketNotificationHandler). bucket.Methods(http.MethodGet).HandlerFunc(metrics.APIStats("listenbucketnotification", h.ListenBucketNotificationHandler)).Queries("events", "{events:.*}").
Queries("events", "{events:.*}").
Name("ListenBucketNotification") Name("ListenBucketNotification")
// ListObjectsV2M // ListObjectsV2M
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.ListObjectsV2MHandler)). m.Handle(metrics.APIStats("listobjectsv2M", h.ListObjectsV2MHandler))).Queries("list-type", "2", "metadata", "true").
Queries("list-type", "2", "metadata", "true").
Name("ListObjectsV2M") Name("ListObjectsV2M")
// ListObjectsV2 // ListObjectsV2
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.ListObjectsV2Handler)). m.Handle(metrics.APIStats("listobjectsv2", h.ListObjectsV2Handler))).Queries("list-type", "2").
Queries("list-type", "2").
Name("ListObjectsV2") Name("ListObjectsV2")
// ListBucketVersions // ListBucketVersions
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.ListBucketObjectVersionsHandler)). m.Handle(metrics.APIStats("listbucketversions", h.ListBucketObjectVersionsHandler))).Queries("versions", "").
Queries("versions", "").
Name("ListBucketVersions") Name("ListBucketVersions")
// ListObjectsV1 (Legacy) // ListObjectsV1 (Legacy)
bucket.Methods(http.MethodGet).HandlerFunc( bucket.Methods(http.MethodGet).HandlerFunc(
m.Handle(h.ListObjectsV1Handler)). m.Handle(metrics.APIStats("listobjectsv1", h.ListObjectsV1Handler))).
Name("ListObjectsV1") Name("ListObjectsV1")
// PutBucketLifecycle // PutBucketLifecycle
bucket.Methods(http.MethodPut).HandlerFunc( bucket.Methods(http.MethodPut).HandlerFunc(
m.Handle(h.PutBucketLifecycleHandler)). m.Handle(metrics.APIStats("putbucketlifecycle", h.PutBucketLifecycleHandler))).Queries("lifecycle", "").
Queries("lifecycle", "").
Name("PutBucketLifecycle") Name("PutBucketLifecycle")
// PutBucketEncryption // PutBucketEncryption
bucket.Methods(http.MethodPut).HandlerFunc( bucket.Methods(http.MethodPut).HandlerFunc(
m.Handle(h.PutBucketEncryptionHandler)). m.Handle(metrics.APIStats("putbucketencryption", h.PutBucketEncryptionHandler))).Queries("encryption", "").
Queries("encryption", "").
Name("PutBucketEncryption") Name("PutBucketEncryption")
// PutBucketPolicy // PutBucketPolicy
bucket.Methods(http.MethodPut).HandlerFunc( bucket.Methods(http.MethodPut).HandlerFunc(
m.Handle(h.PutBucketPolicyHandler)). m.Handle(metrics.APIStats("putbucketpolicy", h.PutBucketPolicyHandler))).Queries("policy", "").
Queries("policy", "").
Name("PutBucketPolicy") Name("PutBucketPolicy")
// PutBucketObjectLockConfig // PutBucketObjectLockConfig
bucket.Methods(http.MethodPut).HandlerFunc( bucket.Methods(http.MethodPut).HandlerFunc(
m.Handle(h.PutBucketObjectLockConfigHandler)). m.Handle(metrics.APIStats("putbucketobjectlockconfig", h.PutBucketObjectLockConfigHandler))).Queries("object-lock", "").
Queries("object-lock", "").
Name("PutBucketObjectLockConfig") Name("PutBucketObjectLockConfig")
// PutBucketTaggingHandler // PutBucketTaggingHandler
bucket.Methods(http.MethodPut).HandlerFunc( bucket.Methods(http.MethodPut).HandlerFunc(
m.Handle(h.PutBucketTaggingHandler)). m.Handle(metrics.APIStats("putbuckettagging", h.PutBucketTaggingHandler))).Queries("tagging", "").
Queries("tagging", "").
Name("PutBucketTagging") Name("PutBucketTagging")
// PutBucketVersioning // PutBucketVersioning
bucket.Methods(http.MethodPut).HandlerFunc( bucket.Methods(http.MethodPut).HandlerFunc(
m.Handle(h.PutBucketVersioningHandler)). m.Handle(metrics.APIStats("putbucketversioning", h.PutBucketVersioningHandler))).Queries("versioning", "").
Queries("versioning", "").
Name("PutBucketVersioning") Name("PutBucketVersioning")
// PutBucketNotification // PutBucketNotification
bucket.Methods(http.MethodPut).HandlerFunc( bucket.Methods(http.MethodPut).HandlerFunc(
m.Handle(h.PutBucketNotificationHandler)). m.Handle(metrics.APIStats("putbucketnotification", h.PutBucketNotificationHandler))).Queries("notification", "").
Queries("notification", "").
Name("PutBucketNotification") Name("PutBucketNotification")
// CreateBucket // CreateBucket
bucket.Methods(http.MethodPut).HandlerFunc( bucket.Methods(http.MethodPut).HandlerFunc(
m.Handle(h.CreateBucketHandler)). m.Handle(metrics.APIStats("createbucket", h.CreateBucketHandler))).
Name("CreateBucket") Name("CreateBucket")
// HeadBucket // HeadBucket
bucket.Methods(http.MethodHead).HandlerFunc( bucket.Methods(http.MethodHead).HandlerFunc(
m.Handle(h.HeadBucketHandler)). m.Handle(metrics.APIStats("headbucket", h.HeadBucketHandler))).
Name("HeadBucket") Name("HeadBucket")
// PostPolicy // PostPolicy
bucket.Methods(http.MethodPost).HeadersRegexp(hdrContentType, "multipart/form-data*").HandlerFunc( bucket.Methods(http.MethodPost).HeadersRegexp(hdrContentType, "multipart/form-data*").HandlerFunc(
m.Handle(h.PostObject)). m.Handle(metrics.APIStats("postobject", h.PostObject))).
Name("PostObject") Name("PostObject")
// DeleteMultipleObjects // DeleteMultipleObjects
bucket.Methods(http.MethodPost).HandlerFunc( bucket.Methods(http.MethodPost).HandlerFunc(
m.Handle(h.DeleteMultipleObjectsHandler)). m.Handle(metrics.APIStats("deletemultipleobjects", h.DeleteMultipleObjectsHandler))).Queries("delete", "").
Queries("delete", "").
Name("DeleteMultipleObjects") Name("DeleteMultipleObjects")
// DeleteBucketPolicy // DeleteBucketPolicy
bucket.Methods(http.MethodDelete).HandlerFunc( bucket.Methods(http.MethodDelete).HandlerFunc(
m.Handle(h.DeleteBucketPolicyHandler)). m.Handle(metrics.APIStats("deletebucketpolicy", h.DeleteBucketPolicyHandler))).Queries("policy", "").
Queries("policy", "").
Name("DeleteBucketPolicy") Name("DeleteBucketPolicy")
// DeleteBucketLifecycle // DeleteBucketLifecycle
bucket.Methods(http.MethodDelete).HandlerFunc( bucket.Methods(http.MethodDelete).HandlerFunc(
m.Handle(h.DeleteBucketLifecycleHandler)). m.Handle(metrics.APIStats("deletebucketlifecycle", h.DeleteBucketLifecycleHandler))).Queries("lifecycle", "").
Queries("lifecycle", "").
Name("DeleteBucketLifecycle") Name("DeleteBucketLifecycle")
// DeleteBucketEncryption // DeleteBucketEncryption
bucket.Methods(http.MethodDelete).HandlerFunc( bucket.Methods(http.MethodDelete).HandlerFunc(
m.Handle(h.DeleteBucketEncryptionHandler)). m.Handle(metrics.APIStats("deletebucketencryption", h.DeleteBucketEncryptionHandler))).Queries("encryption", "").
Queries("encryption", "").
Name("DeleteBucketEncryption") Name("DeleteBucketEncryption")
// DeleteBucket // DeleteBucket
bucket.Methods(http.MethodDelete).HandlerFunc( bucket.Methods(http.MethodDelete).HandlerFunc(
m.Handle(h.DeleteBucketHandler)). m.Handle(metrics.APIStats("deletebucket", h.DeleteBucketHandler))).
Name("DeleteBucket") Name("DeleteBucket")
} }
// Root operation // Root operation
// ListBuckets // ListBuckets
api.Methods(http.MethodGet).Path(SlashSeparator).HandlerFunc( api.Methods(http.MethodGet).Path(SlashSeparator).HandlerFunc(
m.Handle(h.ListBucketsHandler)). m.Handle(metrics.APIStats("listbuckets", h.ListBucketsHandler))).
Name("ListBuckets") Name("ListBuckets")
// S3 browser with signature v4 adds '//' for ListBuckets request, so rather // S3 browser with signature v4 adds '//' for ListBuckets request, so rather
// than failing with UnknownAPIRequest we simply handle it for now. // than failing with UnknownAPIRequest we simply handle it for now.
api.Methods(http.MethodGet).Path(SlashSeparator + SlashSeparator).HandlerFunc( api.Methods(http.MethodGet).Path(SlashSeparator + SlashSeparator).HandlerFunc(
m.Handle(h.ListBucketsHandler)). m.Handle(metrics.APIStats("listbuckets", h.ListBucketsHandler))).
Name("ListBuckets") Name("ListBuckets")
// If none of the routes match, add default error handler routes
api.NotFoundHandler = metrics.APIStats("notfound", errorResponseHandler)
api.MethodNotAllowedHandler = metrics.APIStats("methodnotallowed", errorResponseHandler)
} }

View file

@ -4,8 +4,8 @@ import (
"context" "context"
"net/http" "net/http"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth" "github.com/TrueCloudLab/frostfs-s3-gw/api/auth"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "github.com/TrueCloudLab/frostfs-s3-gw/api/errors"
"github.com/gorilla/mux" "github.com/gorilla/mux"
"go.uber.org/zap" "go.uber.org/zap"
) )
@ -19,9 +19,9 @@ var BoxData = KeyWrapper("__context_box_key")
// ClientTime is an ID used to store client time.Time in a context. // ClientTime is an ID used to store client time.Time in a context.
var ClientTime = KeyWrapper("__context_client_time") var ClientTime = KeyWrapper("__context_client_time")
// AuthMiddleware adds user authentication via center to router using log for logging. // AttachUserAuth adds user authentication via center to router using log for logging.
func AuthMiddleware(log *zap.Logger, center auth.Center) mux.MiddlewareFunc { func AttachUserAuth(router *mux.Router, center auth.Center, log *zap.Logger) {
return func(h http.Handler) http.Handler { router.Use(func(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var ctx context.Context var ctx context.Context
box, err := center.Authenticate(r) box, err := center.Authenticate(r)
@ -46,5 +46,5 @@ func AuthMiddleware(log *zap.Logger, center auth.Center) mux.MiddlewareFunc {
h.ServeHTTP(w, r.WithContext(ctx)) h.ServeHTTP(w, r.WithContext(ctx))
}) })
} })
} }

View file

@ -11,17 +11,17 @@ import (
"os" "os"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/cache" "github.com/TrueCloudLab/frostfs-s3-gw/api/cache"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "github.com/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/tokens" "github.com/TrueCloudLab/frostfs-s3-gw/creds/tokens"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "github.com/TrueCloudLab/frostfs-sdk-go/bearer"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
frostfsecdsa "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/crypto/ecdsa" frostfsecdsa "github.com/TrueCloudLab/frostfs-sdk-go/crypto/ecdsa"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/eacl" "github.com/TrueCloudLab/frostfs-sdk-go/eacl"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" "github.com/TrueCloudLab/frostfs-sdk-go/netmap"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "github.com/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "github.com/TrueCloudLab/frostfs-sdk-go/session"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "github.com/TrueCloudLab/frostfs-sdk-go/user"
"github.com/google/uuid" "github.com/google/uuid"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"go.uber.org/zap" "go.uber.org/zap"

View file

@ -4,9 +4,9 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
apisession "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/session" apisession "github.com/TrueCloudLab/frostfs-api-go/v2/session"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "github.com/TrueCloudLab/frostfs-sdk-go/session"
) )
type ( type (

View file

@ -3,7 +3,7 @@ package authmate
import ( import (
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "github.com/TrueCloudLab/frostfs-sdk-go/session"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )

View file

@ -13,13 +13,13 @@ import (
"syscall" "syscall"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/authmate" "github.com/TrueCloudLab/frostfs-s3-gw/authmate"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs" "github.com/TrueCloudLab/frostfs-s3-gw/internal/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/version" "github.com/TrueCloudLab/frostfs-s3-gw/internal/version"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/wallet" "github.com/TrueCloudLab/frostfs-s3-gw/internal/wallet"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "github.com/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/pool" "github.com/TrueCloudLab/frostfs-sdk-go/pool"
"github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials" "github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/aws/session"

View file

@ -13,20 +13,18 @@ import (
"syscall" "syscall"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "github.com/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth" "github.com/TrueCloudLab/frostfs-s3-gw/api/auth"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/cache" "github.com/TrueCloudLab/frostfs-s3-gw/api/cache"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/handler" "github.com/TrueCloudLab/frostfs-s3-gw/api/handler"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "github.com/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/notifications" "github.com/TrueCloudLab/frostfs-s3-gw/api/notifications"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/resolver" "github.com/TrueCloudLab/frostfs-s3-gw/api/resolver"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs" "github.com/TrueCloudLab/frostfs-s3-gw/internal/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/version" "github.com/TrueCloudLab/frostfs-s3-gw/internal/version"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/wallet" "github.com/TrueCloudLab/frostfs-s3-gw/internal/wallet"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/xml" "github.com/TrueCloudLab/frostfs-sdk-go/netmap"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/metrics" "github.com/TrueCloudLab/frostfs-sdk-go/pool"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/pool"
"github.com/gorilla/mux" "github.com/gorilla/mux"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/spf13/viper" "github.com/spf13/viper"
@ -47,7 +45,7 @@ type (
servers []Server servers []Server
metrics *metrics.AppMetrics metrics *appMetrics
bucketResolver *resolver.BucketResolver bucketResolver *resolver.BucketResolver
services []*Service services []*Service
settings *appSettings settings *appSettings
@ -58,9 +56,8 @@ type (
} }
appSettings struct { appSettings struct {
logLevel zap.AtomicLevel logLevel zap.AtomicLevel
policies *placementPolicy policies *placementPolicy
xmlDecoder *xml.DecoderProvider
} }
Logger struct { Logger struct {
@ -68,6 +65,18 @@ type (
lvl zap.AtomicLevel lvl zap.AtomicLevel
} }
appMetrics struct {
logger *zap.Logger
provider GateMetricsCollector
mu sync.RWMutex
enabled bool
}
GateMetricsCollector interface {
SetHealth(int32)
Unregister()
}
placementPolicy struct { placementPolicy struct {
mu sync.RWMutex mu sync.RWMutex
defaultPolicy netmap.PlacementPolicy defaultPolicy netmap.PlacementPolicy
@ -154,9 +163,8 @@ func newAppSettings(log *Logger, v *viper.Viper) *appSettings {
} }
return &appSettings{ return &appSettings{
logLevel: log.lvl, logLevel: log.lvl,
policies: policies, policies: policies,
xmlDecoder: xml.NewDecoderProvider(v.GetBool(cfgKludgeUseDefaultXMLNSForCompleteMultipartUpload)),
} }
} }
@ -175,7 +183,8 @@ func (a *App) initAPI(ctx context.Context) {
} }
func (a *App) initMetrics() { func (a *App) initMetrics() {
a.metrics = metrics.NewAppMetrics(a.log, frostfs.NewPoolStatistic(a.pool), a.cfg.GetBool(cfgPrometheusEnabled)) gateMetricsProvider := newGateMetrics(frostfs.NewPoolStatistic(a.pool))
a.metrics = newAppMetrics(a.log, gateMetricsProvider, a.cfg.GetBool(cfgPrometheusEnabled))
} }
func (a *App) initResolver() { func (a *App) initResolver() {
@ -264,7 +273,6 @@ func getPool(ctx context.Context, logger *zap.Logger, cfg *viper.Viper) (*pool.P
errorThreshold = defaultPoolErrorThreshold errorThreshold = defaultPoolErrorThreshold
} }
prm.SetErrorThreshold(errorThreshold) prm.SetErrorThreshold(errorThreshold)
prm.SetLogger(logger)
p, err := pool.NewPool(prm) p, err := pool.NewPool(prm)
if err != nil { if err != nil {
@ -335,6 +343,47 @@ func (p *placementPolicy) update(defaultPolicy string, regionPolicyFilepath stri
return nil return nil
} }
func newAppMetrics(logger *zap.Logger, provider GateMetricsCollector, enabled bool) *appMetrics {
if !enabled {
logger.Warn("metrics are disabled")
}
return &appMetrics{
logger: logger,
provider: provider,
}
}
func (m *appMetrics) SetEnabled(enabled bool) {
if !enabled {
m.logger.Warn("metrics are disabled")
}
m.mu.Lock()
m.enabled = enabled
m.mu.Unlock()
}
func (m *appMetrics) SetHealth(status int32) {
m.mu.RLock()
if !m.enabled {
m.mu.RUnlock()
return
}
m.mu.RUnlock()
m.provider.SetHealth(status)
}
func (m *appMetrics) Shutdown() {
m.mu.Lock()
if m.enabled {
m.provider.SetHealth(0)
m.enabled = false
}
m.provider.Unregister()
m.mu.Unlock()
}
func remove(list []string, element string) []string { func remove(list []string, element string) []string {
for i, item := range list { for i, item := range list {
if item == element { if item == element {
@ -372,7 +421,7 @@ func (a *App) Serve(ctx context.Context) {
domains := a.cfg.GetStringSlice(cfgListenDomains) domains := a.cfg.GetStringSlice(cfgListenDomains)
a.log.Info("fetch domains, prepare to use API", zap.Strings("domains", domains)) a.log.Info("fetch domains, prepare to use API", zap.Strings("domains", domains))
router := mux.NewRouter().SkipClean(true).UseEncodedPath() router := mux.NewRouter().SkipClean(true).UseEncodedPath()
api.Attach(router, domains, a.maxClients, a.api, a.ctr, a.log, a.metrics) api.Attach(router, domains, a.maxClients, a.api, a.ctr, a.log)
// Use mux.Router as http.Handler // Use mux.Router as http.Handler
srv := new(http.Server) srv := new(http.Server)
@ -422,11 +471,11 @@ func shutdownContext() (context.Context, context.CancelFunc) {
func (a *App) configReload() { func (a *App) configReload() {
a.log.Info("SIGHUP config reload started") a.log.Info("SIGHUP config reload started")
if !a.cfg.IsSet(cmdConfig) && !a.cfg.IsSet(cmdConfigDir) { if !a.cfg.IsSet(cmdConfig) {
a.log.Warn("failed to reload config because it's missed") a.log.Warn("failed to reload config because it's missed")
return return
} }
if err := readInConfig(a.cfg); err != nil { if err := readConfig(a.cfg); err != nil {
a.log.Warn("failed to reload config", zap.Error(err)) a.log.Warn("failed to reload config", zap.Error(err))
return return
} }
@ -460,8 +509,6 @@ func (a *App) updateSettings() {
if err := a.settings.policies.update(getDefaultPolicyValue(a.cfg), a.cfg.GetString(cfgPolicyRegionMapFile)); err != nil { if err := a.settings.policies.update(getDefaultPolicyValue(a.cfg), a.cfg.GetString(cfgPolicyRegionMapFile)); err != nil {
a.log.Warn("policies won't be updated", zap.Error(err)) a.log.Warn("policies won't be updated", zap.Error(err))
} }
a.settings.xmlDecoder.UseDefaultNamespaceForCompleteMultipart(a.cfg.GetBool(cfgKludgeUseDefaultXMLNSForCompleteMultipartUpload))
} }
func (a *App) startServices() { func (a *App) startServices() {
@ -471,7 +518,7 @@ func (a *App) startServices() {
a.services = append(a.services, pprofService) a.services = append(a.services, pprofService)
go pprofService.Start() go pprofService.Start()
prometheusService := NewPrometheusService(a.cfg, a.log, a.metrics.Handler()) prometheusService := NewPrometheusService(a.cfg, a.log)
a.services = append(a.services, prometheusService) a.services = append(a.services, prometheusService)
go prometheusService.Start() go prometheusService.Start()
} }
@ -479,61 +526,37 @@ func (a *App) startServices() {
func (a *App) initServers(ctx context.Context) { func (a *App) initServers(ctx context.Context) {
serversInfo := fetchServers(a.cfg) serversInfo := fetchServers(a.cfg)
a.servers = make([]Server, 0, len(serversInfo)) a.servers = make([]Server, len(serversInfo))
for _, serverInfo := range serversInfo { for i, serverInfo := range serversInfo {
fields := []zap.Field{ a.log.Info("added server",
zap.String("address", serverInfo.Address), zap.Bool("tls enabled", serverInfo.TLS.Enabled), zap.String("address", serverInfo.Address), zap.Bool("tls enabled", serverInfo.TLS.Enabled),
zap.String("tls cert", serverInfo.TLS.CertFile), zap.String("tls key", serverInfo.TLS.KeyFile), zap.String("tls cert", serverInfo.TLS.CertFile), zap.String("tls key", serverInfo.TLS.KeyFile))
} a.servers[i] = newServer(ctx, serverInfo, a.log)
srv, err := newServer(ctx, serverInfo)
if err != nil {
a.log.Warn("failed to add server", append(fields, zap.Error(err))...)
continue
}
a.servers = append(a.servers, srv)
a.log.Info("add server", fields...)
}
if len(a.servers) == 0 {
a.log.Fatal("no healthy servers")
} }
} }
func (a *App) updateServers() error { func (a *App) updateServers() error {
serversInfo := fetchServers(a.cfg) serversInfo := fetchServers(a.cfg)
var found bool if len(serversInfo) != len(a.servers) {
for _, serverInfo := range serversInfo { return fmt.Errorf("invalid servers configuration: amount mismatch: old '%d', new '%d", len(a.servers), len(serversInfo))
index := a.serverIndex(serverInfo.Address) }
if index == -1 {
continue for i, serverInfo := range serversInfo {
if serverInfo.Address != a.servers[i].Address() {
return fmt.Errorf("invalid servers configuration: addresses mismatch: old '%s', new '%s", a.servers[i].Address(), serverInfo.Address)
} }
if serverInfo.TLS.Enabled { if serverInfo.TLS.Enabled {
if err := a.servers[index].UpdateCert(serverInfo.TLS.CertFile, serverInfo.TLS.KeyFile); err != nil { if err := a.servers[i].UpdateCert(serverInfo.TLS.CertFile, serverInfo.TLS.KeyFile); err != nil {
return fmt.Errorf("failed to update tls certs: %w", err) return fmt.Errorf("failed to update tls certs: %w", err)
} }
} }
found = true
}
if !found {
return fmt.Errorf("invalid servers configuration: no known server found")
} }
return nil return nil
} }
func (a *App) serverIndex(address string) int {
for i := range a.servers {
if a.servers[i].Address() == address {
return i
}
}
return -1
}
func (a *App) stopServices() { func (a *App) stopServices() {
ctx, cancel := shutdownContext() ctx, cancel := shutdownContext()
defer cancel() defer cancel()
@ -630,7 +653,6 @@ func (a *App) initHandler() {
DefaultMaxAge: handler.DefaultMaxAge, DefaultMaxAge: handler.DefaultMaxAge,
NotificatorEnabled: a.cfg.GetBool(cfgEnableNATS), NotificatorEnabled: a.cfg.GetBool(cfgEnableNATS),
CopiesNumber: handler.DefaultCopiesNumber, CopiesNumber: handler.DefaultCopiesNumber,
XMLDecoder: a.settings.xmlDecoder,
} }
if a.cfg.IsSet(cfgDefaultMaxAge) { if a.cfg.IsSet(cfgDefaultMaxAge) {
@ -648,14 +670,6 @@ func (a *App) initHandler() {
cfg.CopiesNumber = val cfg.CopiesNumber = val
} }
cfg.ResolveZoneList = a.cfg.GetStringSlice(cfgResolveBucketAllow)
cfg.IsResolveListAllow = len(cfg.ResolveZoneList) > 0
if !cfg.IsResolveListAllow {
cfg.ResolveZoneList = a.cfg.GetStringSlice(cfgResolveBucketDeny)
}
cfg.CompleteMultipartKeepalive = a.cfg.GetDuration(cfgKludgeCompleteMultipartUploadKeepalive)
var err error var err error
a.api, err = handler.New(a.log, a.obj, a.nc, cfg) a.api, err = handler.New(a.log, a.obj, a.nc, cfg)
if err != nil { if err != nil {

View file

@ -3,12 +3,227 @@ package main
import ( import (
"net/http" "net/http"
"github.com/TrueCloudLab/frostfs-sdk-go/pool"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/spf13/viper" "github.com/spf13/viper"
"go.uber.org/zap" "go.uber.org/zap"
) )
const (
namespace = "frostfs_s3_gw"
stateSubsystem = "state"
poolSubsystem = "pool"
methodGetBalance = "get_balance"
methodPutContainer = "put_container"
methodGetContainer = "get_container"
methodListContainer = "list_container"
methodDeleteContainer = "delete_container"
methodGetContainerEacl = "get_container_eacl"
methodSetContainerEacl = "set_container_eacl"
methodEndpointInfo = "endpoint_info"
methodNetworkInfo = "network_info"
methodPutObject = "put_object"
methodDeleteObject = "delete_object"
methodGetObject = "get_object"
methodHeadObject = "head_object"
methodRangeObject = "range_object"
methodCreateSession = "create_session"
)
type StatisticScraper interface {
Statistic() pool.Statistic
}
type GateMetrics struct {
stateMetrics
poolMetricsCollector
}
type stateMetrics struct {
healthCheck prometheus.Gauge
}
type poolMetricsCollector struct {
poolStatScraper StatisticScraper
overallErrors prometheus.Gauge
overallNodeErrors *prometheus.GaugeVec
overallNodeRequests *prometheus.GaugeVec
currentErrors *prometheus.GaugeVec
requestDuration *prometheus.GaugeVec
}
func newGateMetrics(scraper StatisticScraper) *GateMetrics {
stateMetric := newStateMetrics()
stateMetric.register()
poolMetric := newPoolMetricsCollector(scraper)
poolMetric.register()
return &GateMetrics{
stateMetrics: *stateMetric,
poolMetricsCollector: *poolMetric,
}
}
func (g *GateMetrics) Unregister() {
g.stateMetrics.unregister()
prometheus.Unregister(&g.poolMetricsCollector)
}
func newStateMetrics() *stateMetrics {
return &stateMetrics{
healthCheck: prometheus.NewGauge(prometheus.GaugeOpts{
Namespace: namespace,
Subsystem: stateSubsystem,
Name: "health",
Help: "Current S3 gateway state",
}),
}
}
func (m stateMetrics) register() {
prometheus.MustRegister(m.healthCheck)
}
func (m stateMetrics) unregister() {
prometheus.Unregister(m.healthCheck)
}
func (m stateMetrics) SetHealth(s int32) {
m.healthCheck.Set(float64(s))
}
func newPoolMetricsCollector(scraper StatisticScraper) *poolMetricsCollector {
overallErrors := prometheus.NewGauge(
prometheus.GaugeOpts{
Namespace: namespace,
Subsystem: poolSubsystem,
Name: "overall_errors",
Help: "Total number of errors in pool",
},
)
overallNodeErrors := prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: namespace,
Subsystem: poolSubsystem,
Name: "overall_node_errors",
Help: "Total number of errors for connection in pool",
},
[]string{
"node",
},
)
overallNodeRequests := prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: namespace,
Subsystem: poolSubsystem,
Name: "overall_node_requests",
Help: "Total number of requests to specific node in pool",
},
[]string{
"node",
},
)
currentErrors := prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: namespace,
Subsystem: poolSubsystem,
Name: "current_errors",
Help: "Number of errors on current connections that will be reset after the threshold",
},
[]string{
"node",
},
)
requestsDuration := prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: namespace,
Subsystem: poolSubsystem,
Name: "avg_request_duration",
Help: "Average request duration (in milliseconds) for specific method on node in pool",
},
[]string{
"node",
"method",
},
)
return &poolMetricsCollector{
poolStatScraper: scraper,
overallErrors: overallErrors,
overallNodeErrors: overallNodeErrors,
overallNodeRequests: overallNodeRequests,
currentErrors: currentErrors,
requestDuration: requestsDuration,
}
}
func (m *poolMetricsCollector) Collect(ch chan<- prometheus.Metric) {
m.updateStatistic()
m.overallErrors.Collect(ch)
m.overallNodeErrors.Collect(ch)
m.overallNodeRequests.Collect(ch)
m.currentErrors.Collect(ch)
m.requestDuration.Collect(ch)
}
func (m *poolMetricsCollector) Describe(descs chan<- *prometheus.Desc) {
m.overallErrors.Describe(descs)
m.overallNodeErrors.Describe(descs)
m.overallNodeRequests.Describe(descs)
m.currentErrors.Describe(descs)
m.requestDuration.Describe(descs)
}
func (m *poolMetricsCollector) register() {
prometheus.MustRegister(m)
}
func (m *poolMetricsCollector) updateStatistic() {
stat := m.poolStatScraper.Statistic()
m.overallNodeErrors.Reset()
m.overallNodeRequests.Reset()
m.currentErrors.Reset()
m.requestDuration.Reset()
for _, node := range stat.Nodes() {
m.overallNodeErrors.WithLabelValues(node.Address()).Set(float64(node.OverallErrors()))
m.overallNodeRequests.WithLabelValues(node.Address()).Set(float64(node.Requests()))
m.currentErrors.WithLabelValues(node.Address()).Set(float64(node.CurrentErrors()))
m.updateRequestsDuration(node)
}
m.overallErrors.Set(float64(stat.OverallErrors()))
}
func (m *poolMetricsCollector) updateRequestsDuration(node pool.NodeStatistic) {
m.requestDuration.WithLabelValues(node.Address(), methodGetBalance).Set(float64(node.AverageGetBalance().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodPutContainer).Set(float64(node.AveragePutContainer().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodGetContainer).Set(float64(node.AverageGetContainer().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodListContainer).Set(float64(node.AverageListContainer().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodDeleteContainer).Set(float64(node.AverageDeleteContainer().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodGetContainerEacl).Set(float64(node.AverageGetContainerEACL().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodSetContainerEacl).Set(float64(node.AverageSetContainerEACL().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodEndpointInfo).Set(float64(node.AverageEndpointInfo().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodNetworkInfo).Set(float64(node.AverageNetworkInfo().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodPutObject).Set(float64(node.AveragePutObject().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodDeleteObject).Set(float64(node.AverageDeleteObject().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodGetObject).Set(float64(node.AverageGetObject().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodHeadObject).Set(float64(node.AverageHeadObject().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodRangeObject).Set(float64(node.AverageRangeObject().Milliseconds()))
m.requestDuration.WithLabelValues(node.Address(), methodCreateSession).Set(float64(node.AverageCreateSession().Milliseconds()))
}
// NewPrometheusService creates a new service for gathering prometheus metrics. // NewPrometheusService creates a new service for gathering prometheus metrics.
func NewPrometheusService(v *viper.Viper, log *zap.Logger, handler http.Handler) *Service { func NewPrometheusService(v *viper.Viper, log *zap.Logger) *Service {
if log == nil { if log == nil {
return nil return nil
} }
@ -16,7 +231,7 @@ func NewPrometheusService(v *viper.Viper, log *zap.Logger, handler http.Handler)
return &Service{ return &Service{
Server: &http.Server{ Server: &http.Server{
Addr: v.GetString(cfgPrometheusAddress), Addr: v.GetString(cfgPrometheusAddress),
Handler: handler, Handler: promhttp.Handler(),
}, },
enabled: v.GetBool(cfgPrometheusEnabled), enabled: v.GetBool(cfgPrometheusEnabled),
serviceType: "Prometheus", serviceType: "Prometheus",

View file

@ -3,16 +3,15 @@ package main
import ( import (
"fmt" "fmt"
"os" "os"
"path"
"runtime" "runtime"
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/resolver" "github.com/TrueCloudLab/frostfs-s3-gw/api/resolver"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/version" "github.com/TrueCloudLab/frostfs-s3-gw/internal/version"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/pool" "github.com/TrueCloudLab/frostfs-sdk-go/pool"
"github.com/spf13/pflag" "github.com/spf13/pflag"
"github.com/spf13/viper" "github.com/spf13/viper"
"go.uber.org/zap" "go.uber.org/zap"
@ -113,17 +112,12 @@ const ( // Settings.
// Application. // Application.
cfgApplicationBuildTime = "app.build_time" cfgApplicationBuildTime = "app.build_time"
// Kludge.
cfgKludgeUseDefaultXMLNSForCompleteMultipartUpload = "kludge.use_default_xmlns_for_complete_multipart"
cfgKludgeCompleteMultipartUploadKeepalive = "kludge.complete_multipart_keepalive"
// Command line args. // Command line args.
cmdHelp = "help" cmdHelp = "help"
cmdVersion = "version" cmdVersion = "version"
cmdConfig = "config" cmdConfig = "config"
cmdConfigDir = "config-dir" cmdPProf = "pprof"
cmdPProf = "pprof" cmdMetrics = "metrics"
cmdMetrics = "metrics"
cmdListenAddress = "listen_address" cmdListenAddress = "listen_address"
@ -134,10 +128,6 @@ const ( // Settings.
// List of allowed AccessKeyID prefixes. // List of allowed AccessKeyID prefixes.
cfgAllowedAccessKeyIDPrefixes = "allowed_access_key_id_prefixes" cfgAllowedAccessKeyIDPrefixes = "allowed_access_key_id_prefixes"
// Bucket resolving options.
cfgResolveBucketAllow = "resolve_bucket.allow"
cfgResolveBucketDeny = "resolve_bucket.deny"
// envPrefix is an environment variables prefix used for configuration. // envPrefix is an environment variables prefix used for configuration.
envPrefix = "S3_GW" envPrefix = "S3_GW"
) )
@ -224,8 +214,7 @@ func newSettings() *viper.Viper {
flags.StringP(cmdWallet, "w", "", `path to the wallet`) flags.StringP(cmdWallet, "w", "", `path to the wallet`)
flags.String(cmdAddress, "", `address of wallet account`) flags.String(cmdAddress, "", `address of wallet account`)
flags.StringArray(cmdConfig, nil, "config paths") flags.String(cmdConfig, "", "config path")
flags.String(cmdConfigDir, "", "config dir path")
flags.Duration(cfgHealthcheckTimeout, defaultHealthcheckTimeout, "set timeout to check node health during rebalance") flags.Duration(cfgHealthcheckTimeout, defaultHealthcheckTimeout, "set timeout to check node health during rebalance")
flags.Duration(cfgConnectTimeout, defaultConnectTimeout, "set timeout to connect to FrostFS nodes") flags.Duration(cfgConnectTimeout, defaultConnectTimeout, "set timeout to connect to FrostFS nodes")
@ -257,10 +246,6 @@ func newSettings() *viper.Viper {
v.SetDefault(cfgPProfAddress, "localhost:8085") v.SetDefault(cfgPProfAddress, "localhost:8085")
v.SetDefault(cfgPrometheusAddress, "localhost:8086") v.SetDefault(cfgPrometheusAddress, "localhost:8086")
// kludge
v.SetDefault(cfgKludgeUseDefaultXMLNSForCompleteMultipartUpload, false)
v.SetDefault(cfgKludgeCompleteMultipartUploadKeepalive, 10*time.Second)
// Bind flags // Bind flags
if err := bindFlags(v, flags); err != nil { if err := bindFlags(v, flags); err != nil {
panic(fmt.Errorf("bind flags: %w", err)) panic(fmt.Errorf("bind flags: %w", err))
@ -328,8 +313,10 @@ func newSettings() *viper.Viper {
os.Exit(0) os.Exit(0)
} }
if err := readInConfig(v); err != nil { if v.IsSet(cmdConfig) {
panic(err) if err := readConfig(v); err != nil {
panic(err)
}
} }
return v return v
@ -345,9 +332,6 @@ func bindFlags(v *viper.Viper, flags *pflag.FlagSet) error {
if err := v.BindPFlag(cmdConfig, flags.Lookup(cmdConfig)); err != nil { if err := v.BindPFlag(cmdConfig, flags.Lookup(cmdConfig)); err != nil {
return err return err
} }
if err := v.BindPFlag(cmdConfigDir, flags.Lookup(cmdConfigDir)); err != nil {
return err
}
if err := v.BindPFlag(cfgWalletPath, flags.Lookup(cmdWallet)); err != nil { if err := v.BindPFlag(cfgWalletPath, flags.Lookup(cmdWallet)); err != nil {
return err return err
} }
@ -386,72 +370,17 @@ func bindFlags(v *viper.Viper, flags *pflag.FlagSet) error {
return nil return nil
} }
func readInConfig(v *viper.Viper) error {
if v.IsSet(cmdConfig) {
if err := readConfig(v); err != nil {
return err
}
}
if v.IsSet(cmdConfigDir) {
if err := readConfigDir(v); err != nil {
return err
}
}
return nil
}
func readConfigDir(v *viper.Viper) error {
cfgSubConfigDir := v.GetString(cmdConfigDir)
entries, err := os.ReadDir(cfgSubConfigDir)
if err != nil {
return err
}
for _, entry := range entries {
if entry.IsDir() {
continue
}
ext := path.Ext(entry.Name())
if ext != ".yaml" && ext != ".yml" {
continue
}
if err = mergeConfig(v, path.Join(cfgSubConfigDir, entry.Name())); err != nil {
return err
}
}
return nil
}
func readConfig(v *viper.Viper) error { func readConfig(v *viper.Viper) error {
for _, fileName := range v.GetStringSlice(cmdConfig) { cfgFileName := v.GetString(cmdConfig)
if err := mergeConfig(v, fileName); err != nil { cfgFile, err := os.Open(cfgFileName)
return err
}
}
return nil
}
func mergeConfig(v *viper.Viper, fileName string) error {
cfgFile, err := os.Open(fileName)
if err != nil { if err != nil {
return err return err
} }
if err = v.ReadConfig(cfgFile); err != nil {
defer func() {
if errClose := cfgFile.Close(); errClose != nil {
panic(errClose)
}
}()
if err = v.MergeConfig(cfgFile); err != nil {
return err return err
} }
return nil return cfgFile.Close()
} }
// newLogger constructs a Logger instance for the current application. // newLogger constructs a Logger instance for the current application.

View file

@ -7,6 +7,8 @@ import (
"fmt" "fmt"
"net" "net"
"sync" "sync"
"go.uber.org/zap"
) )
type ( type (
@ -55,11 +57,11 @@ func (s *server) UpdateCert(certFile, keyFile string) error {
return s.tlsProvider.UpdateCert(certFile, keyFile) return s.tlsProvider.UpdateCert(certFile, keyFile)
} }
func newServer(ctx context.Context, serverInfo ServerInfo) (*server, error) { func newServer(ctx context.Context, serverInfo ServerInfo, logger *zap.Logger) *server {
var lic net.ListenConfig var lic net.ListenConfig
ln, err := lic.Listen(ctx, "tcp", serverInfo.Address) ln, err := lic.Listen(ctx, "tcp", serverInfo.Address)
if err != nil { if err != nil {
return nil, fmt.Errorf("could not prepare listener: %w", err) logger.Fatal("could not prepare listener", zap.String("address", serverInfo.Address), zap.Error(err))
} }
tlsProvider := &certProvider{ tlsProvider := &certProvider{
@ -68,7 +70,7 @@ func newServer(ctx context.Context, serverInfo ServerInfo) (*server, error) {
if serverInfo.TLS.Enabled { if serverInfo.TLS.Enabled {
if err = tlsProvider.UpdateCert(serverInfo.TLS.CertFile, serverInfo.TLS.KeyFile); err != nil { if err = tlsProvider.UpdateCert(serverInfo.TLS.CertFile, serverInfo.TLS.KeyFile); err != nil {
return nil, fmt.Errorf("failed to update cert: %w", err) logger.Fatal("failed to update cert", zap.Error(err))
} }
ln = tls.NewListener(ln, &tls.Config{ ln = tls.NewListener(ln, &tls.Config{
@ -80,7 +82,7 @@ func newServer(ctx context.Context, serverInfo ServerInfo) (*server, error) {
address: serverInfo.Address, address: serverInfo.Address,
listener: ln, listener: ln,
tlsProvider: tlsProvider, tlsProvider: tlsProvider,
}, nil }
} }
func (p *certProvider) GetCertificate(*tls.ClientHelloInfo) (*tls.Certificate, error) { func (p *certProvider) GetCertificate(*tls.ClientHelloInfo) (*tls.Certificate, error) {

View file

@ -123,12 +123,3 @@ S3_GW_FROSTFS_SET_COPIES_NUMBER=0
# List of allowed AccessKeyID prefixes # List of allowed AccessKeyID prefixes
# If not set, S3 GW will accept all AccessKeyIDs # If not set, S3 GW will accept all AccessKeyIDs
S3_GW_ALLOWED_ACCESS_KEY_ID_PREFIXES=Ck9BHsgKcnwfCTUSFm6pxhoNS4cBqgN2NQ8zVgPjqZDX 3stjWenX15YwYzczMr88gy3CQr4NYFBQ8P7keGzH5QFn S3_GW_ALLOWED_ACCESS_KEY_ID_PREFIXES=Ck9BHsgKcnwfCTUSFm6pxhoNS4cBqgN2NQ8zVgPjqZDX 3stjWenX15YwYzczMr88gy3CQr4NYFBQ8P7keGzH5QFn
# List of container NNS zones which are allowed or restricted to resolve with HEAD request
S3_GW_RESOLVE_BUCKET_ALLOW=container
# S3_GW_RESOLVE_BUCKET_DENY=
# Enable using default xml namespace `http://s3.amazonaws.com/doc/2006-03-01/` when parse`CompleteMultipartUpload` xml body.
S3_GW_KLUDGE_USE_DEFAULT_XMLNS_FOR_COMPLETE_MULTIPART=false
# Set timeout between whitespace transmissions during CompleteMultipartUpload processing.
S3_GW_KLUDGE_COMPLETE_MULTIPART_KEEPALIVE=10s

View file

@ -55,11 +55,11 @@ resolve_order:
# Metrics # Metrics
pprof: pprof:
enabled: false enabled: true
address: localhost:8085 address: localhost:8085
prometheus: prometheus:
enabled: false enabled: true
address: localhost:8086 address: localhost:8086
# Timeout to connect to a node # Timeout to connect to a node
@ -144,14 +144,3 @@ frostfs:
allowed_access_key_id_prefixes: allowed_access_key_id_prefixes:
- Ck9BHsgKcnwfCTUSFm6pxhoNS4cBqgN2NQ8zVgPjqZDX - Ck9BHsgKcnwfCTUSFm6pxhoNS4cBqgN2NQ8zVgPjqZDX
- 3stjWenX15YwYzczMr88gy3CQr4NYFBQ8P7keGzH5QFn - 3stjWenX15YwYzczMr88gy3CQr4NYFBQ8P7keGzH5QFn
resolve_bucket:
allow:
- container
deny:
kludge:
# Enable using default xml namespace `http://s3.amazonaws.com/doc/2006-03-01/` when parse`CompleteMultipartUpload` xml body.
use_default_xmlns_for_complete_multipart: false
# Set timeout between whitespace transmissions during CompleteMultipartUpload processing.
complete_multipart_keepalive: 10s

View file

@ -1,3 +0,0 @@
pprof:
enabled: true
address: localhost:8085

View file

@ -1,3 +0,0 @@
prometheus:
enabled: true
address: localhost:8086

View file

@ -10,9 +10,9 @@ import (
"fmt" "fmt"
"io" "io"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "github.com/TrueCloudLab/frostfs-sdk-go/bearer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" "github.com/TrueCloudLab/frostfs-sdk-go/netmap"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "github.com/TrueCloudLab/frostfs-sdk-go/session"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"golang.org/x/crypto/chacha20poly1305" "golang.org/x/crypto/chacha20poly1305"
"golang.org/x/crypto/hkdf" "golang.org/x/crypto/hkdf"

View file

@ -1,7 +1,7 @@
// Code generated by protoc-gen-go. DO NOT EDIT. // Code generated by protoc-gen-go. DO NOT EDIT.
// versions: // versions:
// protoc-gen-go v1.28.1 // protoc-gen-go v1.28.1
// protoc v3.21.12 // protoc v3.15.8
// source: creds/accessbox/accessbox.proto // source: creds/accessbox/accessbox.proto
package accessbox package accessbox
@ -290,12 +290,11 @@ var file_creds_accessbox_accessbox_proto_rawDesc = []byte{
0x0c, 0x52, 0x0b, 0x62, 0x65, 0x61, 0x72, 0x65, 0x72, 0x54, 0x6f, 0x6b, 0x65, 0x6e, 0x12, 0x24, 0x0c, 0x52, 0x0b, 0x62, 0x65, 0x61, 0x72, 0x65, 0x72, 0x54, 0x6f, 0x6b, 0x65, 0x6e, 0x12, 0x24,
0x0a, 0x0d, 0x73, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e, 0x54, 0x6f, 0x6b, 0x65, 0x6e, 0x73, 0x18, 0x0a, 0x0d, 0x73, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e, 0x54, 0x6f, 0x6b, 0x65, 0x6e, 0x73, 0x18,
0x03, 0x20, 0x03, 0x28, 0x0c, 0x52, 0x0d, 0x73, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e, 0x54, 0x6f, 0x03, 0x20, 0x03, 0x28, 0x0c, 0x52, 0x0d, 0x73, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e, 0x54, 0x6f,
0x6b, 0x65, 0x6e, 0x73, 0x42, 0x46, 0x5a, 0x44, 0x67, 0x69, 0x74, 0x2e, 0x66, 0x72, 0x6f, 0x73, 0x6b, 0x65, 0x6e, 0x73, 0x42, 0x40, 0x5a, 0x3e, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63,
0x74, 0x66, 0x73, 0x2e, 0x69, 0x6e, 0x66, 0x6f, 0x2f, 0x54, 0x72, 0x75, 0x65, 0x43, 0x6c, 0x6f, 0x6f, 0x6d, 0x2f, 0x54, 0x72, 0x75, 0x65, 0x43, 0x6c, 0x6f, 0x75, 0x64, 0x4c, 0x61, 0x62, 0x2f,
0x75, 0x64, 0x4c, 0x61, 0x62, 0x2f, 0x66, 0x72, 0x6f, 0x73, 0x74, 0x66, 0x73, 0x2d, 0x73, 0x33, 0x66, 0x72, 0x6f, 0x73, 0x74, 0x66, 0x73, 0x2d, 0x73, 0x33, 0x2d, 0x67, 0x77, 0x2f, 0x63, 0x72,
0x2d, 0x67, 0x77, 0x2f, 0x63, 0x72, 0x65, 0x64, 0x73, 0x2f, 0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x62, 0x65, 0x64, 0x73, 0x2f, 0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x62, 0x6f, 0x78, 0x3b, 0x61, 0x63, 0x63,
0x6f, 0x78, 0x3b, 0x61, 0x63, 0x63, 0x65, 0x73, 0x73, 0x62, 0x6f, 0x78, 0x62, 0x06, 0x70, 0x72, 0x65, 0x73, 0x73, 0x62, 0x6f, 0x78, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
0x6f, 0x74, 0x6f, 0x33,
} }
var ( var (

Some files were not shown because too many files have changed in this diff Show more