Compare commits

..

61 commits

Author SHA1 Message Date
a12fea8a5b Release v0.31.0
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-11-20 15:45:07 +03:00
9875307c9b [#556] Check bucket name not only during creation
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-11-20 08:13:27 +00:00
b1775f9478 [#553] authmate: Add retryer to create access box
After using AddChain to provide access to container we have to wait:
* tx with APE chain be accepted by blockchain
* cache in storage node be updated

it takes a while. So we add retry
 (the same as when we add bucket settings during bucket creation)

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-11-19 15:46:00 +03:00
4fa45bdac2 [#553] authmate: Don't use basic acl
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-11-19 15:45:54 +03:00
368c7d2acd [#549] Add tracing attributes
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-11-18 11:55:31 +00:00
31076796ce [#550] Execute CI on push to master
Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2024-11-15 14:31:11 +03:00
eff0de43d5 [#538] Return headers with 304 Not Modified
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-11-13 13:49:09 +00:00
fb00dff83b [#540] Add md5 S3Tests compatability
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2024-11-13 14:50:16 +03:00
d8f126b339 [#539] Fix listing v1 bookmark marker
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-11-12 12:58:09 +00:00
7ab902d8d2 [#536] Add rule ID generation
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-11-12 12:51:02 +00:00
0792fcf456 [#536] Fix error codes in lifecycle configuration check
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-11-12 12:51:02 +00:00
c46ffa8146 [#536] Add prefix to lifecycle rule
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-11-12 12:51:02 +00:00
3260308cc0 [#528] Check owner ID before deleting bucket
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-11-12 12:47:43 +00:00
d6e6a13576 [#542] Stop using obsolete .github directory
This commit is a part of multi-repo cleanup effort:
TrueCloudLab/frostfs-infra#136

Signed-off-by: Vitaliy Potyarkin <v.potyarkin@yadro.com>
2024-11-06 15:31:16 +03:00
17d40245de [#505] docs: Add example of uploading file using presigned URL
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-11-02 08:53:54 +00:00
979d85b046 [#505] authmate: Add flag for headers in generate-presigned-url cmd
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-11-02 08:53:54 +00:00
539dab8680 [#501] Add the trace id to the logs of middlware
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-11-02 08:51:48 +00:00
76008d4ba1 [#501] Consider using request logger in logAndSendError
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-11-02 08:51:48 +00:00
8bc19725ba [#521] Add documentation for multinet settings
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-10-29 15:55:27 +03:00
9e64304499 [#521] Use handler to register dial events
While frostfs-node uses dial handler to udpate metric
value, gateway starts with simple event logging.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-10-29 15:55:27 +03:00
94504e9746 [#521] Use source dialer for gRPC connection to storage
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-10-29 15:55:27 +03:00
a8458dbc27 [#521] Add internal/net package with multinet dialer source
Code is taken from frostfs-node#1422
Author: Dmitrii Stepanov (dstepanov-yadro)

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-10-29 15:55:26 +03:00
424038de6c [#524] Update pool to treat maintenance mode differently
Contains these changes for pool component in SDK:
* frostfs-sdk-go#279 fix mm error counting during search operation
* frostfs-sdk-go#283 immediately mark mm node as unhealthy
* frostfs-sdk-go#278 do not reconnect to mm node

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-10-23 12:17:20 +00:00
3cf27d281d [#509] Support fallback address when getting box
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
3c7cb82553 [#509] Init resolvers before first resolving
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
57b7e83380 [#509] Save isCustom flag into accessbox
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
6a90f4e624 [#509] Update docs
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
cb3753f286 [#509] Add tests for custom credentials
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
81209e308c [#509] Fix tests
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
b78e55e101 [#509] Support custom AWS credentials
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-23 15:01:31 +03:00
25c24f5ce6 [#522] Add waiter to contract clients
Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
2024-10-23 09:22:19 +03:00
09c11262c6 [#516] Check Content-Md5 of lifecycle configuration
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-10-22 14:21:49 +00:00
f120715a37 [#516] Convert expiration date to epoch
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-10-22 14:21:49 +00:00
aaed083d82 [#520] Support the continuous use of interceptors
We can always add interceptors to the grpc
connection to the storage, since the actual
use will be controlled by the configuration
from the frostfs-observability library.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-10-21 11:49:22 +03:00
e35b582fe2 [#506] Deny bucket names with dot
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-10-08 12:50:22 +03:00
39fc7aa3ee [#404] Fix using encoding type in multipart upload listing
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-10-08 09:44:05 +00:00
da41f47826 [#404] Add check of encoding type in object listing
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-10-08 09:44:05 +00:00
9e5fb4be95 [#507] Return not implemented by default in bucket router
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-10-07 05:38:35 +00:00
346243b159 [#450] Fix getPutPayloadSize
If X-Amz-Decoded-Content-Length explicitly set use it even if value is 0

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-03 14:23:06 +03:00
03481274f0 [#467] authmate: Add sign command
Support singing arbitrary data using aws sigv4 algorithm

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-10-02 13:42:22 +00:00
c2adbd758a [#488] middleware/auth: Add frostfs-to-s3 error transformation
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-10-02 13:18:25 +03:00
bc17ab5e47 [#488] middleware/policy: Add frostfs-to-s3 error transformation
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-10-02 12:35:04 +03:00
9fadfbbc2f [#488] Renamed api/errors, layer/frostfs and layer/tree package names
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-10-02 12:35:04 +03:00
827ea1a41e [#488] Move layer/frostfs.go to layer/frostfs/frostfs.go
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-10-02 12:35:04 +03:00
968f10a72f [#488] Move layer/tree_service.go to layer/tree/tree_service.go
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-10-02 12:35:04 +03:00
582e6ac642 [#503] Update SDK to fix error counting
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-09-27 10:36:48 +03:00
99f273f9af [#461] Configure logger sampling policy
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2024-09-26 10:34:44 +03:00
cd96adef36 [#499] Fix of sighup traicing docs
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-09-25 14:34:18 +03:00
738ce14f50 [#434] Remove container on failed bucket creation
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-09-25 07:15:24 +00:00
5358e39f71 [#496] Update frostfs-go-sdk for in-flight requests support
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-09-23 08:57:40 +03:00
34c1426b9f [#484] Add root ca cert for telemetry configuration
Signed-off-by: Aleksey Savaitan <a.savaitan@yadro.com>
2024-09-19 11:07:13 +00:00
8ca73e2079 [#493] Fix of receiving VHS namespaces map
In the process of forming a map with namespaces
for which VHS is enabled, we resolve the alias
of the namespace. The problem is that to resolve,
we need default namespace names, which in turn do
not have time to decide by this time. Therefore,
now the check for the default name takes place
directly in the prepareVHSNamespaces function
based on previously read default names.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-09-17 16:57:05 +03:00
a87c636b4c [#493] Fix X-Frostfs-S3-VHS header processing
It is assumed that the X-Frostfs-S3-VHS header
will have the value enabled/disabled instead
of true/false.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-09-16 14:28:19 +03:00
26baf8a94e [#492] Add panic catchers to fuzzing tests
Signed-off-by: Roman Ognev <r.ognev@yadro.com>
2024-09-16 10:36:30 +00:00
f187141ae5 [#486] Fix PUT object with negative Content-Length
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-09-16 08:45:46 +00:00
3cffc782e9 [#486] Fix PUT encrypted object
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-09-16 08:45:46 +00:00
d0e4d55772 [#460] Add network info cache
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-09-13 09:56:24 +00:00
42e72889a5 [#450] Add test for chunk-encoded object size
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-09-13 12:00:24 +03:00
98815d5473 [#450] Fix aws-chunked header workflow
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2024-09-13 11:59:07 +03:00
62615d7ab7 [#369] Request reproducer
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-09-11 15:25:09 +03:00
575ab4d294 [#369] Enhanced http requests logging
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
2024-09-11 15:25:09 +03:00
149 changed files with 5776 additions and 2003 deletions

View file

@ -3,11 +3,12 @@ FROM golang:1.22 AS builder
ARG BUILD=now ARG BUILD=now
ARG REPO=git.frostfs.info/TrueCloudLab/frostfs-s3-gw ARG REPO=git.frostfs.info/TrueCloudLab/frostfs-s3-gw
ARG VERSION=dev ARG VERSION=dev
ARG GOFLAGS=""
WORKDIR /src WORKDIR /src
COPY . /src COPY . /src
RUN make RUN make GOFLAGS=${GOFLAGS}
# Executable image # Executable image
FROM alpine AS frostfs-s3-gw FROM alpine AS frostfs-s3-gw

View file

@ -1,3 +1,3 @@
.git .git
.cache .cache
.github .forgejo

View file

Before

Width:  |  Height:  |  Size: 5.5 KiB

After

Width:  |  Height:  |  Size: 5.5 KiB

View file

@ -1,4 +1,8 @@
on: [pull_request] on:
pull_request:
push:
branches:
- master
jobs: jobs:
builds: builds:

View file

@ -1,4 +1,8 @@
on: [pull_request] on:
pull_request:
push:
branches:
- master
jobs: jobs:
lint: lint:

View file

@ -1,4 +1,8 @@
on: [pull_request] on:
pull_request:
push:
branches:
- master
jobs: jobs:
vulncheck: vulncheck:

1
.github/CODEOWNERS vendored
View file

@ -1 +0,0 @@
* @alexvanin @dkirillov

View file

@ -4,13 +4,105 @@ This document outlines major changes between releases.
## [Unreleased] ## [Unreleased]
## [0.31.0] - Rongbuk - 2024-11-20
### Fixed
- Docker warnings during image build (#421)
- `PartNumberMarker` in ListMultipart response (#451)
- PostObject handling (#456)
- Tag logging errors (#452)
- Removing of duplicated parts in tree service during split brain (#448)
- Container resolving (#482)
- FrostFS to S3 error transformation (#488)
- Default bucket routing (#507)
- encoding-type in ListBucketObjectVersions (#404)
- SIGHUP support for `tracing.enabled` config parameter (#520)
- `trace_id` parameter in logs (#501)
- Listing marker processing (#539)
- Content-MD5 header check (#540)
- Precondition check (#538)
- Bucket name check during all S3 operations (#556)
### Added ### Added
- Add support for virtual hosted style addressing (#446, #449) - Support for separate container for all CORS settings (#422)
- Support new param `frostfs.graceful_close_on_switch_timeout` (#475) - `X-Amz-Force-Delete-Bucket` header for forced bucket removal (#31)
- Support patch object method (#479) - `Location` support in CompleteMultipart response (#451)
- Tree pool request duration metric (#447)
- Expiration lifecycle configuration support (#42, #412, #459, #460, #516, #536)
- Add support for virtual hosted style addressing (#446, #449, #493)
- Support `frostfs.graceful_close_on_switch_timeout` (#475)
- Vulnerability report document (#413)
- Support patch object method (#462, #473, #466, #479)
- Enhanced logging and request reproducer (#369)
- Root CA configuration for tracing (#484)
- Log sampling policy configuration (#461)
- `sign` command to `frostfs-s3-authmate` (#467)
- Support custom aws credentials (#509)
- Source IP binding configuration for FrostFS requests (#521)
- Tracing attributes (#549)
### Changed ### Changed
- Update go version to go1.19 (#470) - Split `FrostFS` interface into separate read methods (#427)
- golangci-lint v1.60 support (#474)
- Updated Go version to 1.22 (#470)
- Container removal after failed bucket creation (#434)
- Explicit check for `.` symbol in bucket name (#506)
- Transaction waiter in contract clients (#522)
- Avoid maintenance mode storage node during object operations (#524)
- Content-Type does not include in Presigned URL of s3-authmate (#505)
- Check owner ID before deleting bucket (#528)
- S3-Authmate now uses APE instead basic-ACL (#553)
### Removed
- Reduce using mutex when update app settings (#329)
## [0.30.8] - 2024-10-18
### Fixed
- Error handling for correct connection switch in SDK Pool (#517)
## [0.30.7] - 2024-10-03
### Fixed
- Correct aws-chunk encoding size handling (#511)
## [0.30.6] - 2024-09-17
### Fixed
- Object size of objects upload with aws-chunked encoding (#450)
- Object size of objects upload with negative Content-Length (#486)
## [0.30.5] - 2024-09-16
### Fixed
- Panic catchers for fuzzing tests (#492)
## [0.30.4] - 2024-09-03
### Added
- Fuzzing tests (#480)
## [0.30.3] - 2024-08-27
### Fixed
- Empty listing when multipart upload contains more than 1000 parts (#471)
## [0.30.2] - 2024-08-20
### Fixed
- Error counting in pool component before connection switch (#468)
### Added
- Log of endpoint address during tree pool errors (#468)
## [0.30.1] - 2024-07-25
### Fixed
- Redundant system node removal in tree service (#437)
### Added
- Log details on SDK Pool health status change (#439)
## [0.30.0] - Kangshung -2024-07-19 ## [0.30.0] - Kangshung -2024-07-19
@ -241,4 +333,13 @@ To see CHANGELOG for older versions, refer to https://github.com/nspcc-dev/neofs
[0.29.2]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.29.1...v0.29.2 [0.29.2]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.29.1...v0.29.2
[0.29.3]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.29.2...v0.29.3 [0.29.3]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.29.2...v0.29.3
[0.30.0]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.29.3...v0.30.0 [0.30.0]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.29.3...v0.30.0
[Unreleased]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.30.0...master [0.30.1]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.30.0...v0.30.1
[0.30.2]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.30.1...v0.30.2
[0.30.3]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.30.2...v0.30.3
[0.30.4]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.30.3...v0.30.4
[0.30.5]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.30.4...v0.30.5
[0.30.6]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.30.5...v0.30.6
[0.30.7]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.30.6...v0.30.7
[0.30.8]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.30.7...v0.30.8
[0.31.0]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.30.8...v0.31.0
[Unreleased]: https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw/compare/v0.31.0...master

1
CODEOWNERS Normal file
View file

@ -0,0 +1 @@
.* @alexvanin @dkirillov

View file

@ -14,6 +14,8 @@ METRICS_DUMP_OUT ?= ./metrics-dump.json
CMDS = $(addprefix frostfs-, $(notdir $(wildcard cmd/*))) CMDS = $(addprefix frostfs-, $(notdir $(wildcard cmd/*)))
BINS = $(addprefix $(BINDIR)/, $(CMDS)) BINS = $(addprefix $(BINDIR)/, $(CMDS))
GOFLAGS ?=
# Variables for docker # Variables for docker
REPO_BASENAME = $(shell basename `go list -m`) REPO_BASENAME = $(shell basename `go list -m`)
HUB_IMAGE ?= "truecloudlab/$(REPO_BASENAME)" HUB_IMAGE ?= "truecloudlab/$(REPO_BASENAME)"
@ -44,6 +46,7 @@ all: $(BINS)
$(BINS): $(BINDIR) dep $(BINS): $(BINDIR) dep
@echo "⇒ Build $@" @echo "⇒ Build $@"
CGO_ENABLED=0 \ CGO_ENABLED=0 \
GOFLAGS=$(GOFLAGS) \
go build -v -trimpath \ go build -v -trimpath \
-ldflags "-X $(REPO)/internal/version.Version=$(VERSION)" \ -ldflags "-X $(REPO)/internal/version.Version=$(VERSION)" \
-o $@ ./cmd/$(subst frostfs-,,$(notdir $@)) -o $@ ./cmd/$(subst frostfs-,,$(notdir $@))
@ -70,7 +73,7 @@ docker/%:
-w /src \ -w /src \
-u `stat -c "%u:%g" .` \ -u `stat -c "%u:%g" .` \
--env HOME=/src \ --env HOME=/src \
golang:$(GO_VERSION) make $*,\ golang:$(GO_VERSION) make GOFLAGS=$(GOFLAGS) $*,\
@echo "supported docker targets: all $(BINS) lint") @echo "supported docker targets: all $(BINS) lint")
# Run tests # Run tests
@ -121,6 +124,7 @@ image:
@docker build \ @docker build \
--build-arg REPO=$(REPO) \ --build-arg REPO=$(REPO) \
--build-arg VERSION=$(VERSION) \ --build-arg VERSION=$(VERSION) \
--build-arg GOFLAGS=$(GOFLAGS) \
--rm \ --rm \
-f .docker/Dockerfile \ -f .docker/Dockerfile \
-t $(HUB_IMAGE):$(HUB_TAG) . -t $(HUB_IMAGE):$(HUB_TAG) .

View file

@ -1,5 +1,5 @@
<p align="center"> <p align="center">
<img src="./.github/logo.svg" width="500px" alt="FrostFS logo"> <img src="./.forgejo/logo.svg" width="500px" alt="FrostFS logo">
</p> </p>
<p align="center"> <p align="center">
<a href="https://frostfs.info">FrostFS</a> is a decentralized distributed object storage integrated with the <a href="https://neo.org">NEO Blockchain</a>. <a href="https://frostfs.info">FrostFS</a> is a decentralized distributed object storage integrated with the <a href="https://neo.org">NEO Blockchain</a>.

View file

@ -1 +1 @@
v0.30.0 v0.31.0

View file

@ -14,16 +14,17 @@ import (
"time" "time"
v4 "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth/signer/v4" v4 "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth/signer/v4"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/tokens" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/tokens"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/aws/aws-sdk-go/aws/credentials" "github.com/aws/aws-sdk-go/aws/credentials"
) )
// authorizationFieldRegexp -- is regexp for credentials with Base58 encoded cid and oid and '0' (zero) as delimiter. // AuthorizationFieldRegexp -- is regexp for credentials with Base58 encoded cid and oid and '0' (zero) as delimiter.
var authorizationFieldRegexp = regexp.MustCompile(`AWS4-HMAC-SHA256 Credential=(?P<access_key_id>[^/]+)/(?P<date>[^/]+)/(?P<region>[^/]*)/(?P<service>[^/]+)/aws4_request,\s*SignedHeaders=(?P<signed_header_fields>.+),\s*Signature=(?P<v4_signature>.+)`) var AuthorizationFieldRegexp = regexp.MustCompile(`AWS4-HMAC-SHA256 Credential=(?P<access_key_id>[^/]+)/(?P<date>[^/]+)/(?P<region>[^/]*)/(?P<service>[^/]+)/aws4_request,\s*SignedHeaders=(?P<signed_header_fields>.+),\s*Signature=(?P<v4_signature>.+)`)
// postPolicyCredentialRegexp -- is regexp for credentials when uploading file using POST with policy. // postPolicyCredentialRegexp -- is regexp for credentials when uploading file using POST with policy.
var postPolicyCredentialRegexp = regexp.MustCompile(`(?P<access_key_id>[^/]+)/(?P<date>[^/]+)/(?P<region>[^/]*)/(?P<service>[^/]+)/aws4_request`) var postPolicyCredentialRegexp = regexp.MustCompile(`(?P<access_key_id>[^/]+)/(?P<date>[^/]+)/(?P<region>[^/]*)/(?P<service>[^/]+)/aws4_request`)
@ -34,6 +35,11 @@ type (
postReg *RegexpSubmatcher postReg *RegexpSubmatcher
cli tokens.Credentials cli tokens.Credentials
allowedAccessKeyIDPrefixes []string // empty slice means all access key ids are allowed allowedAccessKeyIDPrefixes []string // empty slice means all access key ids are allowed
settings CenterSettings
}
CenterSettings interface {
AccessBoxContainer() (cid.ID, bool)
} }
//nolint:revive //nolint:revive
@ -50,7 +56,6 @@ type (
) )
const ( const (
accessKeyPartsNum = 2
authHeaderPartsNum = 6 authHeaderPartsNum = 6
maxFormSizeMemory = 50 * 1048576 // 50 MB maxFormSizeMemory = 50 * 1048576 // 50 MB
@ -82,24 +87,20 @@ var ContentSHA256HeaderStandardValue = map[string]struct{}{
} }
// New creates an instance of AuthCenter. // New creates an instance of AuthCenter.
func New(creds tokens.Credentials, prefixes []string) *Center { func New(creds tokens.Credentials, prefixes []string, settings CenterSettings) *Center {
return &Center{ return &Center{
cli: creds, cli: creds,
reg: NewRegexpMatcher(authorizationFieldRegexp), reg: NewRegexpMatcher(AuthorizationFieldRegexp),
postReg: NewRegexpMatcher(postPolicyCredentialRegexp), postReg: NewRegexpMatcher(postPolicyCredentialRegexp),
allowedAccessKeyIDPrefixes: prefixes, allowedAccessKeyIDPrefixes: prefixes,
settings: settings,
} }
} }
func (c *Center) parseAuthHeader(header string) (*AuthHeader, error) { func (c *Center) parseAuthHeader(header string) (*AuthHeader, error) {
submatches := c.reg.GetSubmatches(header) submatches := c.reg.GetSubmatches(header)
if len(submatches) != authHeaderPartsNum { if len(submatches) != authHeaderPartsNum {
return nil, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrAuthorizationHeaderMalformed), header) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrAuthorizationHeaderMalformed), header)
}
accessKey := strings.Split(submatches["access_key_id"], "0")
if len(accessKey) != accessKeyPartsNum {
return nil, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrInvalidAccessKeyID), accessKey)
} }
signedFields := strings.Split(submatches["signed_header_fields"], ";") signedFields := strings.Split(submatches["signed_header_fields"], ";")
@ -114,15 +115,6 @@ func (c *Center) parseAuthHeader(header string) (*AuthHeader, error) {
}, nil }, nil
} }
func getAddress(accessKeyID string) (oid.Address, error) {
var addr oid.Address
if err := addr.DecodeString(strings.ReplaceAll(accessKeyID, "0", "/")); err != nil {
return addr, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrInvalidAccessKeyID), accessKeyID)
}
return addr, nil
}
func IsStandardContentSHA256(key string) bool { func IsStandardContentSHA256(key string) bool {
_, ok := ContentSHA256HeaderStandardValue[key] _, ok := ContentSHA256HeaderStandardValue[key]
return ok return ok
@ -181,14 +173,14 @@ func (c *Center) Authenticate(r *http.Request) (*middleware.Box, error) {
return nil, err return nil, err
} }
addr, err := getAddress(authHdr.AccessKeyID) cnrID, err := c.getAccessBoxContainer(authHdr.AccessKeyID)
if err != nil { if err != nil {
return nil, err return nil, err
} }
box, attrs, err := c.cli.GetBox(r.Context(), addr) box, attrs, err := c.cli.GetBox(r.Context(), cnrID, authHdr.AccessKeyID)
if err != nil { if err != nil {
return nil, fmt.Errorf("get box '%s': %w", addr, err) return nil, fmt.Errorf("get box by access key '%s': %w", authHdr.AccessKeyID, err)
} }
if err = checkFormatHashContentSHA256(r.Header.Get(AmzContentSHA256)); err != nil { if err = checkFormatHashContentSHA256(r.Header.Get(AmzContentSHA256)); err != nil {
@ -216,15 +208,29 @@ func (c *Center) Authenticate(r *http.Request) (*middleware.Box, error) {
return result, nil return result, nil
} }
func (c *Center) getAccessBoxContainer(accessKeyID string) (cid.ID, error) {
var addr oid.Address
if err := addr.DecodeString(strings.ReplaceAll(accessKeyID, "0", "/")); err == nil {
return addr.Container(), nil
}
cnrID, ok := c.settings.AccessBoxContainer()
if ok {
return cnrID, nil
}
return cid.ID{}, fmt.Errorf("%w: unknown container for creds '%s'", apierr.GetAPIError(apierr.ErrInvalidAccessKeyID), accessKeyID)
}
func checkFormatHashContentSHA256(hash string) error { func checkFormatHashContentSHA256(hash string) error {
if !IsStandardContentSHA256(hash) { if !IsStandardContentSHA256(hash) {
hashBinary, err := hex.DecodeString(hash) hashBinary, err := hex.DecodeString(hash)
if err != nil { if err != nil {
return fmt.Errorf("%w: decode hash: %s: %s", apiErrors.GetAPIError(apiErrors.ErrContentSHA256Mismatch), return fmt.Errorf("%w: decode hash: %s: %s", apierr.GetAPIError(apierr.ErrContentSHA256Mismatch),
hash, err.Error()) hash, err.Error())
} }
if len(hashBinary) != sha256.Size && len(hash) != 0 { if len(hashBinary) != sha256.Size && len(hash) != 0 {
return fmt.Errorf("%w: invalid hash size %d", apiErrors.GetAPIError(apiErrors.ErrContentSHA256Mismatch), len(hashBinary)) return fmt.Errorf("%w: invalid hash size %d", apierr.GetAPIError(apierr.ErrContentSHA256Mismatch), len(hashBinary))
} }
} }
@ -242,12 +248,12 @@ func (c Center) checkAccessKeyID(accessKeyID string) error {
} }
} }
return fmt.Errorf("%w: accesskeyID prefix isn't allowed", apiErrors.GetAPIError(apiErrors.ErrAccessDenied)) return fmt.Errorf("%w: accesskeyID prefix isn't allowed", apierr.GetAPIError(apierr.ErrAccessDenied))
} }
func (c *Center) checkFormData(r *http.Request) (*middleware.Box, error) { func (c *Center) checkFormData(r *http.Request) (*middleware.Box, error) {
if err := r.ParseMultipartForm(maxFormSizeMemory); err != nil { if err := r.ParseMultipartForm(maxFormSizeMemory); err != nil {
return nil, fmt.Errorf("%w: parse multipart form with max size %d", apiErrors.GetAPIError(apiErrors.ErrInvalidArgument), maxFormSizeMemory) return nil, fmt.Errorf("%w: parse multipart form with max size %d", apierr.GetAPIError(apierr.ErrInvalidArgument), maxFormSizeMemory)
} }
if err := prepareForm(r.MultipartForm); err != nil { if err := prepareForm(r.MultipartForm); err != nil {
@ -262,7 +268,7 @@ func (c *Center) checkFormData(r *http.Request) (*middleware.Box, error) {
creds := MultipartFormValue(r, "x-amz-credential") creds := MultipartFormValue(r, "x-amz-credential")
submatches := c.postReg.GetSubmatches(creds) submatches := c.postReg.GetSubmatches(creds)
if len(submatches) != 4 { if len(submatches) != 4 {
return nil, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrAuthorizationHeaderMalformed), creds) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrAuthorizationHeaderMalformed), creds)
} }
signatureDateTime, err := time.Parse("20060102T150405Z", MultipartFormValue(r, "x-amz-date")) signatureDateTime, err := time.Parse("20060102T150405Z", MultipartFormValue(r, "x-amz-date"))
@ -272,14 +278,14 @@ func (c *Center) checkFormData(r *http.Request) (*middleware.Box, error) {
accessKeyID := submatches["access_key_id"] accessKeyID := submatches["access_key_id"]
addr, err := getAddress(accessKeyID) cnrID, err := c.getAccessBoxContainer(accessKeyID)
if err != nil { if err != nil {
return nil, err return nil, err
} }
box, attrs, err := c.cli.GetBox(r.Context(), addr) box, attrs, err := c.cli.GetBox(r.Context(), cnrID, accessKeyID)
if err != nil { if err != nil {
return nil, fmt.Errorf("get box '%s': %w", addr, err) return nil, fmt.Errorf("get box by accessKeyID '%s': %w", accessKeyID, err)
} }
secret := box.Gate.SecretKey secret := box.Gate.SecretKey
@ -288,7 +294,7 @@ func (c *Center) checkFormData(r *http.Request) (*middleware.Box, error) {
signature := SignStr(secret, service, region, signatureDateTime, policy) signature := SignStr(secret, service, region, signatureDateTime, policy)
reqSignature := MultipartFormValue(r, "x-amz-signature") reqSignature := MultipartFormValue(r, "x-amz-signature")
if signature != reqSignature { if signature != reqSignature {
return nil, fmt.Errorf("%w: %s != %s", apiErrors.GetAPIError(apiErrors.ErrSignatureDoesNotMatch), return nil, fmt.Errorf("%w: %s != %s", apierr.GetAPIError(apierr.ErrSignatureDoesNotMatch),
reqSignature, signature) reqSignature, signature)
} }
@ -333,11 +339,11 @@ func (c *Center) checkSign(authHeader *AuthHeader, box *accessbox.Box, request *
if authHeader.IsPresigned { if authHeader.IsPresigned {
now := time.Now() now := time.Now()
if signatureDateTime.Add(authHeader.Expiration).Before(now) { if signatureDateTime.Add(authHeader.Expiration).Before(now) {
return fmt.Errorf("%w: expired: now %s, signature %s", apiErrors.GetAPIError(apiErrors.ErrExpiredPresignRequest), return fmt.Errorf("%w: expired: now %s, signature %s", apierr.GetAPIError(apierr.ErrExpiredPresignRequest),
now.Format(time.RFC3339), signatureDateTime.Format(time.RFC3339)) now.Format(time.RFC3339), signatureDateTime.Format(time.RFC3339))
} }
if now.Before(signatureDateTime) { if now.Before(signatureDateTime) {
return fmt.Errorf("%w: signature time from the future: now %s, signature %s", apiErrors.GetAPIError(apiErrors.ErrBadRequest), return fmt.Errorf("%w: signature time from the future: now %s, signature %s", apierr.GetAPIError(apierr.ErrBadRequest),
now.Format(time.RFC3339), signatureDateTime.Format(time.RFC3339)) now.Format(time.RFC3339), signatureDateTime.Format(time.RFC3339))
} }
if _, err := signer.Presign(request, nil, authHeader.Service, authHeader.Region, authHeader.Expiration, signatureDateTime); err != nil { if _, err := signer.Presign(request, nil, authHeader.Service, authHeader.Region, authHeader.Expiration, signatureDateTime); err != nil {
@ -352,7 +358,7 @@ func (c *Center) checkSign(authHeader *AuthHeader, box *accessbox.Box, request *
} }
if authHeader.SignatureV4 != signature { if authHeader.SignatureV4 != signature {
return fmt.Errorf("%w: %s != %s: headers %v", apiErrors.GetAPIError(apiErrors.ErrSignatureDoesNotMatch), return fmt.Errorf("%w: %s != %s: headers %v", apierr.GetAPIError(apierr.ErrSignatureDoesNotMatch),
authHeader.SignatureV4, signature, authHeader.SignedFields) authHeader.SignatureV4, signature, authHeader.SignedFields)
} }

View file

@ -72,7 +72,7 @@ func DoFuzzAuthenticate(input []byte) int {
c := &Center{ c := &Center{
cli: mock, cli: mock,
reg: NewRegexpMatcher(authorizationFieldRegexp), reg: NewRegexpMatcher(AuthorizationFieldRegexp),
postReg: NewRegexpMatcher(postPolicyCredentialRegexp), postReg: NewRegexpMatcher(postPolicyCredentialRegexp),
} }

View file

@ -17,8 +17,9 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/tokens" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/tokens"
frostfsErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors" frosterr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
oidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id/test" oidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id/test"
@ -28,11 +29,23 @@ import (
"go.uber.org/zap/zaptest" "go.uber.org/zap/zaptest"
) )
type centerSettingsMock struct {
accessBoxContainer *cid.ID
}
func (c *centerSettingsMock) AccessBoxContainer() (cid.ID, bool) {
if c.accessBoxContainer == nil {
return cid.ID{}, false
}
return *c.accessBoxContainer, true
}
func TestAuthHeaderParse(t *testing.T) { func TestAuthHeaderParse(t *testing.T) {
defaultHeader := "AWS4-HMAC-SHA256 Credential=oid0cid/20210809/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2811ccb9e242f41426738fb1f" defaultHeader := "AWS4-HMAC-SHA256 Credential=oid0cid/20210809/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2811ccb9e242f41426738fb1f"
center := &Center{ center := &Center{
reg: NewRegexpMatcher(authorizationFieldRegexp), reg: NewRegexpMatcher(AuthorizationFieldRegexp),
settings: &centerSettingsMock{},
} }
for _, tc := range []struct { for _, tc := range []struct {
@ -57,11 +70,6 @@ func TestAuthHeaderParse(t *testing.T) {
err: errors.GetAPIError(errors.ErrAuthorizationHeaderMalformed), err: errors.GetAPIError(errors.ErrAuthorizationHeaderMalformed),
expected: nil, expected: nil,
}, },
{
header: strings.ReplaceAll(defaultHeader, "oid0cid", "oidcid"),
err: errors.GetAPIError(errors.ErrInvalidAccessKeyID),
expected: nil,
},
} { } {
authHeader, err := center.parseAuthHeader(tc.header) authHeader, err := center.parseAuthHeader(tc.header)
require.ErrorIs(t, err, tc.err, tc.header) require.ErrorIs(t, err, tc.err, tc.header)
@ -69,43 +77,6 @@ func TestAuthHeaderParse(t *testing.T) {
} }
} }
func TestAuthHeaderGetAddress(t *testing.T) {
defaulErr := errors.GetAPIError(errors.ErrInvalidAccessKeyID)
for _, tc := range []struct {
authHeader *AuthHeader
err error
}{
{
authHeader: &AuthHeader{
AccessKeyID: "vWqF8cMDRbJcvnPLALoQGnABPPhw8NyYMcGsfDPfZJM0HrgjonN8CgFvCZ3kh9BUXw4W2tJ5E7EAGhueSF122HB",
},
err: nil,
},
{
authHeader: &AuthHeader{
AccessKeyID: "vWqF8cMDRbJcvnPLALoQGnABPPhw8NyYMcGsfDPfZJMHrgjonN8CgFvCZ3kh9BUXw4W2tJ5E7EAGhueSF122HB",
},
err: defaulErr,
},
{
authHeader: &AuthHeader{
AccessKeyID: "oid0cid",
},
err: defaulErr,
},
{
authHeader: &AuthHeader{
AccessKeyID: "oidcid",
},
err: defaulErr,
},
} {
_, err := getAddress(tc.authHeader.AccessKeyID)
require.ErrorIs(t, err, tc.err, tc.authHeader.AccessKeyID)
}
}
func TestSignature(t *testing.T) { func TestSignature(t *testing.T) {
secret := "66be461c3cd429941c55daf42fad2b8153e5a2016ba89c9494d97677cc9d3872" secret := "66be461c3cd429941c55daf42fad2b8153e5a2016ba89c9494d97677cc9d3872"
strToSign := "eyAiZXhwaXJhdGlvbiI6ICIyMDE1LTEyLTMwVDEyOjAwOjAwLjAwMFoiLAogICJjb25kaXRpb25zIjogWwogICAgeyJidWNrZXQiOiAiYWNsIn0sCiAgICBbInN0YXJ0cy13aXRoIiwgIiRrZXkiLCAidXNlci91c2VyMS8iXSwKICAgIHsic3VjY2Vzc19hY3Rpb25fcmVkaXJlY3QiOiAiaHR0cDovL2xvY2FsaG9zdDo4MDg0L2FjbCJ9LAogICAgWyJzdGFydHMtd2l0aCIsICIkQ29udGVudC1UeXBlIiwgImltYWdlLyJdLAogICAgeyJ4LWFtei1tZXRhLXV1aWQiOiAiMTQzNjUxMjM2NTEyNzQifSwKICAgIFsic3RhcnRzLXdpdGgiLCAiJHgtYW16LW1ldGEtdGFnIiwgIiJdLAoKICAgIHsiWC1BbXotQ3JlZGVudGlhbCI6ICI4Vmk0MVBIbjVGMXNzY2J4OUhqMXdmMUU2aERUYURpNndxOGhxTU05NllKdTA1QzVDeUVkVlFoV1E2aVZGekFpTkxXaTlFc3BiUTE5ZDRuR3pTYnZVZm10TS8yMDE1MTIyOS91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0sCiAgICB7IngtYW16LWFsZ29yaXRobSI6ICJBV1M0LUhNQUMtU0hBMjU2In0sCiAgICB7IlgtQW16LURhdGUiOiAiMjAxNTEyMjlUMDAwMDAwWiIgfSwKICAgIHsieC1pZ25vcmUtdG1wIjogInNvbWV0aGluZyIgfQogIF0KfQ==" strToSign := "eyAiZXhwaXJhdGlvbiI6ICIyMDE1LTEyLTMwVDEyOjAwOjAwLjAwMFoiLAogICJjb25kaXRpb25zIjogWwogICAgeyJidWNrZXQiOiAiYWNsIn0sCiAgICBbInN0YXJ0cy13aXRoIiwgIiRrZXkiLCAidXNlci91c2VyMS8iXSwKICAgIHsic3VjY2Vzc19hY3Rpb25fcmVkaXJlY3QiOiAiaHR0cDovL2xvY2FsaG9zdDo4MDg0L2FjbCJ9LAogICAgWyJzdGFydHMtd2l0aCIsICIkQ29udGVudC1UeXBlIiwgImltYWdlLyJdLAogICAgeyJ4LWFtei1tZXRhLXV1aWQiOiAiMTQzNjUxMjM2NTEyNzQifSwKICAgIFsic3RhcnRzLXdpdGgiLCAiJHgtYW16LW1ldGEtdGFnIiwgIiJdLAoKICAgIHsiWC1BbXotQ3JlZGVudGlhbCI6ICI4Vmk0MVBIbjVGMXNzY2J4OUhqMXdmMUU2aERUYURpNndxOGhxTU05NllKdTA1QzVDeUVkVlFoV1E2aVZGekFpTkxXaTlFc3BiUTE5ZDRuR3pTYnZVZm10TS8yMDE1MTIyOS91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0sCiAgICB7IngtYW16LWFsZ29yaXRobSI6ICJBV1M0LUhNQUMtU0hBMjU2In0sCiAgICB7IlgtQW16LURhdGUiOiAiMjAxNTEyMjlUMDAwMDAwWiIgfSwKICAgIHsieC1pZ25vcmUtdG1wIjogInNvbWV0aGluZyIgfQogIF0KfQ=="
@ -171,17 +142,17 @@ func TestCheckFormatContentSHA256(t *testing.T) {
} }
type frostFSMock struct { type frostFSMock struct {
objects map[oid.Address]*object.Object objects map[string]*object.Object
} }
func newFrostFSMock() *frostFSMock { func newFrostFSMock() *frostFSMock {
return &frostFSMock{ return &frostFSMock{
objects: map[oid.Address]*object.Object{}, objects: map[string]*object.Object{},
} }
} }
func (f *frostFSMock) GetCredsObject(_ context.Context, address oid.Address) (*object.Object, error) { func (f *frostFSMock) GetCredsObject(_ context.Context, prm tokens.PrmGetCredsObject) (*object.Object, error) {
obj, ok := f.objects[address] obj, ok := f.objects[prm.AccessKeyID]
if !ok { if !ok {
return nil, fmt.Errorf("not found") return nil, fmt.Errorf("not found")
} }
@ -208,7 +179,7 @@ func TestAuthenticate(t *testing.T) {
GateKey: key.PublicKey(), GateKey: key.PublicKey(),
}} }}
accessBox, secret, err := accessbox.PackTokens(gateData, []byte("secret")) accessBox, secret, err := accessbox.PackTokens(gateData, []byte("secret"), false)
require.NoError(t, err) require.NoError(t, err)
data, err := accessBox.Marshal() data, err := accessBox.Marshal()
require.NoError(t, err) require.NoError(t, err)
@ -219,10 +190,10 @@ func TestAuthenticate(t *testing.T) {
obj.SetContainerID(addr.Container()) obj.SetContainerID(addr.Container())
obj.SetID(addr.Object()) obj.SetID(addr.Object())
frostfs := newFrostFSMock() accessKeyID := getAccessKeyID(addr)
frostfs.objects[addr] = &obj
accessKeyID := addr.Container().String() + "0" + addr.Object().String() frostfs := newFrostFSMock()
frostfs.objects[accessKeyID] = &obj
awsCreds := credentials.NewStaticCredentials(accessKeyID, secret.SecretKey, "") awsCreds := credentials.NewStaticCredentials(accessKeyID, secret.SecretKey, "")
defaultSigner := v4.NewSigner(awsCreds) defaultSigner := v4.NewSigner(awsCreds)
@ -413,13 +384,13 @@ func TestAuthenticate(t *testing.T) {
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
creds := tokens.New(bigConfig) creds := tokens.New(bigConfig)
cntr := New(creds, tc.prefixes) cntr := New(creds, tc.prefixes, &centerSettingsMock{})
box, err := cntr.Authenticate(tc.request) box, err := cntr.Authenticate(tc.request)
if tc.err { if tc.err {
require.Error(t, err) require.Error(t, err)
if tc.errCode > 0 { if tc.errCode > 0 {
err = frostfsErrors.UnwrapErr(err) err = frosterr.UnwrapErr(err)
require.Equal(t, errors.GetAPIError(tc.errCode), err) require.Equal(t, errors.GetAPIError(tc.errCode), err)
} }
} else { } else {
@ -455,7 +426,7 @@ func TestHTTPPostAuthenticate(t *testing.T) {
GateKey: key.PublicKey(), GateKey: key.PublicKey(),
}} }}
accessBox, secret, err := accessbox.PackTokens(gateData, []byte("secret")) accessBox, secret, err := accessbox.PackTokens(gateData, []byte("secret"), false)
require.NoError(t, err) require.NoError(t, err)
data, err := accessBox.Marshal() data, err := accessBox.Marshal()
require.NoError(t, err) require.NoError(t, err)
@ -466,10 +437,11 @@ func TestHTTPPostAuthenticate(t *testing.T) {
obj.SetContainerID(addr.Container()) obj.SetContainerID(addr.Container())
obj.SetID(addr.Object()) obj.SetID(addr.Object())
frostfs := newFrostFSMock() accessKeyID := getAccessKeyID(addr)
frostfs.objects[addr] = &obj
frostfs := newFrostFSMock()
frostfs.objects[accessKeyID] = &obj
accessKeyID := addr.Container().String() + "0" + addr.Object().String()
invalidAccessKeyID := oidtest.Address().String() + "0" + oidtest.Address().Object().String() invalidAccessKeyID := oidtest.Address().String() + "0" + oidtest.Address().Object().String()
timeToSign := time.Now() timeToSign := time.Now()
@ -590,13 +562,13 @@ func TestHTTPPostAuthenticate(t *testing.T) {
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
creds := tokens.New(bigConfig) creds := tokens.New(bigConfig)
cntr := New(creds, tc.prefixes) cntr := New(creds, tc.prefixes, &centerSettingsMock{})
box, err := cntr.Authenticate(tc.request) box, err := cntr.Authenticate(tc.request)
if tc.err { if tc.err {
require.Error(t, err) require.Error(t, err)
if tc.errCode > 0 { if tc.errCode > 0 {
err = frostfsErrors.UnwrapErr(err) err = frosterr.UnwrapErr(err)
require.Equal(t, errors.GetAPIError(tc.errCode), err) require.Equal(t, errors.GetAPIError(tc.errCode), err)
} }
} else { } else {
@ -633,3 +605,7 @@ func getRequestWithMultipartForm(t *testing.T, policy, creds, date, sign, fieldN
return req return req
} }
func getAccessKeyID(addr oid.Address) string {
return strings.ReplaceAll(addr.EncodeToString(), "/", "0")
}

View file

@ -23,6 +23,7 @@ type PresignData struct {
Region string Region string
Lifetime time.Duration Lifetime time.Duration
SignTime time.Time SignTime time.Time
Headers map[string]string
} }
// PresignRequest forms pre-signed request to access objects without aws credentials. // PresignRequest forms pre-signed request to access objects without aws credentials.
@ -34,7 +35,10 @@ func PresignRequest(creds *credentials.Credentials, reqData RequestData, presign
} }
req.Header.Set(AmzDate, presignData.SignTime.Format("20060102T150405Z")) req.Header.Set(AmzDate, presignData.SignTime.Format("20060102T150405Z"))
req.Header.Set(ContentTypeHdr, "text/plain")
for k, v := range presignData.Headers {
req.Header.Set(k, v)
}
signer := v4.NewSigner(creds) signer := v4.NewSigner(creds)
signer.DisableURIPathEscaping = true signer.DisableURIPathEscaping = true

View file

@ -29,11 +29,11 @@ func newTokensFrostfsMock() *credentialsMock {
} }
func (m credentialsMock) addBox(addr oid.Address, box *accessbox.Box) { func (m credentialsMock) addBox(addr oid.Address, box *accessbox.Box) {
m.boxes[addr.String()] = box m.boxes[getAccessKeyID(addr)] = box
} }
func (m credentialsMock) GetBox(_ context.Context, addr oid.Address) (*accessbox.Box, []object.Attribute, error) { func (m credentialsMock) GetBox(_ context.Context, _ cid.ID, accessKeyID string) (*accessbox.Box, []object.Attribute, error) {
box, ok := m.boxes[addr.String()] box, ok := m.boxes[accessKeyID]
if !ok { if !ok {
return nil, nil, &apistatus.ObjectNotFound{} return nil, nil, &apistatus.ObjectNotFound{}
} }
@ -41,11 +41,11 @@ func (m credentialsMock) GetBox(_ context.Context, addr oid.Address) (*accessbox
return box, nil, nil return box, nil, nil
} }
func (m credentialsMock) Put(context.Context, cid.ID, tokens.CredentialsParam) (oid.Address, error) { func (m credentialsMock) Put(context.Context, tokens.CredentialsParam) (oid.Address, error) {
return oid.Address{}, nil return oid.Address{}, nil
} }
func (m credentialsMock) Update(context.Context, oid.Address, tokens.CredentialsParam) (oid.Address, error) { func (m credentialsMock) Update(context.Context, tokens.CredentialsParam) (oid.Address, error) {
return oid.Address{}, nil return oid.Address{}, nil
} }
@ -85,8 +85,9 @@ func TestCheckSign(t *testing.T) {
c := &Center{ c := &Center{
cli: mock, cli: mock,
reg: NewRegexpMatcher(authorizationFieldRegexp), reg: NewRegexpMatcher(AuthorizationFieldRegexp),
postReg: NewRegexpMatcher(postPolicyCredentialRegexp), postReg: NewRegexpMatcher(postPolicyCredentialRegexp),
settings: &centerSettingsMock{},
} }
box, err := c.Authenticate(req) box, err := c.Authenticate(req)
require.NoError(t, err) require.NoError(t, err)

View file

@ -30,6 +30,7 @@ type (
Box *accessbox.Box Box *accessbox.Box
Attributes []object.Attribute Attributes []object.Attribute
PutTime time.Time PutTime time.Time
Address *oid.Address
} }
) )
@ -57,8 +58,8 @@ func NewAccessBoxCache(config *Config) *AccessBoxCache {
} }
// Get returns a cached accessbox. // Get returns a cached accessbox.
func (o *AccessBoxCache) Get(address oid.Address) *AccessBoxCacheValue { func (o *AccessBoxCache) Get(accessKeyID string) *AccessBoxCacheValue {
entry, err := o.cache.Get(address) entry, err := o.cache.Get(accessKeyID)
if err != nil { if err != nil {
return nil return nil
} }
@ -74,16 +75,11 @@ func (o *AccessBoxCache) Get(address oid.Address) *AccessBoxCacheValue {
} }
// Put stores an accessbox to cache. // Put stores an accessbox to cache.
func (o *AccessBoxCache) Put(address oid.Address, box *accessbox.Box, attrs []object.Attribute) error { func (o *AccessBoxCache) Put(accessKeyID string, val *AccessBoxCacheValue) error {
val := &AccessBoxCacheValue{ return o.cache.Set(accessKeyID, val)
Box: box,
Attributes: attrs,
PutTime: time.Now(),
}
return o.cache.Set(address, val)
} }
// Delete removes an accessbox from cache. // Delete removes an accessbox from cache.
func (o *AccessBoxCache) Delete(address oid.Address) { func (o *AccessBoxCache) Delete(accessKeyID string) {
o.cache.Remove(address) o.cache.Remove(accessKeyID)
} }

View file

@ -1,13 +1,14 @@
package cache package cache
import ( import (
"strings"
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-contract/frostfsid/client" "git.frostfs.info/TrueCloudLab/frostfs-contract/frostfsid/client"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
cidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id/test" cidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id/test"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
oidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id/test" oidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id/test"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/nspcc-dev/neo-go/pkg/util" "github.com/nspcc-dev/neo-go/pkg/util"
@ -22,18 +23,21 @@ func TestAccessBoxCacheType(t *testing.T) {
addr := oidtest.Address() addr := oidtest.Address()
box := &accessbox.Box{} box := &accessbox.Box{}
var attrs []object.Attribute val := &AccessBoxCacheValue{
Box: box,
}
err := cache.Put(addr, box, attrs) accessKeyID := getAccessKeyID(addr)
err := cache.Put(accessKeyID, val)
require.NoError(t, err) require.NoError(t, err)
val := cache.Get(addr) resVal := cache.Get(accessKeyID)
require.Equal(t, box, val.Box) require.Equal(t, box, resVal.Box)
require.Equal(t, attrs, val.Attributes)
require.Equal(t, 0, observedLog.Len()) require.Equal(t, 0, observedLog.Len())
err = cache.cache.Set(addr, "tmp") err = cache.cache.Set(accessKeyID, "tmp")
require.NoError(t, err) require.NoError(t, err)
assertInvalidCacheEntry(t, cache.Get(addr), observedLog) assertInvalidCacheEntry(t, cache.Get(accessKeyID), observedLog)
} }
func TestBucketsCacheType(t *testing.T) { func TestBucketsCacheType(t *testing.T) {
@ -230,3 +234,7 @@ func getObservedLogger() (*zap.Logger, *observer.ObservedLogs) {
loggerCore, observedLog := observer.New(zap.WarnLevel) loggerCore, observedLog := observer.New(zap.WarnLevel)
return zap.New(loggerCore), observedLog return zap.New(loggerCore), observedLog
} }
func getAccessKeyID(addr oid.Address) string {
return strings.ReplaceAll(addr.EncodeToString(), "/", "0")
}

65
api/cache/network_info.go vendored Normal file
View file

@ -0,0 +1,65 @@
package cache
import (
"fmt"
"time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/bluele/gcache"
"go.uber.org/zap"
)
type (
// NetworkInfoCache provides cache for network info.
NetworkInfoCache struct {
cache gcache.Cache
logger *zap.Logger
}
// NetworkInfoCacheConfig stores expiration params for cache.
NetworkInfoCacheConfig struct {
Lifetime time.Duration
Logger *zap.Logger
}
)
const (
DefaultNetworkInfoCacheLifetime = 1 * time.Minute
networkInfoCacheSize = 1
networkInfoKey = "network_info"
)
// DefaultNetworkInfoConfig returns new default cache expiration values.
func DefaultNetworkInfoConfig(logger *zap.Logger) *NetworkInfoCacheConfig {
return &NetworkInfoCacheConfig{
Lifetime: DefaultNetworkInfoCacheLifetime,
Logger: logger,
}
}
// NewNetworkInfoCache creates an object of NetworkInfoCache.
func NewNetworkInfoCache(config *NetworkInfoCacheConfig) *NetworkInfoCache {
gc := gcache.New(networkInfoCacheSize).LRU().Expiration(config.Lifetime).Build()
return &NetworkInfoCache{cache: gc, logger: config.Logger}
}
func (c *NetworkInfoCache) Get() *netmap.NetworkInfo {
entry, err := c.cache.Get(networkInfoKey)
if err != nil {
return nil
}
result, ok := entry.(netmap.NetworkInfo)
if !ok {
c.logger.Warn(logs.InvalidCacheEntryType, zap.String("actual", fmt.Sprintf("%T", entry)),
zap.String("expected", fmt.Sprintf("%T", result)))
return nil
}
return &result
}
func (c *NetworkInfoCache) Put(info netmap.NetworkInfo) error {
return c.cache.Set(networkInfoKey, info)
}

View file

@ -20,6 +20,7 @@ type (
Filter *LifecycleRuleFilter `xml:"Filter,omitempty"` Filter *LifecycleRuleFilter `xml:"Filter,omitempty"`
ID string `xml:"ID,omitempty"` ID string `xml:"ID,omitempty"`
NonCurrentVersionExpiration *NonCurrentVersionExpiration `xml:"NoncurrentVersionExpiration,omitempty"` NonCurrentVersionExpiration *NonCurrentVersionExpiration `xml:"NoncurrentVersionExpiration,omitempty"`
Prefix string `xml:"Prefix,omitempty"`
} }
AbortIncompleteMultipartUpload struct { AbortIncompleteMultipartUpload struct {
@ -29,6 +30,7 @@ type (
LifecycleExpiration struct { LifecycleExpiration struct {
Date string `xml:"Date,omitempty"` Date string `xml:"Date,omitempty"`
Days *int `xml:"Days,omitempty"` Days *int `xml:"Days,omitempty"`
Epoch *uint64 `xml:"Epoch,omitempty"`
ExpiredObjectDeleteMarker *bool `xml:"ExpiredObjectDeleteMarker,omitempty"` ExpiredObjectDeleteMarker *bool `xml:"ExpiredObjectDeleteMarker,omitempty"`
} }

View file

@ -1,10 +1,13 @@
package errors package errors
import ( import (
"errors"
"fmt" "fmt"
"net/http" "net/http"
frosterrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
frosterr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
) )
type ( type (
@ -1765,7 +1768,7 @@ var errorCodes = errorCodeMap{
// IsS3Error checks if the provided error is a specific s3 error. // IsS3Error checks if the provided error is a specific s3 error.
func IsS3Error(err error, code ErrorCode) bool { func IsS3Error(err error, code ErrorCode) bool {
err = frosterrors.UnwrapErr(err) err = frosterr.UnwrapErr(err)
e, ok := err.(Error) e, ok := err.(Error)
return ok && e.ErrCode == code return ok && e.ErrCode == code
} }
@ -1802,6 +1805,26 @@ func GetAPIErrorWithError(code ErrorCode, err error) Error {
return errorCodes.toAPIErrWithErr(code, err) return errorCodes.toAPIErrWithErr(code, err)
} }
// TransformToS3Error converts FrostFS error to the corresponding S3 error type.
func TransformToS3Error(err error) error {
err = frosterr.UnwrapErr(err) // this wouldn't work with errors.Join
var s3err Error
if errors.As(err, &s3err) {
return err
}
if errors.Is(err, frostfs.ErrAccessDenied) ||
errors.Is(err, tree.ErrNodeAccessDenied) {
return GetAPIError(ErrAccessDenied)
}
if errors.Is(err, frostfs.ErrGatewayTimeout) {
return GetAPIError(ErrGatewayTimeout)
}
return GetAPIError(ErrInternalError)
}
// ObjectError -- error that is linked to a specific object. // ObjectError -- error that is linked to a specific object.
type ObjectError struct { type ObjectError struct {
Err error Err error

View file

@ -2,7 +2,12 @@ package errors
import ( import (
"errors" "errors"
"fmt"
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"github.com/stretchr/testify/require"
) )
func BenchmarkErrCode(b *testing.B) { func BenchmarkErrCode(b *testing.B) {
@ -24,3 +29,56 @@ func BenchmarkErrorsIs(b *testing.B) {
} }
} }
} }
func TestTransformS3Errors(t *testing.T) {
for _, tc := range []struct {
name string
err error
expected ErrorCode
}{
{
name: "simple std error to internal error",
err: errors.New("some error"),
expected: ErrInternalError,
},
{
name: "layer access denied error to s3 access denied error",
err: frostfs.ErrAccessDenied,
expected: ErrAccessDenied,
},
{
name: "wrapped layer access denied error to s3 access denied error",
err: fmt.Errorf("wrap: %w", frostfs.ErrAccessDenied),
expected: ErrAccessDenied,
},
{
name: "layer node access denied error to s3 access denied error",
err: tree.ErrNodeAccessDenied,
expected: ErrAccessDenied,
},
{
name: "layer gateway timeout error to s3 gateway timeout error",
err: frostfs.ErrGatewayTimeout,
expected: ErrGatewayTimeout,
},
{
name: "s3 error to s3 error",
err: GetAPIError(ErrInvalidPart),
expected: ErrInvalidPart,
},
{
name: "wrapped s3 error to s3 error",
err: fmt.Errorf("wrap: %w", GetAPIError(ErrInvalidPart)),
expected: ErrInvalidPart,
},
} {
t.Run(tc.name, func(t *testing.T) {
err := TransformToS3Error(tc.err)
s3err, ok := err.(Error)
require.True(t, ok, "error must be s3 error")
require.Equalf(t, tc.expected, s3err.ErrCode,
"expected: '%s', got: '%s'",
GetAPIError(tc.expected).Code, GetAPIError(s3err.ErrCode).Code)
})
}
}

View file

@ -52,18 +52,18 @@ func (h *handler) GetBucketACLHandler(w http.ResponseWriter, r *http.Request) {
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
settings, err := h.obj.GetBucketSettings(r.Context(), bktInfo) settings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "couldn't get bucket settings", reqInfo, err)
return return
} }
if err = middleware.EncodeToResponse(w, h.encodeBucketCannedACL(ctx, bktInfo, settings)); err != nil { if err = middleware.EncodeToResponse(w, h.encodeBucketCannedACL(ctx, bktInfo, settings)); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
return return
} }
} }
@ -127,17 +127,18 @@ func getTokenIssuerKey(box *accessbox.Box) (*keys.PublicKey, error) {
} }
func (h *handler) PutBucketACLHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) PutBucketACLHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
settings, err := h.obj.GetBucketSettings(r.Context(), bktInfo) settings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "couldn't get bucket settings", reqInfo, err)
return return
} }
@ -149,30 +150,30 @@ func (h *handler) putBucketACLAPEHandler(w http.ResponseWriter, r *http.Request,
defer func() { defer func() {
if errBody := r.Body.Close(); errBody != nil { if errBody := r.Body.Close(); errBody != nil {
h.reqLogger(r.Context()).Warn(logs.CouldNotCloseRequestBody, zap.Error(errBody)) h.reqLogger(ctx).Warn(logs.CouldNotCloseRequestBody, zap.Error(errBody))
} }
}() }()
written, err := io.Copy(io.Discard, r.Body) written, err := io.Copy(io.Discard, r.Body)
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't read request body", reqInfo, err) h.logAndSendError(ctx, w, "couldn't read request body", reqInfo, err)
return return
} }
if written != 0 || len(r.Header.Get(api.AmzACL)) == 0 { if written != 0 || len(r.Header.Get(api.AmzACL)) == 0 {
h.logAndSendError(w, "acl not supported for this bucket", reqInfo, errors.GetAPIError(errors.ErrAccessControlListNotSupported)) h.logAndSendError(ctx, w, "acl not supported for this bucket", reqInfo, errors.GetAPIError(errors.ErrAccessControlListNotSupported))
return return
} }
cannedACL, err := parseCannedACL(r.Header) cannedACL, err := parseCannedACL(r.Header)
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse canned ACL", reqInfo, err) h.logAndSendError(ctx, w, "could not parse canned ACL", reqInfo, err)
return return
} }
chainRules := bucketCannedACLToAPERules(cannedACL, reqInfo, bktInfo.CID) chainRules := bucketCannedACLToAPERules(cannedACL, reqInfo, bktInfo.CID)
if err = h.ape.SaveACLChains(bktInfo.CID.EncodeToString(), chainRules); err != nil { if err = h.ape.SaveACLChains(bktInfo.CID.EncodeToString(), chainRules); err != nil {
h.logAndSendError(w, "failed to add morph rule chains", reqInfo, err) h.logAndSendError(ctx, w, "failed to add morph rule chains", reqInfo, err)
return return
} }
@ -184,7 +185,7 @@ func (h *handler) putBucketACLAPEHandler(w http.ResponseWriter, r *http.Request,
} }
if err = h.obj.PutBucketSettings(ctx, sp); err != nil { if err = h.obj.PutBucketSettings(ctx, sp); err != nil {
h.logAndSendError(w, "couldn't save bucket settings", reqInfo, err, h.logAndSendError(ctx, w, "couldn't save bucket settings", reqInfo, err,
zap.String("container_id", bktInfo.CID.EncodeToString())) zap.String("container_id", bktInfo.CID.EncodeToString()))
return return
} }
@ -198,18 +199,18 @@ func (h *handler) GetObjectACLHandler(w http.ResponseWriter, r *http.Request) {
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
settings, err := h.obj.GetBucketSettings(r.Context(), bktInfo) settings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "couldn't get bucket settings", reqInfo, err)
return return
} }
if err = middleware.EncodeToResponse(w, h.encodePrivateCannedACL(ctx, bktInfo, settings)); err != nil { if err = middleware.EncodeToResponse(w, h.encodePrivateCannedACL(ctx, bktInfo, settings)); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
return return
} }
} }
@ -219,19 +220,20 @@ func (h *handler) PutObjectACLHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(ctx) reqInfo := middleware.GetReqInfo(ctx)
if _, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName); err != nil { if _, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName); err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
h.logAndSendError(w, "acl not supported for this bucket", reqInfo, errors.GetAPIError(errors.ErrAccessControlListNotSupported)) h.logAndSendError(ctx, w, "acl not supported for this bucket", reqInfo, errors.GetAPIError(errors.ErrAccessControlListNotSupported))
} }
func (h *handler) GetBucketPolicyStatusHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketPolicyStatusHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
@ -240,13 +242,13 @@ func (h *handler) GetBucketPolicyStatusHandler(w http.ResponseWriter, r *http.Re
if strings.Contains(err.Error(), "not found") { if strings.Contains(err.Error(), "not found") {
err = fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrNoSuchBucketPolicy), err.Error()) err = fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrNoSuchBucketPolicy), err.Error())
} }
h.logAndSendError(w, "failed to get policy from storage", reqInfo, err) h.logAndSendError(ctx, w, "failed to get policy from storage", reqInfo, err)
return return
} }
var bktPolicy engineiam.Policy var bktPolicy engineiam.Policy
if err = json.Unmarshal(jsonPolicy, &bktPolicy); err != nil { if err = json.Unmarshal(jsonPolicy, &bktPolicy); err != nil {
h.logAndSendError(w, "could not parse bucket policy", reqInfo, err) h.logAndSendError(ctx, w, "could not parse bucket policy", reqInfo, err)
return return
} }
@ -263,17 +265,18 @@ func (h *handler) GetBucketPolicyStatusHandler(w http.ResponseWriter, r *http.Re
} }
if err = middleware.EncodeToResponse(w, policyStatus); err != nil { if err = middleware.EncodeToResponse(w, policyStatus); err != nil {
h.logAndSendError(w, "encode and write response", reqInfo, err) h.logAndSendError(ctx, w, "encode and write response", reqInfo, err)
return return
} }
} }
func (h *handler) GetBucketPolicyHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
@ -282,7 +285,7 @@ func (h *handler) GetBucketPolicyHandler(w http.ResponseWriter, r *http.Request)
if strings.Contains(err.Error(), "not found") { if strings.Contains(err.Error(), "not found") {
err = fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrNoSuchBucketPolicy), err.Error()) err = fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrNoSuchBucketPolicy), err.Error())
} }
h.logAndSendError(w, "failed to get policy from storage", reqInfo, err) h.logAndSendError(ctx, w, "failed to get policy from storage", reqInfo, err)
return return
} }
@ -290,22 +293,23 @@ func (h *handler) GetBucketPolicyHandler(w http.ResponseWriter, r *http.Request)
w.WriteHeader(http.StatusOK) w.WriteHeader(http.StatusOK)
if _, err = w.Write(jsonPolicy); err != nil { if _, err = w.Write(jsonPolicy); err != nil {
h.logAndSendError(w, "write json policy to client", reqInfo, err) h.logAndSendError(ctx, w, "write json policy to client", reqInfo, err)
} }
} }
func (h *handler) DeleteBucketPolicyHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) DeleteBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
chainIDs := []chain.ID{getBucketChainID(chain.S3, bktInfo), getBucketChainID(chain.Ingress, bktInfo)} chainIDs := []chain.ID{getBucketChainID(chain.S3, bktInfo), getBucketChainID(chain.Ingress, bktInfo)}
if err = h.ape.DeleteBucketPolicy(reqInfo.Namespace, bktInfo.CID, chainIDs); err != nil { if err = h.ape.DeleteBucketPolicy(reqInfo.Namespace, bktInfo.CID, chainIDs); err != nil {
h.logAndSendError(w, "failed to delete policy from storage", reqInfo, err) h.logAndSendError(ctx, w, "failed to delete policy from storage", reqInfo, err)
return return
} }
} }
@ -324,40 +328,41 @@ func checkOwner(info *data.BucketInfo, owner string) error {
} }
func (h *handler) PutBucketPolicyHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) PutBucketPolicyHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
jsonPolicy, err := io.ReadAll(r.Body) jsonPolicy, err := io.ReadAll(r.Body)
if err != nil { if err != nil {
h.logAndSendError(w, "read body", reqInfo, err) h.logAndSendError(ctx, w, "read body", reqInfo, err)
return return
} }
var bktPolicy engineiam.Policy var bktPolicy engineiam.Policy
if err = json.Unmarshal(jsonPolicy, &bktPolicy); err != nil { if err = json.Unmarshal(jsonPolicy, &bktPolicy); err != nil {
h.logAndSendError(w, "could not parse bucket policy", reqInfo, err) h.logAndSendError(ctx, w, "could not parse bucket policy", reqInfo, err)
return return
} }
for _, stat := range bktPolicy.Statement { for _, stat := range bktPolicy.Statement {
if len(stat.NotResource) != 0 { if len(stat.NotResource) != 0 {
h.logAndSendError(w, "policy resource mismatched bucket", reqInfo, errors.GetAPIError(errors.ErrMalformedPolicy)) h.logAndSendError(ctx, w, "policy resource mismatched bucket", reqInfo, errors.GetAPIError(errors.ErrMalformedPolicy))
return return
} }
if len(stat.NotPrincipal) != 0 && stat.Effect == engineiam.AllowEffect { if len(stat.NotPrincipal) != 0 && stat.Effect == engineiam.AllowEffect {
h.logAndSendError(w, "invalid NotPrincipal", reqInfo, errors.GetAPIError(errors.ErrMalformedPolicyNotPrincipal)) h.logAndSendError(ctx, w, "invalid NotPrincipal", reqInfo, errors.GetAPIError(errors.ErrMalformedPolicyNotPrincipal))
return return
} }
for _, resource := range stat.Resource { for _, resource := range stat.Resource {
if reqInfo.BucketName != strings.Split(strings.TrimPrefix(resource, arnAwsPrefix), "/")[0] { if reqInfo.BucketName != strings.Split(strings.TrimPrefix(resource, arnAwsPrefix), "/")[0] {
h.logAndSendError(w, "policy resource mismatched bucket", reqInfo, errors.GetAPIError(errors.ErrMalformedPolicy)) h.logAndSendError(ctx, w, "policy resource mismatched bucket", reqInfo, errors.GetAPIError(errors.ErrMalformedPolicy))
return return
} }
} }
@ -365,7 +370,7 @@ func (h *handler) PutBucketPolicyHandler(w http.ResponseWriter, r *http.Request)
s3Chain, err := engineiam.ConvertToS3Chain(bktPolicy, h.frostfsid) s3Chain, err := engineiam.ConvertToS3Chain(bktPolicy, h.frostfsid)
if err != nil { if err != nil {
h.logAndSendError(w, "could not convert s3 policy to chain policy", reqInfo, err) h.logAndSendError(ctx, w, "could not convert s3 policy to chain policy", reqInfo, err)
return return
} }
s3Chain.ID = getBucketChainID(chain.S3, bktInfo) s3Chain.ID = getBucketChainID(chain.S3, bktInfo)
@ -374,10 +379,10 @@ func (h *handler) PutBucketPolicyHandler(w http.ResponseWriter, r *http.Request)
if err == nil { if err == nil {
nativeChain.ID = getBucketChainID(chain.Ingress, bktInfo) nativeChain.ID = getBucketChainID(chain.Ingress, bktInfo)
} else if !stderrors.Is(err, engineiam.ErrActionsNotApplicable) { } else if !stderrors.Is(err, engineiam.ErrActionsNotApplicable) {
h.logAndSendError(w, "could not convert s3 policy to native chain policy", reqInfo, err) h.logAndSendError(ctx, w, "could not convert s3 policy to native chain policy", reqInfo, err)
return return
} else { } else {
h.reqLogger(r.Context()).Warn(logs.PolicyCouldntBeConvertedToNativeRules) h.reqLogger(ctx).Warn(logs.PolicyCouldntBeConvertedToNativeRules)
} }
chainsToSave := []*chain.Chain{s3Chain} chainsToSave := []*chain.Chain{s3Chain}
@ -386,7 +391,7 @@ func (h *handler) PutBucketPolicyHandler(w http.ResponseWriter, r *http.Request)
} }
if err = h.ape.PutBucketPolicy(reqInfo.Namespace, bktInfo.CID, jsonPolicy, chainsToSave); err != nil { if err = h.ape.PutBucketPolicy(reqInfo.Namespace, bktInfo.CID, jsonPolicy, chainsToSave); err != nil {
h.logAndSendError(w, "failed to update policy in contract", reqInfo, err) h.logAndSendError(ctx, w, "failed to update policy in contract", reqInfo, err)
return return
} }
} }

View file

@ -11,7 +11,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
@ -28,12 +28,12 @@ func TestPutObjectACLErrorAPE(t *testing.T) {
info := createBucket(hc, bktName) info := createBucket(hc, bktName)
putObjectWithHeadersAssertS3Error(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPublic}, s3errors.ErrAccessControlListNotSupported) putObjectWithHeadersAssertS3Error(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPublic}, apierr.ErrAccessControlListNotSupported)
putObjectWithHeaders(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPrivate}) // only `private` canned acl is allowed, that is actually ignored putObjectWithHeaders(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPrivate}) // only `private` canned acl is allowed, that is actually ignored
putObjectWithHeaders(hc, bktName, objName, nil) putObjectWithHeaders(hc, bktName, objName, nil)
aclBody := &AccessControlPolicy{} aclBody := &AccessControlPolicy{}
putObjectACLAssertS3Error(hc, bktName, objName, info.Box, nil, aclBody, s3errors.ErrAccessControlListNotSupported) putObjectACLAssertS3Error(hc, bktName, objName, info.Box, nil, aclBody, apierr.ErrAccessControlListNotSupported)
aclRes := getObjectACL(hc, bktName, objName) aclRes := getObjectACL(hc, bktName, objName)
checkPrivateACL(t, aclRes, info.Key.PublicKey()) checkPrivateACL(t, aclRes, info.Key.PublicKey())
@ -49,7 +49,7 @@ func TestCreateObjectACLErrorAPE(t *testing.T) {
copyObject(hc, bktName, objName, objNameCopy, CopyMeta{Headers: map[string]string{api.AmzACL: basicACLPublic}}, http.StatusBadRequest) copyObject(hc, bktName, objName, objNameCopy, CopyMeta{Headers: map[string]string{api.AmzACL: basicACLPublic}}, http.StatusBadRequest)
copyObject(hc, bktName, objName, objNameCopy, CopyMeta{Headers: map[string]string{api.AmzACL: basicACLPrivate}}, http.StatusOK) copyObject(hc, bktName, objName, objNameCopy, CopyMeta{Headers: map[string]string{api.AmzACL: basicACLPrivate}}, http.StatusOK)
createMultipartUploadAssertS3Error(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPublic}, s3errors.ErrAccessControlListNotSupported) createMultipartUploadAssertS3Error(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPublic}, apierr.ErrAccessControlListNotSupported)
createMultipartUpload(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPrivate}) createMultipartUpload(hc, bktName, objName, map[string]string{api.AmzACL: basicACLPrivate})
} }
@ -60,7 +60,7 @@ func TestBucketACLAPE(t *testing.T) {
info := createBucket(hc, bktName) info := createBucket(hc, bktName)
aclBody := &AccessControlPolicy{} aclBody := &AccessControlPolicy{}
putBucketACLAssertS3Error(hc, bktName, info.Box, nil, aclBody, s3errors.ErrAccessControlListNotSupported) putBucketACLAssertS3Error(hc, bktName, info.Box, nil, aclBody, apierr.ErrAccessControlListNotSupported)
aclRes := getBucketACL(hc, bktName) aclRes := getBucketACL(hc, bktName)
checkPrivateACL(t, aclRes, info.Key.PublicKey()) checkPrivateACL(t, aclRes, info.Key.PublicKey())
@ -113,7 +113,7 @@ func TestBucketPolicy(t *testing.T) {
createTestBucket(hc, bktName) createTestBucket(hc, bktName)
getBucketPolicy(hc, bktName, s3errors.ErrNoSuchBucketPolicy) getBucketPolicy(hc, bktName, apierr.ErrNoSuchBucketPolicy)
newPolicy := engineiam.Policy{ newPolicy := engineiam.Policy{
Version: "2012-10-17", Version: "2012-10-17",
@ -125,7 +125,7 @@ func TestBucketPolicy(t *testing.T) {
}}, }},
} }
putBucketPolicy(hc, bktName, newPolicy, s3errors.ErrMalformedPolicy) putBucketPolicy(hc, bktName, newPolicy, apierr.ErrMalformedPolicy)
newPolicy.Statement[0].Resource[0] = arnAwsPrefix + bktName + "/*" newPolicy.Statement[0].Resource[0] = arnAwsPrefix + bktName + "/*"
putBucketPolicy(hc, bktName, newPolicy) putBucketPolicy(hc, bktName, newPolicy)
@ -140,7 +140,7 @@ func TestBucketPolicyStatus(t *testing.T) {
createTestBucket(hc, bktName) createTestBucket(hc, bktName)
getBucketPolicy(hc, bktName, s3errors.ErrNoSuchBucketPolicy) getBucketPolicy(hc, bktName, apierr.ErrNoSuchBucketPolicy)
newPolicy := engineiam.Policy{ newPolicy := engineiam.Policy{
Version: "2012-10-17", Version: "2012-10-17",
@ -152,7 +152,7 @@ func TestBucketPolicyStatus(t *testing.T) {
}}, }},
} }
putBucketPolicy(hc, bktName, newPolicy, s3errors.ErrMalformedPolicyNotPrincipal) putBucketPolicy(hc, bktName, newPolicy, apierr.ErrMalformedPolicyNotPrincipal)
newPolicy.Statement[0].NotPrincipal = nil newPolicy.Statement[0].NotPrincipal = nil
newPolicy.Statement[0].Principal = map[engineiam.PrincipalType][]string{engineiam.Wildcard: {}} newPolicy.Statement[0].Principal = map[engineiam.PrincipalType][]string{engineiam.Wildcard: {}}
@ -191,6 +191,7 @@ func TestDeleteBucketWithPolicy(t *testing.T) {
require.Len(t, hc.h.ape.(*apeMock).policyMap, 1) require.Len(t, hc.h.ape.(*apeMock).policyMap, 1)
require.Len(t, hc.h.ape.(*apeMock).chainMap[engine.ContainerTarget(bi.CID.EncodeToString())], 4) require.Len(t, hc.h.ape.(*apeMock).chainMap[engine.ContainerTarget(bi.CID.EncodeToString())], 4)
hc.owner = bi.Owner
deleteBucket(t, hc, bktName, http.StatusNoContent) deleteBucket(t, hc, bktName, http.StatusNoContent)
require.Empty(t, hc.h.ape.(*apeMock).policyMap) require.Empty(t, hc.h.ape.(*apeMock).policyMap)
@ -221,7 +222,7 @@ func TestPutBucketPolicy(t *testing.T) {
assertStatus(hc.t, w, http.StatusOK) assertStatus(hc.t, w, http.StatusOK)
} }
func getBucketPolicy(hc *handlerContext, bktName string, errCode ...s3errors.ErrorCode) engineiam.Policy { func getBucketPolicy(hc *handlerContext, bktName string, errCode ...apierr.ErrorCode) engineiam.Policy {
w, r := prepareTestRequest(hc, bktName, "", nil) w, r := prepareTestRequest(hc, bktName, "", nil)
hc.Handler().GetBucketPolicyHandler(w, r) hc.Handler().GetBucketPolicyHandler(w, r)
@ -231,13 +232,13 @@ func getBucketPolicy(hc *handlerContext, bktName string, errCode ...s3errors.Err
err := json.NewDecoder(w.Result().Body).Decode(&policy) err := json.NewDecoder(w.Result().Body).Decode(&policy)
require.NoError(hc.t, err) require.NoError(hc.t, err)
} else { } else {
assertS3Error(hc.t, w, s3errors.GetAPIError(errCode[0])) assertS3Error(hc.t, w, apierr.GetAPIError(errCode[0]))
} }
return policy return policy
} }
func getBucketPolicyStatus(hc *handlerContext, bktName string, errCode ...s3errors.ErrorCode) PolicyStatus { func getBucketPolicyStatus(hc *handlerContext, bktName string, errCode ...apierr.ErrorCode) PolicyStatus {
w, r := prepareTestRequest(hc, bktName, "", nil) w, r := prepareTestRequest(hc, bktName, "", nil)
hc.Handler().GetBucketPolicyStatusHandler(w, r) hc.Handler().GetBucketPolicyStatusHandler(w, r)
@ -247,13 +248,13 @@ func getBucketPolicyStatus(hc *handlerContext, bktName string, errCode ...s3erro
err := xml.NewDecoder(w.Result().Body).Decode(&policyStatus) err := xml.NewDecoder(w.Result().Body).Decode(&policyStatus)
require.NoError(hc.t, err) require.NoError(hc.t, err)
} else { } else {
assertS3Error(hc.t, w, s3errors.GetAPIError(errCode[0])) assertS3Error(hc.t, w, apierr.GetAPIError(errCode[0]))
} }
return policyStatus return policyStatus
} }
func putBucketPolicy(hc *handlerContext, bktName string, bktPolicy engineiam.Policy, errCode ...s3errors.ErrorCode) { func putBucketPolicy(hc *handlerContext, bktName string, bktPolicy engineiam.Policy, errCode ...apierr.ErrorCode) {
body, err := json.Marshal(bktPolicy) body, err := json.Marshal(bktPolicy)
require.NoError(hc.t, err) require.NoError(hc.t, err)
@ -263,7 +264,7 @@ func putBucketPolicy(hc *handlerContext, bktName string, bktPolicy engineiam.Pol
if len(errCode) == 0 { if len(errCode) == 0 {
assertStatus(hc.t, w, http.StatusOK) assertStatus(hc.t, w, http.StatusOK)
} else { } else {
assertS3Error(hc.t, w, s3errors.GetAPIError(errCode[0])) assertS3Error(hc.t, w, apierr.GetAPIError(errCode[0]))
} }
} }
@ -312,9 +313,9 @@ func createBucket(hc *handlerContext, bktName string) *createBucketInfo {
} }
} }
func createBucketAssertS3Error(hc *handlerContext, bktName string, box *accessbox.Box, code s3errors.ErrorCode) { func createBucketAssertS3Error(hc *handlerContext, bktName string, box *accessbox.Box, code apierr.ErrorCode) {
w := createBucketBase(hc, bktName, box) w := createBucketBase(hc, bktName, box)
assertS3Error(hc.t, w, s3errors.GetAPIError(code)) assertS3Error(hc.t, w, apierr.GetAPIError(code))
} }
func createBucketBase(hc *handlerContext, bktName string, box *accessbox.Box) *httptest.ResponseRecorder { func createBucketBase(hc *handlerContext, bktName string, box *accessbox.Box) *httptest.ResponseRecorder {
@ -330,9 +331,9 @@ func putBucketACL(hc *handlerContext, bktName string, box *accessbox.Box, header
assertStatus(hc.t, w, http.StatusOK) assertStatus(hc.t, w, http.StatusOK)
} }
func putBucketACLAssertS3Error(hc *handlerContext, bktName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy, code s3errors.ErrorCode) { func putBucketACLAssertS3Error(hc *handlerContext, bktName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy, code apierr.ErrorCode) {
w := putBucketACLBase(hc, bktName, box, header, body) w := putBucketACLBase(hc, bktName, box, header, body)
assertS3Error(hc.t, w, s3errors.GetAPIError(code)) assertS3Error(hc.t, w, apierr.GetAPIError(code))
} }
func putBucketACLBase(hc *handlerContext, bktName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy) *httptest.ResponseRecorder { func putBucketACLBase(hc *handlerContext, bktName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy) *httptest.ResponseRecorder {
@ -360,9 +361,9 @@ func getBucketACLBase(hc *handlerContext, bktName string) *httptest.ResponseReco
return w return w
} }
func putObjectACLAssertS3Error(hc *handlerContext, bktName, objName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy, code s3errors.ErrorCode) { func putObjectACLAssertS3Error(hc *handlerContext, bktName, objName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy, code apierr.ErrorCode) {
w := putObjectACLBase(hc, bktName, objName, box, header, body) w := putObjectACLBase(hc, bktName, objName, box, header, body)
assertS3Error(hc.t, w, s3errors.GetAPIError(code)) assertS3Error(hc.t, w, apierr.GetAPIError(code))
} }
func putObjectACLBase(hc *handlerContext, bktName, objName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy) *httptest.ResponseRecorder { func putObjectACLBase(hc *handlerContext, bktName, objName string, box *accessbox.Box, header map[string]string, body *AccessControlPolicy) *httptest.ResponseRecorder {
@ -396,9 +397,9 @@ func putObjectWithHeaders(hc *handlerContext, bktName, objName string, headers m
return w.Header() return w.Header()
} }
func putObjectWithHeadersAssertS3Error(hc *handlerContext, bktName, objName string, headers map[string]string, code s3errors.ErrorCode) { func putObjectWithHeadersAssertS3Error(hc *handlerContext, bktName, objName string, headers map[string]string, code apierr.ErrorCode) {
w := putObjectWithHeadersBase(hc, bktName, objName, headers, nil, nil) w := putObjectWithHeadersBase(hc, bktName, objName, headers, nil, nil)
assertS3Error(hc.t, w, s3errors.GetAPIError(code)) assertS3Error(hc.t, w, apierr.GetAPIError(code))
} }
func putObjectWithHeadersBase(hc *handlerContext, bktName, objName string, headers map[string]string, box *accessbox.Box, data []byte) *httptest.ResponseRecorder { func putObjectWithHeadersBase(hc *handlerContext, bktName, objName string, headers map[string]string, box *accessbox.Box, data []byte) *httptest.ResponseRecorder {

View file

@ -70,17 +70,18 @@ var validAttributes = map[string]struct{}{
} }
func (h *handler) GetObjectAttributesHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetObjectAttributesHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
params, err := parseGetObjectAttributeArgs(r) params, err := parseGetObjectAttributeArgs(r)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid request", reqInfo, err) h.logAndSendError(ctx, w, "invalid request", reqInfo, err)
return return
} }
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
@ -90,44 +91,44 @@ func (h *handler) GetObjectAttributesHandler(w http.ResponseWriter, r *http.Requ
VersionID: params.VersionID, VersionID: params.VersionID,
} }
extendedInfo, err := h.obj.GetExtendedObjectInfo(r.Context(), p) extendedInfo, err := h.obj.GetExtendedObjectInfo(ctx, p)
if err != nil { if err != nil {
h.logAndSendError(w, "could not fetch object info", reqInfo, err) h.logAndSendError(ctx, w, "could not fetch object info", reqInfo, err)
return return
} }
info := extendedInfo.ObjectInfo info := extendedInfo.ObjectInfo
encryptionParams, err := formEncryptionParams(r) encryptionParams, err := formEncryptionParams(r)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid sse headers", reqInfo, err) h.logAndSendError(ctx, w, "invalid sse headers", reqInfo, err)
return return
} }
if err = encryptionParams.MatchObjectEncryption(layer.FormEncryptionInfo(info.Headers)); err != nil { if err = encryptionParams.MatchObjectEncryption(layer.FormEncryptionInfo(info.Headers)); err != nil {
h.logAndSendError(w, "encryption doesn't match object", reqInfo, errors.GetAPIError(errors.ErrBadRequest), zap.Error(err)) h.logAndSendError(ctx, w, "encryption doesn't match object", reqInfo, errors.GetAPIError(errors.ErrBadRequest), zap.Error(err))
return return
} }
if err = checkPreconditions(info, params.Conditional, h.cfg.MD5Enabled()); err != nil { if err = checkPreconditions(info, params.Conditional, h.cfg.MD5Enabled()); err != nil {
h.logAndSendError(w, "precondition failed", reqInfo, err) h.logAndSendError(ctx, w, "precondition failed", reqInfo, err)
return return
} }
bktSettings, err := h.obj.GetBucketSettings(r.Context(), bktInfo) bktSettings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket settings", reqInfo, err)
return return
} }
response, err := encodeToObjectAttributesResponse(info, params, h.cfg.MD5Enabled()) response, err := encodeToObjectAttributesResponse(info, params, h.cfg.MD5Enabled())
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't encode object info to response", reqInfo, err) h.logAndSendError(ctx, w, "couldn't encode object info to response", reqInfo, err)
return return
} }
writeAttributesHeaders(w.Header(), extendedInfo, bktSettings.Unversioned()) writeAttributesHeaders(w.Header(), extendedInfo, bktSettings.Unversioned())
if err = middleware.EncodeToResponse(w, response); err != nil { if err = middleware.EncodeToResponse(w, response); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
} }
} }

View file

@ -65,7 +65,7 @@ func (h *handler) CopyObjectHandler(w http.ResponseWriter, r *http.Request) {
srcBucket, srcObject, err := path2BucketObject(src) srcBucket, srcObject, err := path2BucketObject(src)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid source copy", reqInfo, err) h.logAndSendError(ctx, w, "invalid source copy", reqInfo, err)
return return
} }
@ -75,74 +75,74 @@ func (h *handler) CopyObjectHandler(w http.ResponseWriter, r *http.Request) {
} }
if srcObjPrm.BktInfo, err = h.getBucketAndCheckOwner(r, srcBucket, api.AmzSourceExpectedBucketOwner); err != nil { if srcObjPrm.BktInfo, err = h.getBucketAndCheckOwner(r, srcBucket, api.AmzSourceExpectedBucketOwner); err != nil {
h.logAndSendError(w, "couldn't get source bucket", reqInfo, err) h.logAndSendError(ctx, w, "couldn't get source bucket", reqInfo, err)
return return
} }
dstBktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) dstBktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't get target bucket", reqInfo, err) h.logAndSendError(ctx, w, "couldn't get target bucket", reqInfo, err)
return return
} }
settings, err := h.obj.GetBucketSettings(ctx, dstBktInfo) settings, err := h.obj.GetBucketSettings(ctx, dstBktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket settings", reqInfo, err)
return return
} }
if cannedACLStatus == aclStatusYes { if cannedACLStatus == aclStatusYes {
h.logAndSendError(w, "acl not supported for this bucket", reqInfo, errors.GetAPIError(errors.ErrAccessControlListNotSupported)) h.logAndSendError(ctx, w, "acl not supported for this bucket", reqInfo, errors.GetAPIError(errors.ErrAccessControlListNotSupported))
return return
} }
extendedSrcObjInfo, err := h.obj.GetExtendedObjectInfo(ctx, srcObjPrm) extendedSrcObjInfo, err := h.obj.GetExtendedObjectInfo(ctx, srcObjPrm)
if err != nil { if err != nil {
h.logAndSendError(w, "could not find object", reqInfo, err) h.logAndSendError(ctx, w, "could not find object", reqInfo, err)
return return
} }
srcObjInfo := extendedSrcObjInfo.ObjectInfo srcObjInfo := extendedSrcObjInfo.ObjectInfo
srcEncryptionParams, err := formCopySourceEncryptionParams(r) srcEncryptionParams, err := formCopySourceEncryptionParams(r)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid sse headers", reqInfo, err) h.logAndSendError(ctx, w, "invalid sse headers", reqInfo, err)
return return
} }
dstEncryptionParams, err := formEncryptionParams(r) dstEncryptionParams, err := formEncryptionParams(r)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid sse headers", reqInfo, err) h.logAndSendError(ctx, w, "invalid sse headers", reqInfo, err)
return return
} }
if err = srcEncryptionParams.MatchObjectEncryption(layer.FormEncryptionInfo(srcObjInfo.Headers)); err != nil { if err = srcEncryptionParams.MatchObjectEncryption(layer.FormEncryptionInfo(srcObjInfo.Headers)); err != nil {
if errors.IsS3Error(err, errors.ErrInvalidEncryptionParameters) || errors.IsS3Error(err, errors.ErrSSEEncryptedObject) || if errors.IsS3Error(err, errors.ErrInvalidEncryptionParameters) || errors.IsS3Error(err, errors.ErrSSEEncryptedObject) ||
errors.IsS3Error(err, errors.ErrInvalidSSECustomerParameters) { errors.IsS3Error(err, errors.ErrInvalidSSECustomerParameters) {
h.logAndSendError(w, "encryption doesn't match object", reqInfo, err, zap.Error(err)) h.logAndSendError(ctx, w, "encryption doesn't match object", reqInfo, err, zap.Error(err))
return return
} }
h.logAndSendError(w, "encryption doesn't match object", reqInfo, errors.GetAPIError(errors.ErrBadRequest), zap.Error(err)) h.logAndSendError(ctx, w, "encryption doesn't match object", reqInfo, errors.GetAPIError(errors.ErrBadRequest), zap.Error(err))
return return
} }
var dstSize uint64 var dstSize uint64
srcSize, err := layer.GetObjectSize(srcObjInfo) srcSize, err := layer.GetObjectSize(srcObjInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "failed to get source object size", reqInfo, err) h.logAndSendError(ctx, w, "failed to get source object size", reqInfo, err)
return return
} else if srcSize > layer.UploadMaxSize { // https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html } else if srcSize > layer.UploadMaxSize { // https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
h.logAndSendError(w, "too bid object to copy with single copy operation, use multipart upload copy instead", reqInfo, errors.GetAPIError(errors.ErrInvalidRequestLargeCopy)) h.logAndSendError(ctx, w, "too bid object to copy with single copy operation, use multipart upload copy instead", reqInfo, errors.GetAPIError(errors.ErrInvalidRequestLargeCopy))
return return
} }
dstSize = srcSize dstSize = srcSize
args, err := parseCopyObjectArgs(r.Header) args, err := parseCopyObjectArgs(r.Header)
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse request params", reqInfo, err) h.logAndSendError(ctx, w, "could not parse request params", reqInfo, err)
return return
} }
if isCopyingToItselfForbidden(reqInfo, srcBucket, srcObject, settings, args) { if isCopyingToItselfForbidden(reqInfo, srcBucket, srcObject, settings, args) {
h.logAndSendError(w, "copying to itself without changing anything", reqInfo, errors.GetAPIError(errors.ErrInvalidCopyDest)) h.logAndSendError(ctx, w, "copying to itself without changing anything", reqInfo, errors.GetAPIError(errors.ErrInvalidCopyDest))
return return
} }
@ -153,7 +153,7 @@ func (h *handler) CopyObjectHandler(w http.ResponseWriter, r *http.Request) {
if args.TaggingDirective == replaceDirective { if args.TaggingDirective == replaceDirective {
tagSet, err = parseTaggingHeader(r.Header) tagSet, err = parseTaggingHeader(r.Header)
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse tagging header", reqInfo, err) h.logAndSendError(ctx, w, "could not parse tagging header", reqInfo, err)
return return
} }
} else { } else {
@ -168,13 +168,13 @@ func (h *handler) CopyObjectHandler(w http.ResponseWriter, r *http.Request) {
_, tagSet, err = h.obj.GetObjectTagging(ctx, tagPrm) _, tagSet, err = h.obj.GetObjectTagging(ctx, tagPrm)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get object tagging", reqInfo, err) h.logAndSendError(ctx, w, "could not get object tagging", reqInfo, err)
return return
} }
} }
if err = checkPreconditions(srcObjInfo, args.Conditional, h.cfg.MD5Enabled()); err != nil { if err = checkPreconditions(srcObjInfo, args.Conditional, h.cfg.MD5Enabled()); err != nil {
h.logAndSendError(w, "precondition failed", reqInfo, errors.GetAPIError(errors.ErrPreconditionFailed)) h.logAndSendError(ctx, w, "precondition failed", reqInfo, errors.GetAPIError(errors.ErrPreconditionFailed))
return return
} }
@ -202,20 +202,20 @@ func (h *handler) CopyObjectHandler(w http.ResponseWriter, r *http.Request) {
params.CopiesNumbers, err = h.pickCopiesNumbers(metadata, reqInfo.Namespace, dstBktInfo.LocationConstraint) params.CopiesNumbers, err = h.pickCopiesNumbers(metadata, reqInfo.Namespace, dstBktInfo.LocationConstraint)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid copies number", reqInfo, err) h.logAndSendError(ctx, w, "invalid copies number", reqInfo, err)
return return
} }
params.Lock, err = formObjectLock(ctx, dstBktInfo, settings.LockConfiguration, r.Header) params.Lock, err = formObjectLock(ctx, dstBktInfo, settings.LockConfiguration, r.Header)
if err != nil { if err != nil {
h.logAndSendError(w, "could not form object lock", reqInfo, err) h.logAndSendError(ctx, w, "could not form object lock", reqInfo, err)
return return
} }
additional := []zap.Field{zap.String("src_bucket_name", srcBucket), zap.String("src_object_name", srcObject)} additional := []zap.Field{zap.String("src_bucket_name", srcBucket), zap.String("src_object_name", srcObject)}
extendedDstObjInfo, err := h.obj.CopyObject(ctx, params) extendedDstObjInfo, err := h.obj.CopyObject(ctx, params)
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't copy object", reqInfo, err, additional...) h.logAndSendError(ctx, w, "couldn't copy object", reqInfo, err, additional...)
return return
} }
dstObjInfo := extendedDstObjInfo.ObjectInfo dstObjInfo := extendedDstObjInfo.ObjectInfo
@ -224,7 +224,7 @@ func (h *handler) CopyObjectHandler(w http.ResponseWriter, r *http.Request) {
LastModified: dstObjInfo.Created.UTC().Format(time.RFC3339), LastModified: dstObjInfo.Created.UTC().Format(time.RFC3339),
ETag: data.Quote(dstObjInfo.ETag(h.cfg.MD5Enabled())), ETag: data.Quote(dstObjInfo.ETag(h.cfg.MD5Enabled())),
}); err != nil { }); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err, additional...) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err, additional...)
return return
} }
@ -239,7 +239,7 @@ func (h *handler) CopyObjectHandler(w http.ResponseWriter, r *http.Request) {
NodeVersion: extendedDstObjInfo.NodeVersion, NodeVersion: extendedDstObjInfo.NodeVersion,
} }
if err = h.obj.PutObjectTagging(ctx, tagPrm); err != nil { if err = h.obj.PutObjectTagging(ctx, tagPrm); err != nil {
h.logAndSendError(w, "could not upload object tagging", reqInfo, err) h.logAndSendError(ctx, w, "could not upload object tagging", reqInfo, err)
return return
} }
} }

View file

@ -20,32 +20,34 @@ const (
) )
func (h *handler) GetBucketCorsHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketCorsHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
cors, err := h.obj.GetBucketCORS(r.Context(), bktInfo) cors, err := h.obj.GetBucketCORS(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get cors", reqInfo, err) h.logAndSendError(ctx, w, "could not get cors", reqInfo, err)
return return
} }
if err = middleware.EncodeToResponse(w, cors); err != nil { if err = middleware.EncodeToResponse(w, cors); err != nil {
h.logAndSendError(w, "could not encode cors to response", reqInfo, err) h.logAndSendError(ctx, w, "could not encode cors to response", reqInfo, err)
return return
} }
} }
func (h *handler) PutBucketCorsHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) PutBucketCorsHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
@ -57,32 +59,33 @@ func (h *handler) PutBucketCorsHandler(w http.ResponseWriter, r *http.Request) {
p.CopiesNumbers, err = h.pickCopiesNumbers(parseMetadata(r), reqInfo.Namespace, bktInfo.LocationConstraint) p.CopiesNumbers, err = h.pickCopiesNumbers(parseMetadata(r), reqInfo.Namespace, bktInfo.LocationConstraint)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid copies number", reqInfo, err) h.logAndSendError(ctx, w, "invalid copies number", reqInfo, err)
return return
} }
if err = h.obj.PutBucketCORS(r.Context(), p); err != nil { if err = h.obj.PutBucketCORS(ctx, p); err != nil {
h.logAndSendError(w, "could not put cors configuration", reqInfo, err) h.logAndSendError(ctx, w, "could not put cors configuration", reqInfo, err)
return return
} }
if err = middleware.WriteSuccessResponseHeadersOnly(w); err != nil { if err = middleware.WriteSuccessResponseHeadersOnly(w); err != nil {
h.logAndSendError(w, "write response", reqInfo, err) h.logAndSendError(ctx, w, "write response", reqInfo, err)
return return
} }
} }
func (h *handler) DeleteBucketCorsHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) DeleteBucketCorsHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
if err = h.obj.DeleteBucketCORS(r.Context(), bktInfo); err != nil { if err = h.obj.DeleteBucketCORS(ctx, bktInfo); err != nil {
h.logAndSendError(w, "could not delete cors", reqInfo, err) h.logAndSendError(ctx, w, "could not delete cors", reqInfo, err)
} }
w.WriteHeader(http.StatusNoContent) w.WriteHeader(http.StatusNoContent)
@ -102,7 +105,7 @@ func (h *handler) AppendCORSHeaders(w http.ResponseWriter, r *http.Request) {
if reqInfo.BucketName == "" { if reqInfo.BucketName == "" {
return return
} }
bktInfo, err := h.obj.GetBucketInfo(ctx, reqInfo.BucketName) bktInfo, err := h.getBucketInfo(ctx, reqInfo.BucketName)
if err != nil { if err != nil {
h.reqLogger(ctx).Warn(logs.GetBucketInfo, zap.Error(err)) h.reqLogger(ctx).Warn(logs.GetBucketInfo, zap.Error(err))
return return
@ -149,21 +152,22 @@ func (h *handler) AppendCORSHeaders(w http.ResponseWriter, r *http.Request) {
} }
func (h *handler) Preflight(w http.ResponseWriter, r *http.Request) { func (h *handler) Preflight(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
bktInfo, err := h.obj.GetBucketInfo(r.Context(), reqInfo.BucketName) reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketInfo(ctx, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
origin := r.Header.Get(api.Origin) origin := r.Header.Get(api.Origin)
if origin == "" { if origin == "" {
h.logAndSendError(w, "origin request header needed", reqInfo, errors.GetAPIError(errors.ErrBadRequest)) h.logAndSendError(ctx, w, "origin request header needed", reqInfo, errors.GetAPIError(errors.ErrBadRequest))
} }
method := r.Header.Get(api.AccessControlRequestMethod) method := r.Header.Get(api.AccessControlRequestMethod)
if method == "" { if method == "" {
h.logAndSendError(w, "Access-Control-Request-Method request header needed", reqInfo, errors.GetAPIError(errors.ErrBadRequest)) h.logAndSendError(ctx, w, "Access-Control-Request-Method request header needed", reqInfo, errors.GetAPIError(errors.ErrBadRequest))
return return
} }
@ -173,9 +177,9 @@ func (h *handler) Preflight(w http.ResponseWriter, r *http.Request) {
headers = strings.Split(requestHeaders, ", ") headers = strings.Split(requestHeaders, ", ")
} }
cors, err := h.obj.GetBucketCORS(r.Context(), bktInfo) cors, err := h.obj.GetBucketCORS(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get cors", reqInfo, err) h.logAndSendError(ctx, w, "could not get cors", reqInfo, err)
return return
} }
@ -204,7 +208,7 @@ func (h *handler) Preflight(w http.ResponseWriter, r *http.Request) {
w.Header().Set(api.AccessControlAllowCredentials, "true") w.Header().Set(api.AccessControlAllowCredentials, "true")
} }
if err = middleware.WriteSuccessResponseHeadersOnly(w); err != nil { if err = middleware.WriteSuccessResponseHeadersOnly(w); err != nil {
h.logAndSendError(w, "write response", reqInfo, err) h.logAndSendError(ctx, w, "write response", reqInfo, err)
return return
} }
return return
@ -213,7 +217,7 @@ func (h *handler) Preflight(w http.ResponseWriter, r *http.Request) {
} }
} }
} }
h.logAndSendError(w, "Forbidden", reqInfo, errors.GetAPIError(errors.ErrAccessDenied)) h.logAndSendError(ctx, w, "Forbidden", reqInfo, errors.GetAPIError(errors.ErrAccessDenied))
} }
func checkSubslice(slice []string, subSlice []string) bool { func checkSubslice(slice []string, subSlice []string) bool {

View file

@ -71,19 +71,19 @@ func (h *handler) DeleteObjectHandler(w http.ResponseWriter, r *http.Request) {
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
bktSettings, err := h.obj.GetBucketSettings(ctx, bktInfo) bktSettings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket settings", reqInfo, err)
return return
} }
networkInfo, err := h.obj.GetNetworkInfo(ctx) networkInfo, err := h.obj.GetNetworkInfo(ctx)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get network info", reqInfo, err) h.logAndSendError(ctx, w, "could not get network info", reqInfo, err)
return return
} }
@ -97,9 +97,9 @@ func (h *handler) DeleteObjectHandler(w http.ResponseWriter, r *http.Request) {
deletedObject := deletedObjects[0] deletedObject := deletedObjects[0]
if deletedObject.Error != nil { if deletedObject.Error != nil {
if isErrObjectLocked(deletedObject.Error) { if isErrObjectLocked(deletedObject.Error) {
h.logAndSendError(w, "object is locked", reqInfo, errors.GetAPIError(errors.ErrAccessDenied)) h.logAndSendError(ctx, w, "object is locked", reqInfo, errors.GetAPIError(errors.ErrAccessDenied))
} else { } else {
h.logAndSendError(w, "could not delete object", reqInfo, deletedObject.Error) h.logAndSendError(ctx, w, "could not delete object", reqInfo, deletedObject.Error)
} }
return return
} }
@ -134,26 +134,26 @@ func (h *handler) DeleteMultipleObjectsHandler(w http.ResponseWriter, r *http.Re
// Content-Md5 is required and should be set // Content-Md5 is required and should be set
// http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html // http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
if _, ok := r.Header[api.ContentMD5]; !ok { if _, ok := r.Header[api.ContentMD5]; !ok {
h.logAndSendError(w, "missing Content-MD5", reqInfo, errors.GetAPIError(errors.ErrMissingContentMD5)) h.logAndSendError(ctx, w, "missing Content-MD5", reqInfo, errors.GetAPIError(errors.ErrMissingContentMD5))
return return
} }
// Content-Length is required and should be non-zero // Content-Length is required and should be non-zero
// http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html // http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
if r.ContentLength <= 0 { if r.ContentLength <= 0 {
h.logAndSendError(w, "missing Content-Length", reqInfo, errors.GetAPIError(errors.ErrMissingContentLength)) h.logAndSendError(ctx, w, "missing Content-Length", reqInfo, errors.GetAPIError(errors.ErrMissingContentLength))
return return
} }
// Unmarshal list of keys to be deleted. // Unmarshal list of keys to be deleted.
requested := &DeleteObjectsRequest{} requested := &DeleteObjectsRequest{}
if err := h.cfg.NewXMLDecoder(r.Body).Decode(requested); err != nil { if err := h.cfg.NewXMLDecoder(r.Body).Decode(requested); err != nil {
h.logAndSendError(w, "couldn't decode body", reqInfo, fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrMalformedXML), err.Error())) h.logAndSendError(ctx, w, "couldn't decode body", reqInfo, fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrMalformedXML), err.Error()))
return return
} }
if len(requested.Objects) == 0 || len(requested.Objects) > maxObjectsToDelete { if len(requested.Objects) == 0 || len(requested.Objects) > maxObjectsToDelete {
h.logAndSendError(w, "number of objects to delete must be greater than 0 and less or equal to 1000", reqInfo, errors.GetAPIError(errors.ErrMalformedXML)) h.logAndSendError(ctx, w, "number of objects to delete must be greater than 0 and less or equal to 1000", reqInfo, errors.GetAPIError(errors.ErrMalformedXML))
return return
} }
@ -178,19 +178,19 @@ func (h *handler) DeleteMultipleObjectsHandler(w http.ResponseWriter, r *http.Re
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
bktSettings, err := h.obj.GetBucketSettings(ctx, bktInfo) bktSettings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket settings", reqInfo, err)
return return
} }
networkInfo, err := h.obj.GetNetworkInfo(ctx) networkInfo, err := h.obj.GetNetworkInfo(ctx)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get network info", reqInfo, err) h.logAndSendError(ctx, w, "could not get network info", reqInfo, err)
return return
} }
@ -231,22 +231,28 @@ func (h *handler) DeleteMultipleObjectsHandler(w http.ResponseWriter, r *http.Re
} }
if err = middleware.EncodeToResponse(w, response); err != nil { if err = middleware.EncodeToResponse(w, response); err != nil {
h.logAndSendError(w, "could not write response", reqInfo, err) h.logAndSendError(ctx, w, "could not write response", reqInfo, err)
return return
} }
} }
func (h *handler) DeleteBucketHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) DeleteBucketHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return
}
if err = checkOwner(bktInfo, reqInfo.User); err != nil {
h.logAndSendError(ctx, w, "request owner id does not match bucket owner id", reqInfo, err)
return return
} }
var sessionToken *session.Container var sessionToken *session.Container
boxData, err := middleware.GetBoxData(r.Context()) boxData, err := middleware.GetBoxData(ctx)
if err == nil { if err == nil {
sessionToken = boxData.Gate.SessionTokenForDelete() sessionToken = boxData.Gate.SessionTokenForDelete()
} }
@ -259,12 +265,12 @@ func (h *handler) DeleteBucketHandler(w http.ResponseWriter, r *http.Request) {
} }
} }
if err = h.obj.DeleteBucket(r.Context(), &layer.DeleteBucketParams{ if err = h.obj.DeleteBucket(ctx, &layer.DeleteBucketParams{
BktInfo: bktInfo, BktInfo: bktInfo,
SessionToken: sessionToken, SessionToken: sessionToken,
SkipCheck: skipObjCheck, SkipCheck: skipObjCheck,
}); err != nil { }); err != nil {
h.logAndSendError(w, "couldn't delete bucket", reqInfo, err) h.logAndSendError(ctx, w, "couldn't delete bucket", reqInfo, err)
return return
} }
@ -275,7 +281,7 @@ func (h *handler) DeleteBucketHandler(w http.ResponseWriter, r *http.Request) {
getBucketCannedChainID(chain.Ingress, bktInfo.CID), getBucketCannedChainID(chain.Ingress, bktInfo.CID),
} }
if err = h.ape.DeleteBucketPolicy(reqInfo.Namespace, bktInfo.CID, chainIDs); err != nil { if err = h.ape.DeleteBucketPolicy(reqInfo.Namespace, bktInfo.CID, chainIDs); err != nil {
h.logAndSendError(w, "failed to delete policy from storage", reqInfo, err) h.logAndSendError(ctx, w, "failed to delete policy from storage", reqInfo, err)
return return
} }

View file

@ -10,7 +10,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
apistatus "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client/status" apistatus "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client/status"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
@ -37,6 +37,7 @@ func TestDeleteBucketOnAlreadyRemovedError(t *testing.T) {
deleteObjects(t, hc, bktName, [][2]string{{objName, emptyVersion}}) deleteObjects(t, hc, bktName, [][2]string{{objName, emptyVersion}})
hc.owner = bktInfo.Owner
deleteBucket(t, hc, bktName, http.StatusNoContent) deleteBucket(t, hc, bktName, http.StatusNoContent)
} }
@ -53,11 +54,12 @@ func TestDeleteBucket(t *testing.T) {
tc := prepareHandlerContext(t) tc := prepareHandlerContext(t)
bktName, objName := "bucket-for-removal", "object-to-delete" bktName, objName := "bucket-for-removal", "object-to-delete"
_, objInfo := createVersionedBucketAndObject(t, tc, bktName, objName) bktInfo, objInfo := createVersionedBucketAndObject(t, tc, bktName, objName)
deleteMarkerVersion, isDeleteMarker := deleteObject(t, tc, bktName, objName, emptyVersion) deleteMarkerVersion, isDeleteMarker := deleteObject(t, tc, bktName, objName, emptyVersion)
require.True(t, isDeleteMarker) require.True(t, isDeleteMarker)
tc.owner = bktInfo.Owner
deleteBucket(t, tc, bktName, http.StatusConflict) deleteBucket(t, tc, bktName, http.StatusConflict)
deleteObject(t, tc, bktName, objName, objInfo.VersionID()) deleteObject(t, tc, bktName, objName, objInfo.VersionID())
deleteBucket(t, tc, bktName, http.StatusConflict) deleteBucket(t, tc, bktName, http.StatusConflict)
@ -82,6 +84,7 @@ func TestDeleteBucketOnNotFoundError(t *testing.T) {
deleteObjects(t, hc, bktName, [][2]string{{objName, emptyVersion}}) deleteObjects(t, hc, bktName, [][2]string{{objName, emptyVersion}})
hc.owner = bktInfo.Owner
deleteBucket(t, hc, bktName, http.StatusNoContent) deleteBucket(t, hc, bktName, http.StatusNoContent)
} }
@ -99,6 +102,7 @@ func TestForceDeleteBucket(t *testing.T) {
addr.SetContainer(bktInfo.CID) addr.SetContainer(bktInfo.CID)
addr.SetObject(nodeVersion.OID) addr.SetObject(nodeVersion.OID)
hc.owner = bktInfo.Owner
deleteBucketForce(t, hc, bktName, http.StatusConflict, "false") deleteBucketForce(t, hc, bktName, http.StatusConflict, "false")
deleteBucketForce(t, hc, bktName, http.StatusNoContent, "true") deleteBucketForce(t, hc, bktName, http.StatusNoContent, "true")
} }
@ -131,7 +135,7 @@ func TestDeleteObjectsError(t *testing.T) {
addr.SetContainer(bktInfo.CID) addr.SetContainer(bktInfo.CID)
addr.SetObject(nodeVersion.OID) addr.SetObject(nodeVersion.OID)
expectedError := apiErrors.GetAPIError(apiErrors.ErrAccessDenied) expectedError := apierr.GetAPIError(apierr.ErrAccessDenied)
hc.tp.SetObjectError(addr, expectedError) hc.tp.SetObjectError(addr, expectedError)
w := deleteObjectsBase(hc, bktName, [][2]string{{objName, nodeVersion.OID.EncodeToString()}}) w := deleteObjectsBase(hc, bktName, [][2]string{{objName, nodeVersion.OID.EncodeToString()}})
@ -457,6 +461,17 @@ func TestDeleteObjectCheckMarkerReturn(t *testing.T) {
require.Equal(t, deleteMarkerVersion, deleteMarkerVersion2) require.Equal(t, deleteMarkerVersion, deleteMarkerVersion2)
} }
func TestDeleteBucketByNotOwner(t *testing.T) {
hc := prepareHandlerContext(t)
bktName := "bucket-name"
bktInfo := createTestBucket(hc, bktName)
deleteBucket(t, hc, bktName, http.StatusForbidden)
hc.owner = bktInfo.Owner
deleteBucket(t, hc, bktName, http.StatusNoContent)
}
func createBucketAndObject(tc *handlerContext, bktName, objName string) (*data.BucketInfo, *data.ObjectInfo) { func createBucketAndObject(tc *handlerContext, bktName, objName string) (*data.BucketInfo, *data.ObjectInfo) {
bktInfo := createTestBucket(tc, bktName) bktInfo := createTestBucket(tc, bktName)
@ -553,9 +568,9 @@ func checkNotFound(t *testing.T, hc *handlerContext, bktName, objName, version s
assertStatus(t, w, http.StatusNotFound) assertStatus(t, w, http.StatusNotFound)
} }
func headObjectAssertS3Error(hc *handlerContext, bktName, objName, version string, code apiErrors.ErrorCode) { func headObjectAssertS3Error(hc *handlerContext, bktName, objName, version string, code apierr.ErrorCode) {
w := headObjectBase(hc, bktName, objName, version) w := headObjectBase(hc, bktName, objName, version)
assertS3Error(hc.t, w, apiErrors.GetAPIError(code)) assertS3Error(hc.t, w, apierr.GetAPIError(code))
} }
func checkFound(t *testing.T, hc *handlerContext, bktName, objName, version string) { func checkFound(t *testing.T, hc *handlerContext, bktName, objName, version string) {
@ -563,6 +578,18 @@ func checkFound(t *testing.T, hc *handlerContext, bktName, objName, version stri
assertStatus(t, w, http.StatusOK) assertStatus(t, w, http.StatusOK)
} }
func headObjectWithHeaders(hc *handlerContext, bktName, objName, version string, headers map[string]string) *httptest.ResponseRecorder {
query := make(url.Values)
query.Add(api.QueryVersionID, version)
w, r := prepareTestFullRequest(hc, bktName, objName, query, nil)
for k, v := range headers {
r.Header.Set(k, v)
}
hc.Handler().HeadObjectHandler(w, r)
return w
}
func headObjectBase(hc *handlerContext, bktName, objName, version string) *httptest.ResponseRecorder { func headObjectBase(hc *handlerContext, bktName, objName, version string) *httptest.ResponseRecorder {
query := make(url.Values) query := make(url.Values)
query.Add(api.QueryVersionID, version) query.Add(api.QueryVersionID, version)

View file

@ -16,6 +16,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@ -37,7 +38,7 @@ func TestSimpleGetEncrypted(t *testing.T) {
objInfo, err := tc.Layer().GetObjectInfo(tc.Context(), &layer.HeadObjectParams{BktInfo: bktInfo, Object: objName}) objInfo, err := tc.Layer().GetObjectInfo(tc.Context(), &layer.HeadObjectParams{BktInfo: bktInfo, Object: objName})
require.NoError(t, err) require.NoError(t, err)
obj, err := tc.MockedPool().GetObject(tc.Context(), layer.PrmObjectGet{Container: bktInfo.CID, Object: objInfo.ID}) obj, err := tc.MockedPool().GetObject(tc.Context(), frostfs.PrmObjectGet{Container: bktInfo.CID, Object: objInfo.ID})
require.NoError(t, err) require.NoError(t, err)
encryptedContent, err := io.ReadAll(obj.Payload) encryptedContent, err := io.ReadAll(obj.Payload)
require.NoError(t, err) require.NoError(t, err)
@ -47,6 +48,25 @@ func TestSimpleGetEncrypted(t *testing.T) {
require.Equal(t, content, string(response)) require.Equal(t, content, string(response))
} }
func TestMD5HeaderBadOrEmpty(t *testing.T) {
tc := prepareHandlerContext(t)
bktName, objName := "bucket-for-sse-c", "object-to-encrypt"
createTestBucket(tc, bktName)
content := "content"
headers := map[string]string{
api.ContentMD5: "",
}
putEncryptedObjectWithHeadersErr(t, tc, bktName, objName, content, headers, errors.ErrInvalidDigest)
headers = map[string]string{
api.ContentMD5: "YWJjMTIzIT8kKiYoKSctPUB+",
}
putEncryptedObjectWithHeadersErr(t, tc, bktName, objName, content, headers, errors.ErrBadDigest)
}
func TestGetEncryptedRange(t *testing.T) { func TestGetEncryptedRange(t *testing.T) {
tc := prepareHandlerContext(t) tc := prepareHandlerContext(t)
@ -359,6 +379,15 @@ func putEncryptedObject(t *testing.T, tc *handlerContext, bktName, objName, cont
assertStatus(t, w, http.StatusOK) assertStatus(t, w, http.StatusOK)
} }
func putEncryptedObjectWithHeadersErr(t *testing.T, tc *handlerContext, bktName, objName, content string, headers map[string]string, code errors.ErrorCode) {
body := bytes.NewReader([]byte(content))
w, r := prepareTestPayloadRequest(tc, bktName, objName, body)
setHeaders(r, headers)
tc.Handler().PutObjectHandler(w, r)
assertS3Error(t, w, errors.GetAPIError(code))
}
func getEncryptedObject(hc *handlerContext, bktName, objName string) ([]byte, http.Header) { func getEncryptedObject(hc *handlerContext, bktName, objName string) ([]byte, http.Header) {
w, r := prepareTestRequest(hc, bktName, objName, nil) w, r := prepareTestRequest(hc, bktName, objName, nil)
setEncryptHeaders(r) setEncryptHeaders(r)
@ -370,6 +399,15 @@ func getObject(hc *handlerContext, bktName, objName string) ([]byte, http.Header
return getObjectBase(hc, w, r) return getObjectBase(hc, w, r)
} }
func getObjectWithHeaders(hc *handlerContext, bktName, objName string, headers map[string]string) *httptest.ResponseRecorder {
w, r := prepareTestRequest(hc, bktName, objName, nil)
for k, v := range headers {
r.Header.Set(k, v)
}
hc.Handler().GetObjectHandler(w, r)
return w
}
func getObjectBase(hc *handlerContext, w *httptest.ResponseRecorder, r *http.Request) ([]byte, http.Header) { func getObjectBase(hc *handlerContext, w *httptest.ResponseRecorder, r *http.Request) ([]byte, http.Header) {
hc.Handler().GetObjectHandler(w, r) hc.Handler().GetObjectHandler(w, r)
assertStatus(hc.t, w, http.StatusOK) assertStatus(hc.t, w, http.StatusOK)

View file

@ -78,6 +78,27 @@ func addSSECHeaders(responseHeader http.Header, requestHeader http.Header) {
responseHeader.Set(api.AmzServerSideEncryptionCustomerKeyMD5, requestHeader.Get(api.AmzServerSideEncryptionCustomerKeyMD5)) responseHeader.Set(api.AmzServerSideEncryptionCustomerKeyMD5, requestHeader.Get(api.AmzServerSideEncryptionCustomerKeyMD5))
} }
func writeNotModifiedHeaders(h http.Header, extendedInfo *data.ExtendedObjectInfo, tagSetLength int, isBucketUnversioned, md5Enabled bool) {
h.Set(api.ETag, data.Quote(extendedInfo.ObjectInfo.ETag(md5Enabled)))
h.Set(api.LastModified, extendedInfo.ObjectInfo.Created.UTC().Format(http.TimeFormat))
h.Set(api.AmzTaggingCount, strconv.Itoa(tagSetLength))
if !isBucketUnversioned {
h.Set(api.AmzVersionID, extendedInfo.Version())
}
if cacheControl := extendedInfo.ObjectInfo.Headers[api.CacheControl]; cacheControl != "" {
h.Set(api.CacheControl, cacheControl)
}
for key, val := range extendedInfo.ObjectInfo.Headers {
if layer.IsSystemHeader(key) {
continue
}
h[api.MetadataPrefix+key] = []string{val}
}
}
func writeHeaders(h http.Header, requestHeader http.Header, extendedInfo *data.ExtendedObjectInfo, tagSetLength int, func writeHeaders(h http.Header, requestHeader http.Header, extendedInfo *data.ExtendedObjectInfo, tagSetLength int,
isBucketUnversioned, md5Enabled bool) { isBucketUnversioned, md5Enabled bool) {
info := extendedInfo.ObjectInfo info := extendedInfo.ObjectInfo
@ -129,18 +150,19 @@ func (h *handler) GetObjectHandler(w http.ResponseWriter, r *http.Request) {
var ( var (
params *layer.RangeParams params *layer.RangeParams
reqInfo = middleware.GetReqInfo(r.Context()) ctx = r.Context()
reqInfo = middleware.GetReqInfo(ctx)
) )
conditional, err := parseConditionalHeaders(r.Header) conditional, err := parseConditionalHeaders(r.Header)
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse request params", reqInfo, err) h.logAndSendError(ctx, w, "could not parse request params", reqInfo, err)
return return
} }
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
@ -150,37 +172,16 @@ func (h *handler) GetObjectHandler(w http.ResponseWriter, r *http.Request) {
VersionID: reqInfo.URL.Query().Get(api.QueryVersionID), VersionID: reqInfo.URL.Query().Get(api.QueryVersionID),
} }
extendedInfo, err := h.obj.GetExtendedObjectInfo(r.Context(), p) extendedInfo, err := h.obj.GetExtendedObjectInfo(ctx, p)
if err != nil { if err != nil {
h.logAndSendError(w, "could not find object", reqInfo, err) h.logAndSendError(ctx, w, "could not find object", reqInfo, err)
return return
} }
info := extendedInfo.ObjectInfo info := extendedInfo.ObjectInfo
if err = checkPreconditions(info, conditional, h.cfg.MD5Enabled()); err != nil { bktSettings, err := h.obj.GetBucketSettings(ctx, bktInfo)
h.logAndSendError(w, "precondition failed", reqInfo, err)
return
}
encryptionParams, err := formEncryptionParams(r)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid sse headers", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket settings", reqInfo, err)
return
}
if err = encryptionParams.MatchObjectEncryption(layer.FormEncryptionInfo(info.Headers)); err != nil {
h.logAndSendError(w, "encryption doesn't match object", reqInfo, errors.GetAPIError(errors.ErrBadRequest), zap.Error(err))
return
}
fullSize, err := layer.GetObjectSize(info)
if err != nil {
h.logAndSendError(w, "invalid size header", reqInfo, errors.GetAPIError(errors.ErrBadRequest))
return
}
if params, err = fetchRangeHeader(r.Header, fullSize); err != nil {
h.logAndSendError(w, "could not parse range header", reqInfo, err)
return return
} }
@ -190,24 +191,48 @@ func (h *handler) GetObjectHandler(w http.ResponseWriter, r *http.Request) {
VersionID: info.VersionID(), VersionID: info.VersionID(),
} }
tagSet, lockInfo, err := h.obj.GetObjectTaggingAndLock(r.Context(), t, extendedInfo.NodeVersion) tagSet, lockInfo, err := h.obj.GetObjectTaggingAndLock(ctx, t, extendedInfo.NodeVersion)
if err != nil && !errors.IsS3Error(err, errors.ErrNoSuchKey) { if err != nil && !errors.IsS3Error(err, errors.ErrNoSuchKey) {
h.logAndSendError(w, "could not get object meta data", reqInfo, err) h.logAndSendError(ctx, w, "could not get object meta data", reqInfo, err)
return return
} }
if layer.IsAuthenticatedRequest(r.Context()) { if err = checkPreconditions(info, conditional, h.cfg.MD5Enabled()); err != nil {
if errors.IsS3Error(err, errors.ErrNotModified) {
writeNotModifiedHeaders(w.Header(), extendedInfo, len(tagSet), bktSettings.Unversioned(), h.cfg.MD5Enabled())
}
h.logAndSendError(ctx, w, "precondition failed", reqInfo, err)
return
}
encryptionParams, err := formEncryptionParams(r)
if err != nil {
h.logAndSendError(ctx, w, "invalid sse headers", reqInfo, err)
return
}
if err = encryptionParams.MatchObjectEncryption(layer.FormEncryptionInfo(info.Headers)); err != nil {
h.logAndSendError(ctx, w, "encryption doesn't match object", reqInfo, errors.GetAPIError(errors.ErrBadRequest), zap.Error(err))
return
}
fullSize, err := layer.GetObjectSize(info)
if err != nil {
h.logAndSendError(ctx, w, "invalid size header", reqInfo, errors.GetAPIError(errors.ErrBadRequest))
return
}
if params, err = fetchRangeHeader(r.Header, fullSize); err != nil {
h.logAndSendError(ctx, w, "could not parse range header", reqInfo, err)
return
}
if layer.IsAuthenticatedRequest(ctx) {
overrideResponseHeaders(w.Header(), reqInfo.URL.Query()) overrideResponseHeaders(w.Header(), reqInfo.URL.Query())
} }
if err = h.setLockingHeaders(bktInfo, lockInfo, w.Header()); err != nil { if err = h.setLockingHeaders(bktInfo, lockInfo, w.Header()); err != nil {
h.logAndSendError(w, "could not get locking info", reqInfo, err) h.logAndSendError(ctx, w, "could not get locking info", reqInfo, err)
return
}
bktSettings, err := h.obj.GetBucketSettings(r.Context(), bktInfo)
if err != nil {
h.logAndSendError(w, "could not get bucket settings", reqInfo, err)
return return
} }
@ -219,9 +244,9 @@ func (h *handler) GetObjectHandler(w http.ResponseWriter, r *http.Request) {
Encryption: encryptionParams, Encryption: encryptionParams,
} }
objPayload, err := h.obj.GetObject(r.Context(), getPayloadParams) objPayload, err := h.obj.GetObject(ctx, getPayloadParams)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get object payload", reqInfo, err) h.logAndSendError(ctx, w, "could not get object payload", reqInfo, err)
return return
} }
@ -233,7 +258,7 @@ func (h *handler) GetObjectHandler(w http.ResponseWriter, r *http.Request) {
} }
if err = objPayload.StreamTo(w); err != nil { if err = objPayload.StreamTo(w); err != nil {
h.logAndSendError(w, "could not stream object payload", reqInfo, err) h.logAndSendError(ctx, w, "could not stream object payload", reqInfo, err)
return return
} }
} }

View file

@ -2,7 +2,7 @@ package handler
import ( import (
"bytes" "bytes"
stderrors "errors" "errors"
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
@ -13,7 +13,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
apistatus "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client/status" apistatus "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client/status"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@ -89,7 +89,7 @@ func TestPreconditions(t *testing.T) {
name: "IfMatch false", name: "IfMatch false",
info: newInfo(etag, today), info: newInfo(etag, today),
args: &conditionalArgs{IfMatch: etag2}, args: &conditionalArgs{IfMatch: etag2},
expected: errors.GetAPIError(errors.ErrPreconditionFailed)}, expected: apierr.GetAPIError(apierr.ErrPreconditionFailed)},
{ {
name: "IfNoneMatch true", name: "IfNoneMatch true",
info: newInfo(etag, today), info: newInfo(etag, today),
@ -99,7 +99,7 @@ func TestPreconditions(t *testing.T) {
name: "IfNoneMatch false", name: "IfNoneMatch false",
info: newInfo(etag, today), info: newInfo(etag, today),
args: &conditionalArgs{IfNoneMatch: etag}, args: &conditionalArgs{IfNoneMatch: etag},
expected: errors.GetAPIError(errors.ErrNotModified)}, expected: apierr.GetAPIError(apierr.ErrNotModified)},
{ {
name: "IfModifiedSince true", name: "IfModifiedSince true",
info: newInfo(etag, today), info: newInfo(etag, today),
@ -109,7 +109,7 @@ func TestPreconditions(t *testing.T) {
name: "IfModifiedSince false", name: "IfModifiedSince false",
info: newInfo(etag, yesterday), info: newInfo(etag, yesterday),
args: &conditionalArgs{IfModifiedSince: &today}, args: &conditionalArgs{IfModifiedSince: &today},
expected: errors.GetAPIError(errors.ErrNotModified)}, expected: apierr.GetAPIError(apierr.ErrNotModified)},
{ {
name: "IfUnmodifiedSince true", name: "IfUnmodifiedSince true",
info: newInfo(etag, yesterday), info: newInfo(etag, yesterday),
@ -119,7 +119,7 @@ func TestPreconditions(t *testing.T) {
name: "IfUnmodifiedSince false", name: "IfUnmodifiedSince false",
info: newInfo(etag, today), info: newInfo(etag, today),
args: &conditionalArgs{IfUnmodifiedSince: &yesterday}, args: &conditionalArgs{IfUnmodifiedSince: &yesterday},
expected: errors.GetAPIError(errors.ErrPreconditionFailed)}, expected: apierr.GetAPIError(apierr.ErrPreconditionFailed)},
{ {
name: "IfMatch true, IfUnmodifiedSince false", name: "IfMatch true, IfUnmodifiedSince false",
@ -131,19 +131,19 @@ func TestPreconditions(t *testing.T) {
name: "IfMatch false, IfUnmodifiedSince true", name: "IfMatch false, IfUnmodifiedSince true",
info: newInfo(etag, yesterday), info: newInfo(etag, yesterday),
args: &conditionalArgs{IfMatch: etag2, IfUnmodifiedSince: &today}, args: &conditionalArgs{IfMatch: etag2, IfUnmodifiedSince: &today},
expected: errors.GetAPIError(errors.ErrPreconditionFailed), expected: apierr.GetAPIError(apierr.ErrPreconditionFailed),
}, },
{ {
name: "IfNoneMatch false, IfModifiedSince true", name: "IfNoneMatch false, IfModifiedSince true",
info: newInfo(etag, today), info: newInfo(etag, today),
args: &conditionalArgs{IfNoneMatch: etag, IfModifiedSince: &yesterday}, args: &conditionalArgs{IfNoneMatch: etag, IfModifiedSince: &yesterday},
expected: errors.GetAPIError(errors.ErrNotModified), expected: apierr.GetAPIError(apierr.ErrNotModified),
}, },
{ {
name: "IfNoneMatch true, IfModifiedSince false", name: "IfNoneMatch true, IfModifiedSince false",
info: newInfo(etag, yesterday), info: newInfo(etag, yesterday),
args: &conditionalArgs{IfNoneMatch: etag2, IfModifiedSince: &today}, args: &conditionalArgs{IfNoneMatch: etag2, IfModifiedSince: &today},
expected: errors.GetAPIError(errors.ErrNotModified), expected: apierr.GetAPIError(apierr.ErrNotModified),
}, },
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
@ -151,7 +151,7 @@ func TestPreconditions(t *testing.T) {
if tc.expected == nil { if tc.expected == nil {
require.NoError(t, actual) require.NoError(t, actual)
} else { } else {
require.True(t, stderrors.Is(actual, tc.expected), tc.expected, actual) require.True(t, errors.Is(actual, tc.expected), tc.expected, actual)
} }
}) })
} }
@ -193,8 +193,8 @@ func TestGetObject(t *testing.T) {
hc.tp.SetObjectError(addr, &apistatus.ObjectNotFound{}) hc.tp.SetObjectError(addr, &apistatus.ObjectNotFound{})
hc.tp.SetObjectError(objInfo.Address(), &apistatus.ObjectNotFound{}) hc.tp.SetObjectError(objInfo.Address(), &apistatus.ObjectNotFound{})
getObjectAssertS3Error(hc, bktName, objName, objInfo.VersionID(), errors.ErrNoSuchVersion) getObjectAssertS3Error(hc, bktName, objName, objInfo.VersionID(), apierr.ErrNoSuchVersion)
getObjectAssertS3Error(hc, bktName, objName, emptyVersion, errors.ErrNoSuchKey) getObjectAssertS3Error(hc, bktName, objName, emptyVersion, apierr.ErrNoSuchKey)
} }
func TestGetObjectEnabledMD5(t *testing.T) { func TestGetObjectEnabledMD5(t *testing.T) {
@ -210,6 +210,27 @@ func TestGetObjectEnabledMD5(t *testing.T) {
require.Equal(t, data.Quote(objInfo.MD5Sum), headers.Get(api.ETag)) require.Equal(t, data.Quote(objInfo.MD5Sum), headers.Get(api.ETag))
} }
func TestGetObjectNotModifiedHeaders(t *testing.T) {
hc := prepareHandlerContextWithMinCache(t)
bktName, objName, metadataHeader := "bucket", "obj", api.MetadataPrefix+"header"
createVersionedBucket(hc, bktName)
header := putObjectWithHeaders(hc, bktName, objName, map[string]string{api.CacheControl: "value", metadataHeader: "value"})
etag, versionID := header.Get(api.ETag), header.Get(api.AmzVersionID)
require.NotEmpty(t, etag)
require.NotEmpty(t, versionID)
putObjectTagging(t, hc, bktName, objName, map[string]string{"key": "value"})
w := getObjectWithHeaders(hc, bktName, objName, map[string]string{api.IfNoneMatch: etag})
require.Equal(t, http.StatusNotModified, w.Code)
require.Equal(t, "1", w.Header().Get(api.AmzTaggingCount))
require.Equal(t, etag, w.Header().Get(api.ETag))
require.NotEmpty(t, w.Header().Get(api.LastModified))
require.Equal(t, versionID, w.Header().Get(api.AmzVersionID))
require.Equal(t, "value", w.Header().Get(api.CacheControl))
require.Equal(t, []string{"value"}, w.Header()[metadataHeader])
}
func putObjectContent(hc *handlerContext, bktName, objName, content string) http.Header { func putObjectContent(hc *handlerContext, bktName, objName, content string) http.Header {
body := bytes.NewReader([]byte(content)) body := bytes.NewReader([]byte(content))
w, r := prepareTestPayloadRequest(hc, bktName, objName, body) w, r := prepareTestPayloadRequest(hc, bktName, objName, body)
@ -236,9 +257,9 @@ func getObjectVersion(tc *handlerContext, bktName, objName, version string) []by
return content return content
} }
func getObjectAssertS3Error(hc *handlerContext, bktName, objName, version string, code errors.ErrorCode) { func getObjectAssertS3Error(hc *handlerContext, bktName, objName, version string, code apierr.ErrorCode) {
w := getObjectBaseResponse(hc, bktName, objName, version) w := getObjectBaseResponse(hc, bktName, objName, version)
assertS3Error(hc.t, w, errors.GetAPIError(code)) assertS3Error(hc.t, w, apierr.GetAPIError(code))
} }
func getObjectBaseResponse(hc *handlerContext, bktName, objName, version string) *httptest.ResponseRecorder { func getObjectBaseResponse(hc *handlerContext, bktName, objName, version string) *httptest.ResponseRecorder {

View file

@ -9,6 +9,7 @@ import (
"encoding/base64" "encoding/base64"
"encoding/hex" "encoding/hex"
"encoding/xml" "encoding/xml"
"errors"
"mime/multipart" "mime/multipart"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
@ -88,6 +89,12 @@ func addMD5Header(tp *utils.TypeProvider, r *http.Request, rawBody []byte) error
} }
if rand == true { if rand == true {
defer func() {
if recover() != nil {
err = errors.New("panic in base64")
}
}()
var dst []byte var dst []byte
base64.StdEncoding.Encode(dst, rawBody) base64.StdEncoding.Encode(dst, rawBody)
hash := md5.Sum(dst) hash := md5.Sum(dst)
@ -584,6 +591,11 @@ func DoFuzzCopyObjectHandler(input []byte) int {
return fuzzFailExitCode return fuzzFailExitCode
} }
defer func() {
if recover() != nil {
err = errors.New("panic in httptest.NewRequest")
}
}()
r = httptest.NewRequest(http.MethodPut, defaultURL+params, nil) r = httptest.NewRequest(http.MethodPut, defaultURL+params, nil)
if r != nil { if r != nil {
return fuzzFailExitCode return fuzzFailExitCode
@ -637,6 +649,11 @@ func DoFuzzDeleteObjectHandler(input []byte) int {
return fuzzFailExitCode return fuzzFailExitCode
} }
defer func() {
if recover() != nil {
err = errors.New("panic in httptest.NewRequest")
}
}()
r = httptest.NewRequest(http.MethodDelete, defaultURL+params, nil) r = httptest.NewRequest(http.MethodDelete, defaultURL+params, nil)
if r != nil { if r != nil {
return fuzzFailExitCode return fuzzFailExitCode
@ -689,6 +706,11 @@ func DoFuzzGetObjectHandler(input []byte) int {
w := httptest.NewRecorder() w := httptest.NewRecorder()
defer func() {
if recover() != nil {
err = errors.New("panic in httptest.NewRequest")
}
}()
r := httptest.NewRequest(http.MethodGet, defaultURL+params, nil) r := httptest.NewRequest(http.MethodGet, defaultURL+params, nil)
if r != nil { if r != nil {
return fuzzFailExitCode return fuzzFailExitCode
@ -914,6 +936,11 @@ func DoFuzzPutObjectRetentionHandler(input []byte) int {
w := httptest.NewRecorder() w := httptest.NewRecorder()
defer func() {
if recover() != nil {
err = errors.New("panic in httptest.NewRequest")
}
}()
r := httptest.NewRequest(http.MethodPut, defaultURL+objName+"?retention", bytes.NewReader(rawBody)) r := httptest.NewRequest(http.MethodPut, defaultURL+objName+"?retention", bytes.NewReader(rawBody))
if r != nil { if r != nil {
return fuzzFailExitCode return fuzzFailExitCode

View file

@ -20,6 +20,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/resolver" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/resolver"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/service/tree" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/service/tree"
@ -243,12 +244,14 @@ func getMinCacheConfig(logger *zap.Logger) *layer.CachesConfig {
Buckets: minCacheCfg, Buckets: minCacheCfg,
System: minCacheCfg, System: minCacheCfg,
AccessControl: minCacheCfg, AccessControl: minCacheCfg,
NetworkInfo: &cache.NetworkInfoCacheConfig{Lifetime: minCacheCfg.Lifetime},
} }
} }
type apeMock struct { type apeMock struct {
chainMap map[engine.Target][]*chain.Chain chainMap map[engine.Target][]*chain.Chain
policyMap map[string][]byte policyMap map[string][]byte
err error
} }
func newAPEMock() *apeMock { func newAPEMock() *apeMock {
@ -292,6 +295,10 @@ func (a *apeMock) DeletePolicy(namespace string, cnrID cid.ID) error {
} }
func (a *apeMock) PutBucketPolicy(ns string, cnrID cid.ID, policy []byte, chain []*chain.Chain) error { func (a *apeMock) PutBucketPolicy(ns string, cnrID cid.ID, policy []byte, chain []*chain.Chain) error {
if a.err != nil {
return a.err
}
if err := a.PutPolicy(ns, cnrID, policy); err != nil { if err := a.PutPolicy(ns, cnrID, policy); err != nil {
return err return err
} }
@ -306,6 +313,10 @@ func (a *apeMock) PutBucketPolicy(ns string, cnrID cid.ID, policy []byte, chain
} }
func (a *apeMock) DeleteBucketPolicy(ns string, cnrID cid.ID, chainIDs []chain.ID) error { func (a *apeMock) DeleteBucketPolicy(ns string, cnrID cid.ID, chainIDs []chain.ID) error {
if a.err != nil {
return a.err
}
if err := a.DeletePolicy(ns, cnrID); err != nil { if err := a.DeletePolicy(ns, cnrID); err != nil {
return err return err
} }
@ -319,6 +330,10 @@ func (a *apeMock) DeleteBucketPolicy(ns string, cnrID cid.ID, chainIDs []chain.I
} }
func (a *apeMock) GetBucketPolicy(ns string, cnrID cid.ID) ([]byte, error) { func (a *apeMock) GetBucketPolicy(ns string, cnrID cid.ID) ([]byte, error) {
if a.err != nil {
return nil, a.err
}
policy, ok := a.policyMap[ns+cnrID.EncodeToString()] policy, ok := a.policyMap[ns+cnrID.EncodeToString()]
if !ok { if !ok {
return nil, errors.New("not found") return nil, errors.New("not found")
@ -328,6 +343,10 @@ func (a *apeMock) GetBucketPolicy(ns string, cnrID cid.ID) ([]byte, error) {
} }
func (a *apeMock) SaveACLChains(cid string, chains []*chain.Chain) error { func (a *apeMock) SaveACLChains(cid string, chains []*chain.Chain) error {
if a.err != nil {
return a.err
}
for i := range chains { for i := range chains {
if err := a.AddChain(engine.ContainerTarget(cid), chains[i]); err != nil { if err := a.AddChain(engine.ContainerTarget(cid), chains[i]); err != nil {
return err return err
@ -369,7 +388,7 @@ func createTestBucket(hc *handlerContext, bktName string) *data.BucketInfo {
} }
func createTestBucketWithLock(hc *handlerContext, bktName string, conf *data.ObjectLockConfiguration) *data.BucketInfo { func createTestBucketWithLock(hc *handlerContext, bktName string, conf *data.ObjectLockConfiguration) *data.BucketInfo {
res, err := hc.MockedPool().CreateContainer(hc.Context(), layer.PrmContainerCreate{ res, err := hc.MockedPool().CreateContainer(hc.Context(), frostfs.PrmContainerCreate{
Creator: hc.owner, Creator: hc.owner,
Name: bktName, Name: bktName,
AdditionalAttributes: [][2]string{{layer.AttributeLockEnabled, "true"}}, AdditionalAttributes: [][2]string{{layer.AttributeLockEnabled, "true"}},
@ -416,7 +435,7 @@ func createTestObject(hc *handlerContext, bktInfo *data.BucketInfo, objName stri
extObjInfo, err := hc.Layer().PutObject(hc.Context(), &layer.PutObjectParams{ extObjInfo, err := hc.Layer().PutObject(hc.Context(), &layer.PutObjectParams{
BktInfo: bktInfo, BktInfo: bktInfo,
Object: objName, Object: objName,
Size: uint64(len(content)), Size: ptr(uint64(len(content))),
Reader: bytes.NewReader(content), Reader: bytes.NewReader(content),
Header: header, Header: header,
Encryption: encryption, Encryption: encryption,
@ -443,6 +462,7 @@ func prepareTestRequestWithQuery(hc *handlerContext, bktName, objName string, qu
r.URL.RawQuery = query.Encode() r.URL.RawQuery = query.Encode()
reqInfo := middleware.NewReqInfo(w, r, middleware.ObjectRequest{Bucket: bktName, Object: objName}, "") reqInfo := middleware.NewReqInfo(w, r, middleware.ObjectRequest{Bucket: bktName, Object: objName}, "")
reqInfo.User = hc.owner.String()
r = r.WithContext(middleware.SetReqInfo(hc.Context(), reqInfo)) r = r.WithContext(middleware.SetReqInfo(hc.Context(), reqInfo))
return w, r return w, r

View file

@ -27,17 +27,18 @@ func getRangeToDetectContentType(maxSize uint64) *layer.RangeParams {
} }
func (h *handler) HeadObjectHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) HeadObjectHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
conditional, err := parseConditionalHeaders(r.Header) conditional, err := parseConditionalHeaders(r.Header)
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse request params", reqInfo, err) h.logAndSendError(ctx, w, "could not parse request params", reqInfo, err)
return return
} }
@ -47,26 +48,27 @@ func (h *handler) HeadObjectHandler(w http.ResponseWriter, r *http.Request) {
VersionID: reqInfo.URL.Query().Get(api.QueryVersionID), VersionID: reqInfo.URL.Query().Get(api.QueryVersionID),
} }
extendedInfo, err := h.obj.GetExtendedObjectInfo(r.Context(), p) extendedInfo, err := h.obj.GetExtendedObjectInfo(ctx, p)
if err != nil { if err != nil {
h.logAndSendError(w, "could not find object", reqInfo, err) h.logAndSendError(ctx, w, "could not find object", reqInfo, err)
return return
} }
info := extendedInfo.ObjectInfo info := extendedInfo.ObjectInfo
encryptionParams, err := formEncryptionParams(r) encryptionParams, err := formEncryptionParams(r)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid sse headers", reqInfo, err) h.logAndSendError(ctx, w, "invalid sse headers", reqInfo, err)
return return
} }
if err = encryptionParams.MatchObjectEncryption(layer.FormEncryptionInfo(info.Headers)); err != nil { if err = encryptionParams.MatchObjectEncryption(layer.FormEncryptionInfo(info.Headers)); err != nil {
h.logAndSendError(w, "encryption doesn't match object", reqInfo, errors.GetAPIError(errors.ErrBadRequest), zap.Error(err)) h.logAndSendError(ctx, w, "encryption doesn't match object", reqInfo, errors.GetAPIError(errors.ErrBadRequest), zap.Error(err))
return return
} }
if err = checkPreconditions(info, conditional, h.cfg.MD5Enabled()); err != nil { bktSettings, err := h.obj.GetBucketSettings(ctx, bktInfo)
h.logAndSendError(w, "precondition failed", reqInfo, err) if err != nil {
h.logAndSendError(ctx, w, "could not get bucket settings", reqInfo, err)
return return
} }
@ -76,9 +78,17 @@ func (h *handler) HeadObjectHandler(w http.ResponseWriter, r *http.Request) {
VersionID: info.VersionID(), VersionID: info.VersionID(),
} }
tagSet, lockInfo, err := h.obj.GetObjectTaggingAndLock(r.Context(), t, extendedInfo.NodeVersion) tagSet, lockInfo, err := h.obj.GetObjectTaggingAndLock(ctx, t, extendedInfo.NodeVersion)
if err != nil && !errors.IsS3Error(err, errors.ErrNoSuchKey) { if err != nil && !errors.IsS3Error(err, errors.ErrNoSuchKey) {
h.logAndSendError(w, "could not get object meta data", reqInfo, err) h.logAndSendError(ctx, w, "could not get object meta data", reqInfo, err)
return
}
if err = checkPreconditions(info, conditional, h.cfg.MD5Enabled()); err != nil {
if errors.IsS3Error(err, errors.ErrNotModified) {
writeNotModifiedHeaders(w.Header(), extendedInfo, len(tagSet), bktSettings.Unversioned(), h.cfg.MD5Enabled())
}
h.logAndSendError(ctx, w, "precondition failed", reqInfo, err)
return return
} }
@ -91,15 +101,15 @@ func (h *handler) HeadObjectHandler(w http.ResponseWriter, r *http.Request) {
BucketInfo: bktInfo, BucketInfo: bktInfo,
} }
objPayload, err := h.obj.GetObject(r.Context(), getParams) objPayload, err := h.obj.GetObject(ctx, getParams)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get object", reqInfo, err, zap.Stringer("oid", info.ID)) h.logAndSendError(ctx, w, "could not get object", reqInfo, err, zap.Stringer("oid", info.ID))
return return
} }
buffer, err := io.ReadAll(objPayload) buffer, err := io.ReadAll(objPayload)
if err != nil { if err != nil {
h.logAndSendError(w, "could not partly read payload to detect content type", reqInfo, err, zap.Stringer("oid", info.ID)) h.logAndSendError(ctx, w, "could not partly read payload to detect content type", reqInfo, err, zap.Stringer("oid", info.ID))
return return
} }
@ -108,13 +118,7 @@ func (h *handler) HeadObjectHandler(w http.ResponseWriter, r *http.Request) {
} }
if err = h.setLockingHeaders(bktInfo, lockInfo, w.Header()); err != nil { if err = h.setLockingHeaders(bktInfo, lockInfo, w.Header()); err != nil {
h.logAndSendError(w, "could not get locking info", reqInfo, err) h.logAndSendError(ctx, w, "could not get locking info", reqInfo, err)
return
}
bktSettings, err := h.obj.GetBucketSettings(r.Context(), bktInfo)
if err != nil {
h.logAndSendError(w, "could not get bucket settings", reqInfo, err)
return return
} }
@ -123,11 +127,12 @@ func (h *handler) HeadObjectHandler(w http.ResponseWriter, r *http.Request) {
} }
func (h *handler) HeadBucketHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) HeadBucketHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
@ -141,7 +146,7 @@ func (h *handler) HeadBucketHandler(w http.ResponseWriter, r *http.Request) {
} }
if err = middleware.WriteResponse(w, http.StatusOK, nil, middleware.MimeNone); err != nil { if err = middleware.WriteResponse(w, http.StatusOK, nil, middleware.MimeNone); err != nil {
h.logAndSendError(w, "write response", reqInfo, err) h.logAndSendError(ctx, w, "write response", reqInfo, err)
return return
} }
} }

View file

@ -6,7 +6,7 @@ import (
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
apistatus "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client/status" apistatus "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client/status"
@ -95,8 +95,29 @@ func TestHeadObject(t *testing.T) {
hc.tp.SetObjectError(addr, &apistatus.ObjectNotFound{}) hc.tp.SetObjectError(addr, &apistatus.ObjectNotFound{})
hc.tp.SetObjectError(objInfo.Address(), &apistatus.ObjectNotFound{}) hc.tp.SetObjectError(objInfo.Address(), &apistatus.ObjectNotFound{})
headObjectAssertS3Error(hc, bktName, objName, objInfo.VersionID(), s3errors.ErrNoSuchVersion) headObjectAssertS3Error(hc, bktName, objName, objInfo.VersionID(), apierr.ErrNoSuchVersion)
headObjectAssertS3Error(hc, bktName, objName, emptyVersion, s3errors.ErrNoSuchKey) headObjectAssertS3Error(hc, bktName, objName, emptyVersion, apierr.ErrNoSuchKey)
}
func TestHeadObjectNotModifiedHeaders(t *testing.T) {
hc := prepareHandlerContextWithMinCache(t)
bktName, objName, metadataHeader := "bucket", "obj", api.MetadataPrefix+"header"
createVersionedBucket(hc, bktName)
header := putObjectWithHeaders(hc, bktName, objName, map[string]string{api.CacheControl: "value", metadataHeader: "value"})
etag, versionID := header.Get(api.ETag), header.Get(api.AmzVersionID)
require.NotEmpty(t, etag)
require.NotEmpty(t, versionID)
putObjectTagging(t, hc, bktName, objName, map[string]string{"key": "value"})
w := headObjectWithHeaders(hc, bktName, objName, emptyVersion, map[string]string{api.IfNoneMatch: etag})
require.Equal(t, http.StatusNotModified, w.Code)
require.Equal(t, "1", w.Header().Get(api.AmzTaggingCount))
require.Equal(t, etag, w.Header().Get(api.ETag))
require.NotEmpty(t, w.Header().Get(api.LastModified))
require.Equal(t, versionID, w.Header().Get(api.AmzVersionID))
require.Equal(t, "value", w.Header().Get(api.CacheControl))
require.Equal(t, []string{"value"}, w.Header()[metadataHeader])
} }
func TestIsAvailableToResolve(t *testing.T) { func TestIsAvailableToResolve(t *testing.T) {

View file

@ -7,15 +7,16 @@ import (
) )
func (h *handler) GetBucketLocationHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketLocationHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
if err = middleware.EncodeToResponse(w, LocationResponse{Location: bktInfo.LocationConstraint}); err != nil { if err = middleware.EncodeToResponse(w, LocationResponse{Location: bktInfo.LocationConstraint}); err != nil {
h.logAndSendError(w, "couldn't encode bucket location response", reqInfo, err) h.logAndSendError(ctx, w, "couldn't encode bucket location response", reqInfo, err)
} }
} }

View file

@ -2,6 +2,9 @@ package handler
import ( import (
"bytes" "bytes"
"context"
"crypto/md5"
"encoding/base64"
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
@ -9,9 +12,12 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/util"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"github.com/google/uuid"
) )
const ( const (
@ -26,18 +32,18 @@ func (h *handler) GetBucketLifecycleHandler(w http.ResponseWriter, r *http.Reque
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
cfg, err := h.obj.GetBucketLifecycleConfiguration(ctx, bktInfo) cfg, err := h.obj.GetBucketLifecycleConfiguration(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket lifecycle configuration", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket lifecycle configuration", reqInfo, err)
return return
} }
if err = middleware.EncodeToResponse(w, cfg); err != nil { if err = middleware.EncodeToResponse(w, cfg); err != nil {
h.logAndSendError(w, "could not encode GetBucketLifecycle response", reqInfo, err) h.logAndSendError(ctx, w, "could not encode GetBucketLifecycle response", reqInfo, err)
return return
} }
} }
@ -52,42 +58,63 @@ func (h *handler) PutBucketLifecycleHandler(w http.ResponseWriter, r *http.Reque
// Content-Md5 is required and should be set // Content-Md5 is required and should be set
// https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html // https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html
if _, ok := r.Header[api.ContentMD5]; !ok { if _, ok := r.Header[api.ContentMD5]; !ok {
h.logAndSendError(w, "missing Content-MD5", reqInfo, apiErr.GetAPIError(apiErr.ErrMissingContentMD5)) h.logAndSendError(ctx, w, "missing Content-MD5", reqInfo, apierr.GetAPIError(apierr.ErrMissingContentMD5))
return return
} }
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) headerMD5, err := base64.StdEncoding.DecodeString(r.Header.Get(api.ContentMD5))
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "invalid Content-MD5", reqInfo, apierr.GetAPIError(apierr.ErrInvalidDigest))
return return
} }
cfg := new(data.LifecycleConfiguration) cfg := new(data.LifecycleConfiguration)
if err = h.cfg.NewXMLDecoder(tee).Decode(cfg); err != nil { if err = h.cfg.NewXMLDecoder(tee).Decode(cfg); err != nil {
h.logAndSendError(w, "could not decode body", reqInfo, fmt.Errorf("%w: %s", apiErr.GetAPIError(apiErr.ErrMalformedXML), err.Error())) h.logAndSendError(ctx, w, "could not decode body", reqInfo, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrMalformedXML), err.Error()))
return return
} }
if err = checkLifecycleConfiguration(cfg); err != nil { bodyMD5, err := getContentMD5(&buf)
h.logAndSendError(w, "invalid lifecycle configuration", reqInfo, fmt.Errorf("%w: %s", apiErr.GetAPIError(apiErr.ErrMalformedXML), err.Error())) if err != nil {
h.logAndSendError(ctx, w, "could not get content md5", reqInfo, err)
return
}
if !bytes.Equal(headerMD5, bodyMD5) {
h.logAndSendError(ctx, w, "Content-MD5 does not match", reqInfo, apierr.GetAPIError(apierr.ErrInvalidDigest))
return
}
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil {
h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return
}
networkInfo, err := h.obj.GetNetworkInfo(ctx)
if err != nil {
h.logAndSendError(ctx, w, "could not get network info", reqInfo, err)
return
}
if err = checkLifecycleConfiguration(ctx, cfg, &networkInfo); err != nil {
h.logAndSendError(ctx, w, "invalid lifecycle configuration", reqInfo, err)
return return
} }
params := &layer.PutBucketLifecycleParams{ params := &layer.PutBucketLifecycleParams{
BktInfo: bktInfo, BktInfo: bktInfo,
LifecycleCfg: cfg, LifecycleCfg: cfg,
LifecycleReader: &buf,
MD5Hash: r.Header.Get(api.ContentMD5),
} }
params.CopiesNumbers, err = h.pickCopiesNumbers(parseMetadata(r), reqInfo.Namespace, bktInfo.LocationConstraint) params.CopiesNumbers, err = h.pickCopiesNumbers(parseMetadata(r), reqInfo.Namespace, bktInfo.LocationConstraint)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid copies number", reqInfo, err) h.logAndSendError(ctx, w, "invalid copies number", reqInfo, err)
return return
} }
if err = h.obj.PutBucketLifecycleConfiguration(ctx, params); err != nil { if err = h.obj.PutBucketLifecycleConfiguration(ctx, params); err != nil {
h.logAndSendError(w, "could not put bucket lifecycle configuration", reqInfo, err) h.logAndSendError(ctx, w, "could not put bucket lifecycle configuration", reqInfo, err)
return return
} }
} }
@ -98,88 +125,118 @@ func (h *handler) DeleteBucketLifecycleHandler(w http.ResponseWriter, r *http.Re
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
if err = h.obj.DeleteBucketLifecycleConfiguration(ctx, bktInfo); err != nil { if err = h.obj.DeleteBucketLifecycleConfiguration(ctx, bktInfo); err != nil {
h.logAndSendError(w, "could not delete bucket lifecycle configuration", reqInfo, err) h.logAndSendError(ctx, w, "could not delete bucket lifecycle configuration", reqInfo, err)
return return
} }
w.WriteHeader(http.StatusNoContent) w.WriteHeader(http.StatusNoContent)
} }
func checkLifecycleConfiguration(cfg *data.LifecycleConfiguration) error { func checkLifecycleConfiguration(ctx context.Context, cfg *data.LifecycleConfiguration, ni *netmap.NetworkInfo) error {
now := layer.TimeNow(ctx)
if len(cfg.Rules) > maxRules { if len(cfg.Rules) > maxRules {
return fmt.Errorf("number of rules cannot be greater than %d", maxRules) return fmt.Errorf("%w: number of rules cannot be greater than %d", apierr.GetAPIError(apierr.ErrInvalidRequest), maxRules)
} }
ids := make(map[string]struct{}, len(cfg.Rules)) ids := make(map[string]struct{}, len(cfg.Rules))
for _, rule := range cfg.Rules { for i, rule := range cfg.Rules {
if _, ok := ids[rule.ID]; ok && rule.ID != "" { if rule.ID == "" {
return fmt.Errorf("duplicate 'ID': %s", rule.ID) id, err := uuid.NewRandom()
if err != nil {
return fmt.Errorf("generate uuid: %w", err)
}
cfg.Rules[i].ID = id.String()
rule.ID = id.String()
}
if _, ok := ids[rule.ID]; ok {
return fmt.Errorf("%w: duplicate 'ID': %s", apierr.GetAPIError(apierr.ErrInvalidArgument), rule.ID)
} }
ids[rule.ID] = struct{}{} ids[rule.ID] = struct{}{}
if len(rule.ID) > maxRuleIDLen { if len(rule.ID) > maxRuleIDLen {
return fmt.Errorf("'ID' value cannot be longer than %d characters", maxRuleIDLen) return fmt.Errorf("%w: 'ID' value cannot be longer than %d characters", apierr.GetAPIError(apierr.ErrInvalidArgument), maxRuleIDLen)
} }
if rule.Status != data.LifecycleStatusEnabled && rule.Status != data.LifecycleStatusDisabled { if rule.Status != data.LifecycleStatusEnabled && rule.Status != data.LifecycleStatusDisabled {
return fmt.Errorf("invalid lifecycle status: %s", rule.Status) return fmt.Errorf("%w: invalid lifecycle status: %s", apierr.GetAPIError(apierr.ErrMalformedXML), rule.Status)
} }
if rule.AbortIncompleteMultipartUpload == nil && rule.Expiration == nil && rule.NonCurrentVersionExpiration == nil { if rule.AbortIncompleteMultipartUpload == nil && rule.Expiration == nil && rule.NonCurrentVersionExpiration == nil {
return fmt.Errorf("at least one action needs to be specified in a rule") return fmt.Errorf("%w: at least one action needs to be specified in a rule", apierr.GetAPIError(apierr.ErrInvalidRequest))
} }
if rule.AbortIncompleteMultipartUpload != nil { if rule.AbortIncompleteMultipartUpload != nil {
if rule.AbortIncompleteMultipartUpload.DaysAfterInitiation != nil && if rule.AbortIncompleteMultipartUpload.DaysAfterInitiation != nil &&
*rule.AbortIncompleteMultipartUpload.DaysAfterInitiation <= 0 { *rule.AbortIncompleteMultipartUpload.DaysAfterInitiation <= 0 {
return fmt.Errorf("days after initiation must be a positive integer: %d", *rule.AbortIncompleteMultipartUpload.DaysAfterInitiation) return fmt.Errorf("%w: days after initiation must be a positive integer", apierr.GetAPIError(apierr.ErrInvalidArgument))
} }
if rule.Filter != nil && (rule.Filter.Tag != nil || (rule.Filter.And != nil && len(rule.Filter.And.Tags) > 0)) { if rule.Filter != nil && (rule.Filter.Tag != nil || (rule.Filter.And != nil && len(rule.Filter.And.Tags) > 0)) {
return fmt.Errorf("abort incomplete multipart upload cannot be specified with tags") return fmt.Errorf("%w: abort incomplete multipart upload cannot be specified with tags", apierr.GetAPIError(apierr.ErrInvalidRequest))
} }
} }
if rule.Expiration != nil { if rule.Expiration != nil {
if rule.Expiration.ExpiredObjectDeleteMarker != nil { if rule.Expiration.ExpiredObjectDeleteMarker != nil {
if rule.Expiration.Days != nil || rule.Expiration.Date != "" { if rule.Expiration.Days != nil || rule.Expiration.Date != "" {
return fmt.Errorf("expired object delete marker cannot be specified with days or date") return fmt.Errorf("%w: expired object delete marker cannot be specified with days or date", apierr.GetAPIError(apierr.ErrMalformedXML))
} }
if rule.Filter != nil && (rule.Filter.Tag != nil || (rule.Filter.And != nil && len(rule.Filter.And.Tags) > 0)) { if rule.Filter != nil && (rule.Filter.Tag != nil || (rule.Filter.And != nil && len(rule.Filter.And.Tags) > 0)) {
return fmt.Errorf("expired object delete marker cannot be specified with tags") return fmt.Errorf("%w: expired object delete marker cannot be specified with tags", apierr.GetAPIError(apierr.ErrInvalidRequest))
} }
} }
if rule.Expiration.Days != nil && *rule.Expiration.Days <= 0 { if rule.Expiration.Days != nil && *rule.Expiration.Days <= 0 {
return fmt.Errorf("expiration days must be a positive integer: %d", *rule.Expiration.Days) return fmt.Errorf("%w: expiration days must be a positive integer", apierr.GetAPIError(apierr.ErrInvalidArgument))
} }
if _, err := time.Parse("2006-01-02T15:04:05Z", rule.Expiration.Date); rule.Expiration.Date != "" && err != nil { if rule.Expiration.Date != "" {
return fmt.Errorf("invalid value of expiration date: %s", rule.Expiration.Date) parsedTime, err := time.Parse("2006-01-02T15:04:05Z", rule.Expiration.Date)
if err != nil {
return fmt.Errorf("%w: invalid value of expiration date: %s", apierr.GetAPIError(apierr.ErrInvalidArgument), rule.Expiration.Date)
}
epoch, err := util.TimeToEpoch(ni, now, parsedTime)
if err != nil {
return fmt.Errorf("convert time to epoch: %w", err)
}
cfg.Rules[i].Expiration.Epoch = &epoch
} }
} }
if rule.NonCurrentVersionExpiration != nil { if rule.NonCurrentVersionExpiration != nil {
if rule.NonCurrentVersionExpiration.NewerNonCurrentVersions != nil && rule.NonCurrentVersionExpiration.NonCurrentDays == nil {
return fmt.Errorf("%w: newer noncurrent versions cannot be specified without noncurrent days", apierr.GetAPIError(apierr.ErrMalformedXML))
}
if rule.NonCurrentVersionExpiration.NewerNonCurrentVersions != nil && if rule.NonCurrentVersionExpiration.NewerNonCurrentVersions != nil &&
(*rule.NonCurrentVersionExpiration.NewerNonCurrentVersions > maxNewerNoncurrentVersions || (*rule.NonCurrentVersionExpiration.NewerNonCurrentVersions > maxNewerNoncurrentVersions ||
*rule.NonCurrentVersionExpiration.NewerNonCurrentVersions <= 0) { *rule.NonCurrentVersionExpiration.NewerNonCurrentVersions <= 0) {
return fmt.Errorf("invalid value of newer noncurrent versions: %d", *rule.NonCurrentVersionExpiration.NewerNonCurrentVersions) return fmt.Errorf("%w: newer noncurrent versions must be a positive integer up to %d", apierr.GetAPIError(apierr.ErrInvalidArgument),
maxNewerNoncurrentVersions)
} }
if rule.NonCurrentVersionExpiration.NonCurrentDays != nil && *rule.NonCurrentVersionExpiration.NonCurrentDays <= 0 { if rule.NonCurrentVersionExpiration.NonCurrentDays != nil && *rule.NonCurrentVersionExpiration.NonCurrentDays <= 0 {
return fmt.Errorf("invalid value of noncurrent days: %d", *rule.NonCurrentVersionExpiration.NonCurrentDays) return fmt.Errorf("%w: noncurrent days must be a positive integer", apierr.GetAPIError(apierr.ErrInvalidArgument))
} }
} }
if err := checkLifecycleRuleFilter(rule.Filter); err != nil { if err := checkLifecycleRuleFilter(rule.Filter); err != nil {
return err return err
} }
if rule.Filter != nil && rule.Filter.Prefix != "" && rule.Prefix != "" {
return fmt.Errorf("%w: rule cannot have two prefixes", apierr.GetAPIError(apierr.ErrMalformedXML))
}
} }
return nil return nil
@ -201,9 +258,14 @@ func checkLifecycleRuleFilter(filter *data.LifecycleRuleFilter) error {
} }
} }
if filter.And.ObjectSizeGreaterThan != nil && filter.And.ObjectSizeLessThan != nil && if filter.And.ObjectSizeLessThan != nil {
*filter.And.ObjectSizeLessThan <= *filter.And.ObjectSizeGreaterThan { if *filter.And.ObjectSizeLessThan == 0 {
return fmt.Errorf("the maximum object size must be larger than the minimum object size") return fmt.Errorf("%w: the maximum object size must be more than 0", apierr.GetAPIError(apierr.ErrInvalidRequest))
}
if filter.And.ObjectSizeGreaterThan != nil && *filter.And.ObjectSizeLessThan <= *filter.And.ObjectSizeGreaterThan {
return fmt.Errorf("%w: the maximum object size must be larger than the minimum object size", apierr.GetAPIError(apierr.ErrInvalidRequest))
}
} }
} }
@ -212,6 +274,9 @@ func checkLifecycleRuleFilter(filter *data.LifecycleRuleFilter) error {
} }
if filter.ObjectSizeLessThan != nil { if filter.ObjectSizeLessThan != nil {
if *filter.ObjectSizeLessThan == 0 {
return fmt.Errorf("%w: the maximum object size must be more than 0", apierr.GetAPIError(apierr.ErrInvalidRequest))
}
fields++ fields++
} }
@ -228,8 +293,17 @@ func checkLifecycleRuleFilter(filter *data.LifecycleRuleFilter) error {
} }
if fields > 1 { if fields > 1 {
return fmt.Errorf("filter cannot have more than one field") return fmt.Errorf("%w: filter cannot have more than one field", apierr.GetAPIError(apierr.ErrMalformedXML))
} }
return nil return nil
} }
func getContentMD5(reader io.Reader) ([]byte, error) {
hash := md5.New()
_, err := io.Copy(hash, reader)
if err != nil {
return nil, err
}
return hash.Sum(nil), nil
}

View file

@ -1,6 +1,7 @@
package handler package handler
import ( import (
"bytes"
"crypto/md5" "crypto/md5"
"crypto/rand" "crypto/rand"
"encoding/base64" "encoding/base64"
@ -14,7 +15,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"github.com/mr-tron/base58" "github.com/mr-tron/base58"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@ -28,17 +29,14 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
for _, tc := range []struct { for _, tc := range []struct {
name string name string
body *data.LifecycleConfiguration body *data.LifecycleConfiguration
error bool errorCode apierr.ErrorCode
}{ }{
{ {
name: "correct configuration", name: "correct configuration",
body: &data.LifecycleConfiguration{ body: &data.LifecycleConfiguration{
XMLName: xml.Name{
Space: `http://s3.amazonaws.com/doc/2006-03-01/`,
Local: "LifecycleConfiguration",
},
Rules: []data.LifecycleRule{ Rules: []data.LifecycleRule{
{ {
ID: "rule-1",
Status: data.LifecycleStatusEnabled, Status: data.LifecycleStatusEnabled,
Expiration: &data.LifecycleExpiration{ Expiration: &data.LifecycleExpiration{
Days: ptr(21), Days: ptr(21),
@ -53,6 +51,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
{ {
ID: "rule-2",
Status: data.LifecycleStatusEnabled, Status: data.LifecycleStatusEnabled,
AbortIncompleteMultipartUpload: &data.AbortIncompleteMultipartUpload{ AbortIncompleteMultipartUpload: &data.AbortIncompleteMultipartUpload{
DaysAfterInitiation: ptr(14), DaysAfterInitiation: ptr(14),
@ -82,7 +81,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
} }
return lifecycle return lifecycle
}(), }(),
error: true, errorCode: apierr.ErrInvalidRequest,
}, },
{ {
name: "duplicate rule ID", name: "duplicate rule ID",
@ -104,7 +103,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
}, },
error: true, errorCode: apierr.ErrInvalidArgument,
}, },
{ {
name: "too long rule ID", name: "too long rule ID",
@ -120,7 +119,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
} }
}(), }(),
error: true, errorCode: apierr.ErrInvalidArgument,
}, },
{ {
name: "invalid status", name: "invalid status",
@ -131,7 +130,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
}, },
error: true, errorCode: apierr.ErrMalformedXML,
}, },
{ {
name: "no actions", name: "no actions",
@ -145,7 +144,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
}, },
error: true, errorCode: apierr.ErrInvalidRequest,
}, },
{ {
name: "invalid days after initiation", name: "invalid days after initiation",
@ -159,7 +158,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
}, },
error: true, errorCode: apierr.ErrInvalidArgument,
}, },
{ {
name: "invalid expired object delete marker declaration", name: "invalid expired object delete marker declaration",
@ -174,7 +173,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
}, },
error: true, errorCode: apierr.ErrMalformedXML,
}, },
{ {
name: "invalid expiration days", name: "invalid expiration days",
@ -188,7 +187,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
}, },
error: true, errorCode: apierr.ErrInvalidArgument,
}, },
{ {
name: "invalid expiration date", name: "invalid expiration date",
@ -202,7 +201,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
}, },
error: true, errorCode: apierr.ErrInvalidArgument,
}, },
{ {
name: "newer noncurrent versions is too small", name: "newer noncurrent versions is too small",
@ -211,12 +210,13 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
{ {
Status: data.LifecycleStatusEnabled, Status: data.LifecycleStatusEnabled,
NonCurrentVersionExpiration: &data.NonCurrentVersionExpiration{ NonCurrentVersionExpiration: &data.NonCurrentVersionExpiration{
NonCurrentDays: ptr(1),
NewerNonCurrentVersions: ptr(0), NewerNonCurrentVersions: ptr(0),
}, },
}, },
}, },
}, },
error: true, errorCode: apierr.ErrInvalidArgument,
}, },
{ {
name: "newer noncurrent versions is too large", name: "newer noncurrent versions is too large",
@ -225,12 +225,13 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
{ {
Status: data.LifecycleStatusEnabled, Status: data.LifecycleStatusEnabled,
NonCurrentVersionExpiration: &data.NonCurrentVersionExpiration{ NonCurrentVersionExpiration: &data.NonCurrentVersionExpiration{
NonCurrentDays: ptr(1),
NewerNonCurrentVersions: ptr(101), NewerNonCurrentVersions: ptr(101),
}, },
}, },
}, },
}, },
error: true, errorCode: apierr.ErrInvalidArgument,
}, },
{ {
name: "invalid noncurrent days", name: "invalid noncurrent days",
@ -244,7 +245,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
}, },
error: true, errorCode: apierr.ErrInvalidArgument,
}, },
{ {
name: "more than one filter field", name: "more than one filter field",
@ -262,7 +263,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
}, },
error: true, errorCode: apierr.ErrMalformedXML,
}, },
{ {
name: "invalid tag in filter", name: "invalid tag in filter",
@ -279,7 +280,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
}, },
error: true, errorCode: apierr.ErrInvalidTagKey,
}, },
{ {
name: "abort incomplete multipart upload with tag", name: "abort incomplete multipart upload with tag",
@ -296,7 +297,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
}, },
error: true, errorCode: apierr.ErrInvalidRequest,
}, },
{ {
name: "expired object delete marker with tag", name: "expired object delete marker with tag",
@ -315,7 +316,7 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
}, },
error: true, errorCode: apierr.ErrInvalidRequest,
}, },
{ {
name: "invalid size range", name: "invalid size range",
@ -335,26 +336,125 @@ func TestPutBucketLifecycleConfiguration(t *testing.T) {
}, },
}, },
}, },
error: true, errorCode: apierr.ErrInvalidRequest,
},
{
name: "two prefixes",
body: &data.LifecycleConfiguration{
Rules: []data.LifecycleRule{
{
Status: data.LifecycleStatusEnabled,
Expiration: &data.LifecycleExpiration{
Days: ptr(21),
},
Filter: &data.LifecycleRuleFilter{
Prefix: "prefix-1/",
},
Prefix: "prefix-2/",
},
},
},
errorCode: apierr.ErrMalformedXML,
},
{
name: "newer noncurrent versions without noncurrent days",
body: &data.LifecycleConfiguration{
Rules: []data.LifecycleRule{
{
Status: data.LifecycleStatusEnabled,
NonCurrentVersionExpiration: &data.NonCurrentVersionExpiration{
NewerNonCurrentVersions: ptr(10),
},
},
},
},
errorCode: apierr.ErrMalformedXML,
},
{
name: "invalid maximum object size in filter",
body: &data.LifecycleConfiguration{
Rules: []data.LifecycleRule{
{
Status: data.LifecycleStatusEnabled,
Expiration: &data.LifecycleExpiration{
Days: ptr(21),
},
Filter: &data.LifecycleRuleFilter{
ObjectSizeLessThan: ptr(uint64(0)),
},
},
},
},
errorCode: apierr.ErrInvalidRequest,
},
{
name: "invalid maximum object size in filter and",
body: &data.LifecycleConfiguration{
Rules: []data.LifecycleRule{
{
Status: data.LifecycleStatusEnabled,
Expiration: &data.LifecycleExpiration{
Days: ptr(21),
},
Filter: &data.LifecycleRuleFilter{
And: &data.LifecycleRuleAndOperator{
Prefix: "prefix/",
ObjectSizeLessThan: ptr(uint64(0)),
},
},
},
},
},
errorCode: apierr.ErrInvalidRequest,
}, },
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
if tc.error { if tc.errorCode > 0 {
putBucketLifecycleConfigurationErr(hc, bktName, tc.body, apiErrors.GetAPIError(apiErrors.ErrMalformedXML)) putBucketLifecycleConfigurationErr(hc, bktName, tc.body, apierr.GetAPIError(tc.errorCode))
return return
} }
putBucketLifecycleConfiguration(hc, bktName, tc.body) putBucketLifecycleConfiguration(hc, bktName, tc.body)
cfg := getBucketLifecycleConfiguration(hc, bktName) cfg := getBucketLifecycleConfiguration(hc, bktName)
require.Equal(t, *tc.body, *cfg) require.Equal(t, tc.body.Rules, cfg.Rules)
deleteBucketLifecycleConfiguration(hc, bktName) deleteBucketLifecycleConfiguration(hc, bktName)
getBucketLifecycleConfigurationErr(hc, bktName, apiErrors.GetAPIError(apiErrors.ErrNoSuchLifecycleConfiguration)) getBucketLifecycleConfigurationErr(hc, bktName, apierr.GetAPIError(apierr.ErrNoSuchLifecycleConfiguration))
}) })
} }
} }
func TestPutBucketLifecycleIDGeneration(t *testing.T) {
hc := prepareHandlerContext(t)
bktName := "bucket-lifecycle-id"
createBucket(hc, bktName)
lifecycle := &data.LifecycleConfiguration{
Rules: []data.LifecycleRule{
{
Status: data.LifecycleStatusEnabled,
Expiration: &data.LifecycleExpiration{
Days: ptr(21),
},
},
{
Status: data.LifecycleStatusEnabled,
AbortIncompleteMultipartUpload: &data.AbortIncompleteMultipartUpload{
DaysAfterInitiation: ptr(14),
},
},
},
}
putBucketLifecycleConfiguration(hc, bktName, lifecycle)
cfg := getBucketLifecycleConfiguration(hc, bktName)
require.Len(t, cfg.Rules, 2)
require.NotEmpty(t, cfg.Rules[0].ID)
require.NotEmpty(t, cfg.Rules[1].ID)
}
func TestPutBucketLifecycleInvalidMD5(t *testing.T) { func TestPutBucketLifecycleInvalidMD5(t *testing.T) {
hc := prepareHandlerContext(t) hc := prepareHandlerContext(t)
@ -374,17 +474,17 @@ func TestPutBucketLifecycleInvalidMD5(t *testing.T) {
w, r := prepareTestRequest(hc, bktName, "", lifecycle) w, r := prepareTestRequest(hc, bktName, "", lifecycle)
hc.Handler().PutBucketLifecycleHandler(w, r) hc.Handler().PutBucketLifecycleHandler(w, r)
assertS3Error(hc.t, w, apiErrors.GetAPIError(apiErrors.ErrMissingContentMD5)) assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrMissingContentMD5))
w, r = prepareTestRequest(hc, bktName, "", lifecycle) w, r = prepareTestRequest(hc, bktName, "", lifecycle)
r.Header.Set(api.ContentMD5, "") r.Header.Set(api.ContentMD5, "")
hc.Handler().PutBucketLifecycleHandler(w, r) hc.Handler().PutBucketLifecycleHandler(w, r)
assertS3Error(hc.t, w, apiErrors.GetAPIError(apiErrors.ErrInvalidDigest)) assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrInvalidDigest))
w, r = prepareTestRequest(hc, bktName, "", lifecycle) w, r = prepareTestRequest(hc, bktName, "", lifecycle)
r.Header.Set(api.ContentMD5, "some-hash") r.Header.Set(api.ContentMD5, "some-hash")
hc.Handler().PutBucketLifecycleHandler(w, r) hc.Handler().PutBucketLifecycleHandler(w, r)
assertS3Error(hc.t, w, apiErrors.GetAPIError(apiErrors.ErrInvalidDigest)) assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrInvalidDigest))
} }
func TestPutBucketLifecycleInvalidXML(t *testing.T) { func TestPutBucketLifecycleInvalidXML(t *testing.T) {
@ -393,10 +493,16 @@ func TestPutBucketLifecycleInvalidXML(t *testing.T) {
bktName := "bucket-lifecycle-invalid-xml" bktName := "bucket-lifecycle-invalid-xml"
createBucket(hc, bktName) createBucket(hc, bktName)
w, r := prepareTestRequest(hc, bktName, "", &data.CORSConfiguration{}) cfg := &data.CORSConfiguration{}
r.Header.Set(api.ContentMD5, "") body, err := xml.Marshal(cfg)
require.NoError(t, err)
contentMD5, err := getContentMD5(bytes.NewReader(body))
require.NoError(t, err)
w, r := prepareTestRequest(hc, bktName, "", cfg)
r.Header.Set(api.ContentMD5, base64.StdEncoding.EncodeToString(contentMD5))
hc.Handler().PutBucketLifecycleHandler(w, r) hc.Handler().PutBucketLifecycleHandler(w, r)
assertS3Error(hc.t, w, apiErrors.GetAPIError(apiErrors.ErrMalformedXML)) assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrMalformedXML))
} }
func putBucketLifecycleConfiguration(hc *handlerContext, bktName string, cfg *data.LifecycleConfiguration) { func putBucketLifecycleConfiguration(hc *handlerContext, bktName string, cfg *data.LifecycleConfiguration) {
@ -404,7 +510,7 @@ func putBucketLifecycleConfiguration(hc *handlerContext, bktName string, cfg *da
assertStatus(hc.t, w, http.StatusOK) assertStatus(hc.t, w, http.StatusOK)
} }
func putBucketLifecycleConfigurationErr(hc *handlerContext, bktName string, cfg *data.LifecycleConfiguration, err apiErrors.Error) { func putBucketLifecycleConfigurationErr(hc *handlerContext, bktName string, cfg *data.LifecycleConfiguration, err apierr.Error) {
w := putBucketLifecycleConfigurationBase(hc, bktName, cfg) w := putBucketLifecycleConfigurationBase(hc, bktName, cfg)
assertS3Error(hc.t, w, err) assertS3Error(hc.t, w, err)
} }
@ -430,7 +536,7 @@ func getBucketLifecycleConfiguration(hc *handlerContext, bktName string) *data.L
return res return res
} }
func getBucketLifecycleConfigurationErr(hc *handlerContext, bktName string, err apiErrors.Error) { func getBucketLifecycleConfigurationErr(hc *handlerContext, bktName string, err apierr.Error) {
w := getBucketLifecycleConfigurationBase(hc, bktName) w := getBucketLifecycleConfigurationBase(hc, bktName)
assertS3Error(hc.t, w, err) assertS3Error(hc.t, w, err)
} }

View file

@ -15,12 +15,13 @@ func (h *handler) ListBucketsHandler(w http.ResponseWriter, r *http.Request) {
var ( var (
own user.ID own user.ID
res *ListBucketsResponse res *ListBucketsResponse
reqInfo = middleware.GetReqInfo(r.Context()) ctx = r.Context()
reqInfo = middleware.GetReqInfo(ctx)
) )
list, err := h.obj.ListBuckets(r.Context()) list, err := h.obj.ListBuckets(ctx)
if err != nil { if err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
return return
} }
@ -43,6 +44,6 @@ func (h *handler) ListBucketsHandler(w http.ResponseWriter, r *http.Request) {
} }
if err = middleware.EncodeToResponse(w, res); err != nil { if err = middleware.EncodeToResponse(w, res); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
} }
} }

View file

@ -9,7 +9,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
) )
@ -26,34 +26,35 @@ const (
) )
func (h *handler) PutBucketObjectLockConfigHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) PutBucketObjectLockConfigHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
if !bktInfo.ObjectLockEnabled { if !bktInfo.ObjectLockEnabled {
h.logAndSendError(w, "couldn't put object locking configuration", reqInfo, h.logAndSendError(ctx, w, "couldn't put object locking configuration", reqInfo,
apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotAllowed)) apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotAllowed))
return return
} }
lockingConf := &data.ObjectLockConfiguration{} lockingConf := &data.ObjectLockConfiguration{}
if err = h.cfg.NewXMLDecoder(r.Body).Decode(lockingConf); err != nil { if err = h.cfg.NewXMLDecoder(r.Body).Decode(lockingConf); err != nil {
h.logAndSendError(w, "couldn't parse locking configuration", reqInfo, err) h.logAndSendError(ctx, w, "couldn't parse locking configuration", reqInfo, err)
return return
} }
if err = checkLockConfiguration(lockingConf); err != nil { if err = checkLockConfiguration(lockingConf); err != nil {
h.logAndSendError(w, "invalid lock configuration", reqInfo, err) h.logAndSendError(ctx, w, "invalid lock configuration", reqInfo, err)
return return
} }
settings, err := h.obj.GetBucketSettings(r.Context(), bktInfo) settings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "couldn't get bucket settings", reqInfo, err)
return return
} }
@ -66,30 +67,31 @@ func (h *handler) PutBucketObjectLockConfigHandler(w http.ResponseWriter, r *htt
Settings: &newSettings, Settings: &newSettings,
} }
if err = h.obj.PutBucketSettings(r.Context(), sp); err != nil { if err = h.obj.PutBucketSettings(ctx, sp); err != nil {
h.logAndSendError(w, "couldn't put bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "couldn't put bucket settings", reqInfo, err)
return return
} }
} }
func (h *handler) GetBucketObjectLockConfigHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketObjectLockConfigHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
if !bktInfo.ObjectLockEnabled { if !bktInfo.ObjectLockEnabled {
h.logAndSendError(w, "object lock disabled", reqInfo, h.logAndSendError(ctx, w, "object lock disabled", reqInfo,
apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound)) apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound))
return return
} }
settings, err := h.obj.GetBucketSettings(r.Context(), bktInfo) settings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "couldn't get bucket settings", reqInfo, err)
return return
} }
@ -101,33 +103,34 @@ func (h *handler) GetBucketObjectLockConfigHandler(w http.ResponseWriter, r *htt
} }
if err = middleware.EncodeToResponse(w, settings.LockConfiguration); err != nil { if err = middleware.EncodeToResponse(w, settings.LockConfiguration); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
} }
} }
func (h *handler) PutObjectLegalHoldHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) PutObjectLegalHoldHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
if !bktInfo.ObjectLockEnabled { if !bktInfo.ObjectLockEnabled {
h.logAndSendError(w, "object lock disabled", reqInfo, h.logAndSendError(ctx, w, "object lock disabled", reqInfo,
apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound)) apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound))
return return
} }
legalHold := &data.LegalHold{} legalHold := &data.LegalHold{}
if err = h.cfg.NewXMLDecoder(r.Body).Decode(legalHold); err != nil { if err = h.cfg.NewXMLDecoder(r.Body).Decode(legalHold); err != nil {
h.logAndSendError(w, "couldn't parse legal hold configuration", reqInfo, err) h.logAndSendError(ctx, w, "couldn't parse legal hold configuration", reqInfo, err)
return return
} }
if legalHold.Status != legalHoldOn && legalHold.Status != legalHoldOff { if legalHold.Status != legalHoldOn && legalHold.Status != legalHoldOff {
h.logAndSendError(w, "invalid legal hold status", reqInfo, h.logAndSendError(ctx, w, "invalid legal hold status", reqInfo,
fmt.Errorf("invalid status %s", legalHold.Status)) fmt.Errorf("invalid status %s", legalHold.Status))
return return
} }
@ -147,28 +150,29 @@ func (h *handler) PutObjectLegalHoldHandler(w http.ResponseWriter, r *http.Reque
p.CopiesNumbers, err = h.pickCopiesNumbers(parseMetadata(r), reqInfo.Namespace, bktInfo.LocationConstraint) p.CopiesNumbers, err = h.pickCopiesNumbers(parseMetadata(r), reqInfo.Namespace, bktInfo.LocationConstraint)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid copies number", reqInfo, err) h.logAndSendError(ctx, w, "invalid copies number", reqInfo, err)
return return
} }
if err = h.obj.PutLockInfo(r.Context(), p); err != nil { if err = h.obj.PutLockInfo(ctx, p); err != nil {
h.logAndSendError(w, "couldn't head put legal hold", reqInfo, err) h.logAndSendError(ctx, w, "couldn't head put legal hold", reqInfo, err)
return return
} }
} }
func (h *handler) GetObjectLegalHoldHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetObjectLegalHoldHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
if !bktInfo.ObjectLockEnabled { if !bktInfo.ObjectLockEnabled {
h.logAndSendError(w, "object lock disabled", reqInfo, h.logAndSendError(ctx, w, "object lock disabled", reqInfo,
apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound)) apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound))
return return
} }
@ -178,9 +182,9 @@ func (h *handler) GetObjectLegalHoldHandler(w http.ResponseWriter, r *http.Reque
VersionID: reqInfo.URL.Query().Get(api.QueryVersionID), VersionID: reqInfo.URL.Query().Get(api.QueryVersionID),
} }
lockInfo, err := h.obj.GetLockInfo(r.Context(), p) lockInfo, err := h.obj.GetLockInfo(ctx, p)
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't head lock object", reqInfo, err) h.logAndSendError(ctx, w, "couldn't head lock object", reqInfo, err)
return return
} }
@ -190,33 +194,34 @@ func (h *handler) GetObjectLegalHoldHandler(w http.ResponseWriter, r *http.Reque
} }
if err = middleware.EncodeToResponse(w, legalHold); err != nil { if err = middleware.EncodeToResponse(w, legalHold); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
} }
} }
func (h *handler) PutObjectRetentionHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) PutObjectRetentionHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
if !bktInfo.ObjectLockEnabled { if !bktInfo.ObjectLockEnabled {
h.logAndSendError(w, "object lock disabled", reqInfo, h.logAndSendError(ctx, w, "object lock disabled", reqInfo,
apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound)) apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound))
return return
} }
retention := &data.Retention{} retention := &data.Retention{}
if err = h.cfg.NewXMLDecoder(r.Body).Decode(retention); err != nil { if err = h.cfg.NewXMLDecoder(r.Body).Decode(retention); err != nil {
h.logAndSendError(w, "couldn't parse object retention", reqInfo, err) h.logAndSendError(ctx, w, "couldn't parse object retention", reqInfo, err)
return return
} }
lock, err := formObjectLockFromRetention(r.Context(), retention, r.Header) lock, err := formObjectLockFromRetention(ctx, retention, r.Header)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid retention configuration", reqInfo, err) h.logAndSendError(ctx, w, "invalid retention configuration", reqInfo, err)
return return
} }
@ -231,28 +236,29 @@ func (h *handler) PutObjectRetentionHandler(w http.ResponseWriter, r *http.Reque
p.CopiesNumbers, err = h.pickCopiesNumbers(parseMetadata(r), reqInfo.Namespace, bktInfo.LocationConstraint) p.CopiesNumbers, err = h.pickCopiesNumbers(parseMetadata(r), reqInfo.Namespace, bktInfo.LocationConstraint)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid copies number", reqInfo, err) h.logAndSendError(ctx, w, "invalid copies number", reqInfo, err)
return return
} }
if err = h.obj.PutLockInfo(r.Context(), p); err != nil { if err = h.obj.PutLockInfo(ctx, p); err != nil {
h.logAndSendError(w, "couldn't put legal hold", reqInfo, err) h.logAndSendError(ctx, w, "couldn't put legal hold", reqInfo, err)
return return
} }
} }
func (h *handler) GetObjectRetentionHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetObjectRetentionHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
if !bktInfo.ObjectLockEnabled { if !bktInfo.ObjectLockEnabled {
h.logAndSendError(w, "object lock disabled", reqInfo, h.logAndSendError(ctx, w, "object lock disabled", reqInfo,
apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound)) apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound))
return return
} }
@ -262,14 +268,14 @@ func (h *handler) GetObjectRetentionHandler(w http.ResponseWriter, r *http.Reque
VersionID: reqInfo.URL.Query().Get(api.QueryVersionID), VersionID: reqInfo.URL.Query().Get(api.QueryVersionID),
} }
lockInfo, err := h.obj.GetLockInfo(r.Context(), p) lockInfo, err := h.obj.GetLockInfo(ctx, p)
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't head lock object", reqInfo, err) h.logAndSendError(ctx, w, "couldn't head lock object", reqInfo, err)
return return
} }
if !lockInfo.IsRetentionSet() { if !lockInfo.IsRetentionSet() {
h.logAndSendError(w, "retention lock isn't set", reqInfo, apiErrors.GetAPIError(apiErrors.ErrNoSuchKey)) h.logAndSendError(ctx, w, "retention lock isn't set", reqInfo, apierr.GetAPIError(apierr.ErrNoSuchKey))
return return
} }
@ -282,7 +288,7 @@ func (h *handler) GetObjectRetentionHandler(w http.ResponseWriter, r *http.Reque
} }
if err = middleware.EncodeToResponse(w, retention); err != nil { if err = middleware.EncodeToResponse(w, retention); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
} }
} }
@ -314,7 +320,7 @@ func checkLockConfiguration(conf *data.ObjectLockConfiguration) error {
func formObjectLock(ctx context.Context, bktInfo *data.BucketInfo, defaultConfig *data.ObjectLockConfiguration, header http.Header) (*data.ObjectLock, error) { func formObjectLock(ctx context.Context, bktInfo *data.BucketInfo, defaultConfig *data.ObjectLockConfiguration, header http.Header) (*data.ObjectLock, error) {
if !bktInfo.ObjectLockEnabled { if !bktInfo.ObjectLockEnabled {
if existLockHeaders(header) { if existLockHeaders(header) {
return nil, apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound) return nil, apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound)
} }
return nil, nil return nil, nil
} }
@ -346,7 +352,7 @@ func formObjectLock(ctx context.Context, bktInfo *data.BucketInfo, defaultConfig
until := header.Get(api.AmzObjectLockRetainUntilDate) until := header.Get(api.AmzObjectLockRetainUntilDate)
if mode != "" && until == "" || mode == "" && until != "" { if mode != "" && until == "" || mode == "" && until != "" {
return nil, apiErrors.GetAPIError(apiErrors.ErrObjectLockInvalidHeaders) return nil, apierr.GetAPIError(apierr.ErrObjectLockInvalidHeaders)
} }
if mode != "" { if mode != "" {
@ -355,7 +361,7 @@ func formObjectLock(ctx context.Context, bktInfo *data.BucketInfo, defaultConfig
} }
if mode != complianceMode && mode != governanceMode { if mode != complianceMode && mode != governanceMode {
return nil, apiErrors.GetAPIError(apiErrors.ErrUnknownWORMModeDirective) return nil, apierr.GetAPIError(apierr.ErrUnknownWORMModeDirective)
} }
objectLock.Retention.IsCompliance = mode == complianceMode objectLock.Retention.IsCompliance = mode == complianceMode
@ -364,7 +370,7 @@ func formObjectLock(ctx context.Context, bktInfo *data.BucketInfo, defaultConfig
if until != "" { if until != "" {
retentionDate, err := time.Parse(time.RFC3339, until) retentionDate, err := time.Parse(time.RFC3339, until)
if err != nil { if err != nil {
return nil, apiErrors.GetAPIError(apiErrors.ErrInvalidRetentionDate) return nil, apierr.GetAPIError(apierr.ErrInvalidRetentionDate)
} }
if objectLock.Retention == nil { if objectLock.Retention == nil {
objectLock.Retention = &data.RetentionLock{} objectLock.Retention = &data.RetentionLock{}
@ -382,7 +388,7 @@ func formObjectLock(ctx context.Context, bktInfo *data.BucketInfo, defaultConfig
} }
if objectLock.Retention.Until.Before(layer.TimeNow(ctx)) { if objectLock.Retention.Until.Before(layer.TimeNow(ctx)) {
return nil, apiErrors.GetAPIError(apiErrors.ErrPastObjectLockRetainDate) return nil, apierr.GetAPIError(apierr.ErrPastObjectLockRetainDate)
} }
} }
@ -397,16 +403,16 @@ func existLockHeaders(header http.Header) bool {
func formObjectLockFromRetention(ctx context.Context, retention *data.Retention, header http.Header) (*data.ObjectLock, error) { func formObjectLockFromRetention(ctx context.Context, retention *data.Retention, header http.Header) (*data.ObjectLock, error) {
if retention.Mode != governanceMode && retention.Mode != complianceMode { if retention.Mode != governanceMode && retention.Mode != complianceMode {
return nil, apiErrors.GetAPIError(apiErrors.ErrMalformedXML) return nil, apierr.GetAPIError(apierr.ErrMalformedXML)
} }
retentionDate, err := time.Parse(time.RFC3339, retention.RetainUntilDate) retentionDate, err := time.Parse(time.RFC3339, retention.RetainUntilDate)
if err != nil { if err != nil {
return nil, apiErrors.GetAPIError(apiErrors.ErrMalformedXML) return nil, apierr.GetAPIError(apierr.ErrMalformedXML)
} }
if retentionDate.Before(layer.TimeNow(ctx)) { if retentionDate.Before(layer.TimeNow(ctx)) {
return nil, apiErrors.GetAPIError(apiErrors.ErrPastObjectLockRetainDate) return nil, apierr.GetAPIError(apierr.ErrPastObjectLockRetainDate)
} }
var bypass bool var bypass bool

View file

@ -12,7 +12,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@ -270,23 +270,24 @@ func TestPutBucketLockConfigurationHandler(t *testing.T) {
for _, tc := range []struct { for _, tc := range []struct {
name string name string
bucket string bucket string
expectedError apiErrors.Error expectedError apierr.Error
noError bool noError bool
configuration *data.ObjectLockConfiguration configuration *data.ObjectLockConfiguration
}{ }{
{ {
name: "bkt not found", name: "bkt not found",
expectedError: apiErrors.GetAPIError(apiErrors.ErrNoSuchBucket), bucket: "not-found-bucket",
expectedError: apierr.GetAPIError(apierr.ErrNoSuchBucket),
}, },
{ {
name: "bkt lock disabled", name: "bkt lock disabled",
bucket: bktLockDisabled, bucket: bktLockDisabled,
expectedError: apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotAllowed), expectedError: apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotAllowed),
}, },
{ {
name: "invalid configuration", name: "invalid configuration",
bucket: bktLockEnabled, bucket: bktLockEnabled,
expectedError: apiErrors.GetAPIError(apiErrors.ErrInternalError), expectedError: apierr.GetAPIError(apierr.ErrInternalError),
configuration: &data.ObjectLockConfiguration{ObjectLockEnabled: "dummy"}, configuration: &data.ObjectLockConfiguration{ObjectLockEnabled: "dummy"},
}, },
{ {
@ -359,18 +360,19 @@ func TestGetBucketLockConfigurationHandler(t *testing.T) {
for _, tc := range []struct { for _, tc := range []struct {
name string name string
bucket string bucket string
expectedError apiErrors.Error expectedError apierr.Error
noError bool noError bool
expectedConf *data.ObjectLockConfiguration expectedConf *data.ObjectLockConfiguration
}{ }{
{ {
name: "bkt not found", name: "bkt not found",
expectedError: apiErrors.GetAPIError(apiErrors.ErrNoSuchBucket), bucket: "not-found-bucket",
expectedError: apierr.GetAPIError(apierr.ErrNoSuchBucket),
}, },
{ {
name: "bkt lock disabled", name: "bkt lock disabled",
bucket: bktLockDisabled, bucket: bktLockDisabled,
expectedError: apiErrors.GetAPIError(apiErrors.ErrObjectLockConfigurationNotFound), expectedError: apierr.GetAPIError(apierr.ErrObjectLockConfigurationNotFound),
}, },
{ {
name: "bkt lock enabled empty default", name: "bkt lock enabled empty default",
@ -407,7 +409,7 @@ func TestGetBucketLockConfigurationHandler(t *testing.T) {
} }
} }
func assertS3Error(t *testing.T, w *httptest.ResponseRecorder, expectedError apiErrors.Error) { func assertS3Error(t *testing.T, w *httptest.ResponseRecorder, expectedError apierr.Error) {
actualErrorResponse := &middleware.ErrorResponse{} actualErrorResponse := &middleware.ErrorResponse{}
err := xml.NewDecoder(w.Result().Body).Decode(actualErrorResponse) err := xml.NewDecoder(w.Result().Body).Decode(actualErrorResponse)
require.NoError(t, err) require.NoError(t, err)
@ -415,7 +417,7 @@ func assertS3Error(t *testing.T, w *httptest.ResponseRecorder, expectedError api
require.Equal(t, expectedError.HTTPStatusCode, w.Code) require.Equal(t, expectedError.HTTPStatusCode, w.Code)
require.Equal(t, expectedError.Code, actualErrorResponse.Code) require.Equal(t, expectedError.Code, actualErrorResponse.Code)
if expectedError.ErrCode != apiErrors.ErrInternalError { if expectedError.ErrCode != apierr.ErrInternalError {
require.Equal(t, expectedError.Description, actualErrorResponse.Message) require.Equal(t, expectedError.Description, actualErrorResponse.Message)
} }
} }
@ -473,33 +475,33 @@ func TestObjectRetention(t *testing.T) {
objName := "obj-for-retention" objName := "obj-for-retention"
createTestObject(hc, bktInfo, objName, encryption.Params{}) createTestObject(hc, bktInfo, objName, encryption.Params{})
getObjectRetention(hc, bktName, objName, nil, apiErrors.ErrNoSuchKey) getObjectRetention(hc, bktName, objName, nil, apierr.ErrNoSuchKey)
retention := &data.Retention{Mode: governanceMode, RetainUntilDate: time.Now().Add(time.Minute).UTC().Format(time.RFC3339)} retention := &data.Retention{Mode: governanceMode, RetainUntilDate: time.Now().Add(time.Minute).UTC().Format(time.RFC3339)}
putObjectRetention(hc, bktName, objName, retention, false, 0) putObjectRetention(hc, bktName, objName, retention, false, 0)
getObjectRetention(hc, bktName, objName, retention, 0) getObjectRetention(hc, bktName, objName, retention, 0)
retention = &data.Retention{Mode: governanceMode, RetainUntilDate: time.Now().UTC().Add(time.Minute).Format(time.RFC3339)} retention = &data.Retention{Mode: governanceMode, RetainUntilDate: time.Now().UTC().Add(time.Minute).Format(time.RFC3339)}
putObjectRetention(hc, bktName, objName, retention, false, apiErrors.ErrInternalError) putObjectRetention(hc, bktName, objName, retention, false, apierr.ErrInternalError)
retention = &data.Retention{Mode: complianceMode, RetainUntilDate: time.Now().Add(time.Minute).UTC().Format(time.RFC3339)} retention = &data.Retention{Mode: complianceMode, RetainUntilDate: time.Now().Add(time.Minute).UTC().Format(time.RFC3339)}
putObjectRetention(hc, bktName, objName, retention, true, 0) putObjectRetention(hc, bktName, objName, retention, true, 0)
getObjectRetention(hc, bktName, objName, retention, 0) getObjectRetention(hc, bktName, objName, retention, 0)
putObjectRetention(hc, bktName, objName, retention, true, apiErrors.ErrInternalError) putObjectRetention(hc, bktName, objName, retention, true, apierr.ErrInternalError)
} }
func getObjectRetention(hc *handlerContext, bktName, objName string, retention *data.Retention, errCode apiErrors.ErrorCode) { func getObjectRetention(hc *handlerContext, bktName, objName string, retention *data.Retention, errCode apierr.ErrorCode) {
w, r := prepareTestRequest(hc, bktName, objName, nil) w, r := prepareTestRequest(hc, bktName, objName, nil)
hc.Handler().GetObjectRetentionHandler(w, r) hc.Handler().GetObjectRetentionHandler(w, r)
if errCode == 0 { if errCode == 0 {
assertRetention(hc.t, w, retention) assertRetention(hc.t, w, retention)
} else { } else {
assertS3Error(hc.t, w, apiErrors.GetAPIError(errCode)) assertS3Error(hc.t, w, apierr.GetAPIError(errCode))
} }
} }
func putObjectRetention(hc *handlerContext, bktName, objName string, retention *data.Retention, byPass bool, errCode apiErrors.ErrorCode) { func putObjectRetention(hc *handlerContext, bktName, objName string, retention *data.Retention, byPass bool, errCode apierr.ErrorCode) {
w, r := prepareTestRequest(hc, bktName, objName, retention) w, r := prepareTestRequest(hc, bktName, objName, retention)
if byPass { if byPass {
r.Header.Set(api.AmzBypassGovernanceRetention, strconv.FormatBool(true)) r.Header.Set(api.AmzBypassGovernanceRetention, strconv.FormatBool(true))
@ -508,7 +510,7 @@ func putObjectRetention(hc *handlerContext, bktName, objName string, retention *
if errCode == 0 { if errCode == 0 {
assertStatus(hc.t, w, http.StatusOK) assertStatus(hc.t, w, http.StatusOK)
} else { } else {
assertS3Error(hc.t, w, apiErrors.GetAPIError(errCode)) assertS3Error(hc.t, w, apierr.GetAPIError(errCode))
} }
} }
@ -572,37 +574,37 @@ func TestPutLockErrors(t *testing.T) {
createTestBucketWithLock(hc, bktName, nil) createTestBucketWithLock(hc, bktName, nil)
headers := map[string]string{api.AmzObjectLockMode: complianceMode} headers := map[string]string{api.AmzObjectLockMode: complianceMode}
putObjectWithLockFailed(t, hc, bktName, objName, headers, apiErrors.ErrObjectLockInvalidHeaders) putObjectWithLockFailed(t, hc, bktName, objName, headers, apierr.ErrObjectLockInvalidHeaders)
delete(headers, api.AmzObjectLockMode) delete(headers, api.AmzObjectLockMode)
headers[api.AmzObjectLockRetainUntilDate] = time.Now().Add(time.Minute).Format(time.RFC3339) headers[api.AmzObjectLockRetainUntilDate] = time.Now().Add(time.Minute).Format(time.RFC3339)
putObjectWithLockFailed(t, hc, bktName, objName, headers, apiErrors.ErrObjectLockInvalidHeaders) putObjectWithLockFailed(t, hc, bktName, objName, headers, apierr.ErrObjectLockInvalidHeaders)
headers[api.AmzObjectLockMode] = "dummy" headers[api.AmzObjectLockMode] = "dummy"
putObjectWithLockFailed(t, hc, bktName, objName, headers, apiErrors.ErrUnknownWORMModeDirective) putObjectWithLockFailed(t, hc, bktName, objName, headers, apierr.ErrUnknownWORMModeDirective)
headers[api.AmzObjectLockMode] = complianceMode headers[api.AmzObjectLockMode] = complianceMode
headers[api.AmzObjectLockRetainUntilDate] = time.Now().Format(time.RFC3339) headers[api.AmzObjectLockRetainUntilDate] = time.Now().Format(time.RFC3339)
putObjectWithLockFailed(t, hc, bktName, objName, headers, apiErrors.ErrPastObjectLockRetainDate) putObjectWithLockFailed(t, hc, bktName, objName, headers, apierr.ErrPastObjectLockRetainDate)
headers[api.AmzObjectLockRetainUntilDate] = "dummy" headers[api.AmzObjectLockRetainUntilDate] = "dummy"
putObjectWithLockFailed(t, hc, bktName, objName, headers, apiErrors.ErrInvalidRetentionDate) putObjectWithLockFailed(t, hc, bktName, objName, headers, apierr.ErrInvalidRetentionDate)
putObject(hc, bktName, objName) putObject(hc, bktName, objName)
retention := &data.Retention{Mode: governanceMode} retention := &data.Retention{Mode: governanceMode}
putObjectRetentionFailed(t, hc, bktName, objName, retention, apiErrors.ErrMalformedXML) putObjectRetentionFailed(t, hc, bktName, objName, retention, apierr.ErrMalformedXML)
retention.Mode = "dummy" retention.Mode = "dummy"
retention.RetainUntilDate = time.Now().Add(time.Minute).UTC().Format(time.RFC3339) retention.RetainUntilDate = time.Now().Add(time.Minute).UTC().Format(time.RFC3339)
putObjectRetentionFailed(t, hc, bktName, objName, retention, apiErrors.ErrMalformedXML) putObjectRetentionFailed(t, hc, bktName, objName, retention, apierr.ErrMalformedXML)
retention.Mode = governanceMode retention.Mode = governanceMode
retention.RetainUntilDate = time.Now().UTC().Format(time.RFC3339) retention.RetainUntilDate = time.Now().UTC().Format(time.RFC3339)
putObjectRetentionFailed(t, hc, bktName, objName, retention, apiErrors.ErrPastObjectLockRetainDate) putObjectRetentionFailed(t, hc, bktName, objName, retention, apierr.ErrPastObjectLockRetainDate)
} }
func putObjectWithLockFailed(t *testing.T, hc *handlerContext, bktName, objName string, headers map[string]string, errCode apiErrors.ErrorCode) { func putObjectWithLockFailed(t *testing.T, hc *handlerContext, bktName, objName string, headers map[string]string, errCode apierr.ErrorCode) {
w, r := prepareTestRequest(hc, bktName, objName, nil) w, r := prepareTestRequest(hc, bktName, objName, nil)
for key, val := range headers { for key, val := range headers {
@ -610,13 +612,13 @@ func putObjectWithLockFailed(t *testing.T, hc *handlerContext, bktName, objName
} }
hc.Handler().PutObjectHandler(w, r) hc.Handler().PutObjectHandler(w, r)
assertS3Error(t, w, apiErrors.GetAPIError(errCode)) assertS3Error(t, w, apierr.GetAPIError(errCode))
} }
func putObjectRetentionFailed(t *testing.T, hc *handlerContext, bktName, objName string, retention *data.Retention, errCode apiErrors.ErrorCode) { func putObjectRetentionFailed(t *testing.T, hc *handlerContext, bktName, objName string, retention *data.Retention, errCode apierr.ErrorCode) {
w, r := prepareTestRequest(hc, bktName, objName, retention) w, r := prepareTestRequest(hc, bktName, objName, retention)
hc.Handler().PutObjectRetentionHandler(w, r) hc.Handler().PutObjectRetentionHandler(w, r)
assertS3Error(t, w, apiErrors.GetAPIError(errCode)) assertS3Error(t, w, apierr.GetAPIError(errCode))
} }
func assertRetentionApproximate(t *testing.T, w *httptest.ResponseRecorder, retention *data.Retention, delta float64) { func assertRetentionApproximate(t *testing.T, w *httptest.ResponseRecorder, retention *data.Retention, delta float64) {

View file

@ -7,6 +7,7 @@ import (
"net/url" "net/url"
"path" "path"
"strconv" "strconv"
"strings"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
@ -103,19 +104,20 @@ const (
) )
func (h *handler) CreateMultipartUploadHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) CreateMultipartUploadHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
uploadID := uuid.New() uploadID := uuid.New()
cannedACLStatus := aclHeadersStatus(r) cannedACLStatus := aclHeadersStatus(r)
additional := []zap.Field{zap.String("uploadID", uploadID.String())} additional := []zap.Field{zap.String("uploadID", uploadID.String())}
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
if cannedACLStatus == aclStatusYes { if cannedACLStatus == aclStatusYes {
h.logAndSendError(w, "acl not supported for this bucket", reqInfo, errors.GetAPIError(errors.ErrAccessControlListNotSupported)) h.logAndSendError(ctx, w, "acl not supported for this bucket", reqInfo, errors.GetAPIError(errors.ErrAccessControlListNotSupported))
return return
} }
@ -131,14 +133,14 @@ func (h *handler) CreateMultipartUploadHandler(w http.ResponseWriter, r *http.Re
if len(r.Header.Get(api.AmzTagging)) > 0 { if len(r.Header.Get(api.AmzTagging)) > 0 {
p.Data.TagSet, err = parseTaggingHeader(r.Header) p.Data.TagSet, err = parseTaggingHeader(r.Header)
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse tagging", reqInfo, err, additional...) h.logAndSendError(ctx, w, "could not parse tagging", reqInfo, err, additional...)
return return
} }
} }
p.Info.Encryption, err = formEncryptionParams(r) p.Info.Encryption, err = formEncryptionParams(r)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid sse headers", reqInfo, err, additional...) h.logAndSendError(ctx, w, "invalid sse headers", reqInfo, err, additional...)
return return
} }
@ -152,12 +154,12 @@ func (h *handler) CreateMultipartUploadHandler(w http.ResponseWriter, r *http.Re
p.CopiesNumbers, err = h.pickCopiesNumbers(p.Header, reqInfo.Namespace, bktInfo.LocationConstraint) p.CopiesNumbers, err = h.pickCopiesNumbers(p.Header, reqInfo.Namespace, bktInfo.LocationConstraint)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid copies number", reqInfo, err, additional...) h.logAndSendError(ctx, w, "invalid copies number", reqInfo, err, additional...)
return return
} }
if err = h.obj.CreateMultipartUpload(r.Context(), p); err != nil { if err = h.obj.CreateMultipartUpload(ctx, p); err != nil {
h.logAndSendError(w, "could create multipart upload", reqInfo, err, additional...) h.logAndSendError(ctx, w, "could create multipart upload", reqInfo, err, additional...)
return return
} }
@ -172,17 +174,18 @@ func (h *handler) CreateMultipartUploadHandler(w http.ResponseWriter, r *http.Re
} }
if err = middleware.EncodeToResponse(w, resp); err != nil { if err = middleware.EncodeToResponse(w, resp); err != nil {
h.logAndSendError(w, "could not encode InitiateMultipartUploadResponse to response", reqInfo, err, additional...) h.logAndSendError(ctx, w, "could not encode InitiateMultipartUploadResponse to response", reqInfo, err, additional...)
return return
} }
} }
func (h *handler) UploadPartHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) UploadPartHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
@ -195,20 +198,17 @@ func (h *handler) UploadPartHandler(w http.ResponseWriter, r *http.Request) {
partNumber, err := strconv.Atoi(partNumStr) partNumber, err := strconv.Atoi(partNumStr)
if err != nil || partNumber < layer.UploadMinPartNumber || partNumber > layer.UploadMaxPartNumber { if err != nil || partNumber < layer.UploadMinPartNumber || partNumber > layer.UploadMaxPartNumber {
h.logAndSendError(w, "invalid part number", reqInfo, errors.GetAPIError(errors.ErrInvalidPartNumber), additional...) h.logAndSendError(ctx, w, "invalid part number", reqInfo, errors.GetAPIError(errors.ErrInvalidPartNumber), additional...)
return return
} }
body, err := h.getBodyReader(r) body, err := h.getBodyReader(r)
if err != nil { if err != nil {
h.logAndSendError(w, "failed to get body reader", reqInfo, err, additional...) h.logAndSendError(ctx, w, "failed to get body reader", reqInfo, err, additional...)
return return
} }
var size uint64 size := h.getPutPayloadSize(r)
if r.ContentLength > 0 {
size = uint64(r.ContentLength)
}
p := &layer.UploadPartParams{ p := &layer.UploadPartParams{
Info: &layer.UploadInfoParams{ Info: &layer.UploadInfoParams{
@ -225,13 +225,13 @@ func (h *handler) UploadPartHandler(w http.ResponseWriter, r *http.Request) {
p.Info.Encryption, err = formEncryptionParams(r) p.Info.Encryption, err = formEncryptionParams(r)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid sse headers", reqInfo, err, additional...) h.logAndSendError(ctx, w, "invalid sse headers", reqInfo, err, additional...)
return return
} }
hash, err := h.obj.UploadPart(r.Context(), p) hash, err := h.obj.UploadPart(ctx, p)
if err != nil { if err != nil {
h.logAndSendError(w, "could not upload a part", reqInfo, err, additional...) h.logAndSendError(ctx, w, "could not upload a part", reqInfo, err, additional...)
return return
} }
@ -241,7 +241,7 @@ func (h *handler) UploadPartHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set(api.ETag, data.Quote(hash)) w.Header().Set(api.ETag, data.Quote(hash))
if err = middleware.WriteSuccessResponseHeadersOnly(w); err != nil { if err = middleware.WriteSuccessResponseHeadersOnly(w); err != nil {
h.logAndSendError(w, "write response", reqInfo, err) h.logAndSendError(ctx, w, "write response", reqInfo, err)
return return
} }
} }
@ -259,7 +259,7 @@ func (h *handler) UploadPartCopy(w http.ResponseWriter, r *http.Request) {
partNumber, err := strconv.Atoi(partNumStr) partNumber, err := strconv.Atoi(partNumStr)
if err != nil || partNumber < layer.UploadMinPartNumber || partNumber > layer.UploadMaxPartNumber { if err != nil || partNumber < layer.UploadMinPartNumber || partNumber > layer.UploadMaxPartNumber {
h.logAndSendError(w, "invalid part number", reqInfo, errors.GetAPIError(errors.ErrInvalidPartNumber), additional...) h.logAndSendError(ctx, w, "invalid part number", reqInfo, errors.GetAPIError(errors.ErrInvalidPartNumber), additional...)
return return
} }
@ -270,26 +270,26 @@ func (h *handler) UploadPartCopy(w http.ResponseWriter, r *http.Request) {
} }
srcBucket, srcObject, err := path2BucketObject(src) srcBucket, srcObject, err := path2BucketObject(src)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid source copy", reqInfo, err, additional...) h.logAndSendError(ctx, w, "invalid source copy", reqInfo, err, additional...)
return return
} }
srcRange, err := parseRange(r.Header.Get(api.AmzCopySourceRange)) srcRange, err := parseRange(r.Header.Get(api.AmzCopySourceRange))
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse copy range", reqInfo, h.logAndSendError(ctx, w, "could not parse copy range", reqInfo,
errors.GetAPIError(errors.ErrInvalidCopyPartRange), additional...) errors.GetAPIError(errors.ErrInvalidCopyPartRange), additional...)
return return
} }
srcBktInfo, err := h.getBucketAndCheckOwner(r, srcBucket, api.AmzSourceExpectedBucketOwner) srcBktInfo, err := h.getBucketAndCheckOwner(r, srcBucket, api.AmzSourceExpectedBucketOwner)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get source bucket info", reqInfo, err, additional...) h.logAndSendError(ctx, w, "could not get source bucket info", reqInfo, err, additional...)
return return
} }
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get target bucket info", reqInfo, err, additional...) h.logAndSendError(ctx, w, "could not get target bucket info", reqInfo, err, additional...)
return return
} }
@ -302,35 +302,35 @@ func (h *handler) UploadPartCopy(w http.ResponseWriter, r *http.Request) {
srcInfo, err := h.obj.GetObjectInfo(ctx, headPrm) srcInfo, err := h.obj.GetObjectInfo(ctx, headPrm)
if err != nil { if err != nil {
if errors.IsS3Error(err, errors.ErrNoSuchKey) && versionID != "" { if errors.IsS3Error(err, errors.ErrNoSuchKey) && versionID != "" {
h.logAndSendError(w, "could not head source object version", reqInfo, h.logAndSendError(ctx, w, "could not head source object version", reqInfo,
errors.GetAPIError(errors.ErrBadRequest), additional...) errors.GetAPIError(errors.ErrBadRequest), additional...)
return return
} }
h.logAndSendError(w, "could not head source object", reqInfo, err, additional...) h.logAndSendError(ctx, w, "could not head source object", reqInfo, err, additional...)
return return
} }
args, err := parseCopyObjectArgs(r.Header) args, err := parseCopyObjectArgs(r.Header)
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse copy object args", reqInfo, h.logAndSendError(ctx, w, "could not parse copy object args", reqInfo,
errors.GetAPIError(errors.ErrInvalidCopyPartRange), additional...) errors.GetAPIError(errors.ErrInvalidCopyPartRange), additional...)
return return
} }
if err = checkPreconditions(srcInfo, args.Conditional, h.cfg.MD5Enabled()); err != nil { if err = checkPreconditions(srcInfo, args.Conditional, h.cfg.MD5Enabled()); err != nil {
h.logAndSendError(w, "precondition failed", reqInfo, errors.GetAPIError(errors.ErrPreconditionFailed), h.logAndSendError(ctx, w, "precondition failed", reqInfo, errors.GetAPIError(errors.ErrPreconditionFailed),
additional...) additional...)
return return
} }
srcEncryptionParams, err := formCopySourceEncryptionParams(r) srcEncryptionParams, err := formCopySourceEncryptionParams(r)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid sse headers", reqInfo, err) h.logAndSendError(ctx, w, "invalid sse headers", reqInfo, err)
return return
} }
if err = srcEncryptionParams.MatchObjectEncryption(layer.FormEncryptionInfo(srcInfo.Headers)); err != nil { if err = srcEncryptionParams.MatchObjectEncryption(layer.FormEncryptionInfo(srcInfo.Headers)); err != nil {
h.logAndSendError(w, "encryption doesn't match object", reqInfo, fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrBadRequest), err), additional...) h.logAndSendError(ctx, w, "encryption doesn't match object", reqInfo, fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrBadRequest), err), additional...)
return return
} }
@ -350,13 +350,13 @@ func (h *handler) UploadPartCopy(w http.ResponseWriter, r *http.Request) {
p.Info.Encryption, err = formEncryptionParams(r) p.Info.Encryption, err = formEncryptionParams(r)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid sse headers", reqInfo, err, additional...) h.logAndSendError(ctx, w, "invalid sse headers", reqInfo, err, additional...)
return return
} }
info, err := h.obj.UploadPartCopy(ctx, p) info, err := h.obj.UploadPartCopy(ctx, p)
if err != nil { if err != nil {
h.logAndSendError(w, "could not upload part copy", reqInfo, err, additional...) h.logAndSendError(ctx, w, "could not upload part copy", reqInfo, err, additional...)
return return
} }
@ -370,22 +370,23 @@ func (h *handler) UploadPartCopy(w http.ResponseWriter, r *http.Request) {
} }
if err = middleware.EncodeToResponse(w, response); err != nil { if err = middleware.EncodeToResponse(w, response); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err, additional...) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err, additional...)
} }
} }
func (h *handler) CompleteMultipartUploadHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) CompleteMultipartUploadHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
settings, err := h.obj.GetBucketSettings(r.Context(), bktInfo) settings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket settings", reqInfo, err)
return return
} }
@ -401,12 +402,12 @@ func (h *handler) CompleteMultipartUploadHandler(w http.ResponseWriter, r *http.
reqBody := new(CompleteMultipartUpload) reqBody := new(CompleteMultipartUpload)
if err = h.cfg.NewXMLDecoder(r.Body).Decode(reqBody); err != nil { if err = h.cfg.NewXMLDecoder(r.Body).Decode(reqBody); err != nil {
h.logAndSendError(w, "could not read complete multipart upload xml", reqInfo, h.logAndSendError(ctx, w, "could not read complete multipart upload xml", reqInfo,
fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrMalformedXML), err.Error()), additional...) fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrMalformedXML), err.Error()), additional...)
return return
} }
if len(reqBody.Parts) == 0 { if len(reqBody.Parts) == 0 {
h.logAndSendError(w, "invalid xml with parts", reqInfo, errors.GetAPIError(errors.ErrMalformedXML), additional...) h.logAndSendError(ctx, w, "invalid xml with parts", reqInfo, errors.GetAPIError(errors.ErrMalformedXML), additional...)
return return
} }
@ -420,7 +421,7 @@ func (h *handler) CompleteMultipartUploadHandler(w http.ResponseWriter, r *http.
objInfo, err := h.completeMultipartUpload(r, c, bktInfo) objInfo, err := h.completeMultipartUpload(r, c, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "complete multipart error", reqInfo, err, additional...) h.logAndSendError(ctx, w, "complete multipart error", reqInfo, err, additional...)
return return
} }
@ -436,7 +437,7 @@ func (h *handler) CompleteMultipartUploadHandler(w http.ResponseWriter, r *http.
} }
if err = middleware.EncodeToResponse(w, response); err != nil { if err = middleware.EncodeToResponse(w, response); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err, additional...) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err, additional...)
} }
} }
@ -495,11 +496,12 @@ func (h *handler) completeMultipartUpload(r *http.Request, c *layer.CompleteMult
} }
func (h *handler) ListMultipartUploadsHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) ListMultipartUploadsHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
@ -512,7 +514,7 @@ func (h *handler) ListMultipartUploadsHandler(w http.ResponseWriter, r *http.Req
if maxUploadsStr != "" { if maxUploadsStr != "" {
val, err := strconv.Atoi(maxUploadsStr) val, err := strconv.Atoi(maxUploadsStr)
if err != nil || val < 1 || val > 1000 { if err != nil || val < 1 || val > 1000 {
h.logAndSendError(w, "invalid maxUploads", reqInfo, errors.GetAPIError(errors.ErrInvalidMaxUploads)) h.logAndSendError(ctx, w, "invalid maxUploads", reqInfo, errors.GetAPIError(errors.ErrInvalidMaxUploads))
return return
} }
maxUploads = val maxUploads = val
@ -528,23 +530,29 @@ func (h *handler) ListMultipartUploadsHandler(w http.ResponseWriter, r *http.Req
UploadIDMarker: queryValues.Get(uploadIDMarkerQueryName), UploadIDMarker: queryValues.Get(uploadIDMarkerQueryName),
} }
list, err := h.obj.ListMultipartUploads(r.Context(), p) if p.EncodingType != "" && strings.ToLower(p.EncodingType) != urlEncodingType {
h.logAndSendError(ctx, w, "invalid encoding type", reqInfo, errors.GetAPIError(errors.ErrInvalidEncodingMethod))
return
}
list, err := h.obj.ListMultipartUploads(ctx, p)
if err != nil { if err != nil {
h.logAndSendError(w, "could not list multipart uploads", reqInfo, err) h.logAndSendError(ctx, w, "could not list multipart uploads", reqInfo, err)
return return
} }
if err = middleware.EncodeToResponse(w, encodeListMultipartUploadsToResponse(list, p)); err != nil { if err = middleware.EncodeToResponse(w, encodeListMultipartUploadsToResponse(list, p)); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
} }
} }
func (h *handler) ListPartsHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) ListPartsHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
@ -553,14 +561,14 @@ func (h *handler) ListPartsHandler(w http.ResponseWriter, r *http.Request) {
queryValues = reqInfo.URL.Query() queryValues = reqInfo.URL.Query()
uploadID = queryValues.Get(uploadIDHeaderName) uploadID = queryValues.Get(uploadIDHeaderName)
additional = []zap.Field{zap.String("uploadID", uploadID), zap.String("Key", reqInfo.ObjectName)} additional = []zap.Field{zap.String("uploadID", uploadID)}
maxParts = layer.MaxSizePartsList maxParts = layer.MaxSizePartsList
) )
if queryValues.Get("max-parts") != "" { if queryValues.Get("max-parts") != "" {
val, err := strconv.Atoi(queryValues.Get("max-parts")) val, err := strconv.Atoi(queryValues.Get("max-parts"))
if err != nil || val < 0 { if err != nil || val < 0 {
h.logAndSendError(w, "invalid MaxParts", reqInfo, errors.GetAPIError(errors.ErrInvalidMaxParts), additional...) h.logAndSendError(ctx, w, "invalid MaxParts", reqInfo, errors.GetAPIError(errors.ErrInvalidMaxParts), additional...)
return return
} }
if val < layer.MaxSizePartsList { if val < layer.MaxSizePartsList {
@ -570,7 +578,7 @@ func (h *handler) ListPartsHandler(w http.ResponseWriter, r *http.Request) {
if queryValues.Get("part-number-marker") != "" { if queryValues.Get("part-number-marker") != "" {
if partNumberMarker, err = strconv.Atoi(queryValues.Get("part-number-marker")); err != nil || partNumberMarker < 0 { if partNumberMarker, err = strconv.Atoi(queryValues.Get("part-number-marker")); err != nil || partNumberMarker < 0 {
h.logAndSendError(w, "invalid PartNumberMarker", reqInfo, err, additional...) h.logAndSendError(ctx, w, "invalid PartNumberMarker", reqInfo, err, additional...)
return return
} }
} }
@ -587,32 +595,33 @@ func (h *handler) ListPartsHandler(w http.ResponseWriter, r *http.Request) {
p.Info.Encryption, err = formEncryptionParams(r) p.Info.Encryption, err = formEncryptionParams(r)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid sse headers", reqInfo, err) h.logAndSendError(ctx, w, "invalid sse headers", reqInfo, err)
return return
} }
list, err := h.obj.ListParts(r.Context(), p) list, err := h.obj.ListParts(ctx, p)
if err != nil { if err != nil {
h.logAndSendError(w, "could not list parts", reqInfo, err, additional...) h.logAndSendError(ctx, w, "could not list parts", reqInfo, err, additional...)
return return
} }
if err = middleware.EncodeToResponse(w, encodeListPartsToResponse(list, p)); err != nil { if err = middleware.EncodeToResponse(w, encodeListPartsToResponse(list, p)); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
} }
} }
func (h *handler) AbortMultipartUploadHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) AbortMultipartUploadHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
uploadID := reqInfo.URL.Query().Get(uploadIDHeaderName) uploadID := reqInfo.URL.Query().Get(uploadIDHeaderName)
additional := []zap.Field{zap.String("uploadID", uploadID), zap.String("Key", reqInfo.ObjectName)} additional := []zap.Field{zap.String("uploadID", uploadID)}
p := &layer.UploadInfoParams{ p := &layer.UploadInfoParams{
UploadID: uploadID, UploadID: uploadID,
@ -622,12 +631,12 @@ func (h *handler) AbortMultipartUploadHandler(w http.ResponseWriter, r *http.Req
p.Encryption, err = formEncryptionParams(r) p.Encryption, err = formEncryptionParams(r)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid sse headers", reqInfo, err) h.logAndSendError(ctx, w, "invalid sse headers", reqInfo, err)
return return
} }
if err = h.obj.AbortMultipartUpload(r.Context(), p); err != nil { if err = h.obj.AbortMultipartUpload(ctx, p); err != nil {
h.logAndSendError(w, "could not abort multipart upload", reqInfo, err, additional...) h.logAndSendError(ctx, w, "could not abort multipart upload", reqInfo, err, additional...)
return return
} }
@ -638,14 +647,14 @@ func encodeListMultipartUploadsToResponse(info *layer.ListMultipartUploadsInfo,
res := ListMultipartUploadsResponse{ res := ListMultipartUploadsResponse{
Bucket: params.Bkt.Name, Bucket: params.Bkt.Name,
CommonPrefixes: fillPrefixes(info.Prefixes, params.EncodingType), CommonPrefixes: fillPrefixes(info.Prefixes, params.EncodingType),
Delimiter: params.Delimiter, Delimiter: s3PathEncode(params.Delimiter, params.EncodingType),
EncodingType: params.EncodingType, EncodingType: params.EncodingType,
IsTruncated: info.IsTruncated, IsTruncated: info.IsTruncated,
KeyMarker: params.KeyMarker, KeyMarker: s3PathEncode(params.KeyMarker, params.EncodingType),
MaxUploads: params.MaxUploads, MaxUploads: params.MaxUploads,
NextKeyMarker: info.NextKeyMarker, NextKeyMarker: s3PathEncode(info.NextKeyMarker, params.EncodingType),
NextUploadIDMarker: info.NextUploadIDMarker, NextUploadIDMarker: info.NextUploadIDMarker,
Prefix: params.Prefix, Prefix: s3PathEncode(params.Prefix, params.EncodingType),
UploadIDMarker: params.UploadIDMarker, UploadIDMarker: params.UploadIDMarker,
} }
@ -657,7 +666,7 @@ func encodeListMultipartUploadsToResponse(info *layer.ListMultipartUploadsInfo,
ID: u.Owner.String(), ID: u.Owner.String(),
DisplayName: u.Owner.String(), DisplayName: u.Owner.String(),
}, },
Key: u.Key, Key: s3PathEncode(u.Key, params.EncodingType),
Owner: Owner{ Owner: Owner{
ID: u.Owner.String(), ID: u.Owner.String(),
DisplayName: u.Owner.String(), DisplayName: u.Owner.String(),

View file

@ -2,19 +2,21 @@ package handler
import ( import (
"crypto/md5" "crypto/md5"
"crypto/rand"
"crypto/tls" "crypto/tls"
"encoding/base64" "encoding/base64"
"encoding/hex" "encoding/hex"
"encoding/xml" "encoding/xml"
"fmt" "fmt"
"net/http" "net/http"
"net/http/httptest"
"net/url" "net/url"
"strconv" "strconv"
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3Errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
@ -39,7 +41,7 @@ func TestMultipartUploadInvalidPart(t *testing.T) {
etag1, _ := uploadPart(hc, bktName, objName, multipartUpload.UploadID, 1, partSize) etag1, _ := uploadPart(hc, bktName, objName, multipartUpload.UploadID, 1, partSize)
etag2, _ := uploadPart(hc, bktName, objName, multipartUpload.UploadID, 2, partSize) etag2, _ := uploadPart(hc, bktName, objName, multipartUpload.UploadID, 2, partSize)
w := completeMultipartUploadBase(hc, bktName, objName, multipartUpload.UploadID, []string{etag1, etag2}) w := completeMultipartUploadBase(hc, bktName, objName, multipartUpload.UploadID, []string{etag1, etag2})
assertS3Error(hc.t, w, s3Errors.GetAPIError(s3Errors.ErrEntityTooSmall)) assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrEntityTooSmall))
} }
func TestDeleteMultipartAllParts(t *testing.T) { func TestDeleteMultipartAllParts(t *testing.T) {
@ -103,7 +105,7 @@ func TestMultipartReUploadPart(t *testing.T) {
require.Equal(t, etag2, list.Parts[1].ETag) require.Equal(t, etag2, list.Parts[1].ETag)
w := completeMultipartUploadBase(hc, bktName, objName, uploadInfo.UploadID, []string{etag1, etag2}) w := completeMultipartUploadBase(hc, bktName, objName, uploadInfo.UploadID, []string{etag1, etag2})
assertS3Error(hc.t, w, s3Errors.GetAPIError(s3Errors.ErrEntityTooSmall)) assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrEntityTooSmall))
etag1, data1 := uploadPart(hc, bktName, objName, uploadInfo.UploadID, 1, partSizeFirst) etag1, data1 := uploadPart(hc, bktName, objName, uploadInfo.UploadID, 1, partSizeFirst)
etag2, data2 := uploadPart(hc, bktName, objName, uploadInfo.UploadID, 2, partSizeLast) etag2, data2 := uploadPart(hc, bktName, objName, uploadInfo.UploadID, 2, partSizeLast)
@ -251,14 +253,14 @@ func TestListMultipartUploads(t *testing.T) {
}) })
t.Run("check max uploads", func(t *testing.T) { t.Run("check max uploads", func(t *testing.T) {
listUploads := listMultipartUploadsBase(hc, bktName, "", "", "", "", 2) listUploads := listMultipartUploads(hc, bktName, "", "", "", "", 2)
require.Len(t, listUploads.Uploads, 2) require.Len(t, listUploads.Uploads, 2)
require.Equal(t, uploadInfo1.UploadID, listUploads.Uploads[0].UploadID) require.Equal(t, uploadInfo1.UploadID, listUploads.Uploads[0].UploadID)
require.Equal(t, uploadInfo2.UploadID, listUploads.Uploads[1].UploadID) require.Equal(t, uploadInfo2.UploadID, listUploads.Uploads[1].UploadID)
}) })
t.Run("check prefix", func(t *testing.T) { t.Run("check prefix", func(t *testing.T) {
listUploads := listMultipartUploadsBase(hc, bktName, "/my", "", "", "", -1) listUploads := listMultipartUploads(hc, bktName, "/my", "", "", "", -1)
require.Len(t, listUploads.Uploads, 2) require.Len(t, listUploads.Uploads, 2)
require.Equal(t, uploadInfo1.UploadID, listUploads.Uploads[0].UploadID) require.Equal(t, uploadInfo1.UploadID, listUploads.Uploads[0].UploadID)
require.Equal(t, uploadInfo2.UploadID, listUploads.Uploads[1].UploadID) require.Equal(t, uploadInfo2.UploadID, listUploads.Uploads[1].UploadID)
@ -266,7 +268,7 @@ func TestListMultipartUploads(t *testing.T) {
t.Run("check markers", func(t *testing.T) { t.Run("check markers", func(t *testing.T) {
t.Run("check only key-marker", func(t *testing.T) { t.Run("check only key-marker", func(t *testing.T) {
listUploads := listMultipartUploadsBase(hc, bktName, "", "", "", objName2, -1) listUploads := listMultipartUploads(hc, bktName, "", "", "", objName2, -1)
require.Len(t, listUploads.Uploads, 1) require.Len(t, listUploads.Uploads, 1)
// If upload-id-marker is not specified, only the keys lexicographically greater than the specified key-marker will be included in the list. // If upload-id-marker is not specified, only the keys lexicographically greater than the specified key-marker will be included in the list.
require.Equal(t, uploadInfo3.UploadID, listUploads.Uploads[0].UploadID) require.Equal(t, uploadInfo3.UploadID, listUploads.Uploads[0].UploadID)
@ -277,7 +279,7 @@ func TestListMultipartUploads(t *testing.T) {
if uploadIDMarker > uploadInfo2.UploadID { if uploadIDMarker > uploadInfo2.UploadID {
uploadIDMarker = uploadInfo2.UploadID uploadIDMarker = uploadInfo2.UploadID
} }
listUploads := listMultipartUploadsBase(hc, bktName, "", "", uploadIDMarker, "", -1) listUploads := listMultipartUploads(hc, bktName, "", "", uploadIDMarker, "", -1)
// If key-marker is not specified, the upload-id-marker parameter is ignored. // If key-marker is not specified, the upload-id-marker parameter is ignored.
require.Len(t, listUploads.Uploads, 3) require.Len(t, listUploads.Uploads, 3)
}) })
@ -285,7 +287,7 @@ func TestListMultipartUploads(t *testing.T) {
t.Run("check key-marker along with upload-id-marker", func(t *testing.T) { t.Run("check key-marker along with upload-id-marker", func(t *testing.T) {
uploadIDMarker := "00000000-0000-0000-0000-000000000000" uploadIDMarker := "00000000-0000-0000-0000-000000000000"
listUploads := listMultipartUploadsBase(hc, bktName, "", "", uploadIDMarker, objName3, -1) listUploads := listMultipartUploads(hc, bktName, "", "", uploadIDMarker, objName3, -1)
require.Len(t, listUploads.Uploads, 1) require.Len(t, listUploads.Uploads, 1)
// If upload-id-marker is specified, any multipart uploads for a key equal to the key-marker might also be included, // If upload-id-marker is specified, any multipart uploads for a key equal to the key-marker might also be included,
// provided those multipart uploads have upload IDs lexicographically greater than the specified upload-id-marker. // provided those multipart uploads have upload IDs lexicographically greater than the specified upload-id-marker.
@ -520,7 +522,7 @@ func TestUploadPartCheckContentSHA256(t *testing.T) {
r.Header.Set(api.AmzContentSha256, tc.hash) r.Header.Set(api.AmzContentSha256, tc.hash)
hc.Handler().UploadPartHandler(w, r) hc.Handler().UploadPartHandler(w, r)
if tc.error { if tc.error {
assertS3Error(t, w, s3Errors.GetAPIError(s3Errors.ErrContentSHA256Mismatch)) assertS3Error(t, w, apierr.GetAPIError(apierr.ErrContentSHA256Mismatch))
list := listParts(hc, bktName, objName, multipartUpload.UploadID, "0", http.StatusOK) list := listParts(hc, bktName, objName, multipartUpload.UploadID, "0", http.StatusOK)
require.Len(t, list.Parts, 1) require.Len(t, list.Parts, 1)
@ -621,6 +623,73 @@ func TestMultipartObjectLocation(t *testing.T) {
} }
} }
func TestUploadPartWithNegativeContentLength(t *testing.T) {
hc := prepareHandlerContext(t)
bktName, objName := "bucket-to-upload-part", "object-multipart"
createTestBucket(hc, bktName)
partSize := 5 * 1024 * 1024
multipartUpload := createMultipartUpload(hc, bktName, objName, map[string]string{})
partBody := make([]byte, partSize)
_, err := rand.Read(partBody)
require.NoError(hc.t, err)
query := make(url.Values)
query.Set(uploadIDQuery, multipartUpload.UploadID)
query.Set(partNumberQuery, "1")
w, r := prepareTestRequestWithQuery(hc, bktName, objName, query, partBody)
r.ContentLength = -1
hc.Handler().UploadPartHandler(w, r)
assertStatus(hc.t, w, http.StatusOK)
completeMultipartUpload(hc, bktName, objName, multipartUpload.UploadID, []string{w.Header().Get(api.ETag)})
res, _ := getObject(hc, bktName, objName)
equalDataSlices(t, partBody, res)
resp := getObjectAttributes(hc, bktName, objName, objectParts)
require.Len(t, resp.ObjectParts.Parts, 1)
require.Equal(t, partSize, resp.ObjectParts.Parts[0].Size)
}
func TestListMultipartUploadsEncoding(t *testing.T) {
hc := prepareHandlerContext(t)
bktName := "bucket-to-list-uploads-encoding"
createTestBucket(hc, bktName)
listAllMultipartUploadsErr(hc, bktName, "invalid", apierr.GetAPIError(apierr.ErrInvalidEncodingMethod))
objects := []string{"foo()/bar", "foo()/bar/xyzzy", "asdf+b"}
for _, objName := range objects {
createMultipartUpload(hc, bktName, objName, nil)
}
listResponse := listMultipartUploadsURL(hc, bktName, "foo(", ")", "", "", -1)
require.Len(t, listResponse.CommonPrefixes, 1)
require.Equal(t, "foo%28%29", listResponse.CommonPrefixes[0].Prefix)
require.Equal(t, "foo%28", listResponse.Prefix)
require.Equal(t, "%29", listResponse.Delimiter)
require.Equal(t, "url", listResponse.EncodingType)
require.Equal(t, maxObjectList, listResponse.MaxUploads)
listResponse = listMultipartUploads(hc, bktName, "", "", "", "", 1)
require.Empty(t, listResponse.EncodingType)
listResponse = listMultipartUploadsURL(hc, bktName, "", "", "", listResponse.NextKeyMarker, 1)
require.Len(t, listResponse.CommonPrefixes, 0)
require.Len(t, listResponse.Uploads, 1)
require.Equal(t, "foo%28%29/bar", listResponse.Uploads[0].Key)
require.Equal(t, "asdf%2Bb", listResponse.KeyMarker)
require.Equal(t, "foo%28%29/bar", listResponse.NextKeyMarker)
require.Equal(t, "url", listResponse.EncodingType)
require.Equal(t, 1, listResponse.MaxUploads)
}
func uploadPartCopy(hc *handlerContext, bktName, objName, uploadID string, num int, srcObj string, start, end int) *UploadPartCopyResponse { func uploadPartCopy(hc *handlerContext, bktName, objName, uploadID string, num int, srcObj string, start, end int) *UploadPartCopyResponse {
return uploadPartCopyBase(hc, bktName, objName, false, uploadID, num, srcObj, start, end) return uploadPartCopyBase(hc, bktName, objName, false, uploadID, num, srcObj, start, end)
} }
@ -646,16 +715,42 @@ func uploadPartCopyBase(hc *handlerContext, bktName, objName string, encrypted b
return uploadPartCopyResponse return uploadPartCopyResponse
} }
func listAllMultipartUploads(hc *handlerContext, bktName string) *ListMultipartUploadsResponse { func listMultipartUploads(hc *handlerContext, bktName, prefix, delimiter, uploadIDMarker, keyMarker string, maxUploads int) *ListMultipartUploadsResponse {
return listMultipartUploadsBase(hc, bktName, "", "", "", "", -1) w := listMultipartUploadsBase(hc, bktName, prefix, delimiter, uploadIDMarker, keyMarker, "", maxUploads)
assertStatus(hc.t, w, http.StatusOK)
res := &ListMultipartUploadsResponse{}
parseTestResponse(hc.t, w, res)
return res
} }
func listMultipartUploadsBase(hc *handlerContext, bktName, prefix, delimiter, uploadIDMarker, keyMarker string, maxUploads int) *ListMultipartUploadsResponse { func listMultipartUploadsURL(hc *handlerContext, bktName, prefix, delimiter, uploadIDMarker, keyMarker string, maxUploads int) *ListMultipartUploadsResponse {
w := listMultipartUploadsBase(hc, bktName, prefix, delimiter, uploadIDMarker, keyMarker, urlEncodingType, maxUploads)
assertStatus(hc.t, w, http.StatusOK)
res := &ListMultipartUploadsResponse{}
parseTestResponse(hc.t, w, res)
return res
}
func listAllMultipartUploads(hc *handlerContext, bktName string) *ListMultipartUploadsResponse {
w := listMultipartUploadsBase(hc, bktName, "", "", "", "", "", -1)
assertStatus(hc.t, w, http.StatusOK)
res := &ListMultipartUploadsResponse{}
parseTestResponse(hc.t, w, res)
return res
}
func listAllMultipartUploadsErr(hc *handlerContext, bktName, encoding string, err apierr.Error) {
w := listMultipartUploadsBase(hc, bktName, "", "", "", "", encoding, -1)
assertS3Error(hc.t, w, err)
}
func listMultipartUploadsBase(hc *handlerContext, bktName, prefix, delimiter, uploadIDMarker, keyMarker, encoding string, maxUploads int) *httptest.ResponseRecorder {
query := make(url.Values) query := make(url.Values)
query.Set(prefixQueryName, prefix) query.Set(prefixQueryName, prefix)
query.Set(delimiterQueryName, delimiter) query.Set(delimiterQueryName, delimiter)
query.Set(uploadIDMarkerQueryName, uploadIDMarker) query.Set(uploadIDMarkerQueryName, uploadIDMarker)
query.Set(keyMarkerQueryName, keyMarker) query.Set(keyMarkerQueryName, keyMarker)
query.Set(encodingTypeQueryName, encoding)
if maxUploads != -1 { if maxUploads != -1 {
query.Set(maxUploadsQueryName, strconv.Itoa(maxUploads)) query.Set(maxUploadsQueryName, strconv.Itoa(maxUploads))
} }
@ -663,10 +758,7 @@ func listMultipartUploadsBase(hc *handlerContext, bktName, prefix, delimiter, up
w, r := prepareTestRequestWithQuery(hc, bktName, "", query, nil) w, r := prepareTestRequestWithQuery(hc, bktName, "", query, nil)
hc.Handler().ListMultipartUploadsHandler(w, r) hc.Handler().ListMultipartUploadsHandler(w, r)
listPartsResponse := &ListMultipartUploadsResponse{} return w
readResponse(hc.t, w, http.StatusOK, listPartsResponse)
return listPartsResponse
} }
func listParts(hc *handlerContext, bktName, objName string, uploadID, partNumberMarker string, status int) *ListPartsResponse { func listParts(hc *handlerContext, bktName, objName string, uploadID, partNumberMarker string, status int) *ListPartsResponse {

View file

@ -8,5 +8,5 @@ import (
) )
func (h *handler) DeleteBucketEncryptionHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) DeleteBucketEncryptionHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not supported", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotSupported)) h.logAndSendError(r.Context(), w, "not supported", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotSupported))
} }

View file

@ -4,6 +4,7 @@ import (
"net/http" "net/http"
"net/url" "net/url"
"strconv" "strconv"
"strings"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
@ -16,26 +17,27 @@ import (
// ListObjectsV1Handler handles objects listing requests for API version 1. // ListObjectsV1Handler handles objects listing requests for API version 1.
func (h *handler) ListObjectsV1Handler(w http.ResponseWriter, r *http.Request) { func (h *handler) ListObjectsV1Handler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
params, err := parseListObjectsArgsV1(reqInfo) params, err := parseListObjectsArgsV1(reqInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "failed to parse arguments", reqInfo, err) h.logAndSendError(ctx, w, "failed to parse arguments", reqInfo, err)
return return
} }
if params.BktInfo, err = h.getBucketAndCheckOwner(r, reqInfo.BucketName); err != nil { if params.BktInfo, err = h.getBucketAndCheckOwner(r, reqInfo.BucketName); err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
list, err := h.obj.ListObjectsV1(r.Context(), params) list, err := h.obj.ListObjectsV1(ctx, params)
if err != nil { if err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
return return
} }
if err = middleware.EncodeToResponse(w, h.encodeV1(params, list)); err != nil { if err = middleware.EncodeToResponse(w, h.encodeV1(params, list)); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
} }
} }
@ -60,26 +62,27 @@ func (h *handler) encodeV1(p *layer.ListObjectsParamsV1, list *layer.ListObjects
// ListObjectsV2Handler handles objects listing requests for API version 2. // ListObjectsV2Handler handles objects listing requests for API version 2.
func (h *handler) ListObjectsV2Handler(w http.ResponseWriter, r *http.Request) { func (h *handler) ListObjectsV2Handler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
params, err := parseListObjectsArgsV2(reqInfo) params, err := parseListObjectsArgsV2(reqInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "failed to parse arguments", reqInfo, err) h.logAndSendError(ctx, w, "failed to parse arguments", reqInfo, err)
return return
} }
if params.BktInfo, err = h.getBucketAndCheckOwner(r, reqInfo.BucketName); err != nil { if params.BktInfo, err = h.getBucketAndCheckOwner(r, reqInfo.BucketName); err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
list, err := h.obj.ListObjectsV2(r.Context(), params) list, err := h.obj.ListObjectsV2(ctx, params)
if err != nil { if err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
return return
} }
if err = middleware.EncodeToResponse(w, h.encodeV2(params, list)); err != nil { if err = middleware.EncodeToResponse(w, h.encodeV2(params, list)); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
} }
} }
@ -153,6 +156,10 @@ func parseListObjectArgs(reqInfo *middleware.ReqInfo) (*layer.ListObjectsParamsC
res.Delimiter = queryValues.Get("delimiter") res.Delimiter = queryValues.Get("delimiter")
res.Encode = queryValues.Get("encoding-type") res.Encode = queryValues.Get("encoding-type")
if res.Encode != "" && strings.ToLower(res.Encode) != urlEncodingType {
return nil, errors.GetAPIError(errors.ErrInvalidEncodingMethod)
}
if queryValues.Get("max-keys") == "" { if queryValues.Get("max-keys") == "" {
res.MaxKeys = maxObjectList res.MaxKeys = maxObjectList
} else if res.MaxKeys, err = strconv.Atoi(queryValues.Get("max-keys")); err != nil || res.MaxKeys < 0 { } else if res.MaxKeys, err = strconv.Atoi(queryValues.Get("max-keys")); err != nil || res.MaxKeys < 0 {
@ -214,27 +221,28 @@ func fillContents(src []*data.ExtendedNodeVersion, encode string, fetchOwner, md
} }
func (h *handler) ListBucketObjectVersionsHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) ListBucketObjectVersionsHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
p, err := parseListObjectVersionsRequest(reqInfo) p, err := parseListObjectVersionsRequest(reqInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "failed to parse request", reqInfo, err) h.logAndSendError(ctx, w, "failed to parse request", reqInfo, err)
return return
} }
if p.BktInfo, err = h.getBucketAndCheckOwner(r, reqInfo.BucketName); err != nil { if p.BktInfo, err = h.getBucketAndCheckOwner(r, reqInfo.BucketName); err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
info, err := h.obj.ListObjectVersions(r.Context(), p) info, err := h.obj.ListObjectVersions(ctx, p)
if err != nil { if err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
return return
} }
response := encodeListObjectVersionsToResponse(p, info, p.BktInfo.Name, h.cfg.MD5Enabled()) response := encodeListObjectVersionsToResponse(p, info, p.BktInfo.Name, h.cfg.MD5Enabled())
if err = middleware.EncodeToResponse(w, response); err != nil { if err = middleware.EncodeToResponse(w, response); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
} }
} }
@ -257,6 +265,10 @@ func parseListObjectVersionsRequest(reqInfo *middleware.ReqInfo) (*layer.ListObj
res.Encode = queryValues.Get("encoding-type") res.Encode = queryValues.Get("encoding-type")
res.VersionIDMarker = queryValues.Get("version-id-marker") res.VersionIDMarker = queryValues.Get("version-id-marker")
if res.Encode != "" && strings.ToLower(res.Encode) != urlEncodingType {
return nil, errors.GetAPIError(errors.ErrInvalidEncodingMethod)
}
if res.VersionIDMarker != "" && res.KeyMarker == "" { if res.VersionIDMarker != "" && res.KeyMarker == "" {
return nil, errors.GetAPIError(errors.VersionIDMarkerWithoutKeyMarker) return nil, errors.GetAPIError(errors.VersionIDMarkerWithoutKeyMarker)
} }

View file

@ -4,6 +4,7 @@ import (
"context" "context"
"fmt" "fmt"
"net/http" "net/http"
"net/http/httptest"
"net/url" "net/url"
"sort" "sort"
"strconv" "strconv"
@ -14,6 +15,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/cache" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/cache"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
@ -679,6 +681,80 @@ func TestS3BucketListDelimiterNotSkipSpecial(t *testing.T) {
} }
} }
func TestS3BucketListMarkerUnreadable(t *testing.T) {
hc := prepareHandlerContext(t)
bktName := "bucket-for-listing"
bktInfo := createTestBucket(hc, bktName)
objects := []string{"bar", "baz", "foo", "quxx"}
for _, objName := range objects {
createTestObject(hc, bktInfo, objName, encryption.Params{})
}
list := listObjectsV1(hc, bktName, "", "", "\x0a", -1)
require.Equal(t, "\x0a", list.Marker)
require.False(t, list.IsTruncated)
require.Len(t, list.Contents, len(objects))
for i := 0; i < len(list.Contents); i++ {
require.Equal(t, objects[i], list.Contents[i].Key)
}
}
func TestS3BucketListMarkerNotInList(t *testing.T) {
hc := prepareHandlerContext(t)
bktName := "bucket-for-listing"
bktInfo := createTestBucket(hc, bktName)
objects := []string{"bar", "baz", "foo", "quxx"}
for _, objName := range objects {
createTestObject(hc, bktInfo, objName, encryption.Params{})
}
list := listObjectsV1(hc, bktName, "", "", "blah", -1)
require.Equal(t, "blah", list.Marker)
expected := []string{"foo", "quxx"}
require.Len(t, list.Contents, len(expected))
for i := 0; i < len(list.Contents); i++ {
require.Equal(t, expected[i], list.Contents[i].Key)
}
}
func TestListTruncatedCacheHit(t *testing.T) {
hc := prepareHandlerContext(t)
bktName := "bucket-for-listing"
bktInfo := createTestBucket(hc, bktName)
objects := []string{"bar", "baz", "foo", "quxx"}
for _, objName := range objects {
createTestObject(hc, bktInfo, objName, encryption.Params{})
}
list := listObjectsV1(hc, bktName, "", "", "", 2)
require.True(t, list.IsTruncated)
require.Len(t, list.Contents, 2)
for i := 0; i < len(list.Contents); i++ {
require.Equal(t, objects[i], list.Contents[i].Key)
}
cacheKey := cache.CreateListSessionCacheKey(bktInfo.CID, "", list.NextMarker)
list = listObjectsV1(hc, bktName, "", "", list.NextMarker, 2)
require.Nil(t, hc.cache.GetListSession(hc.owner, cacheKey))
require.False(t, list.IsTruncated)
require.Len(t, list.Contents, 2)
for i := 0; i < len(list.Contents); i++ {
require.Equal(t, objects[i+2], list.Contents[i].Key)
}
}
func TestMintVersioningListObjectVersionsVersionIDContinuation(t *testing.T) { func TestMintVersioningListObjectVersionsVersionIDContinuation(t *testing.T) {
hc := prepareHandlerContext(t) hc := prepareHandlerContext(t)
@ -755,6 +831,16 @@ func TestListObjectVersionsEncoding(t *testing.T) {
require.Equal(t, 3, listResponse.MaxKeys) require.Equal(t, 3, listResponse.MaxKeys)
} }
func TestListingsWithInvalidEncodingType(t *testing.T) {
hc := prepareHandlerContext(t)
bktName := "bucket-for-listing-invalid-encoding"
createTestBucket(hc, bktName)
listObjectsVersionsErr(hc, bktName, "invalid", apierr.GetAPIError(apierr.ErrInvalidEncodingMethod))
listObjectsV2Err(hc, bktName, "invalid", apierr.GetAPIError(apierr.ErrInvalidEncodingMethod))
listObjectsV1Err(hc, bktName, "invalid", apierr.GetAPIError(apierr.ErrInvalidEncodingMethod))
}
func checkVersionsNames(t *testing.T, versions *ListObjectsVersionsResponse, names []string) { func checkVersionsNames(t *testing.T, versions *ListObjectsVersionsResponse, names []string) {
for i, v := range versions.Version { for i, v := range versions.Version {
require.Equal(t, names[i], v.Key) require.Equal(t, names[i], v.Key)
@ -762,10 +848,19 @@ func checkVersionsNames(t *testing.T, versions *ListObjectsVersionsResponse, nam
} }
func listObjectsV2(hc *handlerContext, bktName, prefix, delimiter, startAfter, continuationToken string, maxKeys int) *ListObjectsV2Response { func listObjectsV2(hc *handlerContext, bktName, prefix, delimiter, startAfter, continuationToken string, maxKeys int) *ListObjectsV2Response {
return listObjectsV2Ext(hc, bktName, prefix, delimiter, startAfter, continuationToken, "", maxKeys) w := listObjectsV2Base(hc, bktName, prefix, delimiter, startAfter, continuationToken, "", maxKeys)
assertStatus(hc.t, w, http.StatusOK)
res := &ListObjectsV2Response{}
parseTestResponse(hc.t, w, res)
return res
} }
func listObjectsV2Ext(hc *handlerContext, bktName, prefix, delimiter, startAfter, continuationToken, encodingType string, maxKeys int) *ListObjectsV2Response { func listObjectsV2Err(hc *handlerContext, bktName, encoding string, err apierr.Error) {
w := listObjectsV2Base(hc, bktName, "", "", "", "", encoding, -1)
assertS3Error(hc.t, w, err)
}
func listObjectsV2Base(hc *handlerContext, bktName, prefix, delimiter, startAfter, continuationToken, encodingType string, maxKeys int) *httptest.ResponseRecorder {
query := prepareCommonListObjectsQuery(prefix, delimiter, maxKeys) query := prepareCommonListObjectsQuery(prefix, delimiter, maxKeys)
query.Add("fetch-owner", "true") query.Add("fetch-owner", "true")
if len(startAfter) != 0 { if len(startAfter) != 0 {
@ -780,10 +875,7 @@ func listObjectsV2Ext(hc *handlerContext, bktName, prefix, delimiter, startAfter
w, r := prepareTestFullRequest(hc, bktName, "", query, nil) w, r := prepareTestFullRequest(hc, bktName, "", query, nil)
hc.Handler().ListObjectsV2Handler(w, r) hc.Handler().ListObjectsV2Handler(w, r)
assertStatus(hc.t, w, http.StatusOK) return w
res := &ListObjectsV2Response{}
parseTestResponse(hc.t, w, res)
return res
} }
func validateListV1(t *testing.T, tc *handlerContext, bktName, prefix, delimiter, marker string, maxKeys int, func validateListV1(t *testing.T, tc *handlerContext, bktName, prefix, delimiter, marker string, maxKeys int,
@ -843,28 +935,54 @@ func prepareCommonListObjectsQuery(prefix, delimiter string, maxKeys int) url.Va
} }
func listObjectsV1(hc *handlerContext, bktName, prefix, delimiter, marker string, maxKeys int) *ListObjectsV1Response { func listObjectsV1(hc *handlerContext, bktName, prefix, delimiter, marker string, maxKeys int) *ListObjectsV1Response {
query := prepareCommonListObjectsQuery(prefix, delimiter, maxKeys) w := listObjectsV1Base(hc, bktName, prefix, delimiter, marker, "", maxKeys)
if len(marker) != 0 {
query.Add("marker", marker)
}
w, r := prepareTestFullRequest(hc, bktName, "", query, nil)
hc.Handler().ListObjectsV1Handler(w, r)
assertStatus(hc.t, w, http.StatusOK) assertStatus(hc.t, w, http.StatusOK)
res := &ListObjectsV1Response{} res := &ListObjectsV1Response{}
parseTestResponse(hc.t, w, res) parseTestResponse(hc.t, w, res)
return res return res
} }
func listObjectsV1Err(hc *handlerContext, bktName, encoding string, err apierr.Error) {
w := listObjectsV1Base(hc, bktName, "", "", "", encoding, -1)
assertS3Error(hc.t, w, err)
}
func listObjectsV1Base(hc *handlerContext, bktName, prefix, delimiter, marker, encoding string, maxKeys int) *httptest.ResponseRecorder {
query := prepareCommonListObjectsQuery(prefix, delimiter, maxKeys)
if len(marker) != 0 {
query.Add("marker", marker)
}
if len(encoding) != 0 {
query.Add("encoding-type", encoding)
}
w, r := prepareTestFullRequest(hc, bktName, "", query, nil)
hc.Handler().ListObjectsV1Handler(w, r)
return w
}
func listObjectsVersions(hc *handlerContext, bktName, prefix, delimiter, keyMarker, versionIDMarker string, maxKeys int) *ListObjectsVersionsResponse { func listObjectsVersions(hc *handlerContext, bktName, prefix, delimiter, keyMarker, versionIDMarker string, maxKeys int) *ListObjectsVersionsResponse {
return listObjectsVersionsBase(hc, bktName, prefix, delimiter, keyMarker, versionIDMarker, maxKeys, false) w := listObjectsVersionsBase(hc, bktName, prefix, delimiter, keyMarker, versionIDMarker, "", maxKeys)
assertStatus(hc.t, w, http.StatusOK)
res := &ListObjectsVersionsResponse{}
parseTestResponse(hc.t, w, res)
return res
} }
func listObjectsVersionsURL(hc *handlerContext, bktName, prefix, delimiter, keyMarker, versionIDMarker string, maxKeys int) *ListObjectsVersionsResponse { func listObjectsVersionsURL(hc *handlerContext, bktName, prefix, delimiter, keyMarker, versionIDMarker string, maxKeys int) *ListObjectsVersionsResponse {
return listObjectsVersionsBase(hc, bktName, prefix, delimiter, keyMarker, versionIDMarker, maxKeys, true) w := listObjectsVersionsBase(hc, bktName, prefix, delimiter, keyMarker, versionIDMarker, urlEncodingType, maxKeys)
assertStatus(hc.t, w, http.StatusOK)
res := &ListObjectsVersionsResponse{}
parseTestResponse(hc.t, w, res)
return res
} }
func listObjectsVersionsBase(hc *handlerContext, bktName, prefix, delimiter, keyMarker, versionIDMarker string, maxKeys int, encode bool) *ListObjectsVersionsResponse { func listObjectsVersionsErr(hc *handlerContext, bktName, encoding string, err apierr.Error) {
w := listObjectsVersionsBase(hc, bktName, "", "", "", "", encoding, -1)
assertS3Error(hc.t, w, err)
}
func listObjectsVersionsBase(hc *handlerContext, bktName, prefix, delimiter, keyMarker, versionIDMarker, encoding string, maxKeys int) *httptest.ResponseRecorder {
query := prepareCommonListObjectsQuery(prefix, delimiter, maxKeys) query := prepareCommonListObjectsQuery(prefix, delimiter, maxKeys)
if len(keyMarker) != 0 { if len(keyMarker) != 0 {
query.Add("key-marker", keyMarker) query.Add("key-marker", keyMarker)
@ -872,14 +990,11 @@ func listObjectsVersionsBase(hc *handlerContext, bktName, prefix, delimiter, key
if len(versionIDMarker) != 0 { if len(versionIDMarker) != 0 {
query.Add("version-id-marker", versionIDMarker) query.Add("version-id-marker", versionIDMarker)
} }
if encode { if len(encoding) != 0 {
query.Add("encoding-type", "url") query.Add("encoding-type", encoding)
} }
w, r := prepareTestFullRequest(hc, bktName, "", query, nil) w, r := prepareTestFullRequest(hc, bktName, "", query, nil)
hc.Handler().ListBucketObjectVersionsHandler(w, r) hc.Handler().ListBucketObjectVersionsHandler(w, r)
assertStatus(hc.t, w, http.StatusOK) return w
res := &ListObjectsVersionsResponse{}
parseTestResponse(hc.t, w, res)
return res
} }

View file

@ -22,30 +22,30 @@ func (h *handler) PatchObjectHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(ctx) reqInfo := middleware.GetReqInfo(ctx)
if _, ok := r.Header[api.ContentRange]; !ok { if _, ok := r.Header[api.ContentRange]; !ok {
h.logAndSendError(w, "missing Content-Range", reqInfo, errors.GetAPIError(errors.ErrMissingContentRange)) h.logAndSendError(ctx, w, "missing Content-Range", reqInfo, errors.GetAPIError(errors.ErrMissingContentRange))
return return
} }
if _, ok := r.Header[api.ContentLength]; !ok { if _, ok := r.Header[api.ContentLength]; !ok {
h.logAndSendError(w, "missing Content-Length", reqInfo, errors.GetAPIError(errors.ErrMissingContentLength)) h.logAndSendError(ctx, w, "missing Content-Length", reqInfo, errors.GetAPIError(errors.ErrMissingContentLength))
return return
} }
conditional, err := parsePatchConditionalHeaders(r.Header) conditional, err := parsePatchConditionalHeaders(r.Header)
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse conditional headers", reqInfo, err) h.logAndSendError(ctx, w, "could not parse conditional headers", reqInfo, err)
return return
} }
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
settings, err := h.obj.GetBucketSettings(ctx, bktInfo) settings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket settings", reqInfo, err)
return return
} }
@ -57,40 +57,40 @@ func (h *handler) PatchObjectHandler(w http.ResponseWriter, r *http.Request) {
extendedSrcObjInfo, err := h.obj.GetExtendedObjectInfo(ctx, srcObjPrm) extendedSrcObjInfo, err := h.obj.GetExtendedObjectInfo(ctx, srcObjPrm)
if err != nil { if err != nil {
h.logAndSendError(w, "could not find object", reqInfo, err) h.logAndSendError(ctx, w, "could not find object", reqInfo, err)
return return
} }
srcObjInfo := extendedSrcObjInfo.ObjectInfo srcObjInfo := extendedSrcObjInfo.ObjectInfo
if err = checkPreconditions(srcObjInfo, conditional, h.cfg.MD5Enabled()); err != nil { if err = checkPreconditions(srcObjInfo, conditional, h.cfg.MD5Enabled()); err != nil {
h.logAndSendError(w, "precondition failed", reqInfo, err) h.logAndSendError(ctx, w, "precondition failed", reqInfo, err)
return return
} }
srcSize, err := layer.GetObjectSize(srcObjInfo) srcSize, err := layer.GetObjectSize(srcObjInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "failed to get source object size", reqInfo, err) h.logAndSendError(ctx, w, "failed to get source object size", reqInfo, err)
return return
} }
byteRange, err := parsePatchByteRange(r.Header.Get(api.ContentRange), srcSize) byteRange, err := parsePatchByteRange(r.Header.Get(api.ContentRange), srcSize)
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse byte range", reqInfo, errors.GetAPIError(errors.ErrInvalidRange), zap.Error(err)) h.logAndSendError(ctx, w, "could not parse byte range", reqInfo, errors.GetAPIError(errors.ErrInvalidRange), zap.Error(err))
return return
} }
if maxPatchSize < byteRange.End-byteRange.Start+1 { if maxPatchSize < byteRange.End-byteRange.Start+1 {
h.logAndSendError(w, "byte range length is longer than allowed", reqInfo, errors.GetAPIError(errors.ErrInvalidRange), zap.Error(err)) h.logAndSendError(ctx, w, "byte range length is longer than allowed", reqInfo, errors.GetAPIError(errors.ErrInvalidRange), zap.Error(err))
return return
} }
if uint64(r.ContentLength) != (byteRange.End - byteRange.Start + 1) { if uint64(r.ContentLength) != (byteRange.End - byteRange.Start + 1) {
h.logAndSendError(w, "content-length must be equal to byte range length", reqInfo, errors.GetAPIError(errors.ErrInvalidRangeLength)) h.logAndSendError(ctx, w, "content-length must be equal to byte range length", reqInfo, errors.GetAPIError(errors.ErrInvalidRangeLength))
return return
} }
if byteRange.Start > srcSize { if byteRange.Start > srcSize {
h.logAndSendError(w, "start byte is greater than object size", reqInfo, errors.GetAPIError(errors.ErrRangeOutOfBounds)) h.logAndSendError(ctx, w, "start byte is greater than object size", reqInfo, errors.GetAPIError(errors.ErrRangeOutOfBounds))
return return
} }
@ -104,16 +104,16 @@ func (h *handler) PatchObjectHandler(w http.ResponseWriter, r *http.Request) {
params.CopiesNumbers, err = h.pickCopiesNumbers(nil, reqInfo.Namespace, bktInfo.LocationConstraint) params.CopiesNumbers, err = h.pickCopiesNumbers(nil, reqInfo.Namespace, bktInfo.LocationConstraint)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid copies number", reqInfo, err) h.logAndSendError(ctx, w, "invalid copies number", reqInfo, err)
return return
} }
extendedObjInfo, err := h.obj.PatchObject(ctx, params) extendedObjInfo, err := h.obj.PatchObject(ctx, params)
if err != nil { if err != nil {
if isErrObjectLocked(err) { if isErrObjectLocked(err) {
h.logAndSendError(w, "object is locked", reqInfo, errors.GetAPIError(errors.ErrAccessDenied)) h.logAndSendError(ctx, w, "object is locked", reqInfo, errors.GetAPIError(errors.ErrAccessDenied))
} else { } else {
h.logAndSendError(w, "could not patch object", reqInfo, err) h.logAndSendError(ctx, w, "could not patch object", reqInfo, err)
} }
return return
} }
@ -132,7 +132,7 @@ func (h *handler) PatchObjectHandler(w http.ResponseWriter, r *http.Request) {
} }
if err = middleware.EncodeToResponse(w, resp); err != nil { if err = middleware.EncodeToResponse(w, resp); err != nil {
h.logAndSendError(w, "could not encode PatchObjectResult to response", reqInfo, err) h.logAndSendError(ctx, w, "could not encode PatchObjectResult to response", reqInfo, err)
return return
} }
} }

View file

@ -18,7 +18,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@ -50,7 +50,7 @@ func TestPatch(t *testing.T) {
name string name string
rng string rng string
headers map[string]string headers map[string]string
code s3errors.ErrorCode code apierr.ErrorCode
}{ }{
{ {
name: "success", name: "success",
@ -63,22 +63,22 @@ func TestPatch(t *testing.T) {
{ {
name: "invalid range syntax", name: "invalid range syntax",
rng: "bytes 0-2", rng: "bytes 0-2",
code: s3errors.ErrInvalidRange, code: apierr.ErrInvalidRange,
}, },
{ {
name: "invalid range length", name: "invalid range length",
rng: "bytes 0-5/*", rng: "bytes 0-5/*",
code: s3errors.ErrInvalidRangeLength, code: apierr.ErrInvalidRangeLength,
}, },
{ {
name: "invalid range start", name: "invalid range start",
rng: "bytes 20-22/*", rng: "bytes 20-22/*",
code: s3errors.ErrRangeOutOfBounds, code: apierr.ErrRangeOutOfBounds,
}, },
{ {
name: "range is too long", name: "range is too long",
rng: "bytes 0-5368709120/*", rng: "bytes 0-5368709120/*",
code: s3errors.ErrInvalidRange, code: apierr.ErrInvalidRange,
}, },
{ {
name: "If-Unmodified-Since precondition are not satisfied", name: "If-Unmodified-Since precondition are not satisfied",
@ -86,7 +86,7 @@ func TestPatch(t *testing.T) {
headers: map[string]string{ headers: map[string]string{
api.IfUnmodifiedSince: created.Add(-24 * time.Hour).Format(http.TimeFormat), api.IfUnmodifiedSince: created.Add(-24 * time.Hour).Format(http.TimeFormat),
}, },
code: s3errors.ErrPreconditionFailed, code: apierr.ErrPreconditionFailed,
}, },
{ {
name: "If-Match precondition are not satisfied", name: "If-Match precondition are not satisfied",
@ -94,7 +94,7 @@ func TestPatch(t *testing.T) {
headers: map[string]string{ headers: map[string]string{
api.IfMatch: "etag", api.IfMatch: "etag",
}, },
code: s3errors.ErrPreconditionFailed, code: apierr.ErrPreconditionFailed,
}, },
} { } {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
@ -102,7 +102,7 @@ func TestPatch(t *testing.T) {
res := patchObject(t, tc, bktName, objName, tt.rng, patchPayload, tt.headers) res := patchObject(t, tc, bktName, objName, tt.rng, patchPayload, tt.headers)
require.Equal(t, data.Quote(hash), res.Object.ETag) require.Equal(t, data.Quote(hash), res.Object.ETag)
} else { } else {
patchObjectErr(t, tc, bktName, objName, tt.rng, patchPayload, tt.headers, tt.code) patchObjectErr(tc, bktName, objName, tt.rng, patchPayload, tt.headers, tt.code)
} }
}) })
} }
@ -377,7 +377,7 @@ func TestPatchEncryptedObject(t *testing.T) {
tc.Handler().PutObjectHandler(w, r) tc.Handler().PutObjectHandler(w, r)
assertStatus(t, w, http.StatusOK) assertStatus(t, w, http.StatusOK)
patchObjectErr(t, tc, bktName, objName, "bytes 2-4/*", []byte("new"), nil, s3errors.ErrInternalError) patchObjectErr(tc, bktName, objName, "bytes 2-4/*", []byte("new"), nil, apierr.ErrInternalError)
} }
func TestPatchMissingHeaders(t *testing.T) { func TestPatchMissingHeaders(t *testing.T) {
@ -393,13 +393,21 @@ func TestPatchMissingHeaders(t *testing.T) {
w = httptest.NewRecorder() w = httptest.NewRecorder()
r = httptest.NewRequest(http.MethodPatch, defaultURL, strings.NewReader("new")) r = httptest.NewRequest(http.MethodPatch, defaultURL, strings.NewReader("new"))
tc.Handler().PatchObjectHandler(w, r) tc.Handler().PatchObjectHandler(w, r)
assertS3Error(t, w, s3errors.GetAPIError(s3errors.ErrMissingContentRange)) assertS3Error(t, w, apierr.GetAPIError(apierr.ErrMissingContentRange))
w = httptest.NewRecorder() w = httptest.NewRecorder()
r = httptest.NewRequest(http.MethodPatch, defaultURL, strings.NewReader("new")) r = httptest.NewRequest(http.MethodPatch, defaultURL, strings.NewReader("new"))
r.Header.Set(api.ContentRange, "bytes 0-2/*") r.Header.Set(api.ContentRange, "bytes 0-2/*")
tc.Handler().PatchObjectHandler(w, r) tc.Handler().PatchObjectHandler(w, r)
assertS3Error(t, w, s3errors.GetAPIError(s3errors.ErrMissingContentLength)) assertS3Error(t, w, apierr.GetAPIError(apierr.ErrMissingContentLength))
}
func TestPatchInvalidBucketName(t *testing.T) {
tc := prepareHandlerContext(t)
bktName, objName := "bucket", "object"
createTestBucket(tc, bktName)
patchObjectErr(tc, "bkt_name", objName, "bytes 2-4/*", []byte("new"), nil, apierr.ErrInvalidBucketName)
} }
func TestParsePatchByteRange(t *testing.T) { func TestParsePatchByteRange(t *testing.T) {
@ -501,9 +509,9 @@ func patchObjectVersion(t *testing.T, tc *handlerContext, bktName, objName, vers
return result return result
} }
func patchObjectErr(t *testing.T, tc *handlerContext, bktName, objName, rng string, payload []byte, headers map[string]string, code s3errors.ErrorCode) { func patchObjectErr(tc *handlerContext, bktName, objName, rng string, payload []byte, headers map[string]string, code apierr.ErrorCode) {
w := patchObjectBase(tc, bktName, objName, "", rng, payload, headers) w := patchObjectBase(tc, bktName, objName, "", rng, payload, headers)
assertS3Error(t, w, s3errors.GetAPIError(code)) assertS3Error(tc.t, w, apierr.GetAPIError(code))
} }
func patchObjectBase(tc *handlerContext, bktName, objName, version, rng string, payload []byte, headers map[string]string) *httptest.ResponseRecorder { func patchObjectBase(tc *handlerContext, bktName, objName, version, rng string, payload []byte, headers map[string]string) *httptest.ResponseRecorder {

View file

@ -2,11 +2,12 @@ package handler
import ( import (
"bytes" "bytes"
"context"
"crypto/md5" "crypto/md5"
"encoding/base64" "encoding/base64"
"encoding/json" "encoding/json"
"encoding/xml" "encoding/xml"
stderrors "errors" "errors"
"fmt" "fmt"
"io" "io"
"mime/multipart" "mime/multipart"
@ -20,7 +21,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
@ -91,11 +92,11 @@ func (p *postPolicy) CheckField(key string, value string) error {
} }
cond := p.condition(key) cond := p.condition(key)
if cond == nil { if cond == nil {
return errors.GetAPIError(errors.ErrPostPolicyConditionInvalidFormat) return apierr.GetAPIError(apierr.ErrPostPolicyConditionInvalidFormat)
} }
if !cond.match(value) { if !cond.match(value) {
return errors.GetAPIError(errors.ErrPostPolicyConditionInvalidFormat) return apierr.GetAPIError(apierr.ErrPostPolicyConditionInvalidFormat)
} }
return nil return nil
@ -192,24 +193,24 @@ func (h *handler) PutObjectHandler(w http.ResponseWriter, r *http.Request) {
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket objInfo", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket objInfo", reqInfo, err)
return return
} }
settings, err := h.obj.GetBucketSettings(ctx, bktInfo) settings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket settings", reqInfo, err)
return return
} }
if cannedACLStatus == aclStatusYes { if cannedACLStatus == aclStatusYes {
h.logAndSendError(w, "acl not supported for this bucket", reqInfo, errors.GetAPIError(errors.ErrAccessControlListNotSupported)) h.logAndSendError(ctx, w, "acl not supported for this bucket", reqInfo, apierr.GetAPIError(apierr.ErrAccessControlListNotSupported))
return return
} }
tagSet, err := parseTaggingHeader(r.Header) tagSet, err := parseTaggingHeader(r.Header)
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse tagging header", reqInfo, err) h.logAndSendError(ctx, w, "could not parse tagging header", reqInfo, err)
return return
} }
@ -229,44 +230,44 @@ func (h *handler) PutObjectHandler(w http.ResponseWriter, r *http.Request) {
encryptionParams, err := formEncryptionParams(r) encryptionParams, err := formEncryptionParams(r)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid sse headers", reqInfo, err) h.logAndSendError(ctx, w, "invalid sse headers", reqInfo, err)
return return
} }
body, err := h.getBodyReader(r) body, err := h.getBodyReader(r)
if err != nil { if err != nil {
h.logAndSendError(w, "failed to get body reader", reqInfo, err) h.logAndSendError(ctx, w, "failed to get body reader", reqInfo, err)
return return
} }
if encodings := r.Header.Get(api.ContentEncoding); len(encodings) > 0 { if encodings := r.Header.Get(api.ContentEncoding); len(encodings) > 0 {
metadata[api.ContentEncoding] = encodings metadata[api.ContentEncoding] = encodings
} }
var size uint64 size := h.getPutPayloadSize(r)
if r.ContentLength > 0 {
size = uint64(r.ContentLength)
}
params := &layer.PutObjectParams{ params := &layer.PutObjectParams{
BktInfo: bktInfo, BktInfo: bktInfo,
Object: reqInfo.ObjectName, Object: reqInfo.ObjectName,
Reader: body, Reader: body,
Size: size,
Header: metadata, Header: metadata,
Encryption: encryptionParams, Encryption: encryptionParams,
ContentMD5: r.Header.Get(api.ContentMD5), ContentMD5: getMD5Header(r),
ContentSHA256Hash: r.Header.Get(api.AmzContentSha256), ContentSHA256Hash: r.Header.Get(api.AmzContentSha256),
} }
if size > 0 {
params.Size = &size
}
params.CopiesNumbers, err = h.pickCopiesNumbers(metadata, reqInfo.Namespace, bktInfo.LocationConstraint) params.CopiesNumbers, err = h.pickCopiesNumbers(metadata, reqInfo.Namespace, bktInfo.LocationConstraint)
if err != nil { if err != nil {
h.logAndSendError(w, "invalid copies number", reqInfo, err) h.logAndSendError(ctx, w, "invalid copies number", reqInfo, err)
return return
} }
params.Lock, err = formObjectLock(ctx, bktInfo, settings.LockConfiguration, r.Header) params.Lock, err = formObjectLock(ctx, bktInfo, settings.LockConfiguration, r.Header)
if err != nil { if err != nil {
h.logAndSendError(w, "could not form object lock", reqInfo, err) h.logAndSendError(ctx, w, "could not form object lock", reqInfo, err)
return return
} }
@ -274,7 +275,7 @@ func (h *handler) PutObjectHandler(w http.ResponseWriter, r *http.Request) {
if err != nil { if err != nil {
_, err2 := io.Copy(io.Discard, body) _, err2 := io.Copy(io.Discard, body)
err3 := body.Close() err3 := body.Close()
h.logAndSendError(w, "could not upload object", reqInfo, err, zap.Errors("body close errors", []error{err2, err3})) h.logAndSendError(ctx, w, "could not upload object", reqInfo, err, zap.Errors("body close errors", []error{err2, err3}))
return return
} }
objInfo := extendedObjInfo.ObjectInfo objInfo := extendedObjInfo.ObjectInfo
@ -289,8 +290,8 @@ func (h *handler) PutObjectHandler(w http.ResponseWriter, r *http.Request) {
TagSet: tagSet, TagSet: tagSet,
NodeVersion: extendedObjInfo.NodeVersion, NodeVersion: extendedObjInfo.NodeVersion,
} }
if err = h.obj.PutObjectTagging(r.Context(), tagPrm); err != nil { if err = h.obj.PutObjectTagging(ctx, tagPrm); err != nil {
h.logAndSendError(w, "could not upload object tagging", reqInfo, err) h.logAndSendError(ctx, w, "could not upload object tagging", reqInfo, err)
return return
} }
} }
@ -305,7 +306,7 @@ func (h *handler) PutObjectHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set(api.ETag, data.Quote(objInfo.ETag(h.cfg.MD5Enabled()))) w.Header().Set(api.ETag, data.Quote(objInfo.ETag(h.cfg.MD5Enabled())))
if err = middleware.WriteSuccessResponseHeadersOnly(w); err != nil { if err = middleware.WriteSuccessResponseHeadersOnly(w); err != nil {
h.logAndSendError(w, "write response", reqInfo, err) h.logAndSendError(ctx, w, "write response", reqInfo, err)
return return
} }
} }
@ -332,16 +333,16 @@ func (h *handler) getBodyReader(r *http.Request) (io.ReadCloser, error) {
if !chunkedEncoding && !h.cfg.BypassContentEncodingInChunks() { if !chunkedEncoding && !h.cfg.BypassContentEncodingInChunks() {
return nil, fmt.Errorf("%w: request is not chunk encoded, encodings '%s'", return nil, fmt.Errorf("%w: request is not chunk encoded, encodings '%s'",
errors.GetAPIError(errors.ErrInvalidEncodingMethod), strings.Join(encodings, ",")) apierr.GetAPIError(apierr.ErrInvalidEncodingMethod), strings.Join(encodings, ","))
} }
decodeContentSize := r.Header.Get(api.AmzDecodedContentLength) decodeContentSize := r.Header.Get(api.AmzDecodedContentLength)
if len(decodeContentSize) == 0 { if len(decodeContentSize) == 0 {
return nil, errors.GetAPIError(errors.ErrMissingContentLength) return nil, apierr.GetAPIError(apierr.ErrMissingContentLength)
} }
if _, err := strconv.Atoi(decodeContentSize); err != nil { if _, err := strconv.Atoi(decodeContentSize); err != nil {
return nil, fmt.Errorf("%w: parse decoded content length: %s", errors.GetAPIError(errors.ErrMissingContentLength), err.Error()) return nil, fmt.Errorf("%w: parse decoded content length: %s", apierr.GetAPIError(apierr.ErrMissingContentLength), err.Error())
} }
chunkReader, err := newSignV4ChunkedReader(r) chunkReader, err := newSignV4ChunkedReader(r)
@ -377,43 +378,43 @@ func formEncryptionParamsBase(r *http.Request, isCopySource bool) (enc encryptio
} }
if r.TLS == nil { if r.TLS == nil {
return enc, errors.GetAPIError(errors.ErrInsecureSSECustomerRequest) return enc, apierr.GetAPIError(apierr.ErrInsecureSSECustomerRequest)
} }
if len(sseCustomerKey) > 0 && len(sseCustomerAlgorithm) == 0 { if len(sseCustomerKey) > 0 && len(sseCustomerAlgorithm) == 0 {
return enc, errors.GetAPIError(errors.ErrMissingSSECustomerAlgorithm) return enc, apierr.GetAPIError(apierr.ErrMissingSSECustomerAlgorithm)
} }
if len(sseCustomerAlgorithm) > 0 && len(sseCustomerKey) == 0 { if len(sseCustomerAlgorithm) > 0 && len(sseCustomerKey) == 0 {
return enc, errors.GetAPIError(errors.ErrMissingSSECustomerKey) return enc, apierr.GetAPIError(apierr.ErrMissingSSECustomerKey)
} }
if sseCustomerAlgorithm != layer.AESEncryptionAlgorithm { if sseCustomerAlgorithm != layer.AESEncryptionAlgorithm {
return enc, errors.GetAPIError(errors.ErrInvalidEncryptionAlgorithm) return enc, apierr.GetAPIError(apierr.ErrInvalidEncryptionAlgorithm)
} }
key, err := base64.StdEncoding.DecodeString(sseCustomerKey) key, err := base64.StdEncoding.DecodeString(sseCustomerKey)
if err != nil { if err != nil {
if isCopySource { if isCopySource {
return enc, errors.GetAPIError(errors.ErrInvalidSSECustomerParameters) return enc, apierr.GetAPIError(apierr.ErrInvalidSSECustomerParameters)
} }
return enc, errors.GetAPIError(errors.ErrInvalidSSECustomerKey) return enc, apierr.GetAPIError(apierr.ErrInvalidSSECustomerKey)
} }
if len(key) != layer.AESKeySize { if len(key) != layer.AESKeySize {
if isCopySource { if isCopySource {
return enc, errors.GetAPIError(errors.ErrInvalidSSECustomerParameters) return enc, apierr.GetAPIError(apierr.ErrInvalidSSECustomerParameters)
} }
return enc, errors.GetAPIError(errors.ErrInvalidSSECustomerKey) return enc, apierr.GetAPIError(apierr.ErrInvalidSSECustomerKey)
} }
keyMD5, err := base64.StdEncoding.DecodeString(sseCustomerKeyMD5) keyMD5, err := base64.StdEncoding.DecodeString(sseCustomerKeyMD5)
if err != nil { if err != nil {
return enc, errors.GetAPIError(errors.ErrSSECustomerKeyMD5Mismatch) return enc, apierr.GetAPIError(apierr.ErrSSECustomerKeyMD5Mismatch)
} }
md5Sum := md5.Sum(key) md5Sum := md5.Sum(key)
if !bytes.Equal(md5Sum[:], keyMD5) { if !bytes.Equal(md5Sum[:], keyMD5) {
return enc, errors.GetAPIError(errors.ErrSSECustomerKeyMD5Mismatch) return enc, apierr.GetAPIError(apierr.ErrSSECustomerKeyMD5Mismatch)
} }
params, err := encryption.NewParams(key) params, err := encryption.NewParams(key)
@ -434,7 +435,7 @@ func (h *handler) PostObject(w http.ResponseWriter, r *http.Request) {
policy, err := checkPostPolicy(r, reqInfo, metadata) policy, err := checkPostPolicy(r, reqInfo, metadata)
if err != nil { if err != nil {
h.logAndSendError(w, "failed check policy", reqInfo, err) h.logAndSendError(ctx, w, "failed check policy", reqInfo, err)
return return
} }
@ -442,31 +443,31 @@ func (h *handler) PostObject(w http.ResponseWriter, r *http.Request) {
buffer := bytes.NewBufferString(tagging) buffer := bytes.NewBufferString(tagging)
tags := new(data.Tagging) tags := new(data.Tagging)
if err = h.cfg.NewXMLDecoder(buffer).Decode(tags); err != nil { if err = h.cfg.NewXMLDecoder(buffer).Decode(tags); err != nil {
h.logAndSendError(w, "could not decode tag set", reqInfo, h.logAndSendError(ctx, w, "could not decode tag set", reqInfo,
fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrMalformedXML), err.Error())) fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrMalformedXML), err.Error()))
return return
} }
tagSet, err = h.readTagSet(tags) tagSet, err = h.readTagSet(tags)
if err != nil { if err != nil {
h.logAndSendError(w, "could not read tag set", reqInfo, err) h.logAndSendError(ctx, w, "could not read tag set", reqInfo, err)
return return
} }
} }
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket objInfo", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket objInfo", reqInfo, err)
return return
} }
settings, err := h.obj.GetBucketSettings(ctx, bktInfo) settings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket settings", reqInfo, err)
return return
} }
if acl := auth.MultipartFormValue(r, "acl"); acl != "" && acl != basicACLPrivate { if acl := auth.MultipartFormValue(r, "acl"); acl != "" && acl != basicACLPrivate {
h.logAndSendError(w, "acl not supported for this bucket", reqInfo, errors.GetAPIError(errors.ErrAccessControlListNotSupported)) h.logAndSendError(ctx, w, "acl not supported for this bucket", reqInfo, apierr.GetAPIError(apierr.ErrAccessControlListNotSupported))
return return
} }
@ -484,7 +485,7 @@ func (h *handler) PostObject(w http.ResponseWriter, r *http.Request) {
if reqInfo.ObjectName == "" || strings.Contains(reqInfo.ObjectName, "${filename}") { if reqInfo.ObjectName == "" || strings.Contains(reqInfo.ObjectName, "${filename}") {
_, head, err := r.FormFile("file") _, head, err := r.FormFile("file")
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse file field", reqInfo, err) h.logAndSendError(ctx, w, "could not parse file field", reqInfo, err)
return return
} }
filename = head.Filename filename = head.Filename
@ -493,7 +494,7 @@ func (h *handler) PostObject(w http.ResponseWriter, r *http.Request) {
var head *multipart.FileHeader var head *multipart.FileHeader
contentReader, head, err = r.FormFile("file") contentReader, head, err = r.FormFile("file")
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse file field", reqInfo, err) h.logAndSendError(ctx, w, "could not parse file field", reqInfo, err)
return return
} }
size = uint64(head.Size) size = uint64(head.Size)
@ -507,12 +508,12 @@ func (h *handler) PostObject(w http.ResponseWriter, r *http.Request) {
} }
if reqInfo.ObjectName == "" { if reqInfo.ObjectName == "" {
h.logAndSendError(w, "missing object name", reqInfo, errors.GetAPIError(errors.ErrInvalidArgument)) h.logAndSendError(ctx, w, "missing object name", reqInfo, apierr.GetAPIError(apierr.ErrInvalidArgument))
return return
} }
if !policy.CheckContentLength(size) { if !policy.CheckContentLength(size) {
h.logAndSendError(w, "invalid content-length", reqInfo, errors.GetAPIError(errors.ErrInvalidArgument)) h.logAndSendError(ctx, w, "invalid content-length", reqInfo, apierr.GetAPIError(apierr.ErrInvalidArgument))
return return
} }
@ -520,13 +521,13 @@ func (h *handler) PostObject(w http.ResponseWriter, r *http.Request) {
BktInfo: bktInfo, BktInfo: bktInfo,
Object: reqInfo.ObjectName, Object: reqInfo.ObjectName,
Reader: contentReader, Reader: contentReader,
Size: size, Size: &size,
Header: metadata, Header: metadata,
} }
extendedObjInfo, err := h.obj.PutObject(ctx, params) extendedObjInfo, err := h.obj.PutObject(ctx, params)
if err != nil { if err != nil {
h.logAndSendError(w, "could not upload object", reqInfo, err) h.logAndSendError(ctx, w, "could not upload object", reqInfo, err)
return return
} }
objInfo := extendedObjInfo.ObjectInfo objInfo := extendedObjInfo.ObjectInfo
@ -542,7 +543,7 @@ func (h *handler) PostObject(w http.ResponseWriter, r *http.Request) {
} }
if err = h.obj.PutObjectTagging(ctx, tagPrm); err != nil { if err = h.obj.PutObjectTagging(ctx, tagPrm); err != nil {
h.logAndSendError(w, "could not upload object tagging", reqInfo, err) h.logAndSendError(ctx, w, "could not upload object tagging", reqInfo, err)
return return
} }
} }
@ -570,10 +571,10 @@ func (h *handler) PostObject(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(status) w.WriteHeader(status)
respData, err := middleware.EncodeResponse(resp) respData, err := middleware.EncodeResponse(resp)
if err != nil { if err != nil {
h.logAndSendError(w, "encode response", reqInfo, err) h.logAndSendError(ctx, w, "encode response", reqInfo, err)
} }
if _, err = w.Write(respData); err != nil { if _, err = w.Write(respData); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
} }
return return
} }
@ -594,13 +595,13 @@ func checkPostPolicy(r *http.Request, reqInfo *middleware.ReqInfo, metadata map[
return nil, fmt.Errorf("could not unmarshal policy: %w", err) return nil, fmt.Errorf("could not unmarshal policy: %w", err)
} }
if policy.Expiration.Before(time.Now()) { if policy.Expiration.Before(time.Now()) {
return nil, fmt.Errorf("policy is expired: %w", errors.GetAPIError(errors.ErrInvalidArgument)) return nil, fmt.Errorf("policy is expired: %w", apierr.GetAPIError(apierr.ErrInvalidArgument))
} }
policy.empty = false policy.empty = false
} }
if r.MultipartForm == nil { if r.MultipartForm == nil {
return nil, stderrors.New("empty multipart form") return nil, errors.New("empty multipart form")
} }
for key, v := range r.MultipartForm.Value { for key, v := range r.MultipartForm.Value {
@ -631,7 +632,7 @@ func checkPostPolicy(r *http.Request, reqInfo *middleware.ReqInfo, metadata map[
for _, cond := range policy.Conditions { for _, cond := range policy.Conditions {
if cond.Key == "bucket" { if cond.Key == "bucket" {
if !cond.match(reqInfo.BucketName) { if !cond.match(reqInfo.BucketName) {
return nil, errors.GetAPIError(errors.ErrPostPolicyConditionInvalidFormat) return nil, apierr.GetAPIError(apierr.ErrPostPolicyConditionInvalidFormat)
} }
} }
} }
@ -673,10 +674,10 @@ func parseTaggingHeader(header http.Header) (map[string]string, error) {
if tagging := header.Get(api.AmzTagging); len(tagging) > 0 { if tagging := header.Get(api.AmzTagging); len(tagging) > 0 {
queries, err := url.ParseQuery(tagging) queries, err := url.ParseQuery(tagging)
if err != nil { if err != nil {
return nil, errors.GetAPIError(errors.ErrInvalidArgument) return nil, apierr.GetAPIError(apierr.ErrInvalidArgument)
} }
if len(queries) > maxTags { if len(queries) > maxTags {
return nil, errors.GetAPIError(errors.ErrInvalidTagsSizeExceed) return nil, apierr.GetAPIError(apierr.ErrInvalidTagsSizeExceed)
} }
tagSet = make(map[string]string, len(queries)) tagSet = make(map[string]string, len(queries))
for k, v := range queries { for k, v := range queries {
@ -726,7 +727,7 @@ func (h *handler) parseCommonCreateBucketParams(reqInfo *middleware.ReqInfo, box
} }
if p.SessionContainerCreation == nil { if p.SessionContainerCreation == nil {
return nil, nil, fmt.Errorf("%w: couldn't find session token for put", errors.GetAPIError(errors.ErrAccessDenied)) return nil, nil, fmt.Errorf("%w: couldn't find session token for put", apierr.GetAPIError(apierr.ErrAccessDenied))
} }
if err := checkBucketName(reqInfo.BucketName); err != nil { if err := checkBucketName(reqInfo.BucketName); err != nil {
@ -758,32 +759,33 @@ func (h *handler) createBucketHandlerPolicy(w http.ResponseWriter, r *http.Reque
boxData, err := middleware.GetBoxData(ctx) boxData, err := middleware.GetBoxData(ctx)
if err != nil { if err != nil {
h.logAndSendError(w, "get access box from request", reqInfo, err) h.logAndSendError(ctx, w, "get access box from request", reqInfo, err)
return return
} }
key, p, err := h.parseCommonCreateBucketParams(reqInfo, boxData, r) key, p, err := h.parseCommonCreateBucketParams(reqInfo, boxData, r)
if err != nil { if err != nil {
h.logAndSendError(w, "parse create bucket params", reqInfo, err) h.logAndSendError(ctx, w, "parse create bucket params", reqInfo, err)
return return
} }
cannedACL, err := parseCannedACL(r.Header) cannedACL, err := parseCannedACL(r.Header)
if err != nil { if err != nil {
h.logAndSendError(w, "could not parse canned ACL", reqInfo, err) h.logAndSendError(ctx, w, "could not parse canned ACL", reqInfo, err)
return return
} }
bktInfo, err := h.obj.CreateBucket(ctx, p) bktInfo, err := h.obj.CreateBucket(ctx, p)
if err != nil { if err != nil {
h.logAndSendError(w, "could not create bucket", reqInfo, err) h.logAndSendError(ctx, w, "could not create bucket", reqInfo, err)
return return
} }
h.reqLogger(ctx).Info(logs.BucketIsCreated, zap.Stringer("container_id", bktInfo.CID)) h.reqLogger(ctx).Info(logs.BucketIsCreated, zap.Stringer("container_id", bktInfo.CID))
chains := bucketCannedACLToAPERules(cannedACL, reqInfo, bktInfo.CID) chains := bucketCannedACLToAPERules(cannedACL, reqInfo, bktInfo.CID)
if err = h.ape.SaveACLChains(bktInfo.CID.EncodeToString(), chains); err != nil { if err = h.ape.SaveACLChains(bktInfo.CID.EncodeToString(), chains); err != nil {
h.logAndSendError(w, "failed to add morph rule chain", reqInfo, err) cleanErr := h.cleanupBucketCreation(ctx, reqInfo, bktInfo, boxData, chains)
h.logAndSendError(ctx, w, "failed to add morph rule chain", reqInfo, err, zap.NamedError("cleanup_error", cleanErr))
return return
} }
@ -804,17 +806,40 @@ func (h *handler) createBucketHandlerPolicy(w http.ResponseWriter, r *http.Reque
return h.obj.PutBucketSettings(ctx, sp) return h.obj.PutBucketSettings(ctx, sp)
}, h.putBucketSettingsRetryer()) }, h.putBucketSettingsRetryer())
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't save bucket settings", reqInfo, err, cleanErr := h.cleanupBucketCreation(ctx, reqInfo, bktInfo, boxData, chains)
zap.String("container_id", bktInfo.CID.EncodeToString())) h.logAndSendError(ctx, w, "couldn't save bucket settings", reqInfo, err,
zap.String("container_id", bktInfo.CID.EncodeToString()), zap.NamedError("cleanup_error", cleanErr))
return return
} }
if err = middleware.WriteSuccessResponseHeadersOnly(w); err != nil { if err = middleware.WriteSuccessResponseHeadersOnly(w); err != nil {
h.logAndSendError(w, "write response", reqInfo, err) h.logAndSendError(ctx, w, "write response", reqInfo, err)
return return
} }
} }
func (h *handler) cleanupBucketCreation(ctx context.Context, reqInfo *middleware.ReqInfo, bktInfo *data.BucketInfo, boxData *accessbox.Box, chains []*chain.Chain) error {
prm := &layer.DeleteBucketParams{
BktInfo: bktInfo,
SessionToken: boxData.Gate.SessionTokenForDelete(),
}
if err := h.obj.DeleteContainer(ctx, prm); err != nil {
return err
}
chainIDs := make([]chain.ID, len(chains))
for i, c := range chains {
chainIDs[i] = c.ID
}
if err := h.ape.DeleteBucketPolicy(reqInfo.Namespace, bktInfo.CID, chainIDs); err != nil {
return fmt.Errorf("delete bucket acl policy: %w", err)
}
return nil
}
func (h *handler) putBucketSettingsRetryer() aws.RetryerV2 { func (h *handler) putBucketSettingsRetryer() aws.RetryerV2 {
return retry.NewStandard(func(options *retry.StandardOptions) { return retry.NewStandard(func(options *retry.StandardOptions) {
options.MaxAttempts = h.cfg.RetryMaxAttempts() options.MaxAttempts = h.cfg.RetryMaxAttempts()
@ -828,7 +853,7 @@ func (h *handler) putBucketSettingsRetryer() aws.RetryerV2 {
} }
options.Retryables = []retry.IsErrorRetryable{retry.IsErrorRetryableFunc(func(err error) aws.Ternary { options.Retryables = []retry.IsErrorRetryable{retry.IsErrorRetryableFunc(func(err error) aws.Ternary {
if stderrors.Is(err, tree.ErrNodeAccessDenied) { if errors.Is(err, tree.ErrNodeAccessDenied) {
return aws.TrueTernary return aws.TrueTernary
} }
return aws.FalseTernary return aws.FalseTernary
@ -957,7 +982,7 @@ func (h handler) setPlacementPolicy(prm *layer.CreateBucketParams, namespace, lo
return nil return nil
} }
return errors.GetAPIError(errors.ErrInvalidLocationConstraint) return apierr.GetAPIError(apierr.ErrInvalidLocationConstraint)
} }
func isLockEnabled(log *zap.Logger, header http.Header) bool { func isLockEnabled(log *zap.Logger, header http.Header) bool {
@ -976,28 +1001,22 @@ func isLockEnabled(log *zap.Logger, header http.Header) bool {
func checkBucketName(bucketName string) error { func checkBucketName(bucketName string) error {
if len(bucketName) < 3 || len(bucketName) > 63 { if len(bucketName) < 3 || len(bucketName) > 63 {
return errors.GetAPIError(errors.ErrInvalidBucketName) return apierr.GetAPIError(apierr.ErrInvalidBucketName)
} }
if strings.HasPrefix(bucketName, "xn--") || strings.HasSuffix(bucketName, "-s3alias") { if strings.HasPrefix(bucketName, "xn--") || strings.HasSuffix(bucketName, "-s3alias") {
return errors.GetAPIError(errors.ErrInvalidBucketName) return apierr.GetAPIError(apierr.ErrInvalidBucketName)
} }
if net.ParseIP(bucketName) != nil { if net.ParseIP(bucketName) != nil {
return errors.GetAPIError(errors.ErrInvalidBucketName) return apierr.GetAPIError(apierr.ErrInvalidBucketName)
} }
labels := strings.Split(bucketName, ".") for i, r := range bucketName {
for _, label := range labels { if r == '.' || (!isAlphaNum(r) && r != '-') {
if len(label) == 0 { return apierr.GetAPIError(apierr.ErrInvalidBucketName)
return errors.GetAPIError(errors.ErrInvalidBucketName)
}
for i, r := range label {
if !isAlphaNum(r) && r != '-' {
return errors.GetAPIError(errors.ErrInvalidBucketName)
}
if (i == 0 || i == len(label)-1) && r == '-' {
return errors.GetAPIError(errors.ErrInvalidBucketName)
} }
if (i == 0 || i == len(bucketName)-1) && r == '-' {
return apierr.GetAPIError(apierr.ErrInvalidBucketName)
} }
} }
@ -1015,7 +1034,17 @@ func (h *handler) parseLocationConstraint(r *http.Request) (*createBucketParams,
params := new(createBucketParams) params := new(createBucketParams)
if err := h.cfg.NewXMLDecoder(r.Body).Decode(params); err != nil { if err := h.cfg.NewXMLDecoder(r.Body).Decode(params); err != nil {
return nil, fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrMalformedXML), err.Error()) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrMalformedXML), err.Error())
} }
return params, nil return params, nil
} }
func getMD5Header(r *http.Request) *string {
var md5Hdr *string
if len(r.Header.Values(api.ContentMD5)) != 0 {
hdr := r.Header.Get(api.ContentMD5)
md5Hdr = &hdr
}
return md5Hdr
}

View file

@ -7,6 +7,7 @@ import (
"encoding/base64" "encoding/base64"
"encoding/hex" "encoding/hex"
"encoding/json" "encoding/json"
"errors"
"io" "io"
"mime/multipart" "mime/multipart"
"net/http" "net/http"
@ -20,7 +21,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth"
v4 "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth/signer/v4" v4 "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth/signer/v4"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
@ -29,6 +30,11 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
const (
awsChunkedRequestExampleDecodedContentLength = 66560
awsChunkedRequestExampleContentLength = 66824
)
func TestCheckBucketName(t *testing.T) { func TestCheckBucketName(t *testing.T) {
for _, tc := range []struct { for _, tc := range []struct {
name string name string
@ -36,10 +42,10 @@ func TestCheckBucketName(t *testing.T) {
}{ }{
{name: "bucket"}, {name: "bucket"},
{name: "2bucket"}, {name: "2bucket"},
{name: "buc.ket"},
{name: "buc-ket"}, {name: "buc-ket"},
{name: "abc"}, {name: "abc"},
{name: "63aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}, {name: "63aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"},
{name: "buc.ket", err: true},
{name: "buc.-ket", err: true}, {name: "buc.-ket", err: true},
{name: "bucket.", err: true}, {name: "bucket.", err: true},
{name: ".bucket", err: true}, {name: ".bucket", err: true},
@ -199,7 +205,7 @@ func TestPostObject(t *testing.T) {
t.Run(tc.key+";"+tc.filename, func(t *testing.T) { t.Run(tc.key+";"+tc.filename, func(t *testing.T) {
w := postObjectBase(hc, ns, bktName, tc.key, tc.filename, tc.content) w := postObjectBase(hc, ns, bktName, tc.key, tc.filename, tc.content)
if tc.err { if tc.err {
assertS3Error(hc.t, w, s3errors.GetAPIError(s3errors.ErrInternalError)) assertS3Error(hc.t, w, apierr.GetAPIError(apierr.ErrInternalError))
return return
} }
assertStatus(hc.t, w, http.StatusNoContent) assertStatus(hc.t, w, http.StatusNoContent)
@ -245,6 +251,10 @@ func TestPutObjectWithNegativeContentLength(t *testing.T) {
tc.Handler().HeadObjectHandler(w, r) tc.Handler().HeadObjectHandler(w, r)
assertStatus(t, w, http.StatusOK) assertStatus(t, w, http.StatusOK)
require.Equal(t, strconv.Itoa(len(content)), w.Header().Get(api.ContentLength)) require.Equal(t, strconv.Itoa(len(content)), w.Header().Get(api.ContentLength))
result := listVersions(t, tc, bktName)
require.Len(t, result.Version, 1)
require.EqualValues(t, len(content), result.Version[0].Size)
} }
func TestPutObjectWithStreamBodyError(t *testing.T) { func TestPutObjectWithStreamBodyError(t *testing.T) {
@ -258,7 +268,7 @@ func TestPutObjectWithStreamBodyError(t *testing.T) {
r.Header.Set(api.AmzContentSha256, api.StreamingContentSHA256) r.Header.Set(api.AmzContentSha256, api.StreamingContentSHA256)
r.Header.Set(api.ContentEncoding, api.AwsChunked) r.Header.Set(api.ContentEncoding, api.AwsChunked)
tc.Handler().PutObjectHandler(w, r) tc.Handler().PutObjectHandler(w, r)
assertS3Error(t, w, s3errors.GetAPIError(s3errors.ErrMissingContentLength)) assertS3Error(t, w, apierr.GetAPIError(apierr.ErrMissingContentLength))
checkNotFound(t, tc, bktName, objName, emptyVersion) checkNotFound(t, tc, bktName, objName, emptyVersion)
} }
@ -274,7 +284,13 @@ func TestPutObjectWithInvalidContentMD5(t *testing.T) {
w, r := prepareTestPayloadRequest(tc, bktName, objName, bytes.NewReader(content)) w, r := prepareTestPayloadRequest(tc, bktName, objName, bytes.NewReader(content))
r.Header.Set(api.ContentMD5, base64.StdEncoding.EncodeToString([]byte("invalid"))) r.Header.Set(api.ContentMD5, base64.StdEncoding.EncodeToString([]byte("invalid")))
tc.Handler().PutObjectHandler(w, r) tc.Handler().PutObjectHandler(w, r)
assertS3Error(t, w, s3errors.GetAPIError(s3errors.ErrInvalidDigest)) assertS3Error(t, w, apierr.GetAPIError(apierr.ErrBadDigest))
content = []byte("content")
w, r = prepareTestPayloadRequest(tc, bktName, objName, bytes.NewReader(content))
r.Header.Set(api.ContentMD5, base64.StdEncoding.EncodeToString([]byte("")))
tc.Handler().PutObjectHandler(w, r)
assertS3Error(t, w, apierr.GetAPIError(apierr.ErrInvalidDigest))
checkNotFound(t, tc, bktName, objName, emptyVersion) checkNotFound(t, tc, bktName, objName, emptyVersion)
} }
@ -337,7 +353,7 @@ func TestPutObjectCheckContentSHA256(t *testing.T) {
hc.Handler().PutObjectHandler(w, r) hc.Handler().PutObjectHandler(w, r)
if tc.error { if tc.error {
assertS3Error(t, w, s3errors.GetAPIError(s3errors.ErrContentSHA256Mismatch)) assertS3Error(t, w, apierr.GetAPIError(apierr.ErrContentSHA256Mismatch))
w, r := prepareTestRequest(hc, bktName, objName, nil) w, r := prepareTestRequest(hc, bktName, objName, nil)
hc.Handler().GetObjectHandler(w, r) hc.Handler().GetObjectHandler(w, r)
@ -361,12 +377,37 @@ func TestPutObjectWithStreamBodyAWSExample(t *testing.T) {
hc.Handler().PutObjectHandler(w, req) hc.Handler().PutObjectHandler(w, req)
assertStatus(t, w, http.StatusOK) assertStatus(t, w, http.StatusOK)
data := getObjectRange(t, hc, bktName, objName, 0, 66824) w, req = prepareTestRequest(hc, bktName, objName, nil)
hc.Handler().HeadObjectHandler(w, req)
assertStatus(t, w, http.StatusOK)
require.Equal(t, strconv.Itoa(awsChunkedRequestExampleDecodedContentLength), w.Header().Get(api.ContentLength))
data := getObjectRange(t, hc, bktName, objName, 0, awsChunkedRequestExampleDecodedContentLength)
for i := range chunk { for i := range chunk {
require.Equal(t, chunk[i], data[i]) require.Equal(t, chunk[i], data[i])
} }
} }
func TestPutObjectWithStreamEmptyBodyAWSExample(t *testing.T) {
hc := prepareHandlerContext(t)
bktName, objName := "dkirillov", "tmp"
createTestBucket(hc, bktName)
w, req := getEmptyChunkedRequest(hc.context, t, bktName, objName)
hc.Handler().PutObjectHandler(w, req)
assertStatus(t, w, http.StatusOK)
w, req = prepareTestRequest(hc, bktName, objName, nil)
hc.Handler().HeadObjectHandler(w, req)
assertStatus(t, w, http.StatusOK)
require.Equal(t, "0", w.Header().Get(api.ContentLength))
res := listObjectsV1(hc, bktName, "", "", "", -1)
require.Len(t, res.Contents, 1)
require.Empty(t, res.Contents[0].Size)
}
func TestPutChunkedTestContentEncoding(t *testing.T) { func TestPutChunkedTestContentEncoding(t *testing.T) {
hc := prepareHandlerContext(t) hc := prepareHandlerContext(t)
@ -385,7 +426,7 @@ func TestPutChunkedTestContentEncoding(t *testing.T) {
w, req, _ = getChunkedRequest(hc.context, t, bktName, objName) w, req, _ = getChunkedRequest(hc.context, t, bktName, objName)
req.Header.Set(api.ContentEncoding, "gzip") req.Header.Set(api.ContentEncoding, "gzip")
hc.Handler().PutObjectHandler(w, req) hc.Handler().PutObjectHandler(w, req)
assertS3Error(t, w, s3errors.GetAPIError(s3errors.ErrInvalidEncodingMethod)) assertS3Error(t, w, apierr.GetAPIError(apierr.ErrInvalidEncodingMethod))
hc.config.bypassContentEncodingInChunks = true hc.config.bypassContentEncodingInChunks = true
w, req, _ = getChunkedRequest(hc.context, t, bktName, objName) w, req, _ = getChunkedRequest(hc.context, t, bktName, objName)
@ -397,6 +438,8 @@ func TestPutChunkedTestContentEncoding(t *testing.T) {
require.Equal(t, "gzip", resp.Header().Get(api.ContentEncoding)) require.Equal(t, "gzip", resp.Header().Get(api.ContentEncoding))
} }
// getChunkedRequest implements request example from
// https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
func getChunkedRequest(ctx context.Context, t *testing.T, bktName, objName string) (*httptest.ResponseRecorder, *http.Request, []byte) { func getChunkedRequest(ctx context.Context, t *testing.T, bktName, objName string) (*httptest.ResponseRecorder, *http.Request, []byte) {
chunk := make([]byte, 65*1024) chunk := make([]byte, 65*1024)
for i := range chunk { for i := range chunk {
@ -424,9 +467,9 @@ func getChunkedRequest(ctx context.Context, t *testing.T, bktName, objName strin
req, err := http.NewRequest("PUT", "https://s3.amazonaws.com/"+bktName+"/"+objName, nil) req, err := http.NewRequest("PUT", "https://s3.amazonaws.com/"+bktName+"/"+objName, nil)
require.NoError(t, err) require.NoError(t, err)
req.Header.Set("content-encoding", "aws-chunked") req.Header.Set("content-encoding", "aws-chunked")
req.Header.Set("content-length", "66824") req.Header.Set("content-length", strconv.Itoa(awsChunkedRequestExampleContentLength))
req.Header.Set("x-amz-content-sha256", "STREAMING-AWS4-HMAC-SHA256-PAYLOAD") req.Header.Set("x-amz-content-sha256", "STREAMING-AWS4-HMAC-SHA256-PAYLOAD")
req.Header.Set("x-amz-decoded-content-length", "66560") req.Header.Set("x-amz-decoded-content-length", strconv.Itoa(awsChunkedRequestExampleDecodedContentLength))
req.Header.Set("x-amz-storage-class", "REDUCED_REDUNDANCY") req.Header.Set("x-amz-storage-class", "REDUCED_REDUNDANCY")
signTime, err := time.Parse("20060102T150405Z", "20130524T000000Z") signTime, err := time.Parse("20060102T150405Z", "20130524T000000Z")
@ -457,15 +500,69 @@ func getChunkedRequest(ctx context.Context, t *testing.T, bktName, objName strin
return w, req, chunk return w, req, chunk
} }
func getEmptyChunkedRequest(ctx context.Context, t *testing.T, bktName, objName string) (*httptest.ResponseRecorder, *http.Request) {
AWSAccessKeyID := "48c1K4PLVb7SvmV3PjDKEuXaMh8yZMXZ8Wx9msrkKcYw06dZeaxeiPe8vyFm2WsoeVaNt7UWEjNsVkagDs8oX4XXh"
AWSSecretAccessKey := "09260955b4eb0279dc017ba20a1ddac909cbd226c86cbb2d868e55534c8e64b0"
reqBody := bytes.NewBufferString("0;chunk-signature=311a7142c8f3a07972c3aca65c36484b513a8fee48ab7178c7225388f2ae9894\r\n\r\n")
req, err := http.NewRequest("PUT", "http://localhost:8084/"+bktName+"/"+objName, reqBody)
require.NoError(t, err)
req.Header.Set("Amz-Sdk-Invocation-Id", "8a8cd4be-aef8-8034-f08d-a6144ade41f9")
req.Header.Set("Amz-Sdk-Request", "attempt=1; max=2")
req.Header.Set(api.Authorization, "AWS4-HMAC-SHA256 Credential=48c1K4PLVb7SvmV3PjDKEuXaMh8yZMXZ8Wx9msrkKcYw06dZeaxeiPe8vyFm2WsoeVaNt7UWEjNsVkagDs8oX4XXh/20241003/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=4b530ab4af2381f214941af591266b209968264a2c94337fa1efc048c7dff352")
req.Header.Set(api.ContentEncoding, "aws-chunked")
req.Header.Set(api.ContentLength, "86")
req.Header.Set(api.ContentType, "text/plain; charset=UTF-8")
req.Header.Set(api.AmzDate, "20241003T100055Z")
req.Header.Set(api.AmzContentSha256, "STREAMING-AWS4-HMAC-SHA256-PAYLOAD")
req.Header.Set(api.AmzDecodedContentLength, "0")
signTime, err := time.Parse("20060102T150405Z", "20241003T100055Z")
require.NoError(t, err)
w := httptest.NewRecorder()
reqInfo := middleware.NewReqInfo(w, req, middleware.ObjectRequest{Bucket: bktName, Object: objName}, "")
req = req.WithContext(middleware.SetReqInfo(ctx, reqInfo))
req = req.WithContext(middleware.SetBox(req.Context(), &middleware.Box{
ClientTime: signTime,
AuthHeaders: &middleware.AuthHeader{
AccessKeyID: AWSAccessKeyID,
SignatureV4: "4b530ab4af2381f214941af591266b209968264a2c94337fa1efc048c7dff352",
Region: "us-east-1",
},
AccessBox: &accessbox.Box{
Gate: &accessbox.GateData{
SecretKey: AWSSecretAccessKey,
},
},
}))
return w, req
}
func TestCreateBucket(t *testing.T) { func TestCreateBucket(t *testing.T) {
hc := prepareHandlerContext(t) hc := prepareHandlerContext(t)
bktName := "bkt-name" bktName := "bkt-name"
info := createBucket(hc, bktName) info := createBucket(hc, bktName)
createBucketAssertS3Error(hc, bktName, info.Box, s3errors.ErrBucketAlreadyOwnedByYou) createBucketAssertS3Error(hc, bktName, info.Box, apierr.ErrBucketAlreadyOwnedByYou)
box2, _ := createAccessBox(t) box2, _ := createAccessBox(t)
createBucketAssertS3Error(hc, bktName, box2, s3errors.ErrBucketAlreadyExists) createBucketAssertS3Error(hc, bktName, box2, apierr.ErrBucketAlreadyExists)
}
func TestCreateBucketWithoutPermissions(t *testing.T) {
hc := prepareHandlerContext(t)
bktName := "bkt-name"
hc.h.ape.(*apeMock).err = errors.New("no permissions")
box, _ := createAccessBox(t)
createBucketAssertS3Error(hc, bktName, box, apierr.ErrInternalError)
_, err := hc.tp.ContainerID(bktName)
require.Errorf(t, err, "container exists after failed creation, but shouldn't")
} }
func TestCreateNamespacedBucket(t *testing.T) { func TestCreateNamespacedBucket(t *testing.T) {

View file

@ -26,13 +26,13 @@ func (h *handler) PutObjectTaggingHandler(w http.ResponseWriter, r *http.Request
tagSet, err := h.readTagSet(reqInfo.Tagging) tagSet, err := h.readTagSet(reqInfo.Tagging)
if err != nil { if err != nil {
h.logAndSendError(w, "could not read tag set", reqInfo, err) h.logAndSendError(ctx, w, "could not read tag set", reqInfo, err)
return return
} }
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
@ -46,7 +46,7 @@ func (h *handler) PutObjectTaggingHandler(w http.ResponseWriter, r *http.Request
} }
if err = h.obj.PutObjectTagging(ctx, tagPrm); err != nil { if err = h.obj.PutObjectTagging(ctx, tagPrm); err != nil {
h.logAndSendError(w, "could not put object tagging", reqInfo, err) h.logAndSendError(ctx, w, "could not put object tagging", reqInfo, err)
return return
} }
@ -54,17 +54,18 @@ func (h *handler) PutObjectTaggingHandler(w http.ResponseWriter, r *http.Request
} }
func (h *handler) GetObjectTaggingHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetObjectTaggingHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
settings, err := h.obj.GetBucketSettings(r.Context(), bktInfo) settings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket settings", reqInfo, err)
return return
} }
@ -76,9 +77,9 @@ func (h *handler) GetObjectTaggingHandler(w http.ResponseWriter, r *http.Request
}, },
} }
versionID, tagSet, err := h.obj.GetObjectTagging(r.Context(), tagPrm) versionID, tagSet, err := h.obj.GetObjectTagging(ctx, tagPrm)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get object tagging", reqInfo, err) h.logAndSendError(ctx, w, "could not get object tagging", reqInfo, err)
return return
} }
@ -86,7 +87,7 @@ func (h *handler) GetObjectTaggingHandler(w http.ResponseWriter, r *http.Request
w.Header().Set(api.AmzVersionID, versionID) w.Header().Set(api.AmzVersionID, versionID)
} }
if err = middleware.EncodeToResponse(w, encodeTagging(tagSet)); err != nil { if err = middleware.EncodeToResponse(w, encodeTagging(tagSet)); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
} }
} }
@ -96,7 +97,7 @@ func (h *handler) DeleteObjectTaggingHandler(w http.ResponseWriter, r *http.Requ
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
@ -107,7 +108,7 @@ func (h *handler) DeleteObjectTaggingHandler(w http.ResponseWriter, r *http.Requ
} }
if err = h.obj.DeleteObjectTagging(ctx, p); err != nil { if err = h.obj.DeleteObjectTagging(ctx, p); err != nil {
h.logAndSendError(w, "could not delete object tagging", reqInfo, err) h.logAndSendError(ctx, w, "could not delete object tagging", reqInfo, err)
return return
} }
@ -115,58 +116,61 @@ func (h *handler) DeleteObjectTaggingHandler(w http.ResponseWriter, r *http.Requ
} }
func (h *handler) PutBucketTaggingHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) PutBucketTaggingHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
tagSet, err := h.readTagSet(reqInfo.Tagging) tagSet, err := h.readTagSet(reqInfo.Tagging)
if err != nil { if err != nil {
h.logAndSendError(w, "could not read tag set", reqInfo, err) h.logAndSendError(ctx, w, "could not read tag set", reqInfo, err)
return return
} }
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
if err = h.obj.PutBucketTagging(r.Context(), bktInfo, tagSet); err != nil { if err = h.obj.PutBucketTagging(ctx, bktInfo, tagSet); err != nil {
h.logAndSendError(w, "could not put object tagging", reqInfo, err) h.logAndSendError(ctx, w, "could not put object tagging", reqInfo, err)
return return
} }
} }
func (h *handler) GetBucketTaggingHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketTaggingHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
tagSet, err := h.obj.GetBucketTagging(r.Context(), bktInfo) tagSet, err := h.obj.GetBucketTagging(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get object tagging", reqInfo, err) h.logAndSendError(ctx, w, "could not get object tagging", reqInfo, err)
return return
} }
if err = middleware.EncodeToResponse(w, encodeTagging(tagSet)); err != nil { if err = middleware.EncodeToResponse(w, encodeTagging(tagSet)); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
return return
} }
} }
func (h *handler) DeleteBucketTaggingHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) DeleteBucketTaggingHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
if err = h.obj.DeleteBucketTagging(r.Context(), bktInfo); err != nil { if err = h.obj.DeleteBucketTagging(ctx, bktInfo); err != nil {
h.logAndSendError(w, "could not delete bucket tagging", reqInfo, err) h.logAndSendError(ctx, w, "could not delete bucket tagging", reqInfo, err)
return return
} }
w.WriteHeader(http.StatusNoContent) w.WriteHeader(http.StatusNoContent)

View file

@ -6,7 +6,7 @@ import (
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@ -98,7 +98,7 @@ func TestPutObjectTaggingCheckUniqueness(t *testing.T) {
middleware.GetReqInfo(r.Context()).Tagging = tc.body middleware.GetReqInfo(r.Context()).Tagging = tc.body
hc.Handler().PutObjectTaggingHandler(w, r) hc.Handler().PutObjectTaggingHandler(w, r)
if tc.error { if tc.error {
assertS3Error(t, w, apiErrors.GetAPIError(apiErrors.ErrInvalidTagKeyUniqueness)) assertS3Error(t, w, apierr.GetAPIError(apierr.ErrInvalidTagKeyUniqueness))
return return
} }
assertStatus(t, w, http.StatusOK) assertStatus(t, w, http.StatusOK)

View file

@ -8,53 +8,53 @@ import (
) )
func (h *handler) SelectObjectContentHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) SelectObjectContentHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented)) h.logAndSendError(r.Context(), w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented))
} }
func (h *handler) GetBucketEncryptionHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketEncryptionHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented)) h.logAndSendError(r.Context(), w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented))
} }
func (h *handler) GetBucketWebsiteHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketWebsiteHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented)) h.logAndSendError(r.Context(), w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented))
} }
func (h *handler) GetBucketAccelerateHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketAccelerateHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented)) h.logAndSendError(r.Context(), w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented))
} }
func (h *handler) GetBucketRequestPaymentHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketRequestPaymentHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented)) h.logAndSendError(r.Context(), w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented))
} }
func (h *handler) GetBucketLoggingHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketLoggingHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented)) h.logAndSendError(r.Context(), w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented))
} }
func (h *handler) GetBucketReplicationHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketReplicationHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented)) h.logAndSendError(r.Context(), w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented))
} }
func (h *handler) DeleteBucketWebsiteHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) DeleteBucketWebsiteHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented)) h.logAndSendError(r.Context(), w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented))
} }
func (h *handler) ListenBucketNotificationHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) ListenBucketNotificationHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented)) h.logAndSendError(r.Context(), w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented))
} }
func (h *handler) ListObjectsV2MHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) ListObjectsV2MHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented)) h.logAndSendError(r.Context(), w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented))
} }
func (h *handler) PutBucketEncryptionHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) PutBucketEncryptionHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented)) h.logAndSendError(r.Context(), w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented))
} }
func (h *handler) PutBucketNotificationHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) PutBucketNotificationHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented)) h.logAndSendError(r.Context(), w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented))
} }
func (h *handler) GetBucketNotificationHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketNotificationHandler(w http.ResponseWriter, r *http.Request) {
h.logAndSendError(w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented)) h.logAndSendError(r.Context(), w, "not implemented", middleware.GetReqInfo(r.Context()), errors.GetAPIError(errors.ErrNotImplemented))
} }

View file

@ -10,13 +10,11 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
frosterrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"go.opentelemetry.io/otel/trace"
"go.uber.org/zap" "go.uber.org/zap"
) )
@ -28,15 +26,14 @@ func (h *handler) reqLogger(ctx context.Context) *zap.Logger {
return h.log return h.log
} }
func (h *handler) logAndSendError(w http.ResponseWriter, logText string, reqInfo *middleware.ReqInfo, err error, additional ...zap.Field) { func (h *handler) logAndSendError(ctx context.Context, w http.ResponseWriter, logText string, reqInfo *middleware.ReqInfo, err error, additional ...zap.Field) {
err = handleDeleteMarker(w, err) err = handleDeleteMarker(w, err)
if code, wrErr := middleware.WriteErrorResponse(w, reqInfo, transformToS3Error(err)); wrErr != nil { if code, wrErr := middleware.WriteErrorResponse(w, reqInfo, apierr.TransformToS3Error(err)); wrErr != nil {
additional = append(additional, zap.NamedError("write_response_error", wrErr)) additional = append(additional, zap.NamedError("write_response_error", wrErr))
} else { } else {
additional = append(additional, zap.Int("status", code)) additional = append(additional, zap.Int("status", code))
} }
fields := []zap.Field{ fields := []zap.Field{
zap.String("request_id", reqInfo.RequestID),
zap.String("method", reqInfo.API), zap.String("method", reqInfo.API),
zap.String("bucket", reqInfo.BucketName), zap.String("bucket", reqInfo.BucketName),
zap.String("object", reqInfo.ObjectName), zap.String("object", reqInfo.ObjectName),
@ -44,10 +41,7 @@ func (h *handler) logAndSendError(w http.ResponseWriter, logText string, reqInfo
zap.String("user", reqInfo.User), zap.String("user", reqInfo.User),
zap.Error(err)} zap.Error(err)}
fields = append(fields, additional...) fields = append(fields, additional...)
if traceID, err := trace.TraceIDFromHex(reqInfo.TraceID); err == nil && traceID.IsValid() { h.reqLogger(ctx).Error(logs.RequestFailed, fields...)
fields = append(fields, zap.String("trace_id", reqInfo.TraceID))
}
h.log.Error(logs.RequestFailed, fields...) // consider using h.reqLogger (it requires accept context.Context or http.Request)
} }
func handleDeleteMarker(w http.ResponseWriter, err error) error { func handleDeleteMarker(w http.ResponseWriter, err error) error {
@ -57,37 +51,26 @@ func handleDeleteMarker(w http.ResponseWriter, err error) error {
} }
w.Header().Set(api.AmzDeleteMarker, "true") w.Header().Set(api.AmzDeleteMarker, "true")
return fmt.Errorf("%w: %s", s3errors.GetAPIError(target.ErrorCode), err) return fmt.Errorf("%w: %s", apierr.GetAPIError(target.ErrorCode), err)
}
func transformToS3Error(err error) error {
err = frosterrors.UnwrapErr(err) // this wouldn't work with errors.Join
if _, ok := err.(s3errors.Error); ok {
return err
}
if errors.Is(err, layer.ErrAccessDenied) ||
errors.Is(err, layer.ErrNodeAccessDenied) {
return s3errors.GetAPIError(s3errors.ErrAccessDenied)
}
if errors.Is(err, layer.ErrGatewayTimeout) {
return s3errors.GetAPIError(s3errors.ErrGatewayTimeout)
}
return s3errors.GetAPIError(s3errors.ErrInternalError)
} }
func (h *handler) ResolveBucket(ctx context.Context, bucket string) (*data.BucketInfo, error) { func (h *handler) ResolveBucket(ctx context.Context, bucket string) (*data.BucketInfo, error) {
return h.obj.GetBucketInfo(ctx, bucket) return h.getBucketInfo(ctx, bucket)
} }
func (h *handler) ResolveCID(ctx context.Context, bucket string) (cid.ID, error) { func (h *handler) ResolveCID(ctx context.Context, bucket string) (cid.ID, error) {
return h.obj.ResolveCID(ctx, bucket) return h.obj.ResolveCID(ctx, bucket)
} }
func (h *handler) getBucketInfo(ctx context.Context, bucket string) (*data.BucketInfo, error) {
if err := checkBucketName(bucket); err != nil {
return nil, err
}
return h.obj.GetBucketInfo(ctx, bucket)
}
func (h *handler) getBucketAndCheckOwner(r *http.Request, bucket string, header ...string) (*data.BucketInfo, error) { func (h *handler) getBucketAndCheckOwner(r *http.Request, bucket string, header ...string) (*data.BucketInfo, error) {
bktInfo, err := h.obj.GetBucketInfo(r.Context(), bucket) bktInfo, err := h.getBucketInfo(r.Context(), bucket)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@ -106,6 +89,19 @@ func (h *handler) getBucketAndCheckOwner(r *http.Request, bucket string, header
return bktInfo, checkOwner(bktInfo, expected) return bktInfo, checkOwner(bktInfo, expected)
} }
func (h *handler) getPutPayloadSize(r *http.Request) uint64 {
decodeContentSize := r.Header.Get(api.AmzDecodedContentLength)
decodedSize, err := strconv.Atoi(decodeContentSize)
if err != nil {
if r.ContentLength >= 0 {
return uint64(r.ContentLength)
}
return 0
}
return uint64(decodedSize)
}
func parseRange(s string) (*layer.RangeParams, error) { func parseRange(s string) (*layer.RangeParams, error) {
if s == "" { if s == "" {
return nil, nil return nil, nil
@ -114,26 +110,26 @@ func parseRange(s string) (*layer.RangeParams, error) {
prefix := "bytes=" prefix := "bytes="
if !strings.HasPrefix(s, prefix) { if !strings.HasPrefix(s, prefix) {
return nil, s3errors.GetAPIError(s3errors.ErrInvalidRange) return nil, apierr.GetAPIError(apierr.ErrInvalidRange)
} }
s = strings.TrimPrefix(s, prefix) s = strings.TrimPrefix(s, prefix)
valuesStr := strings.Split(s, "-") valuesStr := strings.Split(s, "-")
if len(valuesStr) != 2 { if len(valuesStr) != 2 {
return nil, s3errors.GetAPIError(s3errors.ErrInvalidRange) return nil, apierr.GetAPIError(apierr.ErrInvalidRange)
} }
values := make([]uint64, 0, len(valuesStr)) values := make([]uint64, 0, len(valuesStr))
for _, v := range valuesStr { for _, v := range valuesStr {
num, err := strconv.ParseUint(v, 10, 64) num, err := strconv.ParseUint(v, 10, 64)
if err != nil { if err != nil {
return nil, s3errors.GetAPIError(s3errors.ErrInvalidRange) return nil, apierr.GetAPIError(apierr.ErrInvalidRange)
} }
values = append(values, num) values = append(values, num)
} }
if values[0] > values[1] { if values[0] > values[1] {
return nil, s3errors.GetAPIError(s3errors.ErrInvalidRange) return nil, apierr.GetAPIError(apierr.ErrInvalidRange)
} }
return &layer.RangeParams{ return &layer.RangeParams{

View file

@ -1,64 +1 @@
package handler package handler
import (
"errors"
"fmt"
"testing"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer"
"github.com/stretchr/testify/require"
)
func TestTransformS3Errors(t *testing.T) {
for _, tc := range []struct {
name string
err error
expected s3errors.ErrorCode
}{
{
name: "simple std error to internal error",
err: errors.New("some error"),
expected: s3errors.ErrInternalError,
},
{
name: "layer access denied error to s3 access denied error",
err: layer.ErrAccessDenied,
expected: s3errors.ErrAccessDenied,
},
{
name: "wrapped layer access denied error to s3 access denied error",
err: fmt.Errorf("wrap: %w", layer.ErrAccessDenied),
expected: s3errors.ErrAccessDenied,
},
{
name: "layer node access denied error to s3 access denied error",
err: layer.ErrNodeAccessDenied,
expected: s3errors.ErrAccessDenied,
},
{
name: "layer gateway timeout error to s3 gateway timeout error",
err: layer.ErrGatewayTimeout,
expected: s3errors.ErrGatewayTimeout,
},
{
name: "s3 error to s3 error",
err: s3errors.GetAPIError(s3errors.ErrInvalidPart),
expected: s3errors.ErrInvalidPart,
},
{
name: "wrapped s3 error to s3 error",
err: fmt.Errorf("wrap: %w", s3errors.GetAPIError(s3errors.ErrInvalidPart)),
expected: s3errors.ErrInvalidPart,
},
} {
t.Run(tc.name, func(t *testing.T) {
err := transformToS3Error(tc.err)
s3err, ok := err.(s3errors.Error)
require.True(t, ok, "error must be s3 error")
require.Equalf(t, tc.expected, s3err.ErrCode,
"expected: '%s', got: '%s'",
s3errors.GetAPIError(tc.expected).Code, s3errors.GetAPIError(s3err.ErrCode).Code)
})
}
}

View file

@ -10,28 +10,29 @@ import (
) )
func (h *handler) PutBucketVersioningHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) PutBucketVersioningHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
configuration := new(VersioningConfiguration) configuration := new(VersioningConfiguration)
if err := h.cfg.NewXMLDecoder(r.Body).Decode(configuration); err != nil { if err := h.cfg.NewXMLDecoder(r.Body).Decode(configuration); err != nil {
h.logAndSendError(w, "couldn't decode versioning configuration", reqInfo, errors.GetAPIError(errors.ErrIllegalVersioningConfigurationException)) h.logAndSendError(ctx, w, "couldn't decode versioning configuration", reqInfo, errors.GetAPIError(errors.ErrIllegalVersioningConfigurationException))
return return
} }
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
settings, err := h.obj.GetBucketSettings(r.Context(), bktInfo) settings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't get bucket settings", reqInfo, err) h.logAndSendError(ctx, w, "couldn't get bucket settings", reqInfo, err)
return return
} }
if configuration.Status != data.VersioningEnabled && configuration.Status != data.VersioningSuspended { if configuration.Status != data.VersioningEnabled && configuration.Status != data.VersioningSuspended {
h.logAndSendError(w, "invalid versioning configuration", reqInfo, errors.GetAPIError(errors.ErrMalformedXML)) h.logAndSendError(ctx, w, "invalid versioning configuration", reqInfo, errors.GetAPIError(errors.ErrMalformedXML))
return return
} }
@ -45,33 +46,34 @@ func (h *handler) PutBucketVersioningHandler(w http.ResponseWriter, r *http.Requ
} }
if p.Settings.VersioningSuspended() && bktInfo.ObjectLockEnabled { if p.Settings.VersioningSuspended() && bktInfo.ObjectLockEnabled {
h.logAndSendError(w, "couldn't suspend bucket versioning", reqInfo, errors.GetAPIError(errors.ErrObjectLockConfigurationVersioningCannotBeChanged)) h.logAndSendError(ctx, w, "couldn't suspend bucket versioning", reqInfo, errors.GetAPIError(errors.ErrObjectLockConfigurationVersioningCannotBeChanged))
return return
} }
if err = h.obj.PutBucketSettings(r.Context(), p); err != nil { if err = h.obj.PutBucketSettings(ctx, p); err != nil {
h.logAndSendError(w, "couldn't put update versioning settings", reqInfo, err) h.logAndSendError(ctx, w, "couldn't put update versioning settings", reqInfo, err)
} }
} }
// GetBucketVersioningHandler implements bucket versioning getter handler. // GetBucketVersioningHandler implements bucket versioning getter handler.
func (h *handler) GetBucketVersioningHandler(w http.ResponseWriter, r *http.Request) { func (h *handler) GetBucketVersioningHandler(w http.ResponseWriter, r *http.Request) {
reqInfo := middleware.GetReqInfo(r.Context()) ctx := r.Context()
reqInfo := middleware.GetReqInfo(ctx)
bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName) bktInfo, err := h.getBucketAndCheckOwner(r, reqInfo.BucketName)
if err != nil { if err != nil {
h.logAndSendError(w, "could not get bucket info", reqInfo, err) h.logAndSendError(ctx, w, "could not get bucket info", reqInfo, err)
return return
} }
settings, err := h.obj.GetBucketSettings(r.Context(), bktInfo) settings, err := h.obj.GetBucketSettings(ctx, bktInfo)
if err != nil { if err != nil {
h.logAndSendError(w, "couldn't get version settings", reqInfo, err) h.logAndSendError(ctx, w, "couldn't get version settings", reqInfo, err)
return return
} }
if err = middleware.EncodeToResponse(w, formVersioningConfiguration(settings)); err != nil { if err = middleware.EncodeToResponse(w, formVersioningConfiguration(settings)); err != nil {
h.logAndSendError(w, "something went wrong", reqInfo, err) h.logAndSendError(ctx, w, "something went wrong", reqInfo, err)
} }
} }

View file

@ -5,6 +5,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"go.uber.org/zap" "go.uber.org/zap"
@ -19,6 +20,7 @@ type Cache struct {
bucketCache *cache.BucketCache bucketCache *cache.BucketCache
systemCache *cache.SystemCache systemCache *cache.SystemCache
accessCache *cache.AccessControlCache accessCache *cache.AccessControlCache
networkInfoCache *cache.NetworkInfoCache
} }
// CachesConfig contains params for caches. // CachesConfig contains params for caches.
@ -31,6 +33,7 @@ type CachesConfig struct {
Buckets *cache.Config Buckets *cache.Config
System *cache.Config System *cache.Config
AccessControl *cache.Config AccessControl *cache.Config
NetworkInfo *cache.NetworkInfoCacheConfig
} }
// DefaultCachesConfigs returns filled configs. // DefaultCachesConfigs returns filled configs.
@ -44,6 +47,7 @@ func DefaultCachesConfigs(logger *zap.Logger) *CachesConfig {
Buckets: cache.DefaultBucketConfig(logger), Buckets: cache.DefaultBucketConfig(logger),
System: cache.DefaultSystemConfig(logger), System: cache.DefaultSystemConfig(logger),
AccessControl: cache.DefaultAccessControlConfig(logger), AccessControl: cache.DefaultAccessControlConfig(logger),
NetworkInfo: cache.DefaultNetworkInfoConfig(logger),
} }
} }
@ -57,6 +61,7 @@ func NewCache(cfg *CachesConfig) *Cache {
bucketCache: cache.NewBucketCache(cfg.Buckets), bucketCache: cache.NewBucketCache(cfg.Buckets),
systemCache: cache.NewSystemCache(cfg.System), systemCache: cache.NewSystemCache(cfg.System),
accessCache: cache.NewAccessControlCache(cfg.AccessControl), accessCache: cache.NewAccessControlCache(cfg.AccessControl),
networkInfoCache: cache.NewNetworkInfoCache(cfg.NetworkInfo),
} }
} }
@ -283,3 +288,13 @@ func (c *Cache) PutLifecycleConfiguration(owner user.ID, bkt *data.BucketInfo, c
func (c *Cache) DeleteLifecycleConfiguration(bktInfo *data.BucketInfo) { func (c *Cache) DeleteLifecycleConfiguration(bktInfo *data.BucketInfo) {
c.systemCache.Delete(bktInfo.LifecycleConfigurationObjectName()) c.systemCache.Delete(bktInfo.LifecycleConfigurationObjectName())
} }
func (c *Cache) GetNetworkInfo() *netmap.NetworkInfo {
return c.networkInfoCache.Get()
}
func (c *Cache) PutNetworkInfo(info netmap.NetworkInfo) {
if err := c.networkInfoCache.Put(info); err != nil {
c.logger.Warn(logs.CouldntCacheNetworkInfo, zap.Error(err))
}
}

View file

@ -6,7 +6,8 @@ import (
"fmt" "fmt"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
) )
func (n *Layer) GetObjectTaggingAndLock(ctx context.Context, objVersion *data.ObjectVersion, nodeVersion *data.NodeVersion) (map[string]string, data.LockInfo, error) { func (n *Layer) GetObjectTaggingAndLock(ctx context.Context, objVersion *data.ObjectVersion, nodeVersion *data.NodeVersion) (map[string]string, data.LockInfo, error) {
@ -29,8 +30,8 @@ func (n *Layer) GetObjectTaggingAndLock(ctx context.Context, objVersion *data.Ob
tags, lockInfo, err = n.treeService.GetObjectTaggingAndLock(ctx, objVersion.BktInfo, nodeVersion) tags, lockInfo, err = n.treeService.GetObjectTaggingAndLock(ctx, objVersion.BktInfo, nodeVersion)
if err != nil { if err != nil {
if errors.Is(err, ErrNodeNotFound) { if errors.Is(err, tree.ErrNodeNotFound) {
return nil, data.LockInfo{}, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchKey), err.Error()) return nil, data.LockInfo{}, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
} }
return nil, data.LockInfo{}, err return nil, data.LockInfo{}, err
} }

View file

@ -7,7 +7,8 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
@ -20,7 +21,7 @@ const (
AttributeLockEnabled = "LockEnabled" AttributeLockEnabled = "LockEnabled"
) )
func (n *Layer) containerInfo(ctx context.Context, prm PrmContainer) (*data.BucketInfo, error) { func (n *Layer) containerInfo(ctx context.Context, prm frostfs.PrmContainer) (*data.BucketInfo, error) {
var ( var (
err error err error
res *container.Container res *container.Container
@ -37,7 +38,7 @@ func (n *Layer) containerInfo(ctx context.Context, prm PrmContainer) (*data.Buck
res, err = n.frostFS.Container(ctx, prm) res, err = n.frostFS.Container(ctx, prm)
if err != nil { if err != nil {
if client.IsErrContainerNotFound(err) { if client.IsErrContainerNotFound(err) {
return nil, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchBucket), err.Error()) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchBucket), err.Error())
} }
return nil, fmt.Errorf("get frostfs container: %w", err) return nil, fmt.Errorf("get frostfs container: %w", err)
} }
@ -77,7 +78,7 @@ func (n *Layer) containerInfo(ctx context.Context, prm PrmContainer) (*data.Buck
func (n *Layer) containerList(ctx context.Context) ([]*data.BucketInfo, error) { func (n *Layer) containerList(ctx context.Context) ([]*data.BucketInfo, error) {
stoken := n.SessionTokenForRead(ctx) stoken := n.SessionTokenForRead(ctx)
prm := PrmUserContainers{ prm := frostfs.PrmUserContainers{
UserID: n.BearerOwner(ctx), UserID: n.BearerOwner(ctx),
SessionToken: stoken, SessionToken: stoken,
} }
@ -90,7 +91,7 @@ func (n *Layer) containerList(ctx context.Context) ([]*data.BucketInfo, error) {
list := make([]*data.BucketInfo, 0, len(res)) list := make([]*data.BucketInfo, 0, len(res))
for i := range res { for i := range res {
getPrm := PrmContainer{ getPrm := frostfs.PrmContainer{
ContainerID: res[i], ContainerID: res[i],
SessionToken: stoken, SessionToken: stoken,
} }
@ -132,7 +133,7 @@ func (n *Layer) createContainer(ctx context.Context, p *CreateBucketParams) (*da
}) })
} }
res, err := n.frostFS.CreateContainer(ctx, PrmContainerCreate{ res, err := n.frostFS.CreateContainer(ctx, frostfs.PrmContainerCreate{
Creator: bktInfo.Owner, Creator: bktInfo.Owner,
Policy: p.Policy, Policy: p.Policy,
Name: p.Name, Name: p.Name,
@ -140,7 +141,6 @@ func (n *Layer) createContainer(ctx context.Context, p *CreateBucketParams) (*da
SessionToken: p.SessionContainerCreation, SessionToken: p.SessionContainerCreation,
CreationTime: bktInfo.Created, CreationTime: bktInfo.Created,
AdditionalAttributes: attributes, AdditionalAttributes: attributes,
BasicACL: 0, // means APE
}) })
if err != nil { if err != nil {
return nil, fmt.Errorf("create container: %w", err) return nil, fmt.Errorf("create container: %w", err)

View file

@ -3,12 +3,14 @@ package layer
import ( import (
"bytes" "bytes"
"context" "context"
errorsStd "errors" "errors"
"fmt" "fmt"
"io" "io"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
@ -31,14 +33,14 @@ func (n *Layer) PutBucketCORS(ctx context.Context, p *PutCORSParams) error {
} }
if cors.CORSRules == nil { if cors.CORSRules == nil {
return errors.GetAPIError(errors.ErrMalformedXML) return apierr.GetAPIError(apierr.ErrMalformedXML)
} }
if err := checkCORS(cors); err != nil { if err := checkCORS(cors); err != nil {
return err return err
} }
prm := PrmObjectCreate{ prm := frostfs.PrmObjectCreate{
Payload: &buf, Payload: &buf,
Filepath: p.BktInfo.CORSObjectName(), Filepath: p.BktInfo.CORSObjectName(),
CreationTime: TimeNow(ctx), CreationTime: TimeNow(ctx),
@ -61,7 +63,7 @@ func (n *Layer) PutBucketCORS(ctx context.Context, p *PutCORSParams) error {
} }
objsToDelete, err := n.treeService.PutBucketCORS(ctx, p.BktInfo, newAddress(corsBkt.CID, createdObj.ID)) objsToDelete, err := n.treeService.PutBucketCORS(ctx, p.BktInfo, newAddress(corsBkt.CID, createdObj.ID))
objToDeleteNotFound := errorsStd.Is(err, ErrNoNodeToRemove) objToDeleteNotFound := errors.Is(err, tree.ErrNoNodeToRemove)
if err != nil && !objToDeleteNotFound { if err != nil && !objToDeleteNotFound {
return err return err
} }
@ -79,7 +81,7 @@ func (n *Layer) PutBucketCORS(ctx context.Context, p *PutCORSParams) error {
// deleteCORSObject removes object and logs in case of error. // deleteCORSObject removes object and logs in case of error.
func (n *Layer) deleteCORSObject(ctx context.Context, bktInfo *data.BucketInfo, addr oid.Address) { func (n *Layer) deleteCORSObject(ctx context.Context, bktInfo *data.BucketInfo, addr oid.Address) {
var prmAuth PrmAuth var prmAuth frostfs.PrmAuth
corsBkt := bktInfo corsBkt := bktInfo
if !addr.Container().Equals(bktInfo.CID) && !addr.Container().Equals(cid.ID{}) { if !addr.Container().Equals(bktInfo.CID) && !addr.Container().Equals(cid.ID{}) {
corsBkt = &data.BucketInfo{CID: addr.Container()} corsBkt = &data.BucketInfo{CID: addr.Container()}
@ -104,7 +106,7 @@ func (n *Layer) GetBucketCORS(ctx context.Context, bktInfo *data.BucketInfo) (*d
func (n *Layer) DeleteBucketCORS(ctx context.Context, bktInfo *data.BucketInfo) error { func (n *Layer) DeleteBucketCORS(ctx context.Context, bktInfo *data.BucketInfo) error {
objs, err := n.treeService.DeleteBucketCORS(ctx, bktInfo) objs, err := n.treeService.DeleteBucketCORS(ctx, bktInfo)
objNotFound := errorsStd.Is(err, ErrNoNodeToRemove) objNotFound := errors.Is(err, tree.ErrNoNodeToRemove)
if err != nil && !objNotFound { if err != nil && !objNotFound {
return err return err
} }
@ -124,12 +126,12 @@ func checkCORS(cors *data.CORSConfiguration) error {
for _, r := range cors.CORSRules { for _, r := range cors.CORSRules {
for _, m := range r.AllowedMethods { for _, m := range r.AllowedMethods {
if _, ok := supportedMethods[m]; !ok { if _, ok := supportedMethods[m]; !ok {
return errors.GetAPIErrorWithError(errors.ErrCORSUnsupportedMethod, fmt.Errorf("unsupported method is %s", m)) return apierr.GetAPIErrorWithError(apierr.ErrCORSUnsupportedMethod, fmt.Errorf("unsupported method is %s", m))
} }
} }
for _, h := range r.ExposeHeaders { for _, h := range r.ExposeHeaders {
if h == wildcard { if h == wildcard {
return errors.GetAPIError(errors.ErrCORSWildcardExposeHeaders) return apierr.GetAPIError(apierr.ErrCORSWildcardExposeHeaders)
} }
} }
} }

View file

@ -1,4 +1,4 @@
package layer package frostfs
import ( import (
"context" "context"
@ -9,13 +9,13 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/acl"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain"
) )
// PrmContainerCreate groups parameters of FrostFS.CreateContainer operation. // PrmContainerCreate groups parameters of FrostFS.CreateContainer operation.
@ -38,13 +38,19 @@ type PrmContainerCreate struct {
// Token of the container's creation session. Nil means session absence. // Token of the container's creation session. Nil means session absence.
SessionToken *session.Container SessionToken *session.Container
// Basic ACL of the container.
BasicACL acl.Basic
// Attributes for optional parameters. // Attributes for optional parameters.
AdditionalAttributes [][2]string AdditionalAttributes [][2]string
} }
// PrmAddContainerPolicyChain groups parameter of FrostFS.AddContainerPolicyChain operation.
type PrmAddContainerPolicyChain struct {
// ContainerID is a container identifier.
ContainerID cid.ID
// Chain is Access Policy Engine chain that contains rules which provide access to specific actions in container.
Chain chain.Chain
}
// PrmContainer groups parameters of FrostFS.Container operation. // PrmContainer groups parameters of FrostFS.Container operation.
type PrmContainer struct { type PrmContainer struct {
// Container identifier. // Container identifier.
@ -239,6 +245,10 @@ type FrostFS interface {
// prevented the container from being created. // prevented the container from being created.
CreateContainer(context.Context, PrmContainerCreate) (*ContainerCreateResult, error) CreateContainer(context.Context, PrmContainerCreate) (*ContainerCreateResult, error)
// AddContainerPolicyChain create new policy chain for container.
// Can be invoked only by container owner.
AddContainerPolicyChain(context.Context, PrmAddContainerPolicyChain) error
// Container reads a container from FrostFS by ID. // Container reads a container from FrostFS by ID.
// //
// It returns exactly one non-nil value. It returns any error encountered which // It returns exactly one non-nil value. It returns any error encountered which

View file

@ -5,6 +5,7 @@ import (
"context" "context"
"crypto/rand" "crypto/rand"
"crypto/sha256" "crypto/sha256"
"errors"
"fmt" "fmt"
"io" "io"
"strings" "strings"
@ -12,6 +13,7 @@ import (
v2container "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container" v2container "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
objectv2 "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object" objectv2 "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/checksum" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/checksum"
@ -24,6 +26,7 @@ import (
oidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id/test" oidtest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id/test"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
) )
@ -60,13 +63,14 @@ func (k *FeatureSettingsMock) FormContainerZone(ns string) string {
return ns + ".ns" return ns + ".ns"
} }
type TestFrostFS struct { var _ frostfs.FrostFS = (*TestFrostFS)(nil)
FrostFS
type TestFrostFS struct {
objects map[string]*object.Object objects map[string]*object.Object
objectErrors map[string]error objectErrors map[string]error
objectPutErrors map[string]error objectPutErrors map[string]error
containers map[string]*container.Container containers map[string]*container.Container
chains map[string][]chain.Chain
currentEpoch uint64 currentEpoch uint64
key *keys.PrivateKey key *keys.PrivateKey
} }
@ -77,6 +81,7 @@ func NewTestFrostFS(key *keys.PrivateKey) *TestFrostFS {
objectErrors: make(map[string]error), objectErrors: make(map[string]error),
objectPutErrors: make(map[string]error), objectPutErrors: make(map[string]error),
containers: make(map[string]*container.Container), containers: make(map[string]*container.Container),
chains: make(map[string][]chain.Chain),
key: key, key: key,
} }
} }
@ -139,12 +144,11 @@ func (t *TestFrostFS) SetContainer(cnrID cid.ID, cnr *container.Container) {
t.containers[cnrID.EncodeToString()] = cnr t.containers[cnrID.EncodeToString()] = cnr
} }
func (t *TestFrostFS) CreateContainer(_ context.Context, prm PrmContainerCreate) (*ContainerCreateResult, error) { func (t *TestFrostFS) CreateContainer(_ context.Context, prm frostfs.PrmContainerCreate) (*frostfs.ContainerCreateResult, error) {
var cnr container.Container var cnr container.Container
cnr.Init() cnr.Init()
cnr.SetOwner(prm.Creator) cnr.SetOwner(prm.Creator)
cnr.SetPlacementPolicy(prm.Policy) cnr.SetPlacementPolicy(prm.Policy)
cnr.SetBasicACL(prm.BasicACL)
creationTime := prm.CreationTime creationTime := prm.CreationTime
if creationTime.IsZero() { if creationTime.IsZero() {
@ -173,8 +177,9 @@ func (t *TestFrostFS) CreateContainer(_ context.Context, prm PrmContainerCreate)
var id cid.ID var id cid.ID
id.SetSHA256(sha256.Sum256(b)) id.SetSHA256(sha256.Sum256(b))
t.containers[id.EncodeToString()] = &cnr t.containers[id.EncodeToString()] = &cnr
t.chains[id.EncodeToString()] = []chain.Chain{}
return &ContainerCreateResult{ContainerID: id}, nil return &frostfs.ContainerCreateResult{ContainerID: id}, nil
} }
func (t *TestFrostFS) DeleteContainer(_ context.Context, cnrID cid.ID, _ *session.Container) error { func (t *TestFrostFS) DeleteContainer(_ context.Context, cnrID cid.ID, _ *session.Container) error {
@ -183,7 +188,7 @@ func (t *TestFrostFS) DeleteContainer(_ context.Context, cnrID cid.ID, _ *sessio
return nil return nil
} }
func (t *TestFrostFS) Container(_ context.Context, prm PrmContainer) (*container.Container, error) { func (t *TestFrostFS) Container(_ context.Context, prm frostfs.PrmContainer) (*container.Container, error) {
for k, v := range t.containers { for k, v := range t.containers {
if k == prm.ContainerID.EncodeToString() { if k == prm.ContainerID.EncodeToString() {
return v, nil return v, nil
@ -193,7 +198,7 @@ func (t *TestFrostFS) Container(_ context.Context, prm PrmContainer) (*container
return nil, fmt.Errorf("container not found %s", prm.ContainerID) return nil, fmt.Errorf("container not found %s", prm.ContainerID)
} }
func (t *TestFrostFS) UserContainers(context.Context, PrmUserContainers) ([]cid.ID, error) { func (t *TestFrostFS) UserContainers(context.Context, frostfs.PrmUserContainers) ([]cid.ID, error) {
var res []cid.ID var res []cid.ID
for k := range t.containers { for k := range t.containers {
var idCnr cid.ID var idCnr cid.ID
@ -220,7 +225,7 @@ func (t *TestFrostFS) retrieveObject(ctx context.Context, cnrID cid.ID, objID oi
if obj, ok := t.objects[sAddr]; ok { if obj, ok := t.objects[sAddr]; ok {
owner := getBearerOwner(ctx) owner := getBearerOwner(ctx)
if !t.checkAccess(cnrID, owner) { if !t.checkAccess(cnrID, owner) {
return nil, ErrAccessDenied return nil, frostfs.ErrAccessDenied
} }
return obj, nil return obj, nil
@ -229,23 +234,23 @@ func (t *TestFrostFS) retrieveObject(ctx context.Context, cnrID cid.ID, objID oi
return nil, fmt.Errorf("%w: %s", &apistatus.ObjectNotFound{}, addr) return nil, fmt.Errorf("%w: %s", &apistatus.ObjectNotFound{}, addr)
} }
func (t *TestFrostFS) HeadObject(ctx context.Context, prm PrmObjectHead) (*object.Object, error) { func (t *TestFrostFS) HeadObject(ctx context.Context, prm frostfs.PrmObjectHead) (*object.Object, error) {
return t.retrieveObject(ctx, prm.Container, prm.Object) return t.retrieveObject(ctx, prm.Container, prm.Object)
} }
func (t *TestFrostFS) GetObject(ctx context.Context, prm PrmObjectGet) (*Object, error) { func (t *TestFrostFS) GetObject(ctx context.Context, prm frostfs.PrmObjectGet) (*frostfs.Object, error) {
obj, err := t.retrieveObject(ctx, prm.Container, prm.Object) obj, err := t.retrieveObject(ctx, prm.Container, prm.Object)
if err != nil { if err != nil {
return nil, err return nil, err
} }
return &Object{ return &frostfs.Object{
Header: *obj, Header: *obj,
Payload: io.NopCloser(bytes.NewReader(obj.Payload())), Payload: io.NopCloser(bytes.NewReader(obj.Payload())),
}, nil }, nil
} }
func (t *TestFrostFS) RangeObject(ctx context.Context, prm PrmObjectRange) (io.ReadCloser, error) { func (t *TestFrostFS) RangeObject(ctx context.Context, prm frostfs.PrmObjectRange) (io.ReadCloser, error) {
obj, err := t.retrieveObject(ctx, prm.Container, prm.Object) obj, err := t.retrieveObject(ctx, prm.Container, prm.Object)
if err != nil { if err != nil {
return nil, err return nil, err
@ -257,7 +262,7 @@ func (t *TestFrostFS) RangeObject(ctx context.Context, prm PrmObjectRange) (io.R
return io.NopCloser(bytes.NewReader(payload)), nil return io.NopCloser(bytes.NewReader(payload)), nil
} }
func (t *TestFrostFS) CreateObject(_ context.Context, prm PrmObjectCreate) (*CreateObjectResult, error) { func (t *TestFrostFS) CreateObject(_ context.Context, prm frostfs.PrmObjectCreate) (*frostfs.CreateObjectResult, error) {
b := make([]byte, 32) b := make([]byte, 32)
if _, err := io.ReadFull(rand.Reader, b); err != nil { if _, err := io.ReadFull(rand.Reader, b); err != nil {
return nil, err return nil, err
@ -327,13 +332,13 @@ func (t *TestFrostFS) CreateObject(_ context.Context, prm PrmObjectCreate) (*Cre
addr := newAddress(cnrID, objID) addr := newAddress(cnrID, objID)
t.objects[addr.EncodeToString()] = obj t.objects[addr.EncodeToString()] = obj
return &CreateObjectResult{ return &frostfs.CreateObjectResult{
ObjectID: objID, ObjectID: objID,
CreationEpoch: t.currentEpoch - 1, CreationEpoch: t.currentEpoch - 1,
}, nil }, nil
} }
func (t *TestFrostFS) DeleteObject(ctx context.Context, prm PrmObjectDelete) error { func (t *TestFrostFS) DeleteObject(ctx context.Context, prm frostfs.PrmObjectDelete) error {
var addr oid.Address var addr oid.Address
addr.SetContainer(prm.Container) addr.SetContainer(prm.Container)
addr.SetObject(prm.Object) addr.SetObject(prm.Object)
@ -345,7 +350,7 @@ func (t *TestFrostFS) DeleteObject(ctx context.Context, prm PrmObjectDelete) err
if _, ok := t.objects[addr.EncodeToString()]; ok { if _, ok := t.objects[addr.EncodeToString()]; ok {
owner := getBearerOwner(ctx) owner := getBearerOwner(ctx)
if !t.checkAccess(prm.Container, owner) { if !t.checkAccess(prm.Container, owner) {
return ErrAccessDenied return frostfs.ErrAccessDenied
} }
delete(t.objects, addr.EncodeToString()) delete(t.objects, addr.EncodeToString())
@ -372,7 +377,7 @@ func (t *TestFrostFS) AllObjects(cnrID cid.ID) []oid.ID {
return result return result
} }
func (t *TestFrostFS) SearchObjects(_ context.Context, prm PrmObjectSearch) ([]oid.ID, error) { func (t *TestFrostFS) SearchObjects(_ context.Context, prm frostfs.PrmObjectSearch) ([]oid.ID, error) {
filters := object.NewSearchFilters() filters := object.NewSearchFilters()
filters.AddRootFilter() filters.AddRootFilter()
@ -412,11 +417,13 @@ func (t *TestFrostFS) SearchObjects(_ context.Context, prm PrmObjectSearch) ([]o
func (t *TestFrostFS) NetworkInfo(context.Context) (netmap.NetworkInfo, error) { func (t *TestFrostFS) NetworkInfo(context.Context) (netmap.NetworkInfo, error) {
ni := netmap.NetworkInfo{} ni := netmap.NetworkInfo{}
ni.SetCurrentEpoch(t.currentEpoch) ni.SetCurrentEpoch(t.currentEpoch)
ni.SetEpochDuration(60)
ni.SetMsPerBlock(1000)
return ni, nil return ni, nil
} }
func (t *TestFrostFS) PatchObject(ctx context.Context, prm PrmObjectPatch) (oid.ID, error) { func (t *TestFrostFS) PatchObject(ctx context.Context, prm frostfs.PrmObjectPatch) (oid.ID, error) {
obj, err := t.retrieveObject(ctx, prm.Container, prm.Object) obj, err := t.retrieveObject(ctx, prm.Container, prm.Object)
if err != nil { if err != nil {
return oid.ID{}, err return oid.ID{}, err
@ -452,6 +459,17 @@ func (t *TestFrostFS) PatchObject(ctx context.Context, prm PrmObjectPatch) (oid.
return newID, nil return newID, nil
} }
func (t *TestFrostFS) AddContainerPolicyChain(_ context.Context, prm frostfs.PrmAddContainerPolicyChain) error {
list, ok := t.chains[prm.ContainerID.EncodeToString()]
if !ok {
return errors.New("container not found")
}
t.chains[prm.ContainerID.EncodeToString()] = append(list, prm.Chain)
return nil
}
func (t *TestFrostFS) checkAccess(cnrID cid.ID, owner user.ID) bool { func (t *TestFrostFS) checkAccess(cnrID cid.ID, owner user.ID) bool {
cnr, ok := t.containers[cnrID.EncodeToString()] cnr, ok := t.containers[cnrID.EncodeToString()]
if !ok { if !ok {

View file

@ -6,7 +6,7 @@ import (
"crypto/rand" "crypto/rand"
"encoding/json" "encoding/json"
"encoding/xml" "encoding/xml"
stderrors "errors" "errors"
"fmt" "fmt"
"io" "io"
"net/url" "net/url"
@ -17,9 +17,10 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
@ -46,13 +47,13 @@ type (
} }
Layer struct { Layer struct {
frostFS FrostFS frostFS frostfs.FrostFS
gateOwner user.ID gateOwner user.ID
log *zap.Logger log *zap.Logger
anonKey AnonymousKey anonKey AnonymousKey
resolver BucketResolver resolver BucketResolver
cache *Cache cache *Cache
treeService TreeService treeService tree.Service
features FeatureSettings features FeatureSettings
gateKey *keys.PrivateKey gateKey *keys.PrivateKey
corsCnrInfo *data.BucketInfo corsCnrInfo *data.BucketInfo
@ -65,7 +66,7 @@ type (
Cache *Cache Cache *Cache
AnonKey AnonymousKey AnonKey AnonymousKey
Resolver BucketResolver Resolver BucketResolver
TreeService TreeService TreeService tree.Service
Features FeatureSettings Features FeatureSettings
GateKey *keys.PrivateKey GateKey *keys.PrivateKey
CORSCnrInfo *data.BucketInfo CORSCnrInfo *data.BucketInfo
@ -103,14 +104,14 @@ type (
PutObjectParams struct { PutObjectParams struct {
BktInfo *data.BucketInfo BktInfo *data.BucketInfo
Object string Object string
Size uint64 Size *uint64
Reader io.Reader Reader io.Reader
Header map[string]string Header map[string]string
Lock *data.ObjectLock Lock *data.ObjectLock
Encryption encryption.Params Encryption encryption.Params
CopiesNumbers []uint32 CopiesNumbers []uint32
CompleteMD5Hash string CompleteMD5Hash string
ContentMD5 string ContentMD5 *string
ContentSHA256Hash string ContentSHA256Hash string
} }
@ -235,7 +236,7 @@ func (p HeadObjectParams) Versioned() bool {
// NewLayer creates an instance of a Layer. It checks credentials // NewLayer creates an instance of a Layer. It checks credentials
// and establishes gRPC connection with the node. // and establishes gRPC connection with the node.
func NewLayer(log *zap.Logger, frostFS FrostFS, config *Config) *Layer { func NewLayer(log *zap.Logger, frostFS frostfs.FrostFS, config *Config) *Layer {
return &Layer{ return &Layer{
frostFS: frostFS, frostFS: frostFS,
log: log, log: log,
@ -299,7 +300,7 @@ func (n *Layer) reqLogger(ctx context.Context) *zap.Logger {
return n.log return n.log
} }
func (n *Layer) prepareAuthParameters(ctx context.Context, prm *PrmAuth, bktOwner user.ID) { func (n *Layer) prepareAuthParameters(ctx context.Context, prm *frostfs.PrmAuth, bktOwner user.ID) {
if prm.BearerToken != nil || prm.PrivateKey != nil { if prm.BearerToken != nil || prm.PrivateKey != nil {
return return
} }
@ -331,12 +332,12 @@ func (n *Layer) GetBucketInfo(ctx context.Context, name string) (*data.BucketInf
containerID, err := n.ResolveBucket(ctx, zone, name) containerID, err := n.ResolveBucket(ctx, zone, name)
if err != nil { if err != nil {
if strings.Contains(err.Error(), "not found") { if strings.Contains(err.Error(), "not found") {
return nil, fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrNoSuchBucket), err.Error()) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchBucket), err.Error())
} }
return nil, err return nil, err
} }
prm := PrmContainer{ prm := frostfs.PrmContainer{
ContainerID: containerID, ContainerID: containerID,
SessionToken: n.SessionTokenForRead(ctx), SessionToken: n.SessionTokenForRead(ctx),
} }
@ -397,9 +398,9 @@ func (n *Layer) GetObject(ctx context.Context, p *GetObjectParams) (*ObjectPaylo
if err != nil { if err != nil {
if client.IsErrObjectNotFound(err) { if client.IsErrObjectNotFound(err) {
if p.Versioned { if p.Versioned {
err = fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrNoSuchVersion), err.Error()) err = fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchVersion), err.Error())
} else { } else {
err = fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrNoSuchKey), err.Error()) err = fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
} }
} }
@ -528,7 +529,7 @@ func (n *Layer) CopyObject(ctx context.Context, p *CopyObjectParams) (*data.Exte
return n.PutObject(ctx, &PutObjectParams{ return n.PutObject(ctx, &PutObjectParams{
BktInfo: p.DstBktInfo, BktInfo: p.DstBktInfo,
Object: p.DstObject, Object: p.DstObject,
Size: p.DstSize, Size: &p.DstSize,
Reader: objPayload, Reader: objPayload,
Header: p.Header, Header: p.Header,
Encryption: p.DstEncryption, Encryption: p.DstEncryption,
@ -655,22 +656,22 @@ func (n *Layer) handleNotFoundError(bkt *data.BucketInfo, obj *VersionedObject)
} }
func isNotFoundError(err error) bool { func isNotFoundError(err error) bool {
return errors.IsS3Error(err, errors.ErrNoSuchKey) || return apierr.IsS3Error(err, apierr.ErrNoSuchKey) ||
errors.IsS3Error(err, errors.ErrNoSuchVersion) apierr.IsS3Error(err, apierr.ErrNoSuchVersion)
} }
func (n *Layer) getNodeVersionsToDelete(ctx context.Context, bkt *data.BucketInfo, obj *VersionedObject) ([]*data.NodeVersion, error) { func (n *Layer) getNodeVersionsToDelete(ctx context.Context, bkt *data.BucketInfo, obj *VersionedObject) ([]*data.NodeVersion, error) {
var versionsToDelete []*data.NodeVersion var versionsToDelete []*data.NodeVersion
versions, err := n.treeService.GetVersions(ctx, bkt, obj.Name) versions, err := n.treeService.GetVersions(ctx, bkt, obj.Name)
if err != nil { if err != nil {
if stderrors.Is(err, ErrNodeNotFound) { if errors.Is(err, tree.ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchKey), err.Error()) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
} }
return nil, err return nil, err
} }
if len(versions) == 0 { if len(versions) == 0 {
return nil, fmt.Errorf("%w: there isn't tree node with requested version id", s3errors.GetAPIError(s3errors.ErrNoSuchVersion)) return nil, fmt.Errorf("%w: there isn't tree node with requested version id", apierr.GetAPIError(apierr.ErrNoSuchVersion))
} }
sort.Slice(versions, func(i, j int) bool { sort.Slice(versions, func(i, j int) bool {
@ -712,7 +713,7 @@ func (n *Layer) getNodeVersionsToDelete(ctx context.Context, bkt *data.BucketInf
} }
if len(versionsToDelete) == 0 { if len(versionsToDelete) == 0 {
return nil, fmt.Errorf("%w: there isn't tree node with requested version id", s3errors.GetAPIError(s3errors.ErrNoSuchVersion)) return nil, fmt.Errorf("%w: there isn't tree node with requested version id", apierr.GetAPIError(apierr.ErrNoSuchVersion))
} }
n.reqLogger(ctx).Debug(logs.GetTreeNodeToDelete, zap.Stringer("cid", bkt.CID), zap.Strings("oids", oids)) n.reqLogger(ctx).Debug(logs.GetTreeNodeToDelete, zap.Stringer("cid", bkt.CID), zap.Strings("oids", oids))
@ -785,17 +786,17 @@ func (n *Layer) DeleteObjects(ctx context.Context, p *DeleteObjectParams) []*Ver
func (n *Layer) CreateBucket(ctx context.Context, p *CreateBucketParams) (*data.BucketInfo, error) { func (n *Layer) CreateBucket(ctx context.Context, p *CreateBucketParams) (*data.BucketInfo, error) {
bktInfo, err := n.GetBucketInfo(ctx, p.Name) bktInfo, err := n.GetBucketInfo(ctx, p.Name)
if err != nil { if err != nil {
if errors.IsS3Error(err, errors.ErrNoSuchBucket) { if apierr.IsS3Error(err, apierr.ErrNoSuchBucket) {
return n.createContainer(ctx, p) return n.createContainer(ctx, p)
} }
return nil, err return nil, err
} }
if p.SessionContainerCreation != nil && session.IssuedBy(*p.SessionContainerCreation, bktInfo.Owner) { if p.SessionContainerCreation != nil && session.IssuedBy(*p.SessionContainerCreation, bktInfo.Owner) {
return nil, errors.GetAPIError(errors.ErrBucketAlreadyOwnedByYou) return nil, apierr.GetAPIError(apierr.ErrBucketAlreadyOwnedByYou)
} }
return nil, errors.GetAPIError(errors.ErrBucketAlreadyExists) return nil, apierr.GetAPIError(apierr.ErrBucketAlreadyExists)
} }
func (n *Layer) ResolveBucket(ctx context.Context, zone, name string) (cid.ID, error) { func (n *Layer) ResolveBucket(ctx context.Context, zone, name string) (cid.ID, error) {
@ -822,7 +823,7 @@ func (n *Layer) DeleteBucket(ctx context.Context, p *DeleteBucketParams) error {
return err return err
} }
if len(res) != 0 { if len(res) != 0 {
return errors.GetAPIError(errors.ErrBucketNotEmpty) return apierr.GetAPIError(apierr.ErrBucketNotEmpty)
} }
} }
@ -854,11 +855,26 @@ func (n *Layer) DeleteBucket(ctx context.Context, p *DeleteBucketParams) error {
return nil return nil
} }
func (n *Layer) DeleteContainer(ctx context.Context, p *DeleteBucketParams) error {
n.cache.DeleteBucket(p.BktInfo)
if err := n.frostFS.DeleteContainer(ctx, p.BktInfo.CID, p.SessionToken); err != nil {
return fmt.Errorf("delete container: %w", err)
}
return nil
}
func (n *Layer) GetNetworkInfo(ctx context.Context) (netmap.NetworkInfo, error) { func (n *Layer) GetNetworkInfo(ctx context.Context) (netmap.NetworkInfo, error) {
cachedInfo := n.cache.GetNetworkInfo()
if cachedInfo != nil {
return *cachedInfo, nil
}
networkInfo, err := n.frostFS.NetworkInfo(ctx) networkInfo, err := n.frostFS.NetworkInfo(ctx)
if err != nil { if err != nil {
return networkInfo, fmt.Errorf("get network info: %w", err) return netmap.NetworkInfo{}, fmt.Errorf("get network info: %w", err)
} }
n.cache.PutNetworkInfo(networkInfo)
return networkInfo, nil return networkInfo, nil
} }

View file

@ -3,14 +3,14 @@ package layer
import ( import (
"bytes" "bytes"
"context" "context"
"encoding/base64"
"encoding/xml" "encoding/xml"
"errors" "errors"
"fmt" "fmt"
"io"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"go.uber.org/zap" "go.uber.org/zap"
@ -19,14 +19,17 @@ import (
type PutBucketLifecycleParams struct { type PutBucketLifecycleParams struct {
BktInfo *data.BucketInfo BktInfo *data.BucketInfo
LifecycleCfg *data.LifecycleConfiguration LifecycleCfg *data.LifecycleConfiguration
LifecycleReader io.Reader
CopiesNumbers []uint32 CopiesNumbers []uint32
MD5Hash string
} }
func (n *Layer) PutBucketLifecycleConfiguration(ctx context.Context, p *PutBucketLifecycleParams) error { func (n *Layer) PutBucketLifecycleConfiguration(ctx context.Context, p *PutBucketLifecycleParams) error {
prm := PrmObjectCreate{ cfgBytes, err := xml.Marshal(p.LifecycleCfg)
Payload: p.LifecycleReader, if err != nil {
return fmt.Errorf("marshal lifecycle configuration: %w", err)
}
prm := frostfs.PrmObjectCreate{
Payload: bytes.NewReader(cfgBytes),
Filepath: p.BktInfo.LifecycleConfigurationObjectName(), Filepath: p.BktInfo.LifecycleConfigurationObjectName(),
CreationTime: TimeNow(ctx), CreationTime: TimeNow(ctx),
} }
@ -47,19 +50,8 @@ func (n *Layer) PutBucketLifecycleConfiguration(ctx context.Context, p *PutBucke
return fmt.Errorf("put lifecycle object: %w", err) return fmt.Errorf("put lifecycle object: %w", err)
} }
hashBytes, err := base64.StdEncoding.DecodeString(p.MD5Hash)
if err != nil {
return apiErr.GetAPIError(apiErr.ErrInvalidDigest)
}
if !bytes.Equal(hashBytes, createdObj.MD5Sum) {
n.deleteLifecycleObject(ctx, p.BktInfo, newAddress(lifecycleBkt.CID, createdObj.ID))
return apiErr.GetAPIError(apiErr.ErrInvalidDigest)
}
objsToDelete, err := n.treeService.PutBucketLifecycleConfiguration(ctx, p.BktInfo, newAddress(lifecycleBkt.CID, createdObj.ID)) objsToDelete, err := n.treeService.PutBucketLifecycleConfiguration(ctx, p.BktInfo, newAddress(lifecycleBkt.CID, createdObj.ID))
objsToDeleteNotFound := errors.Is(err, ErrNoNodeToRemove) objsToDeleteNotFound := errors.Is(err, tree.ErrNoNodeToRemove)
if err != nil && !objsToDeleteNotFound { if err != nil && !objsToDeleteNotFound {
return err return err
} }
@ -77,7 +69,7 @@ func (n *Layer) PutBucketLifecycleConfiguration(ctx context.Context, p *PutBucke
// deleteLifecycleObject removes object and logs in case of error. // deleteLifecycleObject removes object and logs in case of error.
func (n *Layer) deleteLifecycleObject(ctx context.Context, bktInfo *data.BucketInfo, addr oid.Address) { func (n *Layer) deleteLifecycleObject(ctx context.Context, bktInfo *data.BucketInfo, addr oid.Address) {
var prmAuth PrmAuth var prmAuth frostfs.PrmAuth
lifecycleBkt := bktInfo lifecycleBkt := bktInfo
if !addr.Container().Equals(bktInfo.CID) { if !addr.Container().Equals(bktInfo.CID) {
lifecycleBkt = &data.BucketInfo{CID: addr.Container()} lifecycleBkt = &data.BucketInfo{CID: addr.Container()}
@ -98,16 +90,16 @@ func (n *Layer) GetBucketLifecycleConfiguration(ctx context.Context, bktInfo *da
} }
addr, err := n.treeService.GetBucketLifecycleConfiguration(ctx, bktInfo) addr, err := n.treeService.GetBucketLifecycleConfiguration(ctx, bktInfo)
objNotFound := errors.Is(err, ErrNodeNotFound) objNotFound := errors.Is(err, tree.ErrNodeNotFound)
if err != nil && !objNotFound { if err != nil && !objNotFound {
return nil, err return nil, err
} }
if objNotFound { if objNotFound {
return nil, fmt.Errorf("%w: %s", apiErr.GetAPIError(apiErr.ErrNoSuchLifecycleConfiguration), err.Error()) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchLifecycleConfiguration), err.Error())
} }
var prmAuth PrmAuth var prmAuth frostfs.PrmAuth
lifecycleBkt := bktInfo lifecycleBkt := bktInfo
if !addr.Container().Equals(bktInfo.CID) { if !addr.Container().Equals(bktInfo.CID) {
lifecycleBkt = &data.BucketInfo{CID: addr.Container()} lifecycleBkt = &data.BucketInfo{CID: addr.Container()}
@ -127,12 +119,18 @@ func (n *Layer) GetBucketLifecycleConfiguration(ctx context.Context, bktInfo *da
n.cache.PutLifecycleConfiguration(owner, bktInfo, lifecycleCfg) n.cache.PutLifecycleConfiguration(owner, bktInfo, lifecycleCfg)
for i := range lifecycleCfg.Rules {
if lifecycleCfg.Rules[i].Expiration != nil {
lifecycleCfg.Rules[i].Expiration.Epoch = nil
}
}
return lifecycleCfg, nil return lifecycleCfg, nil
} }
func (n *Layer) DeleteBucketLifecycleConfiguration(ctx context.Context, bktInfo *data.BucketInfo) error { func (n *Layer) DeleteBucketLifecycleConfiguration(ctx context.Context, bktInfo *data.BucketInfo) error {
objs, err := n.treeService.DeleteBucketLifecycleConfiguration(ctx, bktInfo) objs, err := n.treeService.DeleteBucketLifecycleConfiguration(ctx, bktInfo)
objsNotFound := errors.Is(err, ErrNoNodeToRemove) objsNotFound := errors.Is(err, tree.ErrNoNodeToRemove)
if err != nil && !objsNotFound { if err != nil && !objsNotFound {
return err return err
} }

View file

@ -1,15 +1,13 @@
package layer package layer
import ( import (
"bytes"
"crypto/md5" "crypto/md5"
"encoding/base64"
"encoding/xml" "encoding/xml"
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
frostfsErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors" frosterr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@ -36,7 +34,7 @@ func TestBucketLifecycle(t *testing.T) {
hash.Write(lifecycleBytes) hash.Write(lifecycleBytes)
_, err = tc.layer.GetBucketLifecycleConfiguration(tc.ctx, tc.bktInfo) _, err = tc.layer.GetBucketLifecycleConfiguration(tc.ctx, tc.bktInfo)
require.Equal(t, apiErr.GetAPIError(apiErr.ErrNoSuchLifecycleConfiguration), frostfsErrors.UnwrapErr(err)) require.Equal(t, apierr.GetAPIError(apierr.ErrNoSuchLifecycleConfiguration), frosterr.UnwrapErr(err))
err = tc.layer.DeleteBucketLifecycleConfiguration(tc.ctx, tc.bktInfo) err = tc.layer.DeleteBucketLifecycleConfiguration(tc.ctx, tc.bktInfo)
require.NoError(t, err) require.NoError(t, err)
@ -44,8 +42,6 @@ func TestBucketLifecycle(t *testing.T) {
err = tc.layer.PutBucketLifecycleConfiguration(tc.ctx, &PutBucketLifecycleParams{ err = tc.layer.PutBucketLifecycleConfiguration(tc.ctx, &PutBucketLifecycleParams{
BktInfo: tc.bktInfo, BktInfo: tc.bktInfo,
LifecycleCfg: lifecycle, LifecycleCfg: lifecycle,
LifecycleReader: bytes.NewReader(lifecycleBytes),
MD5Hash: base64.StdEncoding.EncodeToString(hash.Sum(nil)),
}) })
require.NoError(t, err) require.NoError(t, err)
@ -57,7 +53,7 @@ func TestBucketLifecycle(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
_, err = tc.layer.GetBucketLifecycleConfiguration(tc.ctx, tc.bktInfo) _, err = tc.layer.GetBucketLifecycleConfiguration(tc.ctx, tc.bktInfo)
require.Equal(t, apiErr.GetAPIError(apiErr.ErrNoSuchLifecycleConfiguration), frostfsErrors.UnwrapErr(err)) require.Equal(t, apierr.GetAPIError(apierr.ErrNoSuchLifecycleConfiguration), frosterr.UnwrapErr(err))
} }
func ptr[T any](t T) *T { func ptr[T any](t T) *T {

View file

@ -11,7 +11,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/cache" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/cache"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"github.com/panjf2000/ants/v2" "github.com/panjf2000/ants/v2"
@ -585,7 +585,7 @@ func shouldSkip(node *data.ExtendedNodeVersion, p commonVersionsListingParams, e
return true return true
} }
if p.Bookmark != "" { if p.Bookmark != "" && p.Bookmark != p.Marker {
if _, ok := existed[continuationToken]; !ok { if _, ok := existed[continuationToken]; !ok {
if p.Bookmark != node.NodeVersion.OID.EncodeToString() { if p.Bookmark != node.NodeVersion.OID.EncodeToString() {
return true return true
@ -691,10 +691,10 @@ func filterVersionsByMarker(objects []*data.ExtendedNodeVersion, p *ListObjectVe
return objects[j+1:], nil return objects[j+1:], nil
} }
} }
return nil, s3errors.GetAPIError(s3errors.ErrInvalidVersion) return nil, apierr.GetAPIError(apierr.ErrInvalidVersion)
} else if obj.NodeVersion.FilePath > p.KeyMarker { } else if obj.NodeVersion.FilePath > p.KeyMarker {
if p.VersionIDMarker != "" { if p.VersionIDMarker != "" {
return nil, s3errors.GetAPIError(s3errors.ErrInvalidVersion) return nil, apierr.GetAPIError(apierr.ErrInvalidVersion)
} }
return objects[i:], nil return objects[i:], nil
} }

View file

@ -17,8 +17,10 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/encryption"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
@ -150,9 +152,9 @@ func (n *Layer) CreateMultipartUpload(ctx context.Context, p *CreateMultipartPar
metaSize += len(p.Data.TagSet) metaSize += len(p.Data.TagSet)
} }
networkInfo, err := n.frostFS.NetworkInfo(ctx) networkInfo, err := n.GetNetworkInfo(ctx)
if err != nil { if err != nil {
return fmt.Errorf("get network info: %w", err) return err
} }
info := &data.MultipartInfo{ info := &data.MultipartInfo{
@ -187,14 +189,14 @@ func (n *Layer) CreateMultipartUpload(ctx context.Context, p *CreateMultipartPar
func (n *Layer) UploadPart(ctx context.Context, p *UploadPartParams) (string, error) { func (n *Layer) UploadPart(ctx context.Context, p *UploadPartParams) (string, error) {
multipartInfo, err := n.treeService.GetMultipartUpload(ctx, p.Info.Bkt, p.Info.Key, p.Info.UploadID) multipartInfo, err := n.treeService.GetMultipartUpload(ctx, p.Info.Bkt, p.Info.Key, p.Info.UploadID)
if err != nil { if err != nil {
if errors.Is(err, ErrNodeNotFound) { if errors.Is(err, tree.ErrNodeNotFound) {
return "", fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchUpload), err.Error()) return "", fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchUpload), err.Error())
} }
return "", err return "", err
} }
if p.Size > UploadMaxSize { if p.Size > UploadMaxSize {
return "", fmt.Errorf("%w: %d/%d", s3errors.GetAPIError(s3errors.ErrEntityTooLarge), p.Size, UploadMaxSize) return "", fmt.Errorf("%w: %d/%d", apierr.GetAPIError(apierr.ErrEntityTooLarge), p.Size, UploadMaxSize)
} }
objInfo, err := n.uploadPart(ctx, multipartInfo, p) objInfo, err := n.uploadPart(ctx, multipartInfo, p)
@ -209,11 +211,11 @@ func (n *Layer) uploadPart(ctx context.Context, multipartInfo *data.MultipartInf
encInfo := FormEncryptionInfo(multipartInfo.Meta) encInfo := FormEncryptionInfo(multipartInfo.Meta)
if err := p.Info.Encryption.MatchObjectEncryption(encInfo); err != nil { if err := p.Info.Encryption.MatchObjectEncryption(encInfo); err != nil {
n.reqLogger(ctx).Warn(logs.MismatchedObjEncryptionInfo, zap.Error(err)) n.reqLogger(ctx).Warn(logs.MismatchedObjEncryptionInfo, zap.Error(err))
return nil, s3errors.GetAPIError(s3errors.ErrInvalidEncryptionParameters) return nil, apierr.GetAPIError(apierr.ErrInvalidEncryptionParameters)
} }
bktInfo := p.Info.Bkt bktInfo := p.Info.Bkt
prm := PrmObjectCreate{ prm := frostfs.PrmObjectCreate{
Container: bktInfo.CID, Container: bktInfo.CID,
Attributes: make([][2]string, 2), Attributes: make([][2]string, 2),
Payload: p.Reader, Payload: p.Reader,
@ -242,10 +244,10 @@ func (n *Layer) uploadPart(ctx context.Context, multipartInfo *data.MultipartInf
if len(p.ContentMD5) > 0 { if len(p.ContentMD5) > 0 {
hashBytes, err := base64.StdEncoding.DecodeString(p.ContentMD5) hashBytes, err := base64.StdEncoding.DecodeString(p.ContentMD5)
if err != nil { if err != nil {
return nil, s3errors.GetAPIError(s3errors.ErrInvalidDigest) return nil, apierr.GetAPIError(apierr.ErrInvalidDigest)
} }
if hex.EncodeToString(hashBytes) != hex.EncodeToString(createdObj.MD5Sum) { if hex.EncodeToString(hashBytes) != hex.EncodeToString(createdObj.MD5Sum) {
prm := PrmObjectDelete{ prm := frostfs.PrmObjectDelete{
Object: createdObj.ID, Object: createdObj.ID,
Container: bktInfo.CID, Container: bktInfo.CID,
} }
@ -254,7 +256,7 @@ func (n *Layer) uploadPart(ctx context.Context, multipartInfo *data.MultipartInf
if err != nil { if err != nil {
n.reqLogger(ctx).Debug(logs.FailedToDeleteObject, zap.Stringer("cid", bktInfo.CID), zap.Stringer("oid", createdObj.ID)) n.reqLogger(ctx).Debug(logs.FailedToDeleteObject, zap.Stringer("cid", bktInfo.CID), zap.Stringer("oid", createdObj.ID))
} }
return nil, s3errors.GetAPIError(s3errors.ErrInvalidDigest) return nil, apierr.GetAPIError(apierr.ErrInvalidDigest)
} }
} }
if p.Info.Encryption.Enabled() { if p.Info.Encryption.Enabled() {
@ -264,14 +266,14 @@ func (n *Layer) uploadPart(ctx context.Context, multipartInfo *data.MultipartInf
if !p.Info.Encryption.Enabled() && len(p.ContentSHA256Hash) > 0 && !auth.IsStandardContentSHA256(p.ContentSHA256Hash) { if !p.Info.Encryption.Enabled() && len(p.ContentSHA256Hash) > 0 && !auth.IsStandardContentSHA256(p.ContentSHA256Hash) {
contentHashBytes, err := hex.DecodeString(p.ContentSHA256Hash) contentHashBytes, err := hex.DecodeString(p.ContentSHA256Hash)
if err != nil { if err != nil {
return nil, s3errors.GetAPIError(s3errors.ErrContentSHA256Mismatch) return nil, apierr.GetAPIError(apierr.ErrContentSHA256Mismatch)
} }
if !bytes.Equal(contentHashBytes, createdObj.HashSum) { if !bytes.Equal(contentHashBytes, createdObj.HashSum) {
err = n.objectDelete(ctx, bktInfo, createdObj.ID) err = n.objectDelete(ctx, bktInfo, createdObj.ID)
if err != nil { if err != nil {
n.reqLogger(ctx).Debug(logs.FailedToDeleteObject, zap.Stringer("cid", bktInfo.CID), zap.Stringer("oid", createdObj.ID)) n.reqLogger(ctx).Debug(logs.FailedToDeleteObject, zap.Stringer("cid", bktInfo.CID), zap.Stringer("oid", createdObj.ID))
} }
return nil, s3errors.GetAPIError(s3errors.ErrContentSHA256Mismatch) return nil, apierr.GetAPIError(apierr.ErrContentSHA256Mismatch)
} }
} }
@ -291,7 +293,7 @@ func (n *Layer) uploadPart(ctx context.Context, multipartInfo *data.MultipartInf
} }
oldPartIDs, err := n.treeService.AddPart(ctx, bktInfo, multipartInfo.ID, partInfo) oldPartIDs, err := n.treeService.AddPart(ctx, bktInfo, multipartInfo.ID, partInfo)
oldPartIDNotFound := errors.Is(err, ErrNoNodeToRemove) oldPartIDNotFound := errors.Is(err, tree.ErrNoNodeToRemove)
if err != nil && !oldPartIDNotFound { if err != nil && !oldPartIDNotFound {
return nil, err return nil, err
} }
@ -323,8 +325,8 @@ func (n *Layer) uploadPart(ctx context.Context, multipartInfo *data.MultipartInf
func (n *Layer) UploadPartCopy(ctx context.Context, p *UploadCopyParams) (*data.ObjectInfo, error) { func (n *Layer) UploadPartCopy(ctx context.Context, p *UploadCopyParams) (*data.ObjectInfo, error) {
multipartInfo, err := n.treeService.GetMultipartUpload(ctx, p.Info.Bkt, p.Info.Key, p.Info.UploadID) multipartInfo, err := n.treeService.GetMultipartUpload(ctx, p.Info.Bkt, p.Info.Key, p.Info.UploadID)
if err != nil { if err != nil {
if errors.Is(err, ErrNodeNotFound) { if errors.Is(err, tree.ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchUpload), err.Error()) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchUpload), err.Error())
} }
return nil, err return nil, err
} }
@ -340,11 +342,11 @@ func (n *Layer) UploadPartCopy(ctx context.Context, p *UploadCopyParams) (*data.
if p.Range != nil { if p.Range != nil {
size = p.Range.End - p.Range.Start + 1 size = p.Range.End - p.Range.Start + 1
if p.Range.End > srcObjectSize { if p.Range.End > srcObjectSize {
return nil, fmt.Errorf("%w: %d-%d/%d", s3errors.GetAPIError(s3errors.ErrInvalidCopyPartRangeSource), p.Range.Start, p.Range.End, srcObjectSize) return nil, fmt.Errorf("%w: %d-%d/%d", apierr.GetAPIError(apierr.ErrInvalidCopyPartRangeSource), p.Range.Start, p.Range.End, srcObjectSize)
} }
} }
if size > UploadMaxSize { if size > UploadMaxSize {
return nil, fmt.Errorf("%w: %d/%d", s3errors.GetAPIError(s3errors.ErrEntityTooLarge), size, UploadMaxSize) return nil, fmt.Errorf("%w: %d/%d", apierr.GetAPIError(apierr.ErrEntityTooLarge), size, UploadMaxSize)
} }
objPayload, err := n.GetObject(ctx, &GetObjectParams{ objPayload, err := n.GetObject(ctx, &GetObjectParams{
@ -371,7 +373,7 @@ func (n *Layer) UploadPartCopy(ctx context.Context, p *UploadCopyParams) (*data.
func (n *Layer) CompleteMultipartUpload(ctx context.Context, p *CompleteMultipartParams) (*UploadData, *data.ExtendedObjectInfo, error) { func (n *Layer) CompleteMultipartUpload(ctx context.Context, p *CompleteMultipartParams) (*UploadData, *data.ExtendedObjectInfo, error) {
for i := 1; i < len(p.Parts); i++ { for i := 1; i < len(p.Parts); i++ {
if p.Parts[i].PartNumber <= p.Parts[i-1].PartNumber { if p.Parts[i].PartNumber <= p.Parts[i-1].PartNumber {
return nil, nil, s3errors.GetAPIError(s3errors.ErrInvalidPartOrder) return nil, nil, apierr.GetAPIError(apierr.ErrInvalidPartOrder)
} }
} }
@ -382,7 +384,7 @@ func (n *Layer) CompleteMultipartUpload(ctx context.Context, p *CompleteMultipar
encInfo := FormEncryptionInfo(multipartInfo.Meta) encInfo := FormEncryptionInfo(multipartInfo.Meta)
if len(partsInfo) < len(p.Parts) { if len(partsInfo) < len(p.Parts) {
return nil, nil, fmt.Errorf("%w: found %d parts, need %d", s3errors.GetAPIError(s3errors.ErrInvalidPart), len(partsInfo), len(p.Parts)) return nil, nil, fmt.Errorf("%w: found %d parts, need %d", apierr.GetAPIError(apierr.ErrInvalidPart), len(partsInfo), len(p.Parts))
} }
var multipartObjetSize uint64 var multipartObjetSize uint64
@ -394,12 +396,12 @@ func (n *Layer) CompleteMultipartUpload(ctx context.Context, p *CompleteMultipar
for i, part := range p.Parts { for i, part := range p.Parts {
partInfo := partsInfo.Extract(part.PartNumber, data.UnQuote(part.ETag), n.features.MD5Enabled()) partInfo := partsInfo.Extract(part.PartNumber, data.UnQuote(part.ETag), n.features.MD5Enabled())
if partInfo == nil { if partInfo == nil {
return nil, nil, fmt.Errorf("%w: unknown part %d or etag mismatched", s3errors.GetAPIError(s3errors.ErrInvalidPart), part.PartNumber) return nil, nil, fmt.Errorf("%w: unknown part %d or etag mismatched", apierr.GetAPIError(apierr.ErrInvalidPart), part.PartNumber)
} }
// for the last part we have no minimum size limit // for the last part we have no minimum size limit
if i != len(p.Parts)-1 && partInfo.Size < UploadMinSize { if i != len(p.Parts)-1 && partInfo.Size < UploadMinSize {
return nil, nil, fmt.Errorf("%w: %d/%d", s3errors.GetAPIError(s3errors.ErrEntityTooSmall), partInfo.Size, UploadMinSize) return nil, nil, fmt.Errorf("%w: %d/%d", apierr.GetAPIError(apierr.ErrEntityTooSmall), partInfo.Size, UploadMinSize)
} }
parts = append(parts, partInfo) parts = append(parts, partInfo)
multipartObjetSize += partInfo.Size // even if encryption is enabled size is actual (decrypted) multipartObjetSize += partInfo.Size // even if encryption is enabled size is actual (decrypted)
@ -460,7 +462,7 @@ func (n *Layer) CompleteMultipartUpload(ctx context.Context, p *CompleteMultipar
Object: p.Info.Key, Object: p.Info.Key,
Reader: bytes.NewReader(partsData), Reader: bytes.NewReader(partsData),
Header: initMetadata, Header: initMetadata,
Size: multipartObjetSize, Size: &multipartObjetSize,
Encryption: p.Info.Encryption, Encryption: p.Info.Encryption,
CopiesNumbers: multipartInfo.CopiesNumbers, CopiesNumbers: multipartInfo.CopiesNumbers,
CompleteMD5Hash: hex.EncodeToString(md5Hash.Sum(nil)) + "-" + strconv.Itoa(len(p.Parts)), CompleteMD5Hash: hex.EncodeToString(md5Hash.Sum(nil)) + "-" + strconv.Itoa(len(p.Parts)),
@ -471,7 +473,7 @@ func (n *Layer) CompleteMultipartUpload(ctx context.Context, p *CompleteMultipar
zap.String("uploadKey", p.Info.Key), zap.String("uploadKey", p.Info.Key),
zap.Error(err)) zap.Error(err))
return nil, nil, s3errors.GetAPIError(s3errors.ErrInternalError) return nil, nil, apierr.GetAPIError(apierr.ErrInternalError)
} }
var addr oid.Address var addr oid.Address
@ -579,7 +581,7 @@ func (n *Layer) ListParts(ctx context.Context, p *ListPartsParams) (*ListPartsIn
encInfo := FormEncryptionInfo(multipartInfo.Meta) encInfo := FormEncryptionInfo(multipartInfo.Meta)
if err = p.Info.Encryption.MatchObjectEncryption(encInfo); err != nil { if err = p.Info.Encryption.MatchObjectEncryption(encInfo); err != nil {
n.reqLogger(ctx).Warn(logs.MismatchedObjEncryptionInfo, zap.Error(err)) n.reqLogger(ctx).Warn(logs.MismatchedObjEncryptionInfo, zap.Error(err))
return nil, s3errors.GetAPIError(s3errors.ErrInvalidEncryptionParameters) return nil, apierr.GetAPIError(apierr.ErrInvalidEncryptionParameters)
} }
res.Owner = multipartInfo.Owner res.Owner = multipartInfo.Owner
@ -646,8 +648,8 @@ func (p PartsInfo) Extract(part int, etag string, md5Enabled bool) *data.PartInf
func (n *Layer) getUploadParts(ctx context.Context, p *UploadInfoParams) (*data.MultipartInfo, PartsInfo, error) { func (n *Layer) getUploadParts(ctx context.Context, p *UploadInfoParams) (*data.MultipartInfo, PartsInfo, error) {
multipartInfo, err := n.treeService.GetMultipartUpload(ctx, p.Bkt, p.Key, p.UploadID) multipartInfo, err := n.treeService.GetMultipartUpload(ctx, p.Bkt, p.Key, p.UploadID)
if err != nil { if err != nil {
if errors.Is(err, ErrNodeNotFound) { if errors.Is(err, tree.ErrNodeNotFound) {
return nil, nil, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchUpload), err.Error()) return nil, nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchUpload), err.Error())
} }
return nil, nil, err return nil, nil, err
} }

View file

@ -12,6 +12,7 @@ import (
"fmt" "fmt"
"io" "io"
"mime" "mime"
"net/http"
"path/filepath" "path/filepath"
"strconv" "strconv"
"strings" "strings"
@ -19,8 +20,11 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/detector"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/client"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
@ -47,7 +51,7 @@ type (
} }
DeleteMarkerError struct { DeleteMarkerError struct {
ErrorCode apiErrors.ErrorCode ErrorCode apierr.ErrorCode
} }
) )
@ -68,7 +72,7 @@ func newAddress(cnr cid.ID, obj oid.ID) oid.Address {
// objectHead returns all object's headers. // objectHead returns all object's headers.
func (n *Layer) objectHead(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID) (*object.Object, error) { func (n *Layer) objectHead(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID) (*object.Object, error) {
prm := PrmObjectHead{ prm := frostfs.PrmObjectHead{
Container: bktInfo.CID, Container: bktInfo.CID,
Object: idObj, Object: idObj,
} }
@ -126,11 +130,11 @@ func (n *Layer) initObjectPayloadReader(ctx context.Context, p getParams) (io.Re
// initializes payload reader of the FrostFS object. // initializes payload reader of the FrostFS object.
// Zero range corresponds to full payload (panics if only offset is set). // Zero range corresponds to full payload (panics if only offset is set).
func (n *Layer) initFrostFSObjectPayloadReader(ctx context.Context, p getFrostFSParams) (io.Reader, error) { func (n *Layer) initFrostFSObjectPayloadReader(ctx context.Context, p getFrostFSParams) (io.Reader, error) {
var prmAuth PrmAuth var prmAuth frostfs.PrmAuth
n.prepareAuthParameters(ctx, &prmAuth, p.bktInfo.Owner) n.prepareAuthParameters(ctx, &prmAuth, p.bktInfo.Owner)
if p.off+p.ln != 0 { if p.off+p.ln != 0 {
prm := PrmObjectRange{ prm := frostfs.PrmObjectRange{
PrmAuth: prmAuth, PrmAuth: prmAuth,
Container: p.bktInfo.CID, Container: p.bktInfo.CID,
Object: p.oid, Object: p.oid,
@ -140,7 +144,7 @@ func (n *Layer) initFrostFSObjectPayloadReader(ctx context.Context, p getFrostFS
return n.frostFS.RangeObject(ctx, prm) return n.frostFS.RangeObject(ctx, prm)
} }
prm := PrmObjectGet{ prm := frostfs.PrmObjectGet{
PrmAuth: prmAuth, PrmAuth: prmAuth,
Container: p.bktInfo.CID, Container: p.bktInfo.CID,
Object: p.oid, Object: p.oid,
@ -155,17 +159,17 @@ func (n *Layer) initFrostFSObjectPayloadReader(ctx context.Context, p getFrostFS
} }
// objectGet returns an object with payload in the object. // objectGet returns an object with payload in the object.
func (n *Layer) objectGet(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID) (*Object, error) { func (n *Layer) objectGet(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID) (*frostfs.Object, error) {
return n.objectGetBase(ctx, bktInfo, objID, PrmAuth{}) return n.objectGetBase(ctx, bktInfo, objID, frostfs.PrmAuth{})
} }
// objectGetWithAuth returns an object with payload in the object. Uses provided PrmAuth. // objectGetWithAuth returns an object with payload in the object. Uses provided PrmAuth.
func (n *Layer) objectGetWithAuth(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID, auth PrmAuth) (*Object, error) { func (n *Layer) objectGetWithAuth(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID, auth frostfs.PrmAuth) (*frostfs.Object, error) {
return n.objectGetBase(ctx, bktInfo, objID, auth) return n.objectGetBase(ctx, bktInfo, objID, auth)
} }
func (n *Layer) objectGetBase(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID, auth PrmAuth) (*Object, error) { func (n *Layer) objectGetBase(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID, auth frostfs.PrmAuth) (*frostfs.Object, error) {
prm := PrmObjectGet{ prm := frostfs.PrmObjectGet{
PrmAuth: auth, PrmAuth: auth,
Container: bktInfo.CID, Container: bktInfo.CID,
Object: objID, Object: objID,
@ -230,40 +234,46 @@ func (n *Layer) PutObject(ctx context.Context, p *PutObjectParams) (*data.Extend
r := p.Reader r := p.Reader
if p.Encryption.Enabled() { if p.Encryption.Enabled() {
p.Header[AttributeDecryptedSize] = strconv.FormatUint(p.Size, 10) var size uint64
if p.Size != nil {
size = *p.Size
}
p.Header[AttributeDecryptedSize] = strconv.FormatUint(size, 10)
if err = addEncryptionHeaders(p.Header, p.Encryption); err != nil { if err = addEncryptionHeaders(p.Header, p.Encryption); err != nil {
return nil, fmt.Errorf("add encryption header: %w", err) return nil, fmt.Errorf("add encryption header: %w", err)
} }
var encSize uint64 var encSize uint64
if r, encSize, err = encryptionReader(p.Reader, p.Size, p.Encryption.Key()); err != nil { if r, encSize, err = encryptionReader(p.Reader, size, p.Encryption.Key()); err != nil {
return nil, fmt.Errorf("create encrypter: %w", err) return nil, fmt.Errorf("create encrypter: %w", err)
} }
p.Size = encSize p.Size = &encSize
} }
if r != nil { if r != nil {
if len(p.Header[api.ContentType]) == 0 { if len(p.Header[api.ContentType]) == 0 {
if contentType := MimeByFilePath(p.Object); len(contentType) == 0 { if contentType := MimeByFilePath(p.Object); len(contentType) == 0 {
d := newDetector(r) d := detector.NewDetector(r, http.DetectContentType)
if contentType, err := d.Detect(); err == nil { if contentType, err := d.Detect(); err == nil {
p.Header[api.ContentType] = contentType p.Header[api.ContentType] = contentType
} }
r = d.MultiReader() r = d.RestoredReader()
} else { } else {
p.Header[api.ContentType] = contentType p.Header[api.ContentType] = contentType
} }
} }
} }
prm := PrmObjectCreate{ prm := frostfs.PrmObjectCreate{
Container: p.BktInfo.CID, Container: p.BktInfo.CID,
PayloadSize: p.Size,
Filepath: p.Object, Filepath: p.Object,
Payload: r, Payload: r,
CreationTime: TimeNow(ctx), CreationTime: TimeNow(ctx),
CopiesNumber: p.CopiesNumbers, CopiesNumber: p.CopiesNumbers,
} }
if p.Size != nil {
prm.PayloadSize = *p.Size
}
prm.Attributes = make([][2]string, 0, len(p.Header)) prm.Attributes = make([][2]string, 0, len(p.Header))
@ -275,31 +285,35 @@ func (n *Layer) PutObject(ctx context.Context, p *PutObjectParams) (*data.Extend
if err != nil { if err != nil {
return nil, err return nil, err
} }
if len(p.ContentMD5) > 0 {
headerMd5Hash, err := base64.StdEncoding.DecodeString(p.ContentMD5) if !p.Encryption.Enabled() && p.ContentMD5 != nil {
if len(*p.ContentMD5) == 0 {
return nil, apierr.GetAPIError(apierr.ErrInvalidDigest)
}
headerMd5Hash, err := base64.StdEncoding.DecodeString(*p.ContentMD5)
if err != nil { if err != nil {
return nil, apiErrors.GetAPIError(apiErrors.ErrInvalidDigest) return nil, apierr.GetAPIError(apierr.ErrInvalidDigest)
} }
if !bytes.Equal(headerMd5Hash, createdObj.MD5Sum) { if !bytes.Equal(headerMd5Hash, createdObj.MD5Sum) {
err = n.objectDelete(ctx, p.BktInfo, createdObj.ID) err = n.objectDelete(ctx, p.BktInfo, createdObj.ID)
if err != nil { if err != nil {
n.reqLogger(ctx).Debug(logs.FailedToDeleteObject, zap.Stringer("cid", p.BktInfo.CID), zap.Stringer("oid", createdObj.ID)) n.reqLogger(ctx).Debug(logs.FailedToDeleteObject, zap.Stringer("cid", p.BktInfo.CID), zap.Stringer("oid", createdObj.ID))
} }
return nil, apiErrors.GetAPIError(apiErrors.ErrInvalidDigest) return nil, apierr.GetAPIError(apierr.ErrBadDigest)
} }
} }
if !p.Encryption.Enabled() && len(p.ContentSHA256Hash) > 0 && !auth.IsStandardContentSHA256(p.ContentSHA256Hash) { if !p.Encryption.Enabled() && len(p.ContentSHA256Hash) > 0 && !auth.IsStandardContentSHA256(p.ContentSHA256Hash) {
contentHashBytes, err := hex.DecodeString(p.ContentSHA256Hash) contentHashBytes, err := hex.DecodeString(p.ContentSHA256Hash)
if err != nil { if err != nil {
return nil, apiErrors.GetAPIError(apiErrors.ErrContentSHA256Mismatch) return nil, apierr.GetAPIError(apierr.ErrContentSHA256Mismatch)
} }
if !bytes.Equal(contentHashBytes, createdObj.HashSum) { if !bytes.Equal(contentHashBytes, createdObj.HashSum) {
err = n.objectDelete(ctx, p.BktInfo, createdObj.ID) err = n.objectDelete(ctx, p.BktInfo, createdObj.ID)
if err != nil { if err != nil {
n.reqLogger(ctx).Debug(logs.FailedToDeleteObject, zap.Stringer("cid", p.BktInfo.CID), zap.Stringer("oid", createdObj.ID)) n.reqLogger(ctx).Debug(logs.FailedToDeleteObject, zap.Stringer("cid", p.BktInfo.CID), zap.Stringer("oid", createdObj.ID))
} }
return nil, apiErrors.GetAPIError(apiErrors.ErrContentSHA256Mismatch) return nil, apierr.GetAPIError(apierr.ErrContentSHA256Mismatch)
} }
} }
@ -310,7 +324,6 @@ func (n *Layer) PutObject(ctx context.Context, p *PutObjectParams) (*data.Extend
OID: createdObj.ID, OID: createdObj.ID,
ETag: hex.EncodeToString(createdObj.HashSum), ETag: hex.EncodeToString(createdObj.HashSum),
FilePath: p.Object, FilePath: p.Object,
Size: p.Size,
Created: &now, Created: &now,
Owner: &n.gateOwner, Owner: &n.gateOwner,
CreationEpoch: createdObj.CreationEpoch, CreationEpoch: createdObj.CreationEpoch,
@ -318,12 +331,19 @@ func (n *Layer) PutObject(ctx context.Context, p *PutObjectParams) (*data.Extend
IsUnversioned: !bktSettings.VersioningEnabled(), IsUnversioned: !bktSettings.VersioningEnabled(),
IsCombined: p.Header[MultipartObjectSize] != "", IsCombined: p.Header[MultipartObjectSize] != "",
} }
if len(p.CompleteMD5Hash) > 0 { if len(p.CompleteMD5Hash) > 0 {
newVersion.MD5 = p.CompleteMD5Hash newVersion.MD5 = p.CompleteMD5Hash
} else { } else {
newVersion.MD5 = hex.EncodeToString(createdObj.MD5Sum) newVersion.MD5 = hex.EncodeToString(createdObj.MD5Sum)
} }
if p.Size != nil {
newVersion.Size = *p.Size
} else {
newVersion.Size = createdObj.Size
}
if newVersion.ID, err = n.treeService.AddVersion(ctx, p.BktInfo, newVersion); err != nil { if newVersion.ID, err = n.treeService.AddVersion(ctx, p.BktInfo, newVersion); err != nil {
return nil, fmt.Errorf("couldn't add new verion to tree service: %w", err) return nil, fmt.Errorf("couldn't add new verion to tree service: %w", err)
} }
@ -380,20 +400,20 @@ func (n *Layer) headLastVersionIfNotDeleted(ctx context.Context, bkt *data.Bucke
node, err := n.treeService.GetLatestVersion(ctx, bkt, objectName) node, err := n.treeService.GetLatestVersion(ctx, bkt, objectName)
if err != nil { if err != nil {
if errors.Is(err, ErrNodeNotFound) { if errors.Is(err, tree.ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrNoSuchKey), err.Error()) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
} }
return nil, err return nil, err
} }
if node.IsDeleteMarker { if node.IsDeleteMarker {
return nil, DeleteMarkerError{ErrorCode: apiErrors.ErrNoSuchKey} return nil, DeleteMarkerError{ErrorCode: apierr.ErrNoSuchKey}
} }
meta, err := n.objectHead(ctx, bkt, node.OID) meta, err := n.objectHead(ctx, bkt, node.OID)
if err != nil { if err != nil {
if client.IsErrObjectNotFound(err) { if client.IsErrObjectNotFound(err) {
return nil, fmt.Errorf("%w: %s; %s", apiErrors.GetAPIError(apiErrors.ErrNoSuchKey), err.Error(), node.OID.EncodeToString()) return nil, fmt.Errorf("%w: %s; %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error(), node.OID.EncodeToString())
} }
return nil, err return nil, err
} }
@ -416,8 +436,8 @@ func (n *Layer) headVersion(ctx context.Context, bkt *data.BucketInfo, p *HeadOb
if p.VersionID == data.UnversionedObjectVersionID { if p.VersionID == data.UnversionedObjectVersionID {
foundVersion, err = n.treeService.GetUnversioned(ctx, bkt, p.Object) foundVersion, err = n.treeService.GetUnversioned(ctx, bkt, p.Object)
if err != nil { if err != nil {
if errors.Is(err, ErrNodeNotFound) { if errors.Is(err, tree.ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrNoSuchVersion), err.Error()) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchVersion), err.Error())
} }
return nil, err return nil, err
} }
@ -434,7 +454,7 @@ func (n *Layer) headVersion(ctx context.Context, bkt *data.BucketInfo, p *HeadOb
} }
} }
if foundVersion == nil { if foundVersion == nil {
return nil, fmt.Errorf("%w: there isn't tree node with requested version id", apiErrors.GetAPIError(apiErrors.ErrNoSuchVersion)) return nil, fmt.Errorf("%w: there isn't tree node with requested version id", apierr.GetAPIError(apierr.ErrNoSuchVersion))
} }
} }
@ -444,13 +464,13 @@ func (n *Layer) headVersion(ctx context.Context, bkt *data.BucketInfo, p *HeadOb
} }
if foundVersion.IsDeleteMarker { if foundVersion.IsDeleteMarker {
return nil, DeleteMarkerError{ErrorCode: apiErrors.ErrMethodNotAllowed} return nil, DeleteMarkerError{ErrorCode: apierr.ErrMethodNotAllowed}
} }
meta, err := n.objectHead(ctx, bkt, foundVersion.OID) meta, err := n.objectHead(ctx, bkt, foundVersion.OID)
if err != nil { if err != nil {
if client.IsErrObjectNotFound(err) { if client.IsErrObjectNotFound(err) {
return nil, fmt.Errorf("%w: %s", apiErrors.GetAPIError(apiErrors.ErrNoSuchVersion), err.Error()) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchVersion), err.Error())
} }
return nil, err return nil, err
} }
@ -469,16 +489,16 @@ func (n *Layer) headVersion(ctx context.Context, bkt *data.BucketInfo, p *HeadOb
// objectDelete puts tombstone object into frostfs. // objectDelete puts tombstone object into frostfs.
func (n *Layer) objectDelete(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID) error { func (n *Layer) objectDelete(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID) error {
return n.objectDeleteBase(ctx, bktInfo, idObj, PrmAuth{}) return n.objectDeleteBase(ctx, bktInfo, idObj, frostfs.PrmAuth{})
} }
// objectDeleteWithAuth puts tombstone object into frostfs. Uses provided PrmAuth. // objectDeleteWithAuth puts tombstone object into frostfs. Uses provided PrmAuth.
func (n *Layer) objectDeleteWithAuth(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID, auth PrmAuth) error { func (n *Layer) objectDeleteWithAuth(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID, auth frostfs.PrmAuth) error {
return n.objectDeleteBase(ctx, bktInfo, idObj, auth) return n.objectDeleteBase(ctx, bktInfo, idObj, auth)
} }
func (n *Layer) objectDeleteBase(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID, auth PrmAuth) error { func (n *Layer) objectDeleteBase(ctx context.Context, bktInfo *data.BucketInfo, idObj oid.ID, auth frostfs.PrmAuth) error {
prm := PrmObjectDelete{ prm := frostfs.PrmObjectDelete{
PrmAuth: auth, PrmAuth: auth,
Container: bktInfo.CID, Container: bktInfo.CID,
Object: idObj, Object: idObj,
@ -492,7 +512,7 @@ func (n *Layer) objectDeleteBase(ctx context.Context, bktInfo *data.BucketInfo,
} }
// objectPutAndHash prepare auth parameters and invoke frostfs.CreateObject. // objectPutAndHash prepare auth parameters and invoke frostfs.CreateObject.
func (n *Layer) objectPutAndHash(ctx context.Context, prm PrmObjectCreate, bktInfo *data.BucketInfo) (*data.CreatedObjectInfo, error) { func (n *Layer) objectPutAndHash(ctx context.Context, prm frostfs.PrmObjectCreate, bktInfo *data.BucketInfo) (*data.CreatedObjectInfo, error) {
n.prepareAuthParameters(ctx, &prm.PrmAuth, bktInfo.Owner) n.prepareAuthParameters(ctx, &prm.PrmAuth, bktInfo.Owner)
prm.ClientCut = n.features.ClientCut() prm.ClientCut = n.features.ClientCut()
prm.BufferMaxSize = n.features.BufferMaxSizeForPut() prm.BufferMaxSize = n.features.BufferMaxSizeForPut()

View file

@ -8,6 +8,7 @@ import (
"io" "io"
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@ -37,7 +38,7 @@ func TestGoroutinesDontLeakInPutAndHash(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
payload := bytes.NewReader(content) payload := bytes.NewReader(content)
prm := PrmObjectCreate{ prm := frostfs.PrmObjectCreate{
Filepath: tc.obj, Filepath: tc.obj,
Payload: payload, Payload: payload,
} }

View file

@ -12,6 +12,7 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
) )
type PatchObjectParams struct { type PatchObjectParams struct {
@ -32,7 +33,7 @@ func (n *Layer) PatchObject(ctx context.Context, p *PatchObjectParams) (*data.Ex
return n.patchMultipartObject(ctx, p) return n.patchMultipartObject(ctx, p)
} }
prmPatch := PrmObjectPatch{ prmPatch := frostfs.PrmObjectPatch{
Container: p.BktInfo.CID, Container: p.BktInfo.CID,
Object: p.Object.ObjectInfo.ID, Object: p.Object.ObjectInfo.ID,
Payload: p.NewBytes, Payload: p.NewBytes,
@ -74,13 +75,13 @@ func (n *Layer) PatchObject(ctx context.Context, p *PatchObjectParams) (*data.Ex
return p.Object, nil return p.Object, nil
} }
func (n *Layer) patchObject(ctx context.Context, p PrmObjectPatch) (*data.CreatedObjectInfo, error) { func (n *Layer) patchObject(ctx context.Context, p frostfs.PrmObjectPatch) (*data.CreatedObjectInfo, error) {
objID, err := n.frostFS.PatchObject(ctx, p) objID, err := n.frostFS.PatchObject(ctx, p)
if err != nil { if err != nil {
return nil, fmt.Errorf("patch object: %w", err) return nil, fmt.Errorf("patch object: %w", err)
} }
prmHead := PrmObjectHead{ prmHead := frostfs.PrmObjectHead{
PrmAuth: p.PrmAuth, PrmAuth: p.PrmAuth,
Container: p.Container, Container: p.Container,
Object: objID, Object: objID,
@ -110,7 +111,7 @@ func (n *Layer) patchMultipartObject(ctx context.Context, p *PatchObjectParams)
return nil, fmt.Errorf("unmarshal combined object parts: %w", err) return nil, fmt.Errorf("unmarshal combined object parts: %w", err)
} }
prmPatch := PrmObjectPatch{ prmPatch := frostfs.PrmObjectPatch{
Container: p.BktInfo.CID, Container: p.BktInfo.CID,
} }
n.prepareAuthParameters(ctx, &prmPatch.PrmAuth, p.BktInfo.Owner) n.prepareAuthParameters(ctx, &prmPatch.PrmAuth, p.BktInfo.Owner)
@ -144,13 +145,13 @@ func (n *Layer) patchMultipartObject(ctx context.Context, p *PatchObjectParams)
} }
// Returns patched part info, updated offset and length. // Returns patched part info, updated offset and length.
func (n *Layer) patchPart(ctx context.Context, part *data.PartInfo, p *PatchObjectParams, prmPatch *PrmObjectPatch, off, ln uint64, lastPart bool) (*data.CreatedObjectInfo, uint64, uint64, error) { func (n *Layer) patchPart(ctx context.Context, part *data.PartInfo, p *PatchObjectParams, prmPatch *frostfs.PrmObjectPatch, off, ln uint64, lastPart bool) (*data.CreatedObjectInfo, uint64, uint64, error) {
if off == 0 && ln >= part.Size { if off == 0 && ln >= part.Size {
curLen := part.Size curLen := part.Size
if lastPart { if lastPart {
curLen = ln curLen = ln
} }
prm := PrmObjectCreate{ prm := frostfs.PrmObjectCreate{
Container: p.BktInfo.CID, Container: p.BktInfo.CID,
Payload: io.LimitReader(p.NewBytes, int64(curLen)), Payload: io.LimitReader(p.NewBytes, int64(curLen)),
CreationTime: part.Created, CreationTime: part.Created,
@ -204,7 +205,7 @@ func (n *Layer) updateCombinedObject(ctx context.Context, parts []*data.PartInfo
headerParts.WriteString(headerPart) headerParts.WriteString(headerPart)
} }
prm := PrmObjectCreate{ prm := frostfs.PrmObjectCreate{
Container: p.BktInfo.CID, Container: p.BktInfo.CID,
PayloadSize: fullObjSize, PayloadSize: fullObjSize,
Filepath: p.Object.ObjectInfo.Name, Filepath: p.Object.ObjectInfo.Name,

View file

@ -3,7 +3,7 @@ package layer
import ( import (
"context" "context"
"encoding/xml" "encoding/xml"
errorsStd "errors" "errors"
"fmt" "fmt"
"math" "math"
"strconv" "strconv"
@ -11,7 +11,9 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object" "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
) )
@ -40,7 +42,7 @@ func (n *Layer) PutLockInfo(ctx context.Context, p *PutLockInfoParams) (err erro
} }
lockInfo, err := n.treeService.GetLock(ctx, p.ObjVersion.BktInfo, versionNode.ID) lockInfo, err := n.treeService.GetLock(ctx, p.ObjVersion.BktInfo, versionNode.ID)
if err != nil && !errorsStd.Is(err, ErrNodeNotFound) { if err != nil && !errors.Is(err, tree.ErrNodeNotFound) {
return err return err
} }
@ -113,7 +115,7 @@ func (n *Layer) getNodeVersionFromCacheOrFrostfs(ctx context.Context, objVersion
} }
func (n *Layer) putLockObject(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID, lock *data.ObjectLock, copiesNumber []uint32) (oid.ID, error) { func (n *Layer) putLockObject(ctx context.Context, bktInfo *data.BucketInfo, objID oid.ID, lock *data.ObjectLock, copiesNumber []uint32) (oid.ID, error) {
prm := PrmObjectCreate{ prm := frostfs.PrmObjectCreate{
Container: bktInfo.CID, Container: bktInfo.CID,
Locks: []oid.ID{objID}, Locks: []oid.ID{objID},
CreationTime: TimeNow(ctx), CreationTime: TimeNow(ctx),
@ -146,7 +148,7 @@ func (n *Layer) GetLockInfo(ctx context.Context, objVersion *data.ObjectVersion)
} }
lockInfo, err := n.treeService.GetLock(ctx, objVersion.BktInfo, versionNode.ID) lockInfo, err := n.treeService.GetLock(ctx, objVersion.BktInfo, versionNode.ID)
if err != nil && !errorsStd.Is(err, ErrNodeNotFound) { if err != nil && !errors.Is(err, tree.ErrNodeNotFound) {
return nil, err return nil, err
} }
if lockInfo == nil { if lockInfo == nil {
@ -165,16 +167,16 @@ func (n *Layer) getCORS(ctx context.Context, bkt *data.BucketInfo) (*data.CORSCo
} }
addr, err := n.treeService.GetBucketCORS(ctx, bkt) addr, err := n.treeService.GetBucketCORS(ctx, bkt)
objNotFound := errorsStd.Is(err, ErrNodeNotFound) objNotFound := errors.Is(err, tree.ErrNodeNotFound)
if err != nil && !objNotFound { if err != nil && !objNotFound {
return nil, err return nil, err
} }
if objNotFound { if objNotFound {
return nil, fmt.Errorf("%w: %s", errors.GetAPIError(errors.ErrNoSuchCORSConfiguration), err.Error()) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchCORSConfiguration), err.Error())
} }
var prmAuth PrmAuth var prmAuth frostfs.PrmAuth
corsBkt := bkt corsBkt := bkt
if !addr.Container().Equals(bkt.CID) && !addr.Container().Equals(cid.ID{}) { if !addr.Container().Equals(bkt.CID) && !addr.Container().Equals(cid.ID{}) {
corsBkt = &data.BucketInfo{CID: addr.Container()} corsBkt = &data.BucketInfo{CID: addr.Container()}
@ -209,7 +211,7 @@ func (n *Layer) GetBucketSettings(ctx context.Context, bktInfo *data.BucketInfo)
settings, err := n.treeService.GetSettingsNode(ctx, bktInfo) settings, err := n.treeService.GetSettingsNode(ctx, bktInfo)
if err != nil { if err != nil {
if !errorsStd.Is(err, ErrNodeNotFound) { if !errors.Is(err, tree.ErrNodeNotFound) {
return nil, err return nil, err
} }
settings = &data.BucketSettings{Versioning: data.VersioningUnversioned} settings = &data.BucketSettings{Versioning: data.VersioningUnversioned}

View file

@ -6,7 +6,8 @@ import (
"fmt" "fmt"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
s3errors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
@ -39,8 +40,8 @@ func (n *Layer) GetObjectTagging(ctx context.Context, p *data.GetObjectTaggingPa
tags, err := n.treeService.GetObjectTagging(ctx, p.ObjectVersion.BktInfo, nodeVersion) tags, err := n.treeService.GetObjectTagging(ctx, p.ObjectVersion.BktInfo, nodeVersion)
if err != nil { if err != nil {
if errors.Is(err, ErrNodeNotFound) { if errors.Is(err, tree.ErrNodeNotFound) {
return "", nil, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchKey), err.Error()) return "", nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
} }
return "", nil, err return "", nil, err
} }
@ -62,8 +63,8 @@ func (n *Layer) PutObjectTagging(ctx context.Context, p *data.PutObjectTaggingPa
err = n.treeService.PutObjectTagging(ctx, p.ObjectVersion.BktInfo, nodeVersion, p.TagSet) err = n.treeService.PutObjectTagging(ctx, p.ObjectVersion.BktInfo, nodeVersion, p.TagSet)
if err != nil { if err != nil {
if errors.Is(err, ErrNodeNotFound) { if errors.Is(err, tree.ErrNodeNotFound) {
return fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchKey), err.Error()) return fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
} }
return err return err
} }
@ -81,8 +82,8 @@ func (n *Layer) DeleteObjectTagging(ctx context.Context, p *data.ObjectVersion)
err = n.treeService.DeleteObjectTagging(ctx, p.BktInfo, version) err = n.treeService.DeleteObjectTagging(ctx, p.BktInfo, version)
if err != nil { if err != nil {
if errors.Is(err, ErrNodeNotFound) { if errors.Is(err, tree.ErrNodeNotFound) {
return fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchKey), err.Error()) return fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
} }
return err return err
} }
@ -102,7 +103,7 @@ func (n *Layer) GetBucketTagging(ctx context.Context, bktInfo *data.BucketInfo)
} }
tags, err := n.treeService.GetBucketTagging(ctx, bktInfo) tags, err := n.treeService.GetBucketTagging(ctx, bktInfo)
if err != nil && !errors.Is(err, ErrNodeNotFound) { if err != nil && !errors.Is(err, tree.ErrNodeNotFound) {
return nil, err return nil, err
} }
@ -155,14 +156,14 @@ func (n *Layer) getNodeVersion(ctx context.Context, objVersion *data.ObjectVersi
} }
} }
if version == nil { if version == nil {
err = fmt.Errorf("%w: there isn't tree node with requested version id", s3errors.GetAPIError(s3errors.ErrNoSuchVersion)) err = fmt.Errorf("%w: there isn't tree node with requested version id", apierr.GetAPIError(apierr.ErrNoSuchVersion))
} }
} }
if err == nil && version.IsDeleteMarker && !objVersion.NoErrorOnDeleteMarker { if err == nil && version.IsDeleteMarker && !objVersion.NoErrorOnDeleteMarker {
return nil, fmt.Errorf("%w: found version is delete marker", s3errors.GetAPIError(s3errors.ErrNoSuchKey)) return nil, fmt.Errorf("%w: found version is delete marker", apierr.GetAPIError(apierr.ErrNoSuchKey))
} else if errors.Is(err, ErrNodeNotFound) { } else if errors.Is(err, tree.ErrNodeNotFound) {
return nil, fmt.Errorf("%w: %s", s3errors.GetAPIError(s3errors.ErrNoSuchKey), err.Error()) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrNoSuchKey), err.Error())
} }
if err == nil && version != nil && !version.IsDeleteMarker { if err == nil && version != nil && !version.IsDeleteMarker {

View file

@ -1,4 +1,4 @@
package layer package tree
import ( import (
"context" "context"
@ -8,8 +8,8 @@ import (
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
) )
// TreeService provide interface to interact with tree service using s3 data models. // Service provide interface to interact with tree service using s3 data models.
type TreeService interface { type Service interface {
// PutSettingsNode update or create new settings node in tree service. // PutSettingsNode update or create new settings node in tree service.
PutSettingsNode(ctx context.Context, bktInfo *data.BucketInfo, settings *data.BucketSettings) error PutSettingsNode(ctx context.Context, bktInfo *data.BucketInfo, settings *data.BucketSettings) error

View file

@ -9,6 +9,7 @@ import (
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/tree"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
) )
@ -105,7 +106,7 @@ func (t *TreeServiceMock) PutSettingsNode(_ context.Context, bktInfo *data.Bucke
func (t *TreeServiceMock) GetSettingsNode(_ context.Context, bktInfo *data.BucketInfo) (*data.BucketSettings, error) { func (t *TreeServiceMock) GetSettingsNode(_ context.Context, bktInfo *data.BucketInfo) (*data.BucketSettings, error) {
settings, ok := t.settings[bktInfo.CID.EncodeToString()] settings, ok := t.settings[bktInfo.CID.EncodeToString()]
if !ok { if !ok {
return nil, ErrNodeNotFound return nil, tree.ErrNodeNotFound
} }
return settings, nil return settings, nil
@ -140,7 +141,7 @@ func (t *TreeServiceMock) PutBucketCORS(_ context.Context, bktInfo *data.BucketI
t.system[bktInfo.CID.EncodeToString()] = systemMap t.system[bktInfo.CID.EncodeToString()] = systemMap
return nil, ErrNoNodeToRemove return nil, tree.ErrNoNodeToRemove
} }
func (t *TreeServiceMock) DeleteBucketCORS(context.Context, *data.BucketInfo) ([]oid.Address, error) { func (t *TreeServiceMock) DeleteBucketCORS(context.Context, *data.BucketInfo) ([]oid.Address, error) {
@ -150,12 +151,12 @@ func (t *TreeServiceMock) DeleteBucketCORS(context.Context, *data.BucketInfo) ([
func (t *TreeServiceMock) GetVersions(_ context.Context, bktInfo *data.BucketInfo, objectName string) ([]*data.NodeVersion, error) { func (t *TreeServiceMock) GetVersions(_ context.Context, bktInfo *data.BucketInfo, objectName string) ([]*data.NodeVersion, error) {
cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()] cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()]
if !ok { if !ok {
return nil, ErrNodeNotFound return nil, tree.ErrNodeNotFound
} }
versions, ok := cnrVersionsMap[objectName] versions, ok := cnrVersionsMap[objectName]
if !ok { if !ok {
return nil, ErrNodeNotFound return nil, tree.ErrNodeNotFound
} }
return versions, nil return versions, nil
@ -164,12 +165,12 @@ func (t *TreeServiceMock) GetVersions(_ context.Context, bktInfo *data.BucketInf
func (t *TreeServiceMock) GetLatestVersion(_ context.Context, bktInfo *data.BucketInfo, objectName string) (*data.NodeVersion, error) { func (t *TreeServiceMock) GetLatestVersion(_ context.Context, bktInfo *data.BucketInfo, objectName string) (*data.NodeVersion, error) {
cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()] cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()]
if !ok { if !ok {
return nil, ErrNodeNotFound return nil, tree.ErrNodeNotFound
} }
versions, ok := cnrVersionsMap[objectName] versions, ok := cnrVersionsMap[objectName]
if !ok { if !ok {
return nil, ErrNodeNotFound return nil, tree.ErrNodeNotFound
} }
sort.Slice(versions, func(i, j int) bool { sort.Slice(versions, func(i, j int) bool {
@ -180,13 +181,13 @@ func (t *TreeServiceMock) GetLatestVersion(_ context.Context, bktInfo *data.Buck
return versions[len(versions)-1], nil return versions[len(versions)-1], nil
} }
return nil, ErrNodeNotFound return nil, tree.ErrNodeNotFound
} }
func (t *TreeServiceMock) InitVersionsByPrefixStream(_ context.Context, bktInfo *data.BucketInfo, prefix string, latestOnly bool) (data.VersionsStream, error) { func (t *TreeServiceMock) InitVersionsByPrefixStream(_ context.Context, bktInfo *data.BucketInfo, prefix string, latestOnly bool) (data.VersionsStream, error) {
cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()] cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()]
if !ok { if !ok {
return nil, ErrNodeNotFound return nil, tree.ErrNodeNotFound
} }
var result []*data.NodeVersion var result []*data.NodeVersion
@ -218,12 +219,12 @@ func (t *TreeServiceMock) InitVersionsByPrefixStream(_ context.Context, bktInfo
func (t *TreeServiceMock) GetUnversioned(_ context.Context, bktInfo *data.BucketInfo, objectName string) (*data.NodeVersion, error) { func (t *TreeServiceMock) GetUnversioned(_ context.Context, bktInfo *data.BucketInfo, objectName string) (*data.NodeVersion, error) {
cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()] cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()]
if !ok { if !ok {
return nil, ErrNodeNotFound return nil, tree.ErrNodeNotFound
} }
versions, ok := cnrVersionsMap[objectName] versions, ok := cnrVersionsMap[objectName]
if !ok { if !ok {
return nil, ErrNodeNotFound return nil, tree.ErrNodeNotFound
} }
for _, version := range versions { for _, version := range versions {
@ -232,7 +233,7 @@ func (t *TreeServiceMock) GetUnversioned(_ context.Context, bktInfo *data.Bucket
} }
} }
return nil, ErrNodeNotFound return nil, tree.ErrNodeNotFound
} }
func (t *TreeServiceMock) AddVersion(_ context.Context, bktInfo *data.BucketInfo, newVersion *data.NodeVersion) (uint64, error) { func (t *TreeServiceMock) AddVersion(_ context.Context, bktInfo *data.BucketInfo, newVersion *data.NodeVersion) (uint64, error) {
@ -278,7 +279,7 @@ func (t *TreeServiceMock) AddVersion(_ context.Context, bktInfo *data.BucketInfo
func (t *TreeServiceMock) RemoveVersion(_ context.Context, bktInfo *data.BucketInfo, nodeID uint64) error { func (t *TreeServiceMock) RemoveVersion(_ context.Context, bktInfo *data.BucketInfo, nodeID uint64) error {
cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()] cnrVersionsMap, ok := t.versions[bktInfo.CID.EncodeToString()]
if !ok { if !ok {
return ErrNodeNotFound return tree.ErrNodeNotFound
} }
for key, versions := range cnrVersionsMap { for key, versions := range cnrVersionsMap {
@ -290,7 +291,7 @@ func (t *TreeServiceMock) RemoveVersion(_ context.Context, bktInfo *data.BucketI
} }
} }
return ErrNodeNotFound return tree.ErrNodeNotFound
} }
func (t *TreeServiceMock) GetAllVersionsByPrefix(_ context.Context, bktInfo *data.BucketInfo, prefix string) ([]*data.NodeVersion, error) { func (t *TreeServiceMock) GetAllVersionsByPrefix(_ context.Context, bktInfo *data.BucketInfo, prefix string) ([]*data.NodeVersion, error) {
@ -334,7 +335,7 @@ func (t *TreeServiceMock) GetMultipartUploadsByPrefix(context.Context, *data.Buc
func (t *TreeServiceMock) GetMultipartUpload(_ context.Context, bktInfo *data.BucketInfo, objectName, uploadID string) (*data.MultipartInfo, error) { func (t *TreeServiceMock) GetMultipartUpload(_ context.Context, bktInfo *data.BucketInfo, objectName, uploadID string) (*data.MultipartInfo, error) {
cnrMultipartsMap, ok := t.multiparts[bktInfo.CID.EncodeToString()] cnrMultipartsMap, ok := t.multiparts[bktInfo.CID.EncodeToString()]
if !ok { if !ok {
return nil, ErrNodeNotFound return nil, tree.ErrNodeNotFound
} }
multiparts := cnrMultipartsMap[objectName] multiparts := cnrMultipartsMap[objectName]
@ -344,7 +345,7 @@ func (t *TreeServiceMock) GetMultipartUpload(_ context.Context, bktInfo *data.Bu
} }
} }
return nil, ErrNodeNotFound return nil, tree.ErrNodeNotFound
} }
func (t *TreeServiceMock) AddPart(ctx context.Context, bktInfo *data.BucketInfo, multipartNodeID uint64, info *data.PartInfo) (oldObjIDsToDelete []oid.ID, err error) { func (t *TreeServiceMock) AddPart(ctx context.Context, bktInfo *data.BucketInfo, multipartNodeID uint64, info *data.PartInfo) (oldObjIDsToDelete []oid.ID, err error) {
@ -387,7 +388,7 @@ LOOP:
} }
if foundMultipart == nil { if foundMultipart == nil {
return nil, ErrNodeNotFound return nil, tree.ErrNodeNotFound
} }
partsMap := t.parts[foundMultipart.UploadID] partsMap := t.parts[foundMultipart.UploadID]
@ -411,18 +412,18 @@ func (t *TreeServiceMock) PutBucketLifecycleConfiguration(_ context.Context, bkt
t.system[bktInfo.CID.EncodeToString()] = systemMap t.system[bktInfo.CID.EncodeToString()] = systemMap
return nil, ErrNoNodeToRemove return nil, tree.ErrNoNodeToRemove
} }
func (t *TreeServiceMock) GetBucketLifecycleConfiguration(_ context.Context, bktInfo *data.BucketInfo) (oid.Address, error) { func (t *TreeServiceMock) GetBucketLifecycleConfiguration(_ context.Context, bktInfo *data.BucketInfo) (oid.Address, error) {
systemMap, ok := t.system[bktInfo.CID.EncodeToString()] systemMap, ok := t.system[bktInfo.CID.EncodeToString()]
if !ok { if !ok {
return oid.Address{}, ErrNodeNotFound return oid.Address{}, tree.ErrNodeNotFound
} }
node, ok := systemMap["lifecycle"] node, ok := systemMap["lifecycle"]
if !ok { if !ok {
return oid.Address{}, ErrNodeNotFound return oid.Address{}, tree.ErrNodeNotFound
} }
return newAddress(bktInfo.CID, node.OID), nil return newAddress(bktInfo.CID, node.OID), nil
@ -431,12 +432,12 @@ func (t *TreeServiceMock) GetBucketLifecycleConfiguration(_ context.Context, bkt
func (t *TreeServiceMock) DeleteBucketLifecycleConfiguration(_ context.Context, bktInfo *data.BucketInfo) ([]oid.Address, error) { func (t *TreeServiceMock) DeleteBucketLifecycleConfiguration(_ context.Context, bktInfo *data.BucketInfo) ([]oid.Address, error) {
systemMap, ok := t.system[bktInfo.CID.EncodeToString()] systemMap, ok := t.system[bktInfo.CID.EncodeToString()]
if !ok { if !ok {
return nil, ErrNoNodeToRemove return nil, tree.ErrNoNodeToRemove
} }
node, ok := systemMap["lifecycle"] node, ok := systemMap["lifecycle"]
if !ok { if !ok {
return nil, ErrNoNodeToRemove return nil, tree.ErrNoNodeToRemove
} }
delete(systemMap, "lifecycle") delete(systemMap, "lifecycle")
@ -461,7 +462,7 @@ LOOP:
} }
if uploadID == "" { if uploadID == "" {
return ErrNodeNotFound return tree.ErrNodeNotFound
} }
delete(t.parts, uploadID) delete(t.parts, uploadID)

View file

@ -7,6 +7,7 @@ import (
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
bearertest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer/test" bearertest "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer/test"
@ -23,7 +24,7 @@ func (tc *testContext) putObject(content []byte) *data.ObjectInfo {
extObjInfo, err := tc.layer.PutObject(tc.ctx, &PutObjectParams{ extObjInfo, err := tc.layer.PutObject(tc.ctx, &PutObjectParams{
BktInfo: tc.bktInfo, BktInfo: tc.bktInfo,
Object: tc.obj, Object: tc.obj,
Size: uint64(len(content)), Size: ptr(uint64(len(content))),
Reader: bytes.NewReader(content), Reader: bytes.NewReader(content),
Header: make(map[string]string), Header: make(map[string]string),
}) })
@ -154,7 +155,7 @@ func prepareContext(t *testing.T, cachesConfig ...*CachesConfig) *testContext {
tp := NewTestFrostFS(key) tp := NewTestFrostFS(key)
bktName := "testbucket1" bktName := "testbucket1"
res, err := tp.CreateContainer(ctx, PrmContainerCreate{ res, err := tp.CreateContainer(ctx, frostfs.PrmContainerCreate{
Name: bktName, Name: bktName,
}) })
require.NoError(t, err) require.NoError(t, err)

View file

@ -3,14 +3,18 @@ package middleware
import ( import (
"net/http" "net/http"
"net/url" "net/url"
"strconv"
"strings" "strings"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"go.uber.org/zap" "go.uber.org/zap"
) )
const wildcardPlaceholder = "<wildcard>" const (
wildcardPlaceholder = "<wildcard>"
enabledVHS = "enabled"
disabledVHS = "disabled"
)
type VHSSettings interface { type VHSSettings interface {
Domains() []string Domains() []string
@ -26,9 +30,9 @@ func PrepareAddressStyle(settings VHSSettings, log *zap.Logger) Func {
ctx := r.Context() ctx := r.Context()
reqInfo := GetReqInfo(ctx) reqInfo := GetReqInfo(ctx)
reqLogger := reqLogOrDefault(ctx, log) reqLogger := reqLogOrDefault(ctx, log)
headerVHSEnabled := r.Header.Get(settings.VHSHeader()) statusVHS := r.Header.Get(settings.VHSHeader())
if isVHSAddress(headerVHSEnabled, settings.GlobalVHS(), settings.VHSNamespacesEnabled(), reqInfo.Namespace) { if isVHSAddress(statusVHS, settings.GlobalVHS(), settings.VHSNamespacesEnabled(), reqInfo.Namespace) {
prepareVHSAddress(reqInfo, r, settings) prepareVHSAddress(reqInfo, r, settings)
} else { } else {
preparePathStyleAddress(reqInfo, r, reqLogger) preparePathStyleAddress(reqInfo, r, reqLogger)
@ -39,11 +43,13 @@ func PrepareAddressStyle(settings VHSSettings, log *zap.Logger) Func {
} }
} }
func isVHSAddress(headerVHSEnabled string, enabledFlag bool, vhsNamespaces map[string]bool, namespace string) bool { func isVHSAddress(statusVHS string, enabledFlag bool, vhsNamespaces map[string]bool, namespace string) bool {
if result, err := strconv.ParseBool(headerVHSEnabled); err == nil { switch statusVHS {
return result case enabledVHS:
} return true
case disabledVHS:
return false
default:
result := enabledFlag result := enabledFlag
if v, ok := vhsNamespaces[namespace]; ok { if v, ok := vhsNamespaces[namespace]; ok {
result = v result = v
@ -51,6 +57,7 @@ func isVHSAddress(headerVHSEnabled string, enabledFlag bool, vhsNamespaces map[s
return result return result
} }
}
func prepareVHSAddress(reqInfo *ReqInfo, r *http.Request, settings VHSSettings) { func prepareVHSAddress(reqInfo *ReqInfo, r *http.Request, settings VHSSettings) {
reqInfo.RequestVHSEnabled = true reqInfo.RequestVHSEnabled = true

View file

@ -42,7 +42,7 @@ func (v *VHSSettingsMock) VHSNamespacesEnabled() map[string]bool {
func TestIsVHSAddress(t *testing.T) { func TestIsVHSAddress(t *testing.T) {
for _, tc := range []struct { for _, tc := range []struct {
name string name string
headerVHSEnabled string headerStatusVHS string
vhsEnabledFlag bool vhsEnabledFlag bool
vhsNamespaced map[string]bool vhsNamespaced map[string]bool
namespace string namespace string
@ -76,7 +76,7 @@ func TestIsVHSAddress(t *testing.T) {
}, },
{ {
name: "vhs enabled (header)", name: "vhs enabled (header)",
headerVHSEnabled: "true", headerStatusVHS: enabledVHS,
vhsEnabledFlag: false, vhsEnabledFlag: false,
vhsNamespaced: map[string]bool{ vhsNamespaced: map[string]bool{
"kapusta": false, "kapusta": false,
@ -86,7 +86,7 @@ func TestIsVHSAddress(t *testing.T) {
}, },
{ {
name: "vhs disabled (header)", name: "vhs disabled (header)",
headerVHSEnabled: "false", headerStatusVHS: disabledVHS,
vhsEnabledFlag: true, vhsEnabledFlag: true,
vhsNamespaced: map[string]bool{ vhsNamespaced: map[string]bool{
"kapusta": true, "kapusta": true,
@ -96,7 +96,7 @@ func TestIsVHSAddress(t *testing.T) {
}, },
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
actual := isVHSAddress(tc.headerVHSEnabled, tc.vhsEnabledFlag, tc.vhsNamespaced, tc.namespace) actual := isVHSAddress(tc.headerStatusVHS, tc.vhsEnabledFlag, tc.vhsNamespaced, tc.namespace)
require.Equal(t, tc.expected, actual) require.Equal(t, tc.expected, actual)
}) })
} }

View file

@ -8,9 +8,8 @@ import (
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/acl" "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/acl"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
frostfsErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
@ -57,9 +56,9 @@ func Auth(center Center, log *zap.Logger) Func {
reqLogOrDefault(ctx, log).Debug(logs.CouldntReceiveAccessBoxForGateKeyRandomKeyWillBeUsed, zap.Error(err)) reqLogOrDefault(ctx, log).Debug(logs.CouldntReceiveAccessBoxForGateKeyRandomKeyWillBeUsed, zap.Error(err))
} else { } else {
reqLogOrDefault(ctx, log).Error(logs.FailedToPassAuthentication, zap.Error(err)) reqLogOrDefault(ctx, log).Error(logs.FailedToPassAuthentication, zap.Error(err))
err = frostfsErrors.UnwrapErr(err) err = apierr.TransformToS3Error(err)
if _, ok := err.(apiErrors.Error); !ok { if err.(apierr.Error).ErrCode == apierr.ErrInternalError {
err = apiErrors.GetAPIError(apiErrors.ErrAccessDenied) err = apierr.GetAPIError(apierr.ErrAccessDenied)
} }
if _, wrErr := WriteErrorResponse(w, GetReqInfo(r.Context()), err); wrErr != nil { if _, wrErr := WriteErrorResponse(w, GetReqInfo(r.Context()), err); wrErr != nil {
reqLogOrDefault(ctx, log).Error(logs.FailedToWriteResponse, zap.Error(wrErr)) reqLogOrDefault(ctx, log).Error(logs.FailedToWriteResponse, zap.Error(wrErr))

View file

@ -107,3 +107,9 @@ const (
PartNumberQuery = "partNumber" PartNumberQuery = "partNumber"
LegalHoldQuery = "legal-hold" LegalHoldQuery = "legal-hold"
) )
const (
StdoutPath = "stdout"
StderrPath = "stderr"
SinkName = "lumberjack"
)

237
api/middleware/log_http.go Normal file
View file

@ -0,0 +1,237 @@
//go:build loghttp
package middleware
import (
"bytes"
"io"
"net/http"
"os"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/detector"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/xmlutils"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
"gopkg.in/natefinch/lumberjack.v2"
)
type (
LogHTTPSettings interface {
LogHTTPConfig() LogHTTPConfig
}
LogHTTPConfig struct {
Enabled bool
MaxBody int64
MaxLogSize int
OutputPath string
UseGzip bool
log *httpLogger
}
httpLogger struct {
*zap.Logger
logRoller *lumberjack.Logger
}
// responseReadWriter helps read http response body.
responseReadWriter struct {
http.ResponseWriter
response *bytes.Buffer
statusCode int
}
)
const (
payloadLabel = "payload"
responseLabel = "response"
)
func (lc *LogHTTPConfig) InitHTTPLogger(log *zap.Logger) {
if err := lc.initHTTPLogger(); err != nil {
log.Error(logs.FailedToInitializeHTTPLogger, zap.Error(err))
}
}
// initHTTPLogger returns registers zap sink and returns new httpLogger.
func (lc *LogHTTPConfig) initHTTPLogger() (err error) {
lc.log = &httpLogger{
Logger: zap.NewNop(),
logRoller: &lumberjack.Logger{},
}
c := newLoggerConfig()
lc.log.Logger, err = c.Build()
if err != nil {
return err
}
lc.setLogOutput()
return nil
}
// newLoggerConfig creates new zap.Config with disabled base fields.
func newLoggerConfig() zap.Config {
c := zap.NewProductionConfig()
c.DisableCaller = true
c.DisableStacktrace = true
c.EncoderConfig = newEncoderConfig()
c.Sampling = nil
return c
}
func (lc *LogHTTPConfig) setLogOutput() {
var output zapcore.WriteSyncer
switch lc.OutputPath {
case "", StdoutPath:
output = zapcore.AddSync(os.Stdout)
case StderrPath:
output = zapcore.AddSync(os.Stderr)
default:
output = zapcore.AddSync(&lumberjack.Logger{
Filename: lc.OutputPath,
MaxSize: lc.MaxLogSize,
Compress: lc.UseGzip,
})
}
// create logger with new sync
lc.log.Logger = lc.log.Logger.WithOptions(zap.WrapCore(func(core zapcore.Core) zapcore.Core {
return zapcore.NewCore(zapcore.NewJSONEncoder(newEncoderConfig()), output, zapcore.InfoLevel)
}))
}
func newEncoderConfig() zapcore.EncoderConfig {
c := zap.NewProductionEncoderConfig()
c.MessageKey = zapcore.OmitKey
c.LevelKey = zapcore.OmitKey
c.TimeKey = zapcore.OmitKey
c.FunctionKey = zapcore.OmitKey
return c
}
func (ww *responseReadWriter) Write(data []byte) (int, error) {
ww.response.Write(data)
return ww.ResponseWriter.Write(data)
}
func (ww *responseReadWriter) WriteHeader(code int) {
ww.statusCode = code
ww.ResponseWriter.WriteHeader(code)
}
func (ww *responseReadWriter) Flush() {
if f, ok := ww.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
// LogHTTP logs http parameters from s3 request.
func LogHTTP(l *zap.Logger, settings LogHTTPSettings) Func {
return func(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
config := settings.LogHTTPConfig()
if !config.Enabled || config.log == nil {
h.ServeHTTP(w, r)
return
}
httplog := config.log.getHTTPLogger(r).
withFieldIfExist("query", r.URL.Query()).
withFieldIfExist("headers", r.Header)
payload := getBody(r.Body, l)
r.Body = io.NopCloser(bytes.NewReader(payload))
payloadReader := io.LimitReader(bytes.NewReader(payload), config.MaxBody)
httplog = httplog.withProcessedBody(payloadLabel, payloadReader, l)
wr := newResponseReadWriter(w)
h.ServeHTTP(wr, r)
respReader := io.LimitReader(wr.response, config.MaxBody)
httplog = httplog.withProcessedBody(responseLabel, respReader, l)
httplog = httplog.with(zap.Int("status", wr.statusCode))
httplog.Info(logs.LogHTTP)
})
}
}
// withFieldIfExist checks whether data is not empty and attach it to log output.
func (lg *httpLogger) withFieldIfExist(label string, data map[string][]string) *httpLogger {
if len(data) != 0 {
return lg.with(zap.Any(label, data))
}
return lg
}
func (lg *httpLogger) with(fields ...zap.Field) *httpLogger {
return &httpLogger{
Logger: lg.Logger.With(fields...),
logRoller: lg.logRoller,
}
}
func (lg *httpLogger) getHTTPLogger(r *http.Request) *httpLogger {
return lg.with(
zap.String("from", r.RemoteAddr),
zap.String("URI", r.RequestURI),
zap.String("method", r.Method),
zap.String("protocol", r.Proto),
)
}
func (lg *httpLogger) withProcessedBody(label string, bodyReader io.Reader, l *zap.Logger) *httpLogger {
resp, err := processBody(bodyReader)
if err != nil {
l.Error(logs.FailedToProcessHTTPBody,
zap.Error(err),
zap.String("body type", payloadLabel))
return lg
}
return lg.with(zap.ByteString(label, resp))
}
func newResponseReadWriter(w http.ResponseWriter) *responseReadWriter {
return &responseReadWriter{
ResponseWriter: w,
response: &bytes.Buffer{},
}
}
func getBody(httpBody io.ReadCloser, l *zap.Logger) []byte {
defer func(httpBody io.ReadCloser) {
if err := httpBody.Close(); err != nil {
l.Error(logs.FailedToCloseHTTPBody, zap.Error(err))
}
}(httpBody)
body, err := io.ReadAll(httpBody)
if err != nil {
l.Error(logs.FailedToReadHTTPBody,
zap.Error(err),
zap.String("body type", payloadLabel))
return nil
}
return body
}
// processBody reads body and base64 encode it if it's not XML.
func processBody(bodyReader io.Reader) ([]byte, error) {
resultBody := &bytes.Buffer{}
detect := detector.NewDetector(bodyReader, xmlutils.DetectXML)
dataType, err := detect.Detect()
if err != nil {
return nil, err
}
writer := xmlutils.ChooseWriter(dataType, resultBody)
if _, err = io.Copy(writer, detect.RestoredReader()); err != nil {
return nil, err
}
if err = writer.Close(); err != nil {
return nil, err
}
return resultBody.Bytes(), nil
}

View file

@ -0,0 +1,36 @@
//go:build !loghttp
package middleware
import (
"net/http"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"go.uber.org/zap"
)
type (
LogHTTPSettings interface {
LogHTTPConfig() LogHTTPConfig
}
LogHTTPConfig struct {
Enabled bool
MaxBody int64
MaxLogSize int
OutputPath string
UseGzip bool
}
)
func LogHTTP(l *zap.Logger, _ LogHTTPSettings) Func {
l.Warn(logs.LogHTTPDisabledInThisBuild)
return func(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
h.ServeHTTP(w, r)
})
}
}
func (*LogHTTPConfig) InitHTTPLogger(*zap.Logger) {
// ignore
}

View file

@ -11,8 +11,7 @@ import (
"strings" "strings"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
frostfsErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain" "git.frostfs.info/TrueCloudLab/policy-engine/pkg/chain"
"git.frostfs.info/TrueCloudLab/policy-engine/pkg/engine" "git.frostfs.info/TrueCloudLab/policy-engine/pkg/engine"
@ -29,7 +28,11 @@ const (
QueryPrefix = "prefix" QueryPrefix = "prefix"
QueryDelimiter = "delimiter" QueryDelimiter = "delimiter"
QueryMaxKeys = "max-keys" QueryMaxKeys = "max-keys"
QueryMarker = "marker"
QueryEncodingType = "encoding-type"
amzTagging = "x-amz-tagging" amzTagging = "x-amz-tagging"
unmatchedBucketOperation = "UnmatchedBucketOperation"
) )
// In these operations we don't check resource tags because // In these operations we don't check resource tags because
@ -85,7 +88,7 @@ func PolicyCheck(cfg PolicyConfig) Func {
ctx := r.Context() ctx := r.Context()
if err := policyCheck(r, cfg); err != nil { if err := policyCheck(r, cfg); err != nil {
reqLogOrDefault(ctx, cfg.Log).Error(logs.PolicyValidationFailed, zap.Error(err)) reqLogOrDefault(ctx, cfg.Log).Error(logs.PolicyValidationFailed, zap.Error(err))
err = frostfsErrors.UnwrapErr(err) err = apierr.TransformToS3Error(err)
if _, wrErr := WriteErrorResponse(w, GetReqInfo(ctx), err); wrErr != nil { if _, wrErr := WriteErrorResponse(w, GetReqInfo(ctx), err); wrErr != nil {
reqLogOrDefault(ctx, cfg.Log).Error(logs.FailedToWriteResponse, zap.Error(wrErr)) reqLogOrDefault(ctx, cfg.Log).Error(logs.FailedToWriteResponse, zap.Error(wrErr))
} }
@ -145,11 +148,11 @@ func policyCheck(r *http.Request, cfg PolicyConfig) error {
case st == chain.Allow: case st == chain.Allow:
return nil return nil
case st != chain.NoRuleFound: case st != chain.NoRuleFound:
return apiErr.GetAPIErrorWithError(apiErr.ErrAccessDenied, fmt.Errorf("policy check: %s", st.String())) return apierr.GetAPIErrorWithError(apierr.ErrAccessDenied, fmt.Errorf("policy check: %s", st.String()))
} }
if cfg.Settings.PolicyDenyByDefault() { if cfg.Settings.PolicyDenyByDefault() {
return apiErr.GetAPIErrorWithError(apiErr.ErrAccessDenied, fmt.Errorf("policy check: %s", st.String())) return apierr.GetAPIErrorWithError(apierr.ErrAccessDenied, fmt.Errorf("policy check: %s", st.String()))
} }
return nil return nil
@ -269,8 +272,17 @@ func determineBucketOperation(r *http.Request) string {
return ListObjectsV2MOperation return ListObjectsV2MOperation
case query.Get(ListTypeQuery) == "2": case query.Get(ListTypeQuery) == "2":
return ListObjectsV2Operation return ListObjectsV2Operation
default: case len(query) == 0 || func() bool {
for key := range query {
if key != QueryDelimiter && key != QueryMaxKeys && key != QueryPrefix && key != QueryMarker && key != QueryEncodingType {
return false
}
}
return true
}():
return ListObjectsV1Operation return ListObjectsV1Operation
default:
return unmatchedBucketOperation
} }
case http.MethodPut: case http.MethodPut:
switch { switch {
@ -292,8 +304,10 @@ func determineBucketOperation(r *http.Request) string {
return PutBucketVersioningOperation return PutBucketVersioningOperation
case query.Has(NotificationQuery): case query.Has(NotificationQuery):
return PutBucketNotificationOperation return PutBucketNotificationOperation
default: case len(query) == 0:
return CreateBucketOperation return CreateBucketOperation
default:
return unmatchedBucketOperation
} }
case http.MethodPost: case http.MethodPost:
switch { switch {
@ -316,12 +330,14 @@ func determineBucketOperation(r *http.Request) string {
return DeleteBucketLifecycleOperation return DeleteBucketLifecycleOperation
case query.Has(EncryptionQuery): case query.Has(EncryptionQuery):
return DeleteBucketEncryptionOperation return DeleteBucketEncryptionOperation
default: case len(query) == 0:
return DeleteBucketOperation return DeleteBucketOperation
default:
return unmatchedBucketOperation
} }
} }
return "UnmatchedBucketOperation" return unmatchedBucketOperation
} }
func determineObjectOperation(r *http.Request) string { func determineObjectOperation(r *http.Request) string {
@ -461,7 +477,7 @@ func determineRequestTags(r *http.Request, decoder XMLDecoder, op string) (map[s
if strings.HasSuffix(op, PutObjectTaggingOperation) || strings.HasSuffix(op, PutBucketTaggingOperation) { if strings.HasSuffix(op, PutObjectTaggingOperation) || strings.HasSuffix(op, PutBucketTaggingOperation) {
tagging := new(data.Tagging) tagging := new(data.Tagging)
if err := decoder.NewXMLDecoder(r.Body).Decode(tagging); err != nil { if err := decoder.NewXMLDecoder(r.Body).Decode(tagging); err != nil {
return nil, fmt.Errorf("%w: %s", apiErr.GetAPIError(apiErr.ErrMalformedXML), err.Error()) return nil, fmt.Errorf("%w: %s", apierr.GetAPIError(apierr.ErrMalformedXML), err.Error())
} }
GetReqInfo(r.Context()).Tagging = tagging GetReqInfo(r.Context()).Tagging = tagging
@ -473,7 +489,7 @@ func determineRequestTags(r *http.Request, decoder XMLDecoder, op string) (map[s
if tagging := r.Header.Get(amzTagging); len(tagging) > 0 { if tagging := r.Header.Get(amzTagging); len(tagging) > 0 {
queries, err := url.ParseQuery(tagging) queries, err := url.ParseQuery(tagging)
if err != nil { if err != nil {
return nil, apiErr.GetAPIError(apiErr.ErrInvalidArgument) return nil, apierr.GetAPIError(apierr.ErrInvalidArgument)
} }
for key := range queries { for key := range queries {
tags[fmt.Sprintf(s3.PropertyKeyFormatRequestTag, key)] = queries.Get(key) tags[fmt.Sprintf(s3.PropertyKeyFormatRequestTag, key)] = queries.Get(key)

View file

@ -152,6 +152,12 @@ func TestDetermineBucketOperation(t *testing.T) {
method: http.MethodGet, method: http.MethodGet,
expected: ListObjectsV1Operation, expected: ListObjectsV1Operation,
}, },
{
name: "UnmatchedBucketOperation GET",
method: http.MethodGet,
queryParam: map[string]string{"query": ""},
expected: unmatchedBucketOperation,
},
{ {
name: "PutBucketCorsOperation", name: "PutBucketCorsOperation",
method: http.MethodPut, method: http.MethodPut,
@ -211,6 +217,12 @@ func TestDetermineBucketOperation(t *testing.T) {
method: http.MethodPut, method: http.MethodPut,
expected: CreateBucketOperation, expected: CreateBucketOperation,
}, },
{
name: "UnmatchedBucketOperation PUT",
method: http.MethodPut,
queryParam: map[string]string{"query": ""},
expected: unmatchedBucketOperation,
},
{ {
name: "DeleteMultipleObjectsOperation", name: "DeleteMultipleObjectsOperation",
method: http.MethodPost, method: http.MethodPost,
@ -263,10 +275,16 @@ func TestDetermineBucketOperation(t *testing.T) {
method: http.MethodDelete, method: http.MethodDelete,
expected: DeleteBucketOperation, expected: DeleteBucketOperation,
}, },
{
name: "UnmatchedBucketOperation DELETE",
method: http.MethodDelete,
queryParam: map[string]string{"query": ""},
expected: unmatchedBucketOperation,
},
{ {
name: "UnmatchedBucketOperation", name: "UnmatchedBucketOperation",
method: "invalid-method", method: "invalid-method",
expected: "UnmatchedBucketOperation", expected: unmatchedBucketOperation,
}, },
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {

View file

@ -187,14 +187,21 @@ func Request(log *zap.Logger, settings RequestSettings) Func {
r = r.WithContext(treepool.SetRequestID(r.Context(), reqInfo.RequestID)) r = r.WithContext(treepool.SetRequestID(r.Context(), reqInfo.RequestID))
reqLogger := log.With(zap.String("request_id", reqInfo.RequestID)) fields := []zap.Field{zap.String("request_id", reqInfo.RequestID)}
r = r.WithContext(SetReqLogger(r.Context(), reqLogger)) ctx, span := StartHTTPServerSpan(r, "REQUEST S3")
if traceID := span.SpanContext().TraceID(); traceID.IsValid() {
fields = append(fields, zap.String("trace_id", traceID.String()))
}
lw := &traceResponseWriter{ResponseWriter: w, ctx: ctx, span: span}
reqLogger := log.With(fields...)
r = r.WithContext(SetReqLogger(ctx, reqLogger))
reqLogger.Info(logs.RequestStart, zap.String("host", r.Host), reqLogger.Info(logs.RequestStart, zap.String("host", r.Host),
zap.String("remote_host", reqInfo.RemoteHost), zap.String("namespace", reqInfo.Namespace)) zap.String("remote_host", reqInfo.RemoteHost), zap.String("namespace", reqInfo.Namespace))
// continue execution // continue execution
h.ServeHTTP(w, r) h.ServeHTTP(lw, r)
}) })
} }
} }

View file

@ -11,7 +11,6 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/version" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/version"
"go.opentelemetry.io/otel/trace"
"go.uber.org/zap" "go.uber.org/zap"
) )
@ -62,7 +61,7 @@ const (
hdrSSE = "X-Amz-Server-Side-Encryption" hdrSSE = "X-Amz-Server-Side-Encryption"
// hdrSSECustomerKey is the HTTP header key referencing the // hdrSSECustomerKey is the HTTP header key referencing the
// SSE-C client-provided key.. // SSE-C client-provided key.
hdrSSECustomerKey = hdrSSE + "-Customer-Key" hdrSSECustomerKey = hdrSSE + "-Customer-Key"
// hdrSSECopyKey is the HTTP header key referencing the SSE-C // hdrSSECopyKey is the HTTP header key referencing the SSE-C
@ -74,7 +73,7 @@ var (
xmlHeader = []byte(xml.Header) xmlHeader = []byte(xml.Header)
) )
// Non exhaustive list of AWS S3 standard error responses - // Non-exhaustive list of AWS S3 standard error responses -
// http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html // http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
var s3ErrorResponseMap = map[string]string{ var s3ErrorResponseMap = map[string]string{
"AccessDenied": "Access Denied.", "AccessDenied": "Access Denied.",
@ -332,10 +331,6 @@ func LogSuccessResponse(l *zap.Logger) Func {
fields = append(fields, zap.String("user", reqInfo.User)) fields = append(fields, zap.String("user", reqInfo.User))
} }
if traceID, err := trace.TraceIDFromHex(reqInfo.TraceID); err == nil && traceID.IsValid() {
fields = append(fields, zap.String("trace_id", reqInfo.TraceID))
}
reqLogger.Info(logs.RequestEnd, fields...) reqLogger.Info(logs.RequestEnd, fields...)
}) })
} }

View file

@ -11,20 +11,6 @@ import (
"go.opentelemetry.io/otel/trace" "go.opentelemetry.io/otel/trace"
) )
// Tracing adds tracing support for requests.
// Must be placed after prepareRequest middleware.
func Tracing() Func {
return func(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
appCtx, span := StartHTTPServerSpan(r, "REQUEST S3")
reqInfo := GetReqInfo(r.Context())
reqInfo.TraceID = span.SpanContext().TraceID().String()
lw := &traceResponseWriter{ResponseWriter: w, ctx: appCtx, span: span}
h.ServeHTTP(lw, r.WithContext(appCtx))
})
}
}
type traceResponseWriter struct { type traceResponseWriter struct {
sync.Once sync.Once
http.ResponseWriter http.ResponseWriter

View file

@ -99,6 +99,7 @@ type Settings interface {
s3middleware.PolicySettings s3middleware.PolicySettings
s3middleware.MetricsSettings s3middleware.MetricsSettings
s3middleware.VHSSettings s3middleware.VHSSettings
s3middleware.LogHTTPSettings
} }
type FrostFSID interface { type FrostFSID interface {
@ -127,11 +128,12 @@ type Config struct {
func NewRouter(cfg Config) *chi.Mux { func NewRouter(cfg Config) *chi.Mux {
api := chi.NewRouter() api := chi.NewRouter()
api.Use( api.Use(
s3middleware.LogHTTP(cfg.Log, cfg.MiddlewareSettings),
s3middleware.Request(cfg.Log, cfg.MiddlewareSettings), s3middleware.Request(cfg.Log, cfg.MiddlewareSettings),
middleware.ThrottleWithOpts(cfg.Throttle), middleware.ThrottleWithOpts(cfg.Throttle),
middleware.Recoverer, middleware.Recoverer,
s3middleware.Tracing(),
s3middleware.Metrics(cfg.Log, cfg.Handler.ResolveCID, cfg.Metrics, cfg.MiddlewareSettings), s3middleware.Metrics(cfg.Log, cfg.Handler.ResolveCID, cfg.Metrics, cfg.MiddlewareSettings),
s3middleware.LogSuccessResponse(cfg.Log), s3middleware.LogSuccessResponse(cfg.Log),
s3middleware.Auth(cfg.Center, cfg.Log), s3middleware.Auth(cfg.Center, cfg.Log),
@ -223,6 +225,28 @@ func errorResponseHandler(w http.ResponseWriter, r *http.Request) {
} }
} }
func notSupportedHandler() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
reqInfo := s3middleware.GetReqInfo(ctx)
_, wrErr := s3middleware.WriteErrorResponse(w, reqInfo, errors.GetAPIError(errors.ErrNotSupported))
if log := s3middleware.GetReqLog(ctx); log != nil {
fields := []zap.Field{
zap.String("http method", r.Method),
zap.String("url", r.RequestURI),
}
if wrErr != nil {
fields = append(fields, zap.NamedError("write_response_error", wrErr))
}
log.Error(logs.NotSupported, fields...)
}
}
}
// attachErrorHandler set NotFoundHandler and MethodNotAllowedHandler for chi.Router. // attachErrorHandler set NotFoundHandler and MethodNotAllowedHandler for chi.Router.
func attachErrorHandler(api *chi.Mux) { func attachErrorHandler(api *chi.Mux) {
errorHandler := http.HandlerFunc(errorResponseHandler) errorHandler := http.HandlerFunc(errorResponseHandler)
@ -310,7 +334,14 @@ func bucketRouter(h Handler) chi.Router {
Add(NewFilter(). Add(NewFilter().
Queries(s3middleware.VersionsQuery). Queries(s3middleware.VersionsQuery).
Handler(named(s3middleware.ListBucketObjectVersionsOperation, h.ListBucketObjectVersionsHandler))). Handler(named(s3middleware.ListBucketObjectVersionsOperation, h.ListBucketObjectVersionsHandler))).
DefaultHandler(listWrapper(h))) Add(NewFilter().
AllowedQueries(s3middleware.QueryDelimiter, s3middleware.QueryMaxKeys, s3middleware.QueryPrefix,
s3middleware.QueryMarker, s3middleware.QueryEncodingType).
Handler(named(s3middleware.ListObjectsV1Operation, h.ListObjectsV1Handler))).
Add(NewFilter().
NoQueries().
Handler(listWrapper(h))).
DefaultHandler(notSupportedHandler()))
}) })
// PUT method handlers // PUT method handlers
@ -343,7 +374,10 @@ func bucketRouter(h Handler) chi.Router {
Add(NewFilter(). Add(NewFilter().
Queries(s3middleware.NotificationQuery). Queries(s3middleware.NotificationQuery).
Handler(named(s3middleware.PutBucketNotificationOperation, h.PutBucketNotificationHandler))). Handler(named(s3middleware.PutBucketNotificationOperation, h.PutBucketNotificationHandler))).
DefaultHandler(named(s3middleware.CreateBucketOperation, h.CreateBucketHandler))) Add(NewFilter().
NoQueries().
Handler(named(s3middleware.CreateBucketOperation, h.CreateBucketHandler))).
DefaultHandler(notSupportedHandler()))
}) })
// POST method handlers // POST method handlers
@ -377,7 +411,10 @@ func bucketRouter(h Handler) chi.Router {
Add(NewFilter(). Add(NewFilter().
Queries(s3middleware.EncryptionQuery). Queries(s3middleware.EncryptionQuery).
Handler(named(s3middleware.DeleteBucketEncryptionOperation, h.DeleteBucketEncryptionHandler))). Handler(named(s3middleware.DeleteBucketEncryptionOperation, h.DeleteBucketEncryptionHandler))).
DefaultHandler(named(s3middleware.DeleteBucketOperation, h.DeleteBucketHandler))) Add(NewFilter().
NoQueries().
Handler(named(s3middleware.DeleteBucketOperation, h.DeleteBucketHandler))).
DefaultHandler(notSupportedHandler()))
}) })
attachErrorHandler(bktRouter) attachErrorHandler(bktRouter)

View file

@ -13,6 +13,8 @@ type HandlerFilters struct {
type Filter struct { type Filter struct {
queries []Pair queries []Pair
headers []Pair headers []Pair
allowedQueries map[string]struct{}
noQueries bool
h http.Handler h http.Handler
} }
@ -105,6 +107,22 @@ func (f *Filter) Queries(queries ...string) *Filter {
return f return f
} }
// NoQueries sets flag indicating that request shouldn't have query parameters.
func (f *Filter) NoQueries() *Filter {
f.noQueries = true
return f
}
// AllowedQueries adds query parameter keys that may be present in request.
func (f *Filter) AllowedQueries(queries ...string) *Filter {
f.allowedQueries = make(map[string]struct{}, len(queries))
for _, query := range queries {
f.allowedQueries[query] = struct{}{}
}
return f
}
func (hf *HandlerFilters) DefaultHandler(handler http.HandlerFunc) *HandlerFilters { func (hf *HandlerFilters) DefaultHandler(handler http.HandlerFunc) *HandlerFilters {
hf.defaultHandler = handler hf.defaultHandler = handler
return hf return hf
@ -122,6 +140,17 @@ func (hf *HandlerFilters) ServeHTTP(w http.ResponseWriter, r *http.Request) {
func (hf *HandlerFilters) match(r *http.Request) http.Handler { func (hf *HandlerFilters) match(r *http.Request) http.Handler {
LOOP: LOOP:
for _, filter := range hf.filters { for _, filter := range hf.filters {
if filter.noQueries && len(r.URL.Query()) > 0 {
continue
}
if len(filter.allowedQueries) > 0 {
queries := r.URL.Query()
for key := range queries {
if _, ok := filter.allowedQueries[key]; !ok {
continue LOOP
}
}
}
for _, header := range filter.headers { for _, header := range filter.headers {
hdrVals := r.Header.Values(header.Key) hdrVals := r.Header.Values(header.Key)
if len(hdrVals) == 0 || header.Value != "" && header.Value != hdrVals[0] { if len(hdrVals) == 0 || header.Value != "" && header.Value != hdrVals[0] {

View file

@ -10,7 +10,7 @@ import (
"testing" "testing"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
@ -40,8 +40,9 @@ type centerMock struct {
t *testing.T t *testing.T
anon bool anon bool
noAuthHeader bool noAuthHeader bool
isError bool err error
attrs []object.Attribute attrs []object.Attribute
key *keys.PrivateKey
} }
func (c *centerMock) Authenticate(*http.Request) (*middleware.Box, error) { func (c *centerMock) Authenticate(*http.Request) (*middleware.Box, error) {
@ -49,8 +50,8 @@ func (c *centerMock) Authenticate(*http.Request) (*middleware.Box, error) {
return nil, middleware.ErrNoAuthorizationHeader return nil, middleware.ErrNoAuthorizationHeader
} }
if c.isError { if c.err != nil {
return nil, fmt.Errorf("some error") return nil, c.err
} }
var token *bearer.Token var token *bearer.Token
@ -58,8 +59,12 @@ func (c *centerMock) Authenticate(*http.Request) (*middleware.Box, error) {
if !c.anon { if !c.anon {
bt := bearertest.Token() bt := bearertest.Token()
token = &bt token = &bt
key, err := keys.NewPrivateKey() key := c.key
if key == nil {
var err error
key, err = keys.NewPrivateKey()
require.NoError(c.t, err) require.NoError(c.t, err)
}
require.NoError(c.t, token.Sign(key.PrivateKey)) require.NoError(c.t, token.Sign(key.PrivateKey))
} }
@ -80,6 +85,7 @@ type middlewareSettingsMock struct {
domains []string domains []string
vhsEnabled bool vhsEnabled bool
vhsNamespacesEnabled map[string]bool vhsNamespacesEnabled map[string]bool
logHTTP middleware.LogHTTPConfig
} }
func (r *middlewareSettingsMock) SourceIPHeader() string { func (r *middlewareSettingsMock) SourceIPHeader() string {
@ -117,6 +123,9 @@ func (r *middlewareSettingsMock) ServernameHeader() string {
func (r *middlewareSettingsMock) VHSNamespacesEnabled() map[string]bool { func (r *middlewareSettingsMock) VHSNamespacesEnabled() map[string]bool {
return r.vhsNamespacesEnabled return r.vhsNamespacesEnabled
} }
func (r *middlewareSettingsMock) LogHTTPConfig() middleware.LogHTTPConfig {
return r.logHTTP
}
type frostFSIDMock struct { type frostFSIDMock struct {
tags map[string]string tags map[string]string
@ -149,20 +158,19 @@ func (m *xmlMock) NewXMLDecoder(r io.Reader) *xml.Decoder {
type resourceTaggingMock struct { type resourceTaggingMock struct {
bucketTags map[string]string bucketTags map[string]string
objectTags map[string]string objectTags map[string]string
noSuchObjectKey bool err error
noSuchBucketKey bool
} }
func (m *resourceTaggingMock) GetBucketTagging(context.Context, *data.BucketInfo) (map[string]string, error) { func (m *resourceTaggingMock) GetBucketTagging(context.Context, *data.BucketInfo) (map[string]string, error) {
if m.noSuchBucketKey { if m.err != nil {
return nil, apiErrors.GetAPIError(apiErrors.ErrNoSuchKey) return nil, m.err
} }
return m.bucketTags, nil return m.bucketTags, nil
} }
func (m *resourceTaggingMock) GetObjectTagging(context.Context, *data.GetObjectTaggingParams) (string, map[string]string, error) { func (m *resourceTaggingMock) GetObjectTagging(context.Context, *data.GetObjectTaggingParams) (string, map[string]string, error) {
if m.noSuchObjectKey { if m.err != nil {
return "", nil, apiErrors.GetAPIError(apiErrors.ErrNoSuchKey) return "", nil, m.err
} }
return "", m.objectTags, nil return "", m.objectTags, nil
} }
@ -569,7 +577,7 @@ func (h *handlerMock) ResolveBucket(ctx context.Context, name string) (*data.Buc
reqInfo := middleware.GetReqInfo(ctx) reqInfo := middleware.GetReqInfo(ctx)
bktInfo, ok := h.buckets[reqInfo.Namespace+name] bktInfo, ok := h.buckets[reqInfo.Namespace+name]
if !ok { if !ok {
return nil, apiErrors.GetAPIError(apiErrors.ErrNoSuchBucket) return nil, apierr.GetAPIError(apierr.ErrNoSuchBucket)
} }
return bktInfo, nil return bktInfo, nil
} }

View file

@ -14,7 +14,8 @@ import (
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/data"
apiErrors "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors" apierr "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/errors"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
s3middleware "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware" s3middleware "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/middleware"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/metrics" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/metrics"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
@ -26,6 +27,7 @@ import (
"git.frostfs.info/TrueCloudLab/policy-engine/schema/s3" "git.frostfs.info/TrueCloudLab/policy-engine/schema/s3"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"go.uber.org/zap/zaptest" "go.uber.org/zap/zaptest"
@ -214,18 +216,18 @@ func TestPolicyChecker(t *testing.T) {
deleteObject(chiRouter, ns2, bktName2, objName2, nil) deleteObject(chiRouter, ns2, bktName2, objName2, nil)
// check we cannot access 'bucket' in custom namespace // check we cannot access 'bucket' in custom namespace
putObjectErr(chiRouter, ns2, bktName1, objName2, nil, apiErrors.ErrAccessDenied) putObjectErr(chiRouter, ns2, bktName1, objName2, nil, apierr.ErrAccessDenied)
deleteObjectErr(chiRouter, ns2, bktName1, objName2, nil, apiErrors.ErrAccessDenied) deleteObjectErr(chiRouter, ns2, bktName1, objName2, nil, apierr.ErrAccessDenied)
} }
func TestPolicyCheckerError(t *testing.T) { func TestPolicyCheckerError(t *testing.T) {
chiRouter := prepareRouter(t) chiRouter := prepareRouter(t)
ns1, bktName1, objName1 := "", "bucket", "object" ns1, bktName1, objName1 := "", "bucket", "object"
putObjectErr(chiRouter, ns1, bktName1, objName1, nil, apiErrors.ErrNoSuchBucket) putObjectErr(chiRouter, ns1, bktName1, objName1, nil, apierr.ErrNoSuchBucket)
chiRouter = prepareRouter(t) chiRouter = prepareRouter(t)
chiRouter.cfg.FrostfsID.(*frostFSIDMock).userGroupsError = true chiRouter.cfg.FrostfsID.(*frostFSIDMock).userGroupsError = true
putObjectErr(chiRouter, ns1, bktName1, objName1, nil, apiErrors.ErrInternalError) putObjectErr(chiRouter, ns1, bktName1, objName1, nil, apierr.ErrInternalError)
} }
func TestPolicyCheckerReqTypeDetermination(t *testing.T) { func TestPolicyCheckerReqTypeDetermination(t *testing.T) {
@ -274,6 +276,36 @@ func TestPolicyCheckerReqTypeDetermination(t *testing.T) {
}) })
} }
func TestPolicyCheckFrostfsErrors(t *testing.T) {
chiRouter := prepareRouter(t)
ns1, bktName1, objName1 := "", "bucket", "object"
createBucket(chiRouter, ns1, bktName1)
key, err := keys.NewPrivateKey()
require.NoError(t, err)
chiRouter.cfg.Center.(*centerMock).key = key
chiRouter.cfg.MiddlewareSettings.(*middlewareSettingsMock).denyByDefault = true
ruleChain := &chain.Chain{
ID: chain.ID("id"),
Rules: []chain.Rule{{
Status: chain.Allow,
Actions: chain.Actions{Names: []string{"*"}},
Resources: chain.Resources{Names: []string{fmt.Sprintf(s3.ResourceFormatS3BucketObjects, bktName1)}},
}},
}
_, _, err = chiRouter.policyChecker.MorphRuleChainStorage().AddMorphRuleChain(chain.S3, engine.UserTarget(ns1+":"+key.Address()), ruleChain)
require.NoError(t, err)
// check we can access 'bucket' in default namespace
putObject(chiRouter, ns1, bktName1, objName1, nil)
chiRouter.cfg.Center.(*centerMock).anon = true
chiRouter.cfg.Tagging.(*resourceTaggingMock).err = frostfs.ErrAccessDenied
getObjectErr(chiRouter, ns1, bktName1, objName1, apierr.ErrAccessDenied)
}
func TestDefaultBehaviorPolicyChecker(t *testing.T) { func TestDefaultBehaviorPolicyChecker(t *testing.T) {
chiRouter := prepareRouter(t) chiRouter := prepareRouter(t)
ns, bktName := "", "bucket" ns, bktName := "", "bucket"
@ -283,7 +315,7 @@ func TestDefaultBehaviorPolicyChecker(t *testing.T) {
// check we cannot access if rules not found when settings is enabled // check we cannot access if rules not found when settings is enabled
chiRouter.middlewareSettings.denyByDefault = true chiRouter.middlewareSettings.denyByDefault = true
createBucketErr(chiRouter, ns, bktName, nil, apiErrors.ErrAccessDenied) createBucketErr(chiRouter, ns, bktName, nil, apierr.ErrAccessDenied)
} }
func TestDefaultPolicyCheckerWithUserTags(t *testing.T) { func TestDefaultPolicyCheckerWithUserTags(t *testing.T) {
@ -294,7 +326,7 @@ func TestDefaultPolicyCheckerWithUserTags(t *testing.T) {
allowOperations(router, ns, []string{"s3:CreateBucket"}, engineiam.Conditions{ allowOperations(router, ns, []string{"s3:CreateBucket"}, engineiam.Conditions{
engineiam.CondStringEquals: engineiam.Condition{fmt.Sprintf(common.PropertyKeyFormatFrostFSIDUserClaim, "tag-test"): []string{"test"}}, engineiam.CondStringEquals: engineiam.Condition{fmt.Sprintf(common.PropertyKeyFormatFrostFSIDUserClaim, "tag-test"): []string{"test"}},
}) })
createBucketErr(router, ns, bktName, nil, apiErrors.ErrAccessDenied) createBucketErr(router, ns, bktName, nil, apierr.ErrAccessDenied)
tags := make(map[string]string) tags := make(map[string]string)
tags["tag-test"] = "test" tags["tag-test"] = "test"
@ -321,8 +353,8 @@ func TestRequestParametersCheck(t *testing.T) {
}) })
listObjectsV1(router, ns, bktName, prefix, "", "") listObjectsV1(router, ns, bktName, prefix, "", "")
listObjectsV1Err(router, ns, bktName, "", "", "", apiErrors.ErrAccessDenied) listObjectsV1Err(router, ns, bktName, "", "", "", apierr.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "invalid", "", "", apiErrors.ErrAccessDenied) listObjectsV1Err(router, ns, bktName, "invalid", "", "", apierr.ErrAccessDenied)
}) })
t.Run("delimiter parameter, prohibit specific value", func(t *testing.T) { t.Run("delimiter parameter, prohibit specific value", func(t *testing.T) {
@ -344,7 +376,7 @@ func TestRequestParametersCheck(t *testing.T) {
listObjectsV1(router, ns, bktName, "", "", "") listObjectsV1(router, ns, bktName, "", "", "")
listObjectsV1(router, ns, bktName, "", "some-delimiter", "") listObjectsV1(router, ns, bktName, "", "some-delimiter", "")
listObjectsV1Err(router, ns, bktName, "", delimiter, "", apiErrors.ErrAccessDenied) listObjectsV1Err(router, ns, bktName, "", delimiter, "", apierr.ErrAccessDenied)
}) })
t.Run("max-keys parameter, allow specific value", func(t *testing.T) { t.Run("max-keys parameter, allow specific value", func(t *testing.T) {
@ -365,9 +397,9 @@ func TestRequestParametersCheck(t *testing.T) {
}) })
listObjectsV1(router, ns, bktName, "", "", strconv.Itoa(maxKeys)) listObjectsV1(router, ns, bktName, "", "", strconv.Itoa(maxKeys))
listObjectsV1Err(router, ns, bktName, "", "", "", apiErrors.ErrAccessDenied) listObjectsV1Err(router, ns, bktName, "", "", "", apierr.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "", "", strconv.Itoa(maxKeys-1), apiErrors.ErrAccessDenied) listObjectsV1Err(router, ns, bktName, "", "", strconv.Itoa(maxKeys-1), apierr.ErrAccessDenied)
listObjectsV1Err(router, ns, bktName, "", "", "invalid", apiErrors.ErrAccessDenied) listObjectsV1Err(router, ns, bktName, "", "", "invalid", apierr.ErrAccessDenied)
}) })
t.Run("max-keys parameter, allow range of values", func(t *testing.T) { t.Run("max-keys parameter, allow range of values", func(t *testing.T) {
@ -389,7 +421,7 @@ func TestRequestParametersCheck(t *testing.T) {
listObjectsV1(router, ns, bktName, "", "", strconv.Itoa(maxKeys)) listObjectsV1(router, ns, bktName, "", "", strconv.Itoa(maxKeys))
listObjectsV1(router, ns, bktName, "", "", strconv.Itoa(maxKeys-1)) listObjectsV1(router, ns, bktName, "", "", strconv.Itoa(maxKeys-1))
listObjectsV1Err(router, ns, bktName, "", "", strconv.Itoa(maxKeys+1), apiErrors.ErrAccessDenied) listObjectsV1Err(router, ns, bktName, "", "", strconv.Itoa(maxKeys+1), apierr.ErrAccessDenied)
}) })
t.Run("max-keys parameter, prohibit specific value", func(t *testing.T) { t.Run("max-keys parameter, prohibit specific value", func(t *testing.T) {
@ -411,7 +443,7 @@ func TestRequestParametersCheck(t *testing.T) {
listObjectsV1(router, ns, bktName, "", "", "") listObjectsV1(router, ns, bktName, "", "", "")
listObjectsV1(router, ns, bktName, "", "", strconv.Itoa(maxKeys-1)) listObjectsV1(router, ns, bktName, "", "", strconv.Itoa(maxKeys-1))
listObjectsV1Err(router, ns, bktName, "", "", strconv.Itoa(maxKeys), apiErrors.ErrAccessDenied) listObjectsV1Err(router, ns, bktName, "", "", strconv.Itoa(maxKeys), apierr.ErrAccessDenied)
}) })
} }
@ -439,10 +471,10 @@ func TestRequestTagsCheck(t *testing.T) {
tagging, err = xml.Marshal(data.Tagging{TagSet: []data.Tag{{Key: "key", Value: tagValue}}}) tagging, err = xml.Marshal(data.Tagging{TagSet: []data.Tag{{Key: "key", Value: tagValue}}})
require.NoError(t, err) require.NoError(t, err)
putBucketTaggingErr(router, ns, bktName, tagging, apiErrors.ErrAccessDenied) putBucketTaggingErr(router, ns, bktName, tagging, apierr.ErrAccessDenied)
tagging = nil tagging = nil
putBucketTaggingErr(router, ns, bktName, tagging, apiErrors.ErrMalformedXML) putBucketTaggingErr(router, ns, bktName, tagging, apierr.ErrMalformedXML)
}) })
t.Run("put object with tag", func(t *testing.T) { t.Run("put object with tag", func(t *testing.T) {
@ -464,7 +496,7 @@ func TestRequestTagsCheck(t *testing.T) {
putObject(router, ns, bktName, objName, &data.Tag{Key: tagKey, Value: tagValue}) putObject(router, ns, bktName, objName, &data.Tag{Key: tagKey, Value: tagValue})
putObjectErr(router, ns, bktName, objName, &data.Tag{Key: "key", Value: tagValue}, apiErrors.ErrAccessDenied) putObjectErr(router, ns, bktName, objName, &data.Tag{Key: "key", Value: tagValue}, apierr.ErrAccessDenied)
}) })
} }
@ -490,7 +522,7 @@ func TestResourceTagsCheck(t *testing.T) {
listObjectsV1(router, ns, bktName, "", "", "") listObjectsV1(router, ns, bktName, "", "", "")
router.cfg.Tagging.(*resourceTaggingMock).bucketTags = map[string]string{} router.cfg.Tagging.(*resourceTaggingMock).bucketTags = map[string]string{}
listObjectsV1Err(router, ns, bktName, "", "", "", apiErrors.ErrAccessDenied) listObjectsV1Err(router, ns, bktName, "", "", "", apierr.ErrAccessDenied)
}) })
t.Run("object tagging", func(t *testing.T) { t.Run("object tagging", func(t *testing.T) {
@ -515,22 +547,21 @@ func TestResourceTagsCheck(t *testing.T) {
getObject(router, ns, bktName, objName) getObject(router, ns, bktName, objName)
router.cfg.Tagging.(*resourceTaggingMock).objectTags = map[string]string{} router.cfg.Tagging.(*resourceTaggingMock).objectTags = map[string]string{}
getObjectErr(router, ns, bktName, objName, apiErrors.ErrAccessDenied) getObjectErr(router, ns, bktName, objName, apierr.ErrAccessDenied)
}) })
t.Run("non-existent resources", func(t *testing.T) { t.Run("non-existent resources", func(t *testing.T) {
router := prepareRouter(t) router := prepareRouter(t)
ns, bktName, objName := "", "bucket", "object" ns, bktName, objName := "", "bucket", "object"
listObjectsV1Err(router, ns, bktName, "", "", "", apiErrors.ErrNoSuchBucket) listObjectsV1Err(router, ns, bktName, "", "", "", apierr.ErrNoSuchBucket)
router.cfg.Tagging.(*resourceTaggingMock).noSuchBucketKey = true router.cfg.Tagging.(*resourceTaggingMock).err = apierr.GetAPIError(apierr.ErrNoSuchKey)
createBucket(router, ns, bktName) createBucket(router, ns, bktName)
getBucketErr(router, ns, bktName, apiErrors.ErrNoSuchKey) getBucketErr(router, ns, bktName, apierr.ErrNoSuchKey)
router.cfg.Tagging.(*resourceTaggingMock).noSuchObjectKey = true
createBucket(router, ns, bktName) createBucket(router, ns, bktName)
getObjectErr(router, ns, bktName, objName, apiErrors.ErrNoSuchKey) getObjectErr(router, ns, bktName, objName, apierr.ErrNoSuchKey)
}) })
} }
@ -548,7 +579,7 @@ func TestAccessBoxAttributesCheck(t *testing.T) {
engineiam.CondBool: engineiam.Condition{fmt.Sprintf(s3.PropertyKeyFormatAccessBoxAttr, attrKey): []string{attrValue}}, engineiam.CondBool: engineiam.Condition{fmt.Sprintf(s3.PropertyKeyFormatAccessBoxAttr, attrKey): []string{attrValue}},
}) })
listObjectsV1Err(router, ns, bktName, "", "", "", apiErrors.ErrAccessDenied) listObjectsV1Err(router, ns, bktName, "", "", "", apierr.ErrAccessDenied)
var attr object.Attribute var attr object.Attribute
attr.SetKey(attrKey) attr.SetKey(attrKey)
@ -570,7 +601,7 @@ func TestSourceIPCheck(t *testing.T) {
router.middlewareSettings.sourceIPHeader = hdr router.middlewareSettings.sourceIPHeader = hdr
header := map[string][]string{hdr: {"192.0.3.0"}} header := map[string][]string{hdr: {"192.0.3.0"}}
createBucketErr(router, ns, bktName, header, apiErrors.ErrAccessDenied) createBucketErr(router, ns, bktName, header, apierr.ErrAccessDenied)
router.middlewareSettings.sourceIPHeader = "" router.middlewareSettings.sourceIPHeader = ""
createBucket(router, ns, bktName) createBucket(router, ns, bktName)
@ -586,7 +617,7 @@ func TestMFAPolicy(t *testing.T) {
denyOperations(router, ns, []string{"s3:CreateBucket"}, engineiam.Conditions{ denyOperations(router, ns, []string{"s3:CreateBucket"}, engineiam.Conditions{
engineiam.CondBool: engineiam.Condition{s3.PropertyKeyAccessBoxAttrMFA: []string{"false"}}, engineiam.CondBool: engineiam.Condition{s3.PropertyKeyAccessBoxAttrMFA: []string{"false"}},
}) })
createBucketErr(router, ns, bktName, nil, apiErrors.ErrAccessDenied) createBucketErr(router, ns, bktName, nil, apierr.ErrAccessDenied)
var attr object.Attribute var attr object.Attribute
attr.SetKey("IAM-MFA") attr.SetKey("IAM-MFA")
@ -630,7 +661,7 @@ func createBucket(router *routerMock, namespace, bktName string) {
require.Equal(router.t, s3middleware.CreateBucketOperation, resp.Method) require.Equal(router.t, s3middleware.CreateBucketOperation, resp.Method)
} }
func createBucketErr(router *routerMock, namespace, bktName string, header http.Header, errCode apiErrors.ErrorCode) { func createBucketErr(router *routerMock, namespace, bktName string, header http.Header, errCode apierr.ErrorCode) {
w := createBucketBase(router, namespace, bktName, header) w := createBucketBase(router, namespace, bktName, header)
assertAPIError(router.t, w, errCode) assertAPIError(router.t, w, errCode)
} }
@ -646,7 +677,7 @@ func createBucketBase(router *routerMock, namespace, bktName string, header http
return w return w
} }
func getBucketErr(router *routerMock, namespace, bktName string, errCode apiErrors.ErrorCode) { func getBucketErr(router *routerMock, namespace, bktName string, errCode apierr.ErrorCode) {
w := getBucketBase(router, namespace, bktName) w := getBucketBase(router, namespace, bktName)
assertAPIError(router.t, w, errCode) assertAPIError(router.t, w, errCode)
} }
@ -665,7 +696,7 @@ func putObject(router *routerMock, namespace, bktName, objName string, tag *data
return resp return resp
} }
func putObjectErr(router *routerMock, namespace, bktName, objName string, tag *data.Tag, errCode apiErrors.ErrorCode) { func putObjectErr(router *routerMock, namespace, bktName, objName string, tag *data.Tag, errCode apierr.ErrorCode) {
w := putObjectBase(router, namespace, bktName, objName, tag) w := putObjectBase(router, namespace, bktName, objName, tag)
assertAPIError(router.t, w, errCode) assertAPIError(router.t, w, errCode)
} }
@ -690,7 +721,7 @@ func deleteObject(router *routerMock, namespace, bktName, objName string, tag *d
return resp return resp
} }
func deleteObjectErr(router *routerMock, namespace, bktName, objName string, tag *data.Tag, errCode apiErrors.ErrorCode) { func deleteObjectErr(router *routerMock, namespace, bktName, objName string, tag *data.Tag, errCode apierr.ErrorCode) {
w := deleteObjectBase(router, namespace, bktName, objName, tag) w := deleteObjectBase(router, namespace, bktName, objName, tag)
assertAPIError(router.t, w, errCode) assertAPIError(router.t, w, errCode)
} }
@ -715,7 +746,7 @@ func putBucketTagging(router *routerMock, namespace, bktName string, tagging []b
return resp return resp
} }
func putBucketTaggingErr(router *routerMock, namespace, bktName string, tagging []byte, errCode apiErrors.ErrorCode) { func putBucketTaggingErr(router *routerMock, namespace, bktName string, tagging []byte, errCode apierr.ErrorCode) {
w := putBucketTaggingBase(router, namespace, bktName, tagging) w := putBucketTaggingBase(router, namespace, bktName, tagging)
assertAPIError(router.t, w, errCode) assertAPIError(router.t, w, errCode)
} }
@ -738,7 +769,7 @@ func getObject(router *routerMock, namespace, bktName, objName string) handlerRe
return resp return resp
} }
func getObjectErr(router *routerMock, namespace, bktName, objName string, errCode apiErrors.ErrorCode) { func getObjectErr(router *routerMock, namespace, bktName, objName string, errCode apierr.ErrorCode) {
w := getObjectBase(router, namespace, bktName, objName) w := getObjectBase(router, namespace, bktName, objName)
assertAPIError(router.t, w, errCode) assertAPIError(router.t, w, errCode)
} }
@ -757,7 +788,7 @@ func listObjectsV1(router *routerMock, namespace, bktName, prefix, delimiter, ma
return resp return resp
} }
func listObjectsV1Err(router *routerMock, namespace, bktName, prefix, delimiter, maxKeys string, errCode apiErrors.ErrorCode) { func listObjectsV1Err(router *routerMock, namespace, bktName, prefix, delimiter, maxKeys string, errCode apierr.ErrorCode) {
w := listObjectsV1Base(router, namespace, bktName, prefix, delimiter, maxKeys) w := listObjectsV1Base(router, namespace, bktName, prefix, delimiter, maxKeys)
assertAPIError(router.t, w, errCode) assertAPIError(router.t, w, errCode)
} }
@ -826,8 +857,17 @@ func TestAuthenticate(t *testing.T) {
createBucket(chiRouter, "", "bkt-2") createBucket(chiRouter, "", "bkt-2")
chiRouter = prepareRouter(t) chiRouter = prepareRouter(t)
chiRouter.cfg.Center.(*centerMock).isError = true chiRouter.cfg.Center.(*centerMock).err = apierr.GetAPIError(apierr.ErrAccessDenied)
createBucketErr(chiRouter, "", "bkt-3", nil, apiErrors.ErrAccessDenied) createBucketErr(chiRouter, "", "bkt-3", nil, apierr.ErrAccessDenied)
chiRouter.cfg.Center.(*centerMock).err = frostfs.ErrGatewayTimeout
createBucketErr(chiRouter, "", "bkt-3", nil, apierr.ErrGatewayTimeout)
chiRouter.cfg.Center.(*centerMock).err = apierr.GetAPIError(apierr.ErrInternalError)
createBucketErr(chiRouter, "", "bkt-3", nil, apierr.ErrAccessDenied)
chiRouter.cfg.Center.(*centerMock).err = apierr.GetAPIError(apierr.ErrBadRequest)
createBucketErr(chiRouter, "", "bkt-3", nil, apierr.ErrBadRequest)
} }
func TestFrostFSIDValidation(t *testing.T) { func TestFrostFSIDValidation(t *testing.T) {
@ -843,7 +883,7 @@ func TestFrostFSIDValidation(t *testing.T) {
// frostFSID validation failed // frostFSID validation failed
chiRouter = prepareRouter(t, frostFSIDValidation(true)) chiRouter = prepareRouter(t, frostFSIDValidation(true))
chiRouter.cfg.FrostfsID.(*frostFSIDMock).validateError = true chiRouter.cfg.FrostfsID.(*frostFSIDMock).validateError = true
createBucketErr(chiRouter, "", "bkt-3", nil, apiErrors.ErrInternalError) createBucketErr(chiRouter, "", "bkt-3", nil, apierr.ErrInternalError)
} }
func TestRouterListObjectsV2Domains(t *testing.T) { func TestRouterListObjectsV2Domains(t *testing.T) {
@ -882,17 +922,17 @@ func readResponse(t *testing.T, w *httptest.ResponseRecorder) handlerResult {
return res return res
} }
func assertAPIError(t *testing.T, w *httptest.ResponseRecorder, expectedErrorCode apiErrors.ErrorCode) { func assertAPIError(t *testing.T, w *httptest.ResponseRecorder, expectedErrorCode apierr.ErrorCode) {
actualErrorResponse := &s3middleware.ErrorResponse{} actualErrorResponse := &s3middleware.ErrorResponse{}
err := xml.NewDecoder(w.Result().Body).Decode(actualErrorResponse) err := xml.NewDecoder(w.Result().Body).Decode(actualErrorResponse)
require.NoError(t, err) require.NoError(t, err)
expectedError := apiErrors.GetAPIError(expectedErrorCode) expectedError := apierr.GetAPIError(expectedErrorCode)
require.Equal(t, expectedError.HTTPStatusCode, w.Code) require.Equal(t, expectedError.HTTPStatusCode, w.Code)
require.Equal(t, expectedError.Code, actualErrorResponse.Code) require.Equal(t, expectedError.Code, actualErrorResponse.Code)
if expectedError.ErrCode != apiErrors.ErrInternalError { if expectedError.ErrCode != apierr.ErrInternalError {
require.Contains(t, actualErrorResponse.Message, expectedError.Description) require.Contains(t, actualErrorResponse.Message, expectedError.Description)
} }
} }

View file

@ -14,9 +14,12 @@ import (
"git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/acl" "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/acl"
sessionv2 "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/session" sessionv2 "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/session"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/cache" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/cache"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/handler"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/layer/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/accessbox"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/tokens" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/creds/tokens"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/pkg/retryer"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/bearer"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
frostfsecdsa "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/crypto/ecdsa" frostfsecdsa "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/crypto/ecdsa"
@ -25,6 +28,8 @@ import (
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id" oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/session"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user" "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/user"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/aws/retry"
"github.com/google/uuid" "github.com/google/uuid"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"go.uber.org/zap" "go.uber.org/zap"
@ -33,7 +38,7 @@ import (
// PrmContainerCreate groups parameters of containers created by authmate. // PrmContainerCreate groups parameters of containers created by authmate.
type PrmContainerCreate struct { type PrmContainerCreate struct {
// FrostFS identifier of the container creator. // FrostFS identifier of the container creator.
Owner user.ID Owner *keys.PublicKey
// Container placement policy. // Container placement policy.
Policy netmap.PlacementPolicy Policy netmap.PlacementPolicy
@ -85,11 +90,56 @@ type FrostFS interface {
type Agent struct { type Agent struct {
frostFS FrostFS frostFS FrostFS
log *zap.Logger log *zap.Logger
cfg *config
}
type config struct {
RetryMaxAttempts int
RetryMaxBackoff time.Duration
RetryStrategy handler.RetryStrategy
}
func defaultConfig() *config {
return &config{
RetryMaxAttempts: 4,
RetryMaxBackoff: 30 * time.Second,
RetryStrategy: handler.RetryStrategyExponential,
}
}
type Option func(cfg *config)
func WithRetryMaxAttempts(attempts int) func(*config) {
return func(cfg *config) {
cfg.RetryMaxAttempts = attempts
}
}
func WithRetryMaxBackoff(backoff time.Duration) func(*config) {
return func(cfg *config) {
cfg.RetryMaxBackoff = backoff
}
}
func WithRetryStrategy(strategy handler.RetryStrategy) func(*config) {
return func(cfg *config) {
cfg.RetryStrategy = strategy
}
} }
// New creates an object of type Agent that consists of Client and logger. // New creates an object of type Agent that consists of Client and logger.
func New(log *zap.Logger, frostFS FrostFS) *Agent { func New(log *zap.Logger, frostFS FrostFS, options ...Option) *Agent {
return &Agent{log: log, frostFS: frostFS} cfg := defaultConfig()
for _, opt := range options {
opt(cfg)
}
return &Agent{
log: log,
frostFS: frostFS,
cfg: cfg,
}
} }
type ( type (
@ -98,7 +148,9 @@ type (
// IssueSecretOptions contains options for passing to Agent.IssueSecret method. // IssueSecretOptions contains options for passing to Agent.IssueSecret method.
IssueSecretOptions struct { IssueSecretOptions struct {
Container ContainerOptions Container cid.ID
AccessKeyID string
SecretAccessKey string
FrostFSKey *keys.PrivateKey FrostFSKey *keys.PrivateKey
GatesPublicKeys []*keys.PublicKey GatesPublicKeys []*keys.PublicKey
Impersonate bool Impersonate bool
@ -114,7 +166,9 @@ type (
UpdateSecretOptions struct { UpdateSecretOptions struct {
FrostFSKey *keys.PrivateKey FrostFSKey *keys.PrivateKey
GatesPublicKeys []*keys.PublicKey GatesPublicKeys []*keys.PublicKey
Address oid.Address IsCustom bool
AccessKeyID string
ContainerID cid.ID
GatePrivateKey *keys.PrivateKey GatePrivateKey *keys.PrivateKey
CustomAttributes []object.Attribute CustomAttributes []object.Attribute
} }
@ -141,7 +195,8 @@ type (
// ObtainSecretOptions contains options for passing to Agent.ObtainSecret method. // ObtainSecretOptions contains options for passing to Agent.ObtainSecret method.
ObtainSecretOptions struct { ObtainSecretOptions struct {
SecretAddress string Container cid.ID
AccessKeyID string
GatePrivateKey *keys.PrivateKey GatePrivateKey *keys.PrivateKey
} }
) )
@ -168,32 +223,9 @@ type (
} }
) )
func (a *Agent) checkContainer(ctx context.Context, opts ContainerOptions, idOwner user.ID) (cid.ID, error) { func (a *Agent) checkContainer(ctx context.Context, cnrID cid.ID) error {
if !opts.ID.Equals(cid.ID{}) { a.log.Info(logs.CheckContainer, zap.Stringer("cid", cnrID))
a.log.Info(logs.CheckContainer, zap.Stringer("cid", opts.ID)) return a.frostFS.ContainerExists(ctx, cnrID)
return opts.ID, a.frostFS.ContainerExists(ctx, opts.ID)
}
a.log.Info(logs.CreateContainer,
zap.String("friendly_name", opts.FriendlyName),
zap.String("placement_policy", opts.PlacementPolicy))
var prm PrmContainerCreate
err := prm.Policy.DecodeString(opts.PlacementPolicy)
if err != nil {
return cid.ID{}, fmt.Errorf("failed to build placement policy: %w", err)
}
prm.Owner = idOwner
prm.FriendlyName = opts.FriendlyName
cnrID, err := a.frostFS.CreateContainer(ctx, prm)
if err != nil {
return cid.ID{}, fmt.Errorf("create container in FrostFS: %w", err)
}
return cnrID, nil
} }
func checkPolicy(policyString string) (*netmap.PlacementPolicy, error) { func checkPolicy(policyString string) (*netmap.PlacementPolicy, error) {
@ -255,20 +287,24 @@ func (a *Agent) IssueSecret(ctx context.Context, w io.Writer, options *IssueSecr
return fmt.Errorf("create tokens: %w", err) return fmt.Errorf("create tokens: %w", err)
} }
box, secrets, err := accessbox.PackTokens(gatesData, nil) var secret []byte
isCustom := options.AccessKeyID != ""
if isCustom {
secret = []byte(options.SecretAccessKey)
}
box, secrets, err := accessbox.PackTokens(gatesData, secret, isCustom)
if err != nil { if err != nil {
return fmt.Errorf("pack tokens: %w", err) return fmt.Errorf("pack tokens: %w", err)
} }
box.ContainerPolicy = policies box.ContainerPolicy = policies
var idOwner user.ID if err = a.checkContainer(ctx, options.Container); err != nil {
user.IDFromKey(&idOwner, options.FrostFSKey.PrivateKey.PublicKey)
id, err := a.checkContainer(ctx, options.Container, idOwner)
if err != nil {
return fmt.Errorf("check container: %w", err) return fmt.Errorf("check container: %w", err)
} }
var idOwner user.ID
user.IDFromKey(&idOwner, options.FrostFSKey.PrivateKey.PublicKey)
a.log.Info(logs.StoreBearerTokenIntoFrostFS, a.log.Info(logs.StoreBearerTokenIntoFrostFS,
zap.Stringer("owner_tkn", idOwner)) zap.Stringer("owner_tkn", idOwner))
@ -281,26 +317,37 @@ func (a *Agent) IssueSecret(ctx context.Context, w io.Writer, options *IssueSecr
creds := tokens.New(cfg) creds := tokens.New(cfg)
prm := tokens.CredentialsParam{ prm := tokens.CredentialsParam{
OwnerID: idOwner, Container: options.Container,
AccessKeyID: options.AccessKeyID,
AccessBox: box, AccessBox: box,
Expiration: lifetime.Exp, Expiration: lifetime.Exp,
Keys: options.GatesPublicKeys, Keys: options.GatesPublicKeys,
CustomAttributes: options.CustomAttributes, CustomAttributes: options.CustomAttributes,
} }
addr, err := creds.Put(ctx, id, prm) var addr oid.Address
err = retryer.MakeWithRetry(ctx, func() error {
var inErr error
addr, inErr = creds.Put(ctx, prm)
return inErr
}, a.credsPutRetryer())
if err != nil { if err != nil {
return fmt.Errorf("failed to put creds: %w", err) return fmt.Errorf("failed to put creds: %w", err)
} }
accessKeyID := accessKeyIDFromAddr(addr) accessKeyID := options.AccessKeyID
if accessKeyID == "" {
accessKeyID = accessKeyIDFromAddr(addr)
}
ir := &issuingResult{ ir := &issuingResult{
InitialAccessKeyID: accessKeyID, InitialAccessKeyID: accessKeyID,
AccessKeyID: accessKeyID, AccessKeyID: accessKeyID,
SecretAccessKey: secrets.SecretKey, SecretAccessKey: secrets.SecretKey,
OwnerPrivateKey: hex.EncodeToString(secrets.EphemeralKey.Bytes()), OwnerPrivateKey: hex.EncodeToString(secrets.EphemeralKey.Bytes()),
WalletPublicKey: hex.EncodeToString(options.FrostFSKey.PublicKey().Bytes()), WalletPublicKey: hex.EncodeToString(options.FrostFSKey.PublicKey().Bytes()),
ContainerID: id.EncodeToString(), ContainerID: options.Container.EncodeToString(),
} }
enc := json.NewEncoder(w) enc := json.NewEncoder(w)
@ -337,13 +384,15 @@ func (a *Agent) UpdateSecret(ctx context.Context, w io.Writer, options *UpdateSe
creds := tokens.New(cfg) creds := tokens.New(cfg)
box, _, err := creds.GetBox(ctx, options.Address) box, _, err := creds.GetBox(ctx, options.ContainerID, options.AccessKeyID)
if err != nil { if err != nil {
return fmt.Errorf("get accessbox: %w", err) return fmt.Errorf("get accessbox: %w", err)
} }
secret, err := hex.DecodeString(box.Gate.SecretKey) var secret []byte
if err != nil { if options.IsCustom {
secret = []byte(box.Gate.SecretKey)
} else if secret, err = hex.DecodeString(box.Gate.SecretKey); err != nil {
return fmt.Errorf("failed to decode secret key access box: %w", err) return fmt.Errorf("failed to decode secret key access box: %w", err)
} }
@ -360,7 +409,7 @@ func (a *Agent) UpdateSecret(ctx context.Context, w io.Writer, options *UpdateSe
return fmt.Errorf("create tokens: %w", err) return fmt.Errorf("create tokens: %w", err)
} }
updatedBox, secrets, err := accessbox.PackTokens(gatesData, secret) updatedBox, secrets, err := accessbox.PackTokens(gatesData, secret, options.IsCustom)
if err != nil { if err != nil {
return fmt.Errorf("pack tokens: %w", err) return fmt.Errorf("pack tokens: %w", err)
} }
@ -371,22 +420,26 @@ func (a *Agent) UpdateSecret(ctx context.Context, w io.Writer, options *UpdateSe
zap.Stringer("owner_tkn", idOwner)) zap.Stringer("owner_tkn", idOwner))
prm := tokens.CredentialsParam{ prm := tokens.CredentialsParam{
OwnerID: idOwner, Container: options.ContainerID,
AccessBox: updatedBox, AccessBox: updatedBox,
Expiration: lifetime.Exp, Expiration: lifetime.Exp,
Keys: options.GatesPublicKeys, Keys: options.GatesPublicKeys,
CustomAttributes: options.CustomAttributes, CustomAttributes: options.CustomAttributes,
} }
oldAddr := options.Address addr, err := creds.Update(ctx, prm)
addr, err := creds.Update(ctx, oldAddr, prm)
if err != nil { if err != nil {
return fmt.Errorf("failed to update creds: %w", err) return fmt.Errorf("failed to update creds: %w", err)
} }
accessKeyID := options.AccessKeyID
if !options.IsCustom {
accessKeyID = accessKeyIDFromAddr(addr)
}
ir := &issuingResult{ ir := &issuingResult{
AccessKeyID: accessKeyIDFromAddr(addr), AccessKeyID: accessKeyID,
InitialAccessKeyID: accessKeyIDFromAddr(oldAddr), InitialAccessKeyID: options.AccessKeyID,
SecretAccessKey: secrets.SecretKey, SecretAccessKey: secrets.SecretKey,
OwnerPrivateKey: hex.EncodeToString(secrets.EphemeralKey.Bytes()), OwnerPrivateKey: hex.EncodeToString(secrets.EphemeralKey.Bytes()),
WalletPublicKey: hex.EncodeToString(options.FrostFSKey.PublicKey().Bytes()), WalletPublicKey: hex.EncodeToString(options.FrostFSKey.PublicKey().Bytes()),
@ -419,12 +472,7 @@ func (a *Agent) ObtainSecret(ctx context.Context, w io.Writer, options *ObtainSe
bearerCreds := tokens.New(cfg) bearerCreds := tokens.New(cfg)
var addr oid.Address box, _, err := bearerCreds.GetBox(ctx, options.Container, options.AccessKeyID)
if err := addr.DecodeString(options.SecretAddress); err != nil {
return fmt.Errorf("failed to parse secret address: %w", err)
}
box, _, err := bearerCreds.GetBox(ctx, addr)
if err != nil { if err != nil {
return fmt.Errorf("failed to get tokens: %w", err) return fmt.Errorf("failed to get tokens: %w", err)
} }
@ -439,6 +487,27 @@ func (a *Agent) ObtainSecret(ctx context.Context, w io.Writer, options *ObtainSe
return enc.Encode(or) return enc.Encode(or)
} }
func (a *Agent) credsPutRetryer() aws.RetryerV2 {
return retry.NewStandard(func(options *retry.StandardOptions) {
options.MaxAttempts = a.cfg.RetryMaxAttempts
options.MaxBackoff = a.cfg.RetryMaxBackoff
if a.cfg.RetryStrategy == handler.RetryStrategyExponential {
options.Backoff = retry.NewExponentialJitterBackoff(options.MaxBackoff)
} else {
options.Backoff = retry.BackoffDelayerFunc(func(int, error) (time.Duration, error) {
return options.MaxBackoff, nil
})
}
options.Retryables = []retry.IsErrorRetryable{retry.IsErrorRetryableFunc(func(err error) aws.Ternary {
if errors.Is(err, frostfs.ErrAccessDenied) {
return aws.TrueTernary
}
return aws.FalseTernary
})}
})
}
func buildBearerToken(key *keys.PrivateKey, impersonate bool, lifetime lifetimeOptions, gateKey *keys.PublicKey) (*bearer.Token, error) { func buildBearerToken(key *keys.PrivateKey, impersonate bool, lifetime lifetimeOptions, gateKey *keys.PublicKey) (*bearer.Token, error) {
var ownerID user.ID var ownerID user.ID
user.IDFromKey(&ownerID, (ecdsa.PublicKey)(*gateKey)) user.IDFromKey(&ownerID, (ecdsa.PublicKey)(*gateKey))

View file

@ -4,6 +4,7 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"os" "os"
"strings"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/auth"
@ -21,7 +22,7 @@ var generatePresignedURLCmd = &cobra.Command{
You provide profile to load using --profile flag or explicitly provide credentials and region using You provide profile to load using --profile flag or explicitly provide credentials and region using
--aws-access-key-id, --aws-secret-access-key, --region. --aws-access-key-id, --aws-secret-access-key, --region.
Note to override credentials you must provide both access key and secret key.`, Note to override credentials you must provide both access key and secret key.`,
Example: `frostfs-s3-authmate generate-presigned-url --method put --bucket my-bucket --object my-object --endpoint http://localhost:8084 --lifetime 12h --region ru --aws-access-key-id ETaA2CadPcA7bAkLsML2PbTudXY8uRt2PDjCCwkvRv9s0FDCxWDXYc1SA1vKv8KbyCNsLY2AmAjJ92Vz5rgvsFCy --aws-secret-access-key c2d65ef2980f03f4f495bdebedeeae760496697880d61d106bb9a4e5cd2e0607`, Example: `frostfs-s3-authmate generate-presigned-url --method put --bucket my-bucket --object my-object --endpoint http://localhost:8084 --lifetime 12h --region ru --aws-access-key-id ETaA2CadPcA7bAkLsML2PbTudXY8uRt2PDjCCwkvRv9s0FDCxWDXYc1SA1vKv8KbyCNsLY2AmAjJ92Vz5rgvsFCy --aws-secret-access-key c2d65ef2980f03f4f495bdebedeeae760496697880d61d106bb9a4e5cd2e0607 --header 'Content-Type: text/plain'`,
RunE: runGeneratePresignedURLCmd, RunE: runGeneratePresignedURLCmd,
} }
@ -36,6 +37,7 @@ const (
regionFlag = "region" regionFlag = "region"
awsAccessKeyIDFlag = "aws-access-key-id" awsAccessKeyIDFlag = "aws-access-key-id"
awsSecretAccessKeyFlag = "aws-secret-access-key" awsSecretAccessKeyFlag = "aws-secret-access-key"
headerFlag = "header"
) )
func initGeneratePresignedURLCmd() { func initGeneratePresignedURLCmd() {
@ -48,6 +50,7 @@ func initGeneratePresignedURLCmd() {
generatePresignedURLCmd.Flags().String(regionFlag, "", "AWS region to use in signature (default is taken from ~/.aws/config)") generatePresignedURLCmd.Flags().String(regionFlag, "", "AWS region to use in signature (default is taken from ~/.aws/config)")
generatePresignedURLCmd.Flags().String(awsAccessKeyIDFlag, "", "AWS access key id to sign the URL (default is taken from ~/.aws/credentials)") generatePresignedURLCmd.Flags().String(awsAccessKeyIDFlag, "", "AWS access key id to sign the URL (default is taken from ~/.aws/credentials)")
generatePresignedURLCmd.Flags().String(awsSecretAccessKeyFlag, "", "AWS secret access key to sign the URL (default is taken from ~/.aws/credentials)") generatePresignedURLCmd.Flags().String(awsSecretAccessKeyFlag, "", "AWS secret access key to sign the URL (default is taken from ~/.aws/credentials)")
generatePresignedURLCmd.Flags().StringSlice(headerFlag, nil, "Header in form of 'Key: value' to use in presigned URL (use flags repeatedly for multiple headers or separate them by comma)")
_ = generatePresignedURLCmd.MarkFlagRequired(endpointFlag) _ = generatePresignedURLCmd.MarkFlagRequired(endpointFlag)
_ = generatePresignedURLCmd.MarkFlagRequired(bucketFlag) _ = generatePresignedURLCmd.MarkFlagRequired(bucketFlag)
@ -92,6 +95,12 @@ func runGeneratePresignedURLCmd(*cobra.Command, []string) error {
SignTime: time.Now().UTC(), SignTime: time.Now().UTC(),
} }
headers, err := parseHeaders()
if err != nil {
return wrapPreparationError(fmt.Errorf("failed to parse headers: %s", err))
}
presignData.Headers = headers
req, err := auth.PresignRequest(sess.Config.Credentials, reqData, presignData) req, err := auth.PresignRequest(sess.Config.Credentials, reqData, presignData)
if err != nil { if err != nil {
return wrapBusinessLogicError(err) return wrapBusinessLogicError(err)
@ -110,3 +119,21 @@ func runGeneratePresignedURLCmd(*cobra.Command, []string) error {
} }
return nil return nil
} }
func parseHeaders() (map[string]string, error) {
headers := viper.GetStringSlice(headerFlag)
if len(headers) == 0 {
return nil, nil
}
result := make(map[string]string, len(headers))
for _, header := range headers {
k, v, found := strings.Cut(header, ":")
if !found {
return nil, fmt.Errorf("invalid header format: %s", header)
}
result[strings.Trim(k, " ")] = strings.Trim(v, " ")
}
return result, nil
}

View file

@ -4,22 +4,34 @@ import (
"context" "context"
"fmt" "fmt"
"os" "os"
"strings"
"time" "time"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/api/handler"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/authmate" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/authmate"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/util"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/logs"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/wallet" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/wallet"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id" cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
oid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/id"
"github.com/nspcc-dev/neo-go/pkg/crypto/keys" "github.com/nspcc-dev/neo-go/pkg/crypto/keys"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"github.com/spf13/viper" "github.com/spf13/viper"
"go.uber.org/zap"
) )
var issueSecretCmd = &cobra.Command{ var issueSecretCmd = &cobra.Command{
Use: "issue-secret", Use: "issue-secret",
Short: "Issue a secret in FrostFS network", Short: "Issue a secret in FrostFS network",
Long: "Creates new s3 credentials to use with frostfs-s3-gw", Long: "Creates new s3 credentials to use with frostfs-s3-gw",
Example: `frostfs-s3-authmate issue-secret --wallet wallet.json --peer s01.frostfs.devenv:8080 --gate-public-key 031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a Example: `To create new s3 credentials use:
frostfs-s3-authmate issue-secret --wallet wallet.json --peer s01.frostfs.devenv:8080 --gate-public-key 031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a --attributes LOGIN=NUUb82KR2JrVByHs2YSKgtK29gKnF5q6Vt`, frostfs-s3-authmate issue-secret --wallet wallet.json --peer s01.frostfs.devenv:8080 --gate-public-key 031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a
frostfs-s3-authmate issue-secret --wallet wallet.json --peer s01.frostfs.devenv:8080 --gate-public-key 031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a --attributes LOGIN=NUUb82KR2JrVByHs2YSKgtK29gKnF5q6Vt
To create new s3 credentials using specific access key id and secret access key use:
frostfs-s3-authmate issue-secret --wallet wallet.json --peer s01.frostfs.devenv:8080 --gate-public-key 031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a --access-key-id my-access-key-id --secret-access-key my-secret-key --container-id BpExV76416Vo7GrkJsGwXGoLM35xsBwup8voedDZR3c6
`,
RunE: runIssueSecretCmd, RunE: runIssueSecretCmd,
} }
@ -36,6 +48,9 @@ const (
containerPolicyFlag = "container-policy" containerPolicyFlag = "container-policy"
awsCLICredentialFlag = "aws-cli-credentials" awsCLICredentialFlag = "aws-cli-credentials"
attributesFlag = "attributes" attributesFlag = "attributes"
retryMaxAttemptsFlag = "retry-max-attempts"
retryMaxBackoffFlag = "retry-max-backoff"
retryStrategyFlag = "retry-strategy"
) )
const walletPassphraseCfg = "wallet.passphrase" const walletPassphraseCfg = "wallet.passphrase"
@ -47,6 +62,10 @@ const (
defaultPoolHealthcheckTimeout = 5 * time.Second defaultPoolHealthcheckTimeout = 5 * time.Second
defaultPoolRebalanceInterval = 30 * time.Second defaultPoolRebalanceInterval = 30 * time.Second
defaultPoolStreamTimeout = 10 * time.Second defaultPoolStreamTimeout = 10 * time.Second
defaultRetryMaxAttempts = 4
defaultRetryMaxBackoff = 30 * time.Second
defaultRetryStrategy = handler.RetryStrategyExponential
) )
const ( const (
@ -54,6 +73,9 @@ const (
poolHealthcheckTimeoutFlag = "pool-healthcheck-timeout" poolHealthcheckTimeoutFlag = "pool-healthcheck-timeout"
poolRebalanceIntervalFlag = "pool-rebalance-interval" poolRebalanceIntervalFlag = "pool-rebalance-interval"
poolStreamTimeoutFlag = "pool-stream-timeout" poolStreamTimeoutFlag = "pool-stream-timeout"
accessKeyIDFlag = "access-key-id"
secretAccessKeyFlag = "secret-access-key"
) )
func initIssueSecretCmd() { func initIssueSecretCmd() {
@ -73,6 +95,12 @@ func initIssueSecretCmd() {
issueSecretCmd.Flags().Duration(poolRebalanceIntervalFlag, defaultPoolRebalanceInterval, "Interval for updating nodes health status") issueSecretCmd.Flags().Duration(poolRebalanceIntervalFlag, defaultPoolRebalanceInterval, "Interval for updating nodes health status")
issueSecretCmd.Flags().Duration(poolStreamTimeoutFlag, defaultPoolStreamTimeout, "Timeout for individual operation in streaming RPC") issueSecretCmd.Flags().Duration(poolStreamTimeoutFlag, defaultPoolStreamTimeout, "Timeout for individual operation in streaming RPC")
issueSecretCmd.Flags().String(attributesFlag, "", "User attributes in form of Key1=Value1,Key2=Value2 (note: you cannot override system attributes)") issueSecretCmd.Flags().String(attributesFlag, "", "User attributes in form of Key1=Value1,Key2=Value2 (note: you cannot override system attributes)")
issueSecretCmd.Flags().String(accessKeyIDFlag, "", "Access key id of s3 credential that must be created")
issueSecretCmd.Flags().String(secretAccessKeyFlag, "", "Secret access key of s3 credential that must be used")
issueSecretCmd.Flags().String(rpcEndpointFlag, "", "NEO node RPC address (must be provided if container-id is nns name)")
issueSecretCmd.Flags().Int(retryMaxAttemptsFlag, defaultRetryMaxAttempts, "Max amount of request attempts")
issueSecretCmd.Flags().Duration(retryMaxBackoffFlag, defaultRetryMaxBackoff, "Max delay before next attempt")
issueSecretCmd.Flags().String(retryStrategyFlag, defaultRetryStrategy, "Backoff strategy. `exponential` and `constant` are allowed")
_ = issueSecretCmd.MarkFlagRequired(walletFlag) _ = issueSecretCmd.MarkFlagRequired(walletFlag)
_ = issueSecretCmd.MarkFlagRequired(peerFlag) _ = issueSecretCmd.MarkFlagRequired(peerFlag)
@ -91,14 +119,6 @@ func runIssueSecretCmd(cmd *cobra.Command, _ []string) error {
return wrapPreparationError(fmt.Errorf("failed to load frostfs private key: %s", err)) return wrapPreparationError(fmt.Errorf("failed to load frostfs private key: %s", err))
} }
var cnrID cid.ID
containerID := viper.GetString(containerIDFlag)
if len(containerID) > 0 {
if err = cnrID.DecodeString(containerID); err != nil {
return wrapPreparationError(fmt.Errorf("failed to parse auth container id: %s", err))
}
}
var gatesPublicKeys []*keys.PublicKey var gatesPublicKeys []*keys.PublicKey
for _, keyStr := range viper.GetStringSlice(gatePublicKeyFlag) { for _, keyStr := range viper.GetStringSlice(gatePublicKeyFlag) {
gpk, err := keys.NewPublicKeyFromString(keyStr) gpk, err := keys.NewPublicKeyFromString(keyStr)
@ -137,17 +157,29 @@ func runIssueSecretCmd(cmd *cobra.Command, _ []string) error {
return wrapFrostFSInitError(fmt.Errorf("failed to create FrostFS component: %s", err)) return wrapFrostFSInitError(fmt.Errorf("failed to create FrostFS component: %s", err))
} }
var accessBox cid.ID
if viper.IsSet(containerIDFlag) {
if accessBox, err = util.ResolveContainerID(viper.GetString(containerIDFlag), viper.GetString(rpcEndpointFlag)); err != nil {
return wrapPreparationError(fmt.Errorf("resolve accessbox container id (make sure you provided %s): %w", rpcEndpointFlag, err))
}
} else if accessBox, err = createAccessBox(ctx, frostFS, key, log); err != nil {
return wrapPreparationError(err)
}
accessKeyID, secretAccessKey, err := parseAccessKeys()
if err != nil {
return wrapPreparationError(err)
}
customAttrs, err := parseObjectAttrs(viper.GetString(attributesFlag)) customAttrs, err := parseObjectAttrs(viper.GetString(attributesFlag))
if err != nil { if err != nil {
return wrapPreparationError(fmt.Errorf("failed to parse attributes: %s", err)) return wrapPreparationError(fmt.Errorf("failed to parse attributes: %s", err))
} }
issueSecretOptions := &authmate.IssueSecretOptions{ issueSecretOptions := &authmate.IssueSecretOptions{
Container: authmate.ContainerOptions{ Container: accessBox,
ID: cnrID, AccessKeyID: accessKeyID,
FriendlyName: viper.GetString(containerFriendlyNameFlag), SecretAccessKey: secretAccessKey,
PlacementPolicy: viper.GetString(containerPlacementPolicyFlag),
},
FrostFSKey: key, FrostFSKey: key,
GatesPublicKeys: gatesPublicKeys, GatesPublicKeys: gatesPublicKeys,
Impersonate: true, Impersonate: true,
@ -159,8 +191,69 @@ func runIssueSecretCmd(cmd *cobra.Command, _ []string) error {
CustomAttributes: customAttrs, CustomAttributes: customAttrs,
} }
if err = authmate.New(log, frostFS).IssueSecret(ctx, os.Stdout, issueSecretOptions); err != nil { options := []authmate.Option{
authmate.WithRetryMaxAttempts(viper.GetInt(retryMaxAttemptsFlag)),
authmate.WithRetryMaxBackoff(viper.GetDuration(retryMaxBackoffFlag)),
authmate.WithRetryStrategy(handler.RetryStrategy(viper.GetString(retryStrategyFlag))),
}
if err = authmate.New(log, frostFS, options...).IssueSecret(ctx, os.Stdout, issueSecretOptions); err != nil {
return wrapBusinessLogicError(fmt.Errorf("failed to issue secret: %s", err)) return wrapBusinessLogicError(fmt.Errorf("failed to issue secret: %s", err))
} }
return nil return nil
} }
func parseAccessKeys() (accessKeyID, secretAccessKey string, err error) {
accessKeyID = viper.GetString(accessKeyIDFlag)
secretAccessKey = viper.GetString(secretAccessKeyFlag)
if accessKeyID == "" && secretAccessKey != "" || accessKeyID != "" && secretAccessKey == "" {
return "", "", fmt.Errorf("flags %s and %s must be both provided or not", accessKeyIDFlag, secretAccessKeyFlag)
}
if accessKeyID != "" {
if !isCustomCreds(accessKeyID) {
return "", "", fmt.Errorf("invalid custom AccessKeyID format: %s", accessKeyID)
}
if !checkAccessKeyLength(accessKeyID) {
return "", "", fmt.Errorf("invalid custom AccessKeyID length: %s", accessKeyID)
}
if !checkAccessKeyLength(secretAccessKey) {
return "", "", fmt.Errorf("invalid custom SecretAccessKey length: %s", secretAccessKey)
}
}
return accessKeyID, secretAccessKey, nil
}
func isCustomCreds(accessKeyID string) bool {
var addr oid.Address
return addr.DecodeString(strings.ReplaceAll(accessKeyID, "0", "/")) != nil
}
func checkAccessKeyLength(key string) bool {
return 4 <= len(key) && len(key) <= 128
}
func createAccessBox(ctx context.Context, frostFS *frostfs.AuthmateFrostFS, key *keys.PrivateKey, log *zap.Logger) (cid.ID, error) {
friendlyName := viper.GetString(containerFriendlyNameFlag)
placementPolicy := viper.GetString(containerPlacementPolicyFlag)
log.Info(logs.CreateContainer, zap.String("friendly_name", friendlyName), zap.String("placement_policy", placementPolicy))
prm := authmate.PrmContainerCreate{
FriendlyName: friendlyName,
Owner: key.PublicKey(),
}
if err := prm.Policy.DecodeString(placementPolicy); err != nil {
return cid.ID{}, fmt.Errorf("failed to build placement policy: %w", err)
}
accessBox, err := frostFS.CreateContainer(ctx, prm)
if err != nil {
return cid.ID{}, fmt.Errorf("create container in FrostFS: %w", err)
}
return accessBox, nil
}

View file

@ -4,7 +4,6 @@ import (
"context" "context"
"fmt" "fmt"
"os" "os"
"strings"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/authmate" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/authmate"
"git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/wallet" "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/wallet"
@ -24,7 +23,6 @@ var obtainSecretCmd = &cobra.Command{
const ( const (
gateWalletFlag = "gate-wallet" gateWalletFlag = "gate-wallet"
gateAddressFlag = "gate-address" gateAddressFlag = "gate-address"
accessKeyIDFlag = "access-key-id"
) )
const ( const (
@ -38,10 +36,12 @@ func initObtainSecretCmd() {
obtainSecretCmd.Flags().String(gateWalletFlag, "", "Path to the s3 gateway wallet to decrypt accessbox") obtainSecretCmd.Flags().String(gateWalletFlag, "", "Path to the s3 gateway wallet to decrypt accessbox")
obtainSecretCmd.Flags().String(gateAddressFlag, "", "Address of the s3 gateway wallet account") obtainSecretCmd.Flags().String(gateAddressFlag, "", "Address of the s3 gateway wallet account")
obtainSecretCmd.Flags().String(accessKeyIDFlag, "", "Access key id of s3 credential for which secret must be obtained") obtainSecretCmd.Flags().String(accessKeyIDFlag, "", "Access key id of s3 credential for which secret must be obtained")
obtainSecretCmd.Flags().String(containerIDFlag, "", "CID or NNS name of auth container that contains provided credential (must be provided if custom access key id is used)")
obtainSecretCmd.Flags().Duration(poolDialTimeoutFlag, defaultPoolDialTimeout, "Timeout for connection to the node in pool to be established") obtainSecretCmd.Flags().Duration(poolDialTimeoutFlag, defaultPoolDialTimeout, "Timeout for connection to the node in pool to be established")
obtainSecretCmd.Flags().Duration(poolHealthcheckTimeoutFlag, defaultPoolHealthcheckTimeout, "Timeout for request to node to decide if it is alive") obtainSecretCmd.Flags().Duration(poolHealthcheckTimeoutFlag, defaultPoolHealthcheckTimeout, "Timeout for request to node to decide if it is alive")
obtainSecretCmd.Flags().Duration(poolRebalanceIntervalFlag, defaultPoolRebalanceInterval, "Interval for updating nodes health status") obtainSecretCmd.Flags().Duration(poolRebalanceIntervalFlag, defaultPoolRebalanceInterval, "Interval for updating nodes health status")
obtainSecretCmd.Flags().Duration(poolStreamTimeoutFlag, defaultPoolStreamTimeout, "Timeout for individual operation in streaming RPC") obtainSecretCmd.Flags().Duration(poolStreamTimeoutFlag, defaultPoolStreamTimeout, "Timeout for individual operation in streaming RPC")
obtainSecretCmd.Flags().String(rpcEndpointFlag, "", "NEO node RPC address (must be provided if container-id is nns name)")
_ = obtainSecretCmd.MarkFlagRequired(walletFlag) _ = obtainSecretCmd.MarkFlagRequired(walletFlag)
_ = obtainSecretCmd.MarkFlagRequired(peerFlag) _ = obtainSecretCmd.MarkFlagRequired(peerFlag)
@ -81,8 +81,14 @@ func runObtainSecretCmd(cmd *cobra.Command, _ []string) error {
return wrapFrostFSInitError(cli.Exit(fmt.Sprintf("failed to create FrostFS component: %s", err), 2)) return wrapFrostFSInitError(cli.Exit(fmt.Sprintf("failed to create FrostFS component: %s", err), 2))
} }
accessBox, accessKeyID, _, err := getAccessBoxID()
if err != nil {
return wrapPreparationError(err)
}
obtainSecretOptions := &authmate.ObtainSecretOptions{ obtainSecretOptions := &authmate.ObtainSecretOptions{
SecretAddress: strings.Replace(viper.GetString(accessKeyIDFlag), "0", "/", 1), Container: accessBox,
AccessKeyID: accessKeyID,
GatePrivateKey: gateKey, GatePrivateKey: gateKey,
} }

View file

@ -7,6 +7,7 @@ import (
"os" "os"
"strings" "strings"
"git.frostfs.info/TrueCloudLab/frostfs-contract/commonclient"
"git.frostfs.info/TrueCloudLab/frostfs-contract/policy" "git.frostfs.info/TrueCloudLab/frostfs-contract/policy"
ffsidContract "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/frostfsid/contract" ffsidContract "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/frostfsid/contract"
policyContact "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/policy/contract" policyContact "git.frostfs.info/TrueCloudLab/frostfs-s3-gw/internal/frostfs/policy/contract"
@ -116,6 +117,10 @@ func initFrostFSIDContract(ctx context.Context, log *zap.Logger, key *keys.Priva
Contract: viper.GetString(frostfsIDContractFlag), Contract: viper.GetString(frostfsIDContractFlag),
ProxyContract: viper.GetString(proxyContractFlag), ProxyContract: viper.GetString(proxyContractFlag),
Key: key, Key: key,
Waiter: commonclient.WaiterOptions{
IgnoreAlreadyExistsError: false,
VerifyExecResults: true,
},
} }
cli, err := ffsidContract.New(ctx, cfg) cli, err := ffsidContract.New(ctx, cfg)

Some files were not shown because too many files have changed in this diff Show more