FrostFS S3 Protocol Gateway with fuzzing tests

Find a file

Alex Vanin 036662feab Merge pull request #222 from KirillovDenis/feature/221-unify_source_filenames [#221] Unify source file naming		2021-08-24 12:44:23 +03:00
.github/workflows	*: Add docker to workflows	2021-05-24 16:11:46 +03:00
api	Merge pull request #222 from KirillovDenis/feature/221-unify_source_filenames	2021-08-24 12:44:23 +03:00
authmate	Merge pull request #170 from KirillovDenis/feature/89-placement_policy	2021-07-26 13:43:38 +03:00
cmd	[#221 ] Unify source file naming	2021-08-20 15:19:49 +03:00
creds	[#89 ] Add placement policy	2021-07-16 16:01:35 +03:00
docs	[#206 ] docs: Language check	2021-08-19 17:10:14 +03:00
internal	[#104 ] Support NEP-6 for authmate	2021-06-25 12:16:24 +03:00
.dockerignore	Initial commit based on https://github.com/minio/minio/releases/tag/RELEASE.2020-07-02T00-15-09Z	2020-07-03 15:03:06 +03:00
.gitignore	gitignore: more ignores	2021-05-13 22:08:20 +03:00
.golangci.yml	golangci: add configuration	2021-05-13 23:26:05 +03:00
CHANGELOG.md	CHANGELOG: Add latest changes of v0.16.0 release	2021-07-16 15:49:49 +03:00
Dockerfile	[#57 ] *: Fix docker builds	2021-05-24 15:07:08 +03:00
go.mod	[#206 ] api: Add objects cache	2021-08-18 10:16:27 +03:00
go.sum	[#206 ] api: Add objects cache	2021-08-18 10:16:27 +03:00
help.mk	Refactoring Makefile	2021-02-08 12:45:18 +03:00
LICENSE	Initial commit based on https://github.com/minio/minio/releases/tag/RELEASE.2020-07-02T00-15-09Z	2020-07-03 15:03:06 +03:00
Makefile	[#48 ] creda: Add accessbox in protobuf format	2021-06-14 16:38:37 +03:00
README.md	[#210 ] *: Split README into docs	2021-08-19 17:10:14 +03:00

README.md

NeoFS S3 Gateway

NeoFS S3 gateway provides API compatible with Amazon S3 cloud storage service.

Installation

go get -u github.com/nspcc-dev/neofs-s3-gw

Or you can call make to build it from the cloned repository (the binary will end up in bin/neofs-s3-gw with authmate helper in bin/neofs-authmate).

Notable make targets:

dep          Check and ensure dependencies
image        Build clean docker image
dirty-image  Build dirty docker image with host-built binaries
format       Run all code formatters
lint         Run linters
version      Show current version

Or you can also use a Docker image provided for released (and occasionally unreleased) versions of gateway (:latest points to the latest stable release).

Execution

Minimalistic S3 gateway setup needs:

NeoFS node(s) address (S3 gateway itself is not a NeoFS node) Passed via -p parameter or via S3_GW_PEERS_<N>_ADDRESS and S3_GW_PEERS_<N>_WEIGHT environment variables (gateway supports multiple NeoFS nodes with weighted load balancing).
a wallet used to fetch key and communicate with NeoFS nodes Passed via --wallet parameter or S3_GW_WALLET environment variable.

These two commands are functionally equivalent, they run the gate with one backend node, some keys and otherwise default settings:

$ neofs-s3-gw -p 192.168.130.72:8080 --wallet wallet.json

$ S3_GW_PEERS_0_ADDRESS=192.168.130.72:8080 \
  S3_GW_WALLET=wallet.json \
  neofs-s3-gw

It's also possible to specify uri scheme (grpc or grpcs) when using -p or environment variables:

$ neofs-s3-gw -p grpc://192.168.130.72:8080 --wallet wallet.json

$ S3_GW_PEERS_0_ADDRESS=grpcs://192.168.130.72:8080 \
  S3_GW_WALLET=wallet.json \
  neofs-s3-gw

Documentation

S3 API supported

Reference:

AWS S3 API Reference

Limitations

ACL

For now there are some restrictions:

Bucket policy support only one Principal (type AWS) per Statement. To refer all users use "AWS": "*"
AWS conditions and wildcard are not supported in resources
Only CanonicalUser (with hex encoded public key) and All Users Group are supported in ACL

Object

Method	Status
CopyObject	Supported
DeleteObject	Supported
DeleteObjects	Supported, aka DeleteMultipleObjects
GetObject	Supported
GetObjectTorrent	Unsupported, won't be
HeadObject	Supported
ListObjectParts	Unsupported
ListObjects	Supported
ListObjectsV2	Supported
PutObject	Supported (Content-MD5 option is not supported)
SelectObjectContent	Unsupported
WriteGetObjectResponse	Unsupported

ACL

Method	Status
GetObjectAcl	Supported
PutObjectAcl	Supported

Locking

Method	Status
GetObjectLegalHold	Unsupported
GetObjectLockConfiguration	Unsupported, aka GetBucketObjectLockConfig
GetObjectRetention	Unsupported
PutObjectLegalHold	Unsupported
PutObjectLockConfiguration	Unsupported, aka PutBucketObjectLockConfig
PutObjectRetention	Unsupported

Multipart

Should be supported eventually.

Method	Status
AbortMultipartUpload	Unsupported
CompleteMultipartUpload	Unsupported
CreateMultipartUpload	Unsupported, aka InitiateMultipartUpload and NewMultipartUpload
ListMultipartUploads	Unsupported
ListParts	Unsupported
UploadPart	Unsupported, aka PutObjectPart
UploadPartCopy	Unsupported, aka CopyObjectPart

Tagging

Also passed in PutObject parameters. We can support adding via PutObject and getting via GetBucketTagging, but deleting and putting can't be supported normally.

Method	Status
DeleteObjectTagging	Unsupported
GetObjectTagging	Unsupported
PutObjectTagging	Unsupported

Versioning

See also GetObject and other method parameters.

Method	Status
ListObjectVersions	Supported (null-versioning), aka ListBucketObjectVersions
RestoreObject	Unsupported

Bucket

Method	Status
CreateBucket	Supported, aka PutBucket
DeleteBucket	Supported
GetBucketLocation	Unsupported
HeadBucket	Supported
ListBuckets	Supported
PutPublicAccessBlock	Unsupported

Acceleration

Method	Status
GetBucketAccelerateConfiguration	Unsupported, aka GetBucketAccelerate
PutBucketAccelerateConfiguration	Unsupported

ACL

Method	Status
GetBucketAcl	Supported
PutBucketAcl	Supported

Analytics

Method	Status
DeleteBucketAnalyticsConfiguration	Unsupported
GetBucketAnalyticsConfiguration	Unsupported
ListBucketAnalyticsConfigurations	Unsupported
PutBucketAnalyticsConfiguration	Unsupported

Cors

Method	Status
DeleteBucketCors	Unsupported
GetBucketCors	Unsupported
PutBucketCors	Unsupported

Encryption

Method	Status
DeleteBucketEncryption	Unsupported
GetBucketEncryption	Unsupported
PutBucketEncryption	Unsupported

Inventory

Method	Status
DeleteBucketInventoryConfiguration	Unsupported
GetBucketInventoryConfiguration	Unsupported
ListBucketInventoryConfigurations	Unsupported
PutBucketInventoryConfiguration	Unsupported

Lifecycle

Method	Status
DeleteBucketLifecycle	Unsupported
GetBucketLifecycle	Unsupported
GetBucketLifecycleConfiguration	Unsupported
PutBucketLifecycle	Unsupported
PutBucketLifecycleConfiguration	Unsupported

Logging

Method	Status
GetBucketLogging	Unsupported
PutBucketLogging	Unsupported

Metrics

Method	Status
DeleteBucketMetricsConfiguration	Unsupported
GetBucketMetricsConfiguration	Unsupported
ListBucketMetricsConfigurations	Unsupported
PutBucketMetricsConfiguration	Unsupported

Notifications

Method	Status
GetBucketNotification	Unsupported
GetBucketNotificationConfiguration	Unsupported
ListenBucketNotification	Unsupported, non-standard?
PutBucketNotification	Unsupported
PutBucketNotificationConfiguration	Unsupported

Ownership controls

Method	Status
DeleteBucketOwnershipControls	Unsupported
GetBucketOwnershipControls	Unsupported
PutBucketOwnershipControls	Unsupported

Policy and replication

Method	Status
DeleteBucketPolicy	Unsupported
DeleteBucketReplication	Unsupported
DeletePublicAccessBlock	Unsupported
GetBucketPolicy	Supported
GetBucketPolicyStatus	Unsupported
GetBucketReplication	Unsupported
PostPolicyBucket	Unsupported, non-standard?
PutBucketPolicy	Supported
PutBucketReplication	Unsupported

Request payment

Method	Status
GetBucketRequestPayment	Unsupported
PutBucketRequestPayment	Unsupported

Tagging

Method	Status
DeleteBucketTagging	Unsupported
GetBucketTagging	Unsupported
PutBucketTagging	Unsupported

Tiering

Method	Status
DeleteBucketIntelligentTieringConfiguration	Unsupported
GetBucketIntelligentTieringConfiguration	Unsupported
ListBucketIntelligentTieringConfigurations	Unsupported
PutBucketIntelligentTieringConfiguration	Unsupported

Versioning

Method	Status
GetBucketVersioning	Unsupported
PutBucketVersioning	Unsupported

Website

Method	Status
DeleteBucketWebsite	Unsupported
GetBucketWebsite	Unsupported
PutBucketWebsite	Unsupported