TrueCloudLab/frostfs-s3-gw
17
3
Fork 19
Code Issues 28 Pull requests 6 Projects Releases 18 Packages Activity Actions

[#343] docs: Actualize s3 compatibility table
All checks were successful
/ Builds (1.20) (pull_request) Successful in 13m52s
Details
/ Builds (1.21) (pull_request) Successful in 13m40s
Details
/ Lint (pull_request) Successful in 19m2s
Details
/ Tests (1.20) (pull_request) Successful in 14m18s
Details
/ Tests (1.21) (pull_request) Successful in 14m23s
Details
/ DCO (pull_request) Successful in 2m55s
Details
/ Vulncheck (pull_request) Successful in 1m9s
Details

Browse source 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
This commit is contained in:
Denis Kirillov 2024-04-02 10:00:44 +03:00
parent 348126b3b8
commit 6b8095182e
1 changed files with 176 additions and 178 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

354
docs/aws_s3_compat.md
View file

@ -1,73 +1,58 @@
# S3 API support
Reference:
* [AWS S3 API Reference](https://docs.aws.amazon.com/AmazonS3/latest/API/s3-api.pdf)
| | Legend |
|----|-------------------------------------------|
| 🟢 | Supported |
| 🟡 | Partially supported |
| 🔵 | Not supported yet, but will be in future |
| 🔴 | Not applicable or will never be supported |
| | Legend |
|-----|-------------------------------------------|
| 🟢 | Supported |
| 🟡 | Partially supported |
| 🔵 | Not supported yet, but will be in future |
| 🔴 | Not applicable or will never be supported |
## Object
| | Method | Comments |
|----|------------------------|-----------------------------------------|
| 🟢 | CopyObject | Done on gateway side |
| 🟢 | DeleteObject | |
| 🟢 | DeleteObjects | aka DeleteMultipleObjects |
| 🟢 | GetObject | |
| 🔴 | GetObjectTorrent | We don't plan implementing BT gateway |
| 🟢 | HeadObject | |
| 🟢 | ListParts | Parts loaded with MultipartUpload |
| 🟢 | ListObjects | |
| 🟢 | ListObjectsV2 | |
| 🟢 | PutObject | Content-MD5 header deprecated |
| 🔵 | SelectObjectContent | Need to have some Lambda to execute SQL |
| 🔵 | WriteGetObjectResponse | Waiting for Lambda to be developed |
| 🟢 | GetObjectAttributes | |
| | Method | Comments |
|-----|------------------------|-----------------------------------------|
| 🟢 | CopyObject | Done on gateway side |
| 🟢 | DeleteObject | |
| 🟢 | DeleteObjects | aka DeleteMultipleObjects |
| 🟢 | GetObject | |
| 🔴 | GetObjectTorrent | We don't plan implementing BT gateway |
| 🟢 | HeadObject | |
| 🟢 | ListParts | Parts loaded with MultipartUpload |
| 🟢 | ListObjects | |
| 🟢 | ListObjectsV2 | |
| 🟢 | PutObject | Content-MD5 header deprecated |
| 🔵 | SelectObjectContent | Need to have some Lambda to execute SQL |
| 🔵 | WriteGetObjectResponse | Waiting for Lambda to be developed |
| 🟢 | GetObjectAttributes | |
## ACL
For now there are some limitations:
* [Bucket policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-policies.html) supports only one `Principal` per `Statement`.
Principal must be `"AWS": "*"` (to refer all users) or `"CanonicalUser": "0313b1ac3a8076e155a7e797b24f0b650cccad5941ea59d7cfd51a024a8b2a06bf"` (hex encoded public key of desired user).
* Resource in bucket policy is an array. Each item MUST contain bucket name, CAN contain object name (wildcards are not supported):
```json
{
"Statement": [
{
"Resource": [
"arn:aws:s3:::bucket",
"arn:aws:s3:::bucket/some/object"
]
}
]
}
```
* AWS conditions and wildcard are not supported in [resources](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-arn-format.html)
* Only `CanonicalUser` (with hex encoded public key) and `All Users Group` are supported in [ACL](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html)
| | Method | Comments |
|----|--------------|-----------------|
| 🟡 | GetObjectAcl | See Limitations |
| 🟡 | PutObjectAcl | See Limitations |
| | Method | Comments |
|-----|--------------|-----------------------------------|
| 🟢 | GetObjectAcl | Objects can have only private acl |
| 🔴 | PutObjectAcl | Use PutBucketPolicy instead |
## Locking
For now there are some limitations:
* Retention period can't be shortened, only extended.
* You can't delete locks or object with unexpired lock.
| | Method | Comments |
|-----|----------------------------|---------------------------|
| 🟡 | GetObjectLegalHold | |
| 🟢 | GetObjectLockConfiguration | GetBucketObjectLockConfig |
| 🟡 | GetObjectRetention | |
| 🟡 | PutObjectLegalHold | |
| 🟢 | PutObjectLockConfiguration | PutBucketObjectLockConfig |
| 🟡 | PutObjectRetention | |
| | Method | Comments |
|-----|----------------------------|-------------------------------|
| 🟡 | GetObjectLegalHold | |
| 🟢 | GetObjectLockConfiguration | aka GetBucketObjectLockConfig |
| 🟡 | GetObjectRetention | |
| 🟡 | PutObjectLegalHold | |
| 🟢 | PutObjectLockConfiguration | aka PutBucketObjectLockConfig |
| 🟡 | PutObjectRetention | |
## Multipart
@ -75,206 +60,219 @@ CompleteMultipartUpload operations may take long time to complete. Gateway
sends whitespace characters to keep connection with the client alive. In this
case, gateway is unable to set proper HTTP headers like `X-Amz-Version-Id`.
| | Method | Comments |
|----|-------------------------|----------|
| 🟢 | AbortMultipartUpload | |
| 🟢 | CompleteMultipartUpload | |
| 🟢 | CreateMultipartUpload | |
| 🟢 | ListMultipartUploads | |
| 🟢 | ListParts | |
| 🟢 | UploadPart | |
| 🟢 | UploadPartCopy | |
| | Method | Comments |
|-----|-------------------------|----------|
| 🟢 | AbortMultipartUpload | |
| 🟢 | CompleteMultipartUpload | |
| 🟢 | CreateMultipartUpload | |
| 🟢 | ListMultipartUploads | |
| 🟢 | ListParts | |
| 🟢 | UploadPart | |
| 🟢 | UploadPartCopy | |
## Tagging
| | Method | Comments |
|----|---------------------|----------|
| 🟢 | DeleteObjectTagging | |
| 🟢 | GetObjectTagging | |
| 🟢 | PutObjectTagging | |
| | Method | Comments |
|-----|---------------------|----------|
| 🟢 | DeleteObjectTagging | |
| 🟢 | GetObjectTagging | |
| 🟢 | PutObjectTagging | |
## Versioning
See also `GetObject` and other method parameters.
| | Method | Comments |
|----|--------------------|--------------------------|
| 🟢 | ListObjectVersions | ListBucketObjectVersions |
| 🔵 | RestoreObject | |
| | Method | Comments |
|-----|--------------------|--------------------------|
| 🟢 | ListObjectVersions | ListBucketObjectVersions |
| 🔵 | RestoreObject | |
## Bucket
| | Method | Comments |
|----|----------------------|-----------|
| 🟢 | CreateBucket | PutBucket |
| 🟢 | DeleteBucket | |
| 🟢 | GetBucketLocation | |
| 🟢 | HeadBucket | |
| 🟢 | ListBuckets | |
| 🔵 | PutPublicAccessBlock | |
| | Method | Comments |
|-----|----------------------|-----------|
| 🟢 | CreateBucket | PutBucket |
| 🟢 | DeleteBucket | |
| 🟢 | GetBucketLocation | |
| 🟢 | HeadBucket | |
| 🟢 | ListBuckets | |
| 🔵 | PutPublicAccessBlock | |
## Acceleration
| | Method | Comments |
|----|----------------------------------|---------------------|
| 🔴 | GetBucketAccelerateConfiguration | GetBucketAccelerate |
| 🔴 | PutBucketAccelerateConfiguration | |
| | Method | Comments |
|-----|----------------------------------|---------------------|
| 🔴 | GetBucketAccelerateConfiguration | GetBucketAccelerate |
| 🔴 | PutBucketAccelerateConfiguration | |
## ACL
| | Method | Comments |
|----|--------------|---------------------|
| 🟡 | GetBucketAcl | See ACL limitations |
| 🟡 | PutBucketAcl | See ACL Limitations |
| | Method | Comments |
|-----|--------------|------------------------------|
| 🟡 | GetBucketAcl | Only canned acl is supported |
| 🟡 | PutBucketAcl | Only canned acl is supported |
## Analytics
| | Method | Comments |
|----|------------------------------------|----------|
| 🔵 | DeleteBucketAnalyticsConfiguration | |
| 🔵 | GetBucketAnalyticsConfiguration | |
| 🔵 | ListBucketAnalyticsConfigurations | |
| 🔵 | PutBucketAnalyticsConfiguration | |
| | Method | Comments |
|-----|------------------------------------|----------|
| 🔵 | DeleteBucketAnalyticsConfiguration | |
| 🔵 | GetBucketAnalyticsConfiguration | |
| 🔵 | ListBucketAnalyticsConfigurations | |
| 🔵 | PutBucketAnalyticsConfiguration | |
## CORS
| | Method | Comments |
|----|------------------|----------|
| 🟢 | DeleteBucketCors | |
| 🟢 | GetBucketCors | |
| 🟢 | PutBucketCors | |
| | Method | Comments |
|-----|------------------|----------|
| 🟢 | DeleteBucketCors | |
| 🟢 | GetBucketCors | |
| 🟢 | PutBucketCors | |
## Encryption
| | Method | Comments |
|----|------------------------|----------|
| 🔵 | DeleteBucketEncryption | |
| 🔵 | GetBucketEncryption | |
| 🔵 | PutBucketEncryption | |
| | Method | Comments |
|-----|------------------------|----------|
| 🔵 | DeleteBucketEncryption | |
| 🔵 | GetBucketEncryption | |
| 🔵 | PutBucketEncryption | |
## Inventory
| | Method | Comments |
|----|------------------------------------|----------|
| 🔵 | DeleteBucketInventoryConfiguration | |
| 🔵 | GetBucketInventoryConfiguration | |
| 🔵 | ListBucketInventoryConfigurations | |
| 🔵 | PutBucketInventoryConfiguration | |
| | Method | Comments |
|-----|------------------------------------|----------|
| 🔵 | DeleteBucketInventoryConfiguration | |
| 🔵 | GetBucketInventoryConfiguration | |
| 🔵 | ListBucketInventoryConfigurations | |
| 🔵 | PutBucketInventoryConfiguration | |
## Lifecycle
| | Method | Comments |
|----|---------------------------------|----------|
| 🔵 | DeleteBucketLifecycle | |
| 🔵 | GetBucketLifecycle | |
| 🔵 | GetBucketLifecycleConfiguration | |
| 🔵 | PutBucketLifecycle | |
| 🔵 | PutBucketLifecycleConfiguration | |
| | Method | Comments |
|-----|---------------------------------|----------|
| 🔵 | DeleteBucketLifecycle | |
| 🔵 | GetBucketLifecycle | |
| 🔵 | GetBucketLifecycleConfiguration | |
| 🔵 | PutBucketLifecycle | |
| 🔵 | PutBucketLifecycleConfiguration | |
## Logging
| | Method | Comments |
|----|------------------|----------|
| 🔵 | GetBucketLogging | |
| 🔵 | PutBucketLogging | |
| | Method | Comments |
|-----|------------------|----------|
| 🔵 | GetBucketLogging | |
| 🔵 | PutBucketLogging | |
## Metrics
| | Method | Comments |
|----|----------------------------------|----------|
| 🔵 | DeleteBucketMetricsConfiguration | |
| 🔵 | GetBucketMetricsConfiguration | |
| 🔵 | ListBucketMetricsConfigurations | |
| 🔵 | PutBucketMetricsConfiguration | |
| | Method | Comments |
|-----|----------------------------------|----------|
| 🔵 | DeleteBucketMetricsConfiguration | |
| 🔵 | GetBucketMetricsConfiguration | |
| 🔵 | ListBucketMetricsConfigurations | |
| 🔵 | PutBucketMetricsConfiguration | |
## Notifications
| | Method | Comments |
|----|------------------------------------|---------------|
| 🔵 | GetBucketNotification | |
| 🔵 | GetBucketNotificationConfiguration | |
| 🔵 | ListenBucketNotification | non-standard? |
| 🔵 | PutBucketNotification | |
| 🔵 | PutBucketNotificationConfiguration | |
| | Method | Comments |
|-----|------------------------------------|---------------|
| 🔵 | GetBucketNotification | |
| 🔵 | GetBucketNotificationConfiguration | |
| 🔵 | ListenBucketNotification | non-standard? |
| 🔵 | PutBucketNotification | |
| 🔵 | PutBucketNotificationConfiguration | |
## Ownership controls
| | Method | Comments |
|----|-------------------------------|----------|
| 🔵 | DeleteBucketOwnershipControls | |
| 🔵 | GetBucketOwnershipControls | |
| 🔵 | PutBucketOwnershipControls | |
| | Method | Comments |
|-----|-------------------------------|----------|
| 🔵 | DeleteBucketOwnershipControls | |
| 🔵 | GetBucketOwnershipControls | |
| 🔵 | PutBucketOwnershipControls | |
## Policy and replication
Bucket policy has the following limitations
* Supports only AWS principals in format `arn:aws:iam::<namespace>:user/<user>` or wildcard `*`.
* No complex conditions (only conditions for groups now supported)
Simple valid policy example:
```json
{
"Version": "2012-10-17",
"Statement": [{
"Principal": {"AWS": ["arn:aws:iam::111122223333:role/JohnDoe"]},
"Effect": "Allow",
"Action": ["s3:GetObject","s3:GetObjectVersion"],
"Resource": ["arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"]
}]
"Version": "2012-10-17",
"Statement": [
{
"Principal": {
"AWS": [
"arn:aws:iam::111122223333:role/JohnDoe"
]
},
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion"
],
"Resource": [
"arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"
]
}
]
}
```
Bucket policy status determines using the following scheme:
* If policy has statement with principal that is wildcard (`*`) then policy is considered as public
| | Method | Comments |
|-----|-------------------------|-----------------------------|
| 🟡 | DeleteBucketPolicy | See Policy limitations |
| 🔵 | DeleteBucketReplication | |
| 🔵 | DeletePublicAccessBlock | |
| 🟡 | GetBucketPolicy | See Policy limitations |
| 🟡 | GetBucketPolicyStatus | |
| 🔵 | GetBucketReplication | |
| 🟢 | PostPolicyBucket | Upload file using POST form |
| 🟡 | PutBucketPolicy | See Policy limitations |
| 🔵 | PutBucketReplication | |
| | Method | Comments |
|-----|-------------------------|---------------------------------------------------|
| 🟢 | DeleteBucketPolicy | See Policy limitations |
| 🔵 | DeleteBucketReplication | |
| 🔵 | DeletePublicAccessBlock | |
| 🟢 | GetBucketPolicy | See Policy limitations |
| 🟢 | GetBucketPolicyStatus | See rule determining status in Policy limitations |
| 🔵 | GetBucketReplication | |
| 🟢 | PostPolicyBucket | Upload file using POST form |
| 🟡 | PutBucketPolicy | See Policy limitations |
| 🔵 | PutBucketReplication | |
## Request payment
| | Method | Comments |
|----|-------------------------|----------|
| 🔴 | GetBucketRequestPayment | |
| 🔴 | PutBucketRequestPayment | |
| | Method | Comments |
|-----|-------------------------|----------|
| 🔴 | GetBucketRequestPayment | |
| 🔴 | PutBucketRequestPayment | |
## Tagging
| | Method | Comments |
|----|---------------------|----------|
| 🟢 | DeleteBucketTagging | |
| 🟢 | GetBucketTagging | |
| 🟢 | PutBucketTagging | |
| | Method | Comments |
|-----|---------------------|----------|
| 🟢 | DeleteBucketTagging | |
| 🟢 | GetBucketTagging | |
| 🟢 | PutBucketTagging | |
## Tiering
| | Method | Comments |
|----|---------------------------------------------|----------|
| 🔵 | DeleteBucketIntelligentTieringConfiguration | |
| 🔵 | GetBucketIntelligentTieringConfiguration | |
| 🔵 | ListBucketIntelligentTieringConfigurations | |
| 🔵 | PutBucketIntelligentTieringConfiguration | |
| | Method | Comments |
|-----|---------------------------------------------|----------|
| 🔵 | DeleteBucketIntelligentTieringConfiguration | |
| 🔵 | GetBucketIntelligentTieringConfiguration | |
| 🔵 | ListBucketIntelligentTieringConfigurations | |
| 🔵 | PutBucketIntelligentTieringConfiguration | |
## Versioning
| | Method | Comments |
|----|---------------------|----------|
| 🟢 | GetBucketVersioning | |
| 🟢 | PutBucketVersioning | |
| | Method | Comments |
|-----|---------------------|----------|
| 🟢 | GetBucketVersioning | |
| 🟢 | PutBucketVersioning | |
## Website
| | Method | Comments |
|----|---------------------|----------|
| 🔵 | DeleteBucketWebsite | |
| 🔵 | GetBucketWebsite | |
| 🔵 | PutBucketWebsite | |
| | Method | Comments |
|-----|---------------------|----------|
| 🔵 | DeleteBucketWebsite | |
| 🔵 | GetBucketWebsite | |
| 🔵 | PutBucketWebsite | |