Commit graph

492 commits

Author SHA1 Message Date
3dc989d7fe [] Support Location in CompleteMultipart response
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-08-06 15:45:09 +03:00
69e77aecc9 [] Don't skip [Next]PartNumberMarker in response
In ListParts response we should always keep PartNumberMarker (even if it's zero)
and NextPartNumberMarker (even if response isn't truncated)

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-08-06 15:44:56 +03:00
51c5c227c2 [] Add force bucket delete flag
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2024-07-25 14:04:54 +03:00
6cb0026007 [] layer: Split FrostFS ReadObject to separate methods
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-23 16:53:59 +03:00
3213dd7236 [] Add processing of encoding-type in versions listing
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-07-23 12:10:35 +00:00
c0011ebb8d [] tree: Fix multipart having system name
Previously if multipart key has the same name as some system node
(e.g. bucket-settings, bucket-cors etc.) it shadows real system node
and bucket started to be unversioned again for example.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-19 11:24:50 +03:00
456319d2f1 [] Fix split tree
Update tree service to fix split tree problem.
Tree intermediate nodes can be duplicated, so we must handle this.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-19 11:24:46 +03:00
f86b82351a [] Fix parameter parsing in bucket retryer
RetryStrategyExponential should use jitter backoff
instead of constant delay function

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-07-03 13:42:24 +03:00
465eaa816a [] Drop [e]ACL related code
Always consider buckets as APE compatible

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-01 16:58:44 +03:00
77f8bdac58 [] Drop kludge.acl_enabled flag
Now only APE container can be created using s3-gw

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-07-01 16:26:19 +03:00
91541a432d [] Check uniqueness in DeleteMultipleObjects
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-26 16:39:06 +03:00
414f3943e2 [] Drop layer.Client interface
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-25 15:57:55 +03:00
9432782ce6 [] Drop notifications
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-25 15:49:37 +03:00
280d11c794 [] Don't set full_control for bucket owner
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-19 10:55:24 +03:00
bb81afc14a [] Support retryer
Add two strategy for PutBucketSettings request retryer:
* exponential backoff (increasing up to `max_backoff` delays with jitter)
* constant backoff (always the same `max_backoff` delay between requests)

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-06-06 13:02:17 +00:00
e25dc90c20 [] Add OPTIONS method for object operations
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-06-04 12:59:45 +00:00
b5fae316cf [] Add user to response
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2024-06-04 09:37:55 +00:00
9152b084ec [] Fix typo
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-05-22 15:06:02 +00:00
87b9e97a80 [] Do not proceed on bucket remove error
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-05-17 20:38:39 +03:00
d62d8f3874 [] Support the renaming of ObjectRequest and ObjectContainer
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2024-05-14 16:51:36 +03:00
db05021786 [] Add Iana CharsetReader for Oracle integration
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2024-04-25 17:44:38 +03:00
45f77de8c8 [] Add custom Source IP header configuration
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-04-22 07:42:45 +00:00
e22ff52165 [] Add check of AccessBox attributes
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-04-19 06:25:26 +00:00
3ff027587c [] Add check of request and resource tags
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-04-17 07:06:58 +00:00
8307c73fef [] Fix removing combined object
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-04-12 14:56:38 +03:00
d8889fca56 [] Fix encode object acl
In the process of encode the acl of an object,
we use a map. As a result, when traversing the
map, we can get a different sequence of permissions
each time. Therefore, a list is used instead of a map.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-04-11 09:28:30 +00:00
6da1acc554 [] Use 'c' prefix for bucket policies instead of 'n'
With 'c' prefix, acl chains become shorter, thus gateway
receives shorter results and avoids sessions to neo-go.

There is still issue with many IAM rules.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-04-10 17:56:47 +03:00
9c012d0a66 [] Remove policies when delete bucket
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-04-09 15:49:46 +00:00
8669bf6b50 [] acl: Update APE and fix using
* Remove native policy when remove bucket policy
* Allow policies that contain only s3 compatible statements
(now deny rules cannot be converted to native rules)

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-04-02 12:43:04 +00:00
fbe7a784e8 [] Support GetBucketPolicyStatus
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-28 09:13:25 +03:00
80c7b73eb9 [] In APE buckets forbid canned acl except private
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-19 16:57:26 +03:00
62cc5a04a7 [] Log error on failed response writing
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-15 11:02:26 +03:00
4ee3648183 [] Log invalid lock enabled header
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-04 15:09:51 +03:00
f958eef2b3 [] Use default empty data.LockInfo in get/head in case of error
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-04 09:49:41 +00:00
8050ca2d51 [] Use session token for container read operations
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-01 18:14:33 +03:00
c12e264697 [] Simplify cid resolver for metrics
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-01 17:46:16 +03:00
937367caaf [] Fix panic on invalid multipart form
Previously, simple 'curl -X POST http://localhost:8084/test' leads to panic because of wrong handle matching

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-29 17:30:28 +03:00
31da31862a [] Update error logging in DeleteMultipleObjects
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-02-29 14:24:32 +00:00
56b50f2075 [] Remove flag to disable policy contract
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
8f89f275bd [] Save bucket policy as native chain
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
ff15f9f28a [] Fix update settings for buckets without owner key in tree
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
c868af8a62 [] Add flag to enable old ACL bucket creation
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
bac1b3fb2d [] Use zero basic acl to mark APE containers
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
c452d58ce2 [] Reduce number of policy contract invocations
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
d9d12debc3 [] Add tests
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
3d0d2032c6 [] acl: Handle put/get acl for APE buckets
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
1f2cf0ed67 [] Use APE instead of eACL on bucket creation
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
37be8851b3 [] Simplify namespaces configuration
Resolve ns alias at the beginning of the request just once.
Keep in ns map only one default ns key.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
c4c199defe [] Use s3 as chain id prefix to be consistent
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
563c1d9bd7 [] Fix linter issues
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-02-16 18:25:06 +03:00
c334adeb6d [] Sort nodes in ServiceClientMemory
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:53:12 +03:00
a74d498df2 [] Return sort after HEAD in listing
We have to sort object after HEAD because we make request in different goroutines,
so the order is not deterministic.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:15:08 +03:00
69297a4a38 [] Delete object from tree in case of storage error
Extend storage node errors in case of which we continue deleting from tree
with 'object not found' error

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:15:08 +03:00
71d82d1cc8 [] Fix lint issues
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:15:08 +03:00
fafe4af529 [] Fix real object size in listing
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:15:03 +03:00
88f1acbdfc [] Cancel context in outdated list session
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
4e15452853 [] Fix lint errors
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
da642a498a [] Listing fix data race
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
093de13f54 [] Add stream listing tests
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
cf4fc3b602 [] Extend error on getting listing containers not in current namespace
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
739a6ec9df [] Support latest only stream listing
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
c7ee628ab0 [] Fix versions listing
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
4ad84b9b94 [] listing: Use NodeVersion instead of ObjectInfo
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
29ac91dfd5 [] Support streaming listing
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
84af85ed67 [] Update APE to support chain id as bytes
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 12:53:45 +03:00
43cae9ee04 [] Correct object versions response markers
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-21 10:53:50 +03:00
a17ff66975 [] policy: Use prefixes to distinguish s3/iam actions/resources
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-20 10:41:15 +03:00
38c5503a02 [] alc: Remove unused
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-18 15:49:58 +03:00
8273af8bf8 [] Make PutBucketPolicy handler use policy contract
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-18 15:49:54 +03:00
28c6bb4cb8 [] Support per namespace placement policies configuration
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-11-28 16:12:42 +03:00
ff1ec56d24 [] Use namespace as domain when create bucket
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-11-27 15:30:12 +03:00
cf7254f8cd [] Refactor api/auth/center.go
Move the Center interface to middleware package where it's used

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-11-23 11:00:09 +03:00
861454e499 [] Add check content sha256 header
The X-Amz-Content-Sha256 header check is done only for unencrypted payload.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-11-22 11:33:52 +00:00
b28ecef43b [] Return ETag in quotes
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-11-22 11:12:32 +00:00
fe796ba538 [] Consider Copy-Source-SSE-* headers during copy
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-11-13 13:22:58 +00:00
5ee73fad6a [] Correct NextVersionIDMarker in listing versions
Despite the spec https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html#API_ListObjectVersions_ResponseElements
says that
"When the number of responses exceeds the value of MaxKeys,
NextVersionIdMarker specifies the first object version not returned
 that satisfies the search criteria. Use this value for the
 version-id-marker request parameter in a subsequent request."
 the actual behavior of AWS S3 is returning NextVersionIdMarker as the last returned object version

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-31 17:36:24 +03:00
890a8ed237 [] Add versionID header after complete multipart
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-31 14:07:08 +00:00
0bed25816c [] Add conditional escaping for object name
Chi gives inconsistent results in terms of whether
the strings returned are URL coded or not
See:
* https://github.com/go-chi/chi/issues/641
* https://github.com/go-chi/chi/issues/642

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-31 13:58:51 +00:00
b169c5e6c3 [] Update test for check goroutines leak
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-31 13:51:23 +00:00
cf13aae342 [] Add default storage class to responses
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-31 13:37:07 +00:00
0938d7ee82 [] Fix status code in GET/HEAD delete marker
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-27 10:58:57 +03:00
4f5f5fb5c8 [] Fix marshaling errors in DeleteObjects method
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-25 14:54:02 +00:00
25bb581fee [] Add md5 checksum in header
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-25 11:04:19 +03:00
8d6aa0d40a [] Fix list object versions marker param
According to https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html
we have to use `key-marker`

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-18 10:35:47 +03:00
7e91f62c28 [] Add store content language
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-10-17 14:42:02 +00:00
01323ca8e0 [] Add check tag key uniqueness
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-10-17 14:40:29 +00:00
298662df9d [] Expand xmlns field ignore
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-13 16:21:13 +03:00
7de73f6b73 [] Disable homomorphic hash for PUT
Disable TZ hash for PUT if it's disabled for container itself

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-11 12:32:48 +00:00
e3f2d59565 [] Rename access key to secret key
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-10-06 09:20:39 +00:00
b8c93ed391 [] Convert handler config to interface
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-04 11:01:27 +00:00
12cf29aed2 [] Fix part-number-marker handling
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-09-19 12:43:07 +03:00
066b9a0250 [] Add trace ID into log when tracing is enabled
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-09-07 14:19:37 +03:00
9120e97ac5 [] Add go1.21 to CI
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-08-31 15:26:07 +03:00
b5fce5c8d2 [] Skip only invalid policies and copies instead of ignoring all of them
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-25 12:05:30 +03:00
6617adc22b [] Use correct object size when object is combined or encrypted
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-24 18:03:08 +03:00
8898c2ec08 [] Add tests for list multipart uploads
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-24 18:01:08 +03:00
8efcc957ea [] Move log messages to constants
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-08-23 18:32:31 +03:00
6b728fef87 [] Add tests to make sure client_cut flag is passed to sdk
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-23 06:26:55 +00:00
fcf1c45ad2 [] Fix url escaping
Url escaping has already been done in `net/http/request.go`

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-22 11:27:39 +03:00
94a42fa273 [] Update frostfs-sdk-go
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-08-17 14:26:25 +03:00