e22ff52165
[ #367 ] Add check of AccessBox attributes
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-04-19 06:25:26 +00:00
3ff027587c
[ #357 ] Add check of request and resource tags
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-04-17 07:06:58 +00:00
8307c73fef
[ #364 ] Fix removing combined object
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-04-12 14:56:38 +03:00
d8889fca56
[ #340 ] Fix encode object acl
...
In the process of encode the acl of an object,
we use a map. As a result, when traversing the
map, we can get a different sequence of permissions
each time. Therefore, a list is used instead of a map.
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2024-04-11 09:28:30 +00:00
6da1acc554
[ #360 ] Use 'c' prefix for bucket policies instead of 'n'
...
With 'c' prefix, acl chains become shorter, thus gateway
receives shorter results and avoids sessions to neo-go.
There is still issue with many IAM rules.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-04-10 17:56:47 +03:00
9c012d0a66
[ #355 ] Remove policies when delete bucket
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-04-09 15:49:46 +00:00
8669bf6b50
[ #346 ] acl: Update APE and fix using
...
* Remove native policy when remove bucket policy
* Allow policies that contain only s3 compatible statements
(now deny rules cannot be converted to native rules)
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-04-02 12:43:04 +00:00
fbe7a784e8
[ #301 ] Support GetBucketPolicyStatus
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-28 09:13:25 +03:00
80c7b73eb9
[ #306 ] In APE buckets forbid canned acl except private
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-19 16:57:26 +03:00
62cc5a04a7
[ #328 ] Log error on failed response writing
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-15 11:02:26 +03:00
4ee3648183
[ #328 ] Log invalid lock enabled header
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-04 15:09:51 +03:00
f958eef2b3
[ #325 ] Use default empty data.LockInfo in get/head in case of error
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-04 09:49:41 +00:00
8050ca2d51
[ #306 ] Use session token for container read operations
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-01 18:14:33 +03:00
c12e264697
[ #306 ] Simplify cid resolver for metrics
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-03-01 17:46:16 +03:00
937367caaf
[ #318 ] Fix panic on invalid multipart form
...
Previously, simple 'curl -X POST http://localhost:8084/test ' leads to panic because of wrong handle matching
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-29 17:30:28 +03:00
31da31862a
[ #300 ] Update error logging in DeleteMultipleObjects
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2024-02-29 14:24:32 +00:00
56b50f2075
[ #306 ] Remove flag to disable policy contract
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
8f89f275bd
[ #306 ] Save bucket policy as native chain
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
ff15f9f28a
[ #306 ] Fix update settings for buckets without owner key in tree
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
c868af8a62
[ #306 ] Add flag to enable old ACL bucket creation
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
bac1b3fb2d
[ #306 ] Use zero basic acl to mark APE containers
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
c452d58ce2
[ #306 ] Reduce number of policy contract invocations
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
d9d12debc3
[ #306 ] Add tests
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
3d0d2032c6
[ #306 ] acl: Handle put/get acl for APE buckets
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
1f2cf0ed67
[ #306 ] Use APE instead of eACL on bucket creation
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
37be8851b3
[ #306 ] Simplify namespaces configuration
...
Resolve ns alias at the beginning of the request just once.
Keep in ns map only one default ns key.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
c4c199defe
[ #306 ] Use s3 as chain id prefix to be consistent
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-28 17:50:08 +03:00
563c1d9bd7
[ #308 ] Fix linter issues
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2024-02-16 18:25:06 +03:00
c334adeb6d
[ #165 ] Sort nodes in ServiceClientMemory
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:53:12 +03:00
a74d498df2
[ #165 ] Return sort after HEAD in listing
...
We have to sort object after HEAD because we make request in different goroutines,
so the order is not deterministic.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:15:08 +03:00
69297a4a38
[ #165 ] Delete object from tree in case of storage error
...
Extend storage node errors in case of which we continue deleting from tree
with 'object not found' error
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:15:08 +03:00
71d82d1cc8
[ #165 ] Fix lint issues
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:15:08 +03:00
fafe4af529
[ #165 ] Fix real object size in listing
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:15:03 +03:00
88f1acbdfc
[ #165 ] Cancel context in outdated list session
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
4e15452853
[ #165 ] Fix lint errors
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
da642a498a
[ #165 ] Listing fix data race
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
093de13f54
[ #165 ] Add stream listing tests
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
cf4fc3b602
[ #165 ] Extend error on getting listing containers not in current namespace
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
739a6ec9df
[ #165 ] Support latest only stream listing
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
c7ee628ab0
[ #165 ] Fix versions listing
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
4ad84b9b94
[ #165 ] listing: Use NodeVersion instead of ObjectInfo
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
29ac91dfd5
[ #165 ] Support streaming listing
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
84af85ed67
[ #302 ] Update APE to support chain id as bytes
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 12:53:45 +03:00
43cae9ee04
[ #248 ] Correct object versions response markers
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-21 10:53:50 +03:00
a17ff66975
[ #282 ] policy: Use prefixes to distinguish s3/iam actions/resources
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-20 10:41:15 +03:00
38c5503a02
[ #261 ] alc: Remove unused
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-18 15:49:58 +03:00
8273af8bf8
[ #261 ] Make PutBucketPolicy handler use policy contract
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-18 15:49:54 +03:00
28c6bb4cb8
[ #266 ] Support per namespace placement policies configuration
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-11-28 16:12:42 +03:00
ff1ec56d24
[ #260 ] Use namespace as domain when create bucket
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-11-27 15:30:12 +03:00
cf7254f8cd
[ #260 ] Refactor api/auth/center.go
...
Move the Center interface to middleware package where it's used
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-11-23 11:00:09 +03:00
861454e499
[ #218 ] Add check content sha256 header
...
The X-Amz-Content-Sha256 header check is done only for unencrypted payload.
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-11-22 11:33:52 +00:00
b28ecef43b
[ #219 ] Return ETag in quotes
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-11-22 11:12:32 +00:00
fe796ba538
[ #217 ] Consider Copy-Source-SSE-* headers during copy
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-11-13 13:22:58 +00:00
5ee73fad6a
[ #248 ] Correct NextVersionIDMarker in listing versions
...
Despite the spec https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html#API_ListObjectVersions_ResponseElements
says that
"When the number of responses exceeds the value of MaxKeys,
NextVersionIdMarker specifies the first object version not returned
that satisfies the search criteria. Use this value for the
version-id-marker request parameter in a subsequent request."
the actual behavior of AWS S3 is returning NextVersionIdMarker as the last returned object version
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-31 17:36:24 +03:00
890a8ed237
[ #227 ] Add versionID header after complete multipart
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-31 14:07:08 +00:00
0bed25816c
[ #224 ] Add conditional escaping for object name
...
Chi gives inconsistent results in terms of whether
the strings returned are URL coded or not
See:
* https://github.com/go-chi/chi/issues/641
* https://github.com/go-chi/chi/issues/642
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-31 13:58:51 +00:00
b169c5e6c3
[ #239 ] Update test for check goroutines leak
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-31 13:51:23 +00:00
cf13aae342
[ #225 ] Add default storage class to responses
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-31 13:37:07 +00:00
0938d7ee82
[ #226 ] Fix status code in GET/HEAD delete marker
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-27 10:58:57 +03:00
4f5f5fb5c8
[ #222 ] Fix marshaling errors in DeleteObjects method
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-25 14:54:02 +00:00
25bb581fee
[ #205 ] Add md5 checksum in header
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-25 11:04:19 +03:00
8d6aa0d40a
[ #243 ] Fix list object versions marker param
...
According to https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html
we have to use `key-marker`
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-18 10:35:47 +03:00
7e91f62c28
[ #223 ] Add store content language
...
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-10-17 14:42:02 +00:00
01323ca8e0
[ #216 ] Add check tag key uniqueness
...
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-10-17 14:40:29 +00:00
298662df9d
[ #221 ] Expand xmlns field ignore
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-13 16:21:13 +03:00
7de73f6b73
[ #197 ] Disable homomorphic hash for PUT
...
Disable TZ hash for PUT if it's disabled for container itself
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-11 12:32:48 +00:00
e3f2d59565
[ #154 ] Rename access key to secret key
...
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-10-06 09:20:39 +00:00
b8c93ed391
[ #172 ] Convert handler config to interface
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-04 11:01:27 +00:00
12cf29aed2
[ #207 ] Fix part-number-marker handling
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-09-19 12:43:07 +03:00
066b9a0250
[ #142 ] Add trace ID into log when tracing is enabled
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-09-07 14:19:37 +03:00
9120e97ac5
[ #203 ] Add go1.21 to CI
...
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-08-31 15:26:07 +03:00
b5fce5c8d2
[ #168 ] Skip only invalid policies and copies instead of ignoring all of them
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-25 12:05:30 +03:00
6617adc22b
[ #185 ] Use correct object size when object is combined or encrypted
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-24 18:03:08 +03:00
8898c2ec08
[ #185 ] Add tests for list multipart uploads
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-24 18:01:08 +03:00
8efcc957ea
[ #96 ] Move log messages to constants
...
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-08-23 18:32:31 +03:00
6b728fef87
[ #192 ] Add tests to make sure client_cut flag is passed to sdk
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-23 06:26:55 +00:00
fcf1c45ad2
[ #188 ] Fix url escaping
...
Url escaping has already been done in `net/http/request.go`
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-22 11:27:39 +03:00
94a42fa273
[ #51 ] Update frostfs-sdk-go
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-08-17 14:26:25 +03:00
40d7f844e3
[ #137 ] Refactor context data retrievers
...
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-08-16 14:05:38 +00:00
18878b66d3
[ #175 ] Use gate owner as object owner
...
This is required because node check session token owner
TrueCloudLab/frostfs-node#528
For client cut TrueCloudLab/frostfs-sdk-go#114
such owner will be gate owner
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-08 12:22:33 +03:00
fe897ec588
[ #178 ] wrapReader: Fix goroutine leak
...
In case of error in FrostFS.CreateObject wrapped reader
can be blocked because of synchronous pipe. We have to read out all payload in such case.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-01 17:59:34 +03:00
52931663e1
[ #176 ] multipart: Replace part on re-upload
...
We want to have exactly one object and tree node for each part number
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-21 16:20:04 +03:00
631b7f67b4
[ #63 ] multipart: Log upload id for every failed request
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 14:00:15 +03:00
bcf5a85aab
[ #63 ] multipart: Fix copying
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 14:00:15 +03:00
ad81b599dd
[ #63 ] Add fast multipart upload
...
Add new flag to object tree meta `isCombined` that means
the object payload is list of parts that forms real payload.
Set this attribute when complete multipart upload not to do unnecessary copying.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 14:00:15 +03:00
b59aa06637
[ #146 ] Add kludge.bypass_content_encoding_check_in_chunks flag
...
Flag allows to skip checking `Content-Encoding` for `aws-chunked` value
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-18 14:49:52 +03:00
d62aa7b979
[ #146 ] Fix preconditions: trim quotes in etags
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-18 14:49:03 +03:00
751a9be7cc
[ #146 ] Move getting chunk payload reader to separate function
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-18 14:49:03 +03:00
14ef9ff091
[ #158 ] Separate init object reader from read itself
...
To be able to handle cases and return appropriate http status code
when object missed in storage but gate cache contains its metadata
we need write code after init object reader.
So we separate init reader from actual reading.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:32:05 +03:00
fc90981c03
[ #149 ] Update inner imports after moving middlewares
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:25:09 +03:00
d531b13866
[ #143 ] Add more context to some s3 errors
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-30 12:08:33 +03:00
f921bc8af5
[ #143 ] Fix typo
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-30 12:04:52 +03:00
be03c5178f
[ #143 ] Fix NoSuchKey error on get/head
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-30 12:04:52 +03:00
9df8695463
[ #143 ] Fix transformToS3Error function
...
Unwrap error before checking for s3 error
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-21 17:16:40 +03:00
614d703726
[ #106 ] Add chunk uploading
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-06-21 10:23:57 +03:00
23593eee3d
[ #111 ] Use request scope logger
...
Store child zap logger with request scope fields into context.
Request scoped fields: request_id, api/method, bucket, object
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-19 13:54:51 +03:00
462589fc0c
[ #103 ] Return 504 http code on timeout errors
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-14 09:16:21 +03:00
4a6e3a19ce
[ #125 ] Handle negative Content-Length on put
...
Add computing actual object size during calculating hash on put.
Use this actual value to save in tree and cache
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-02 14:16:19 +00:00
868edfdb31
[ #78 ] Add test of bucket removal with object not found error
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-06-01 16:13:28 +03:00
a0f0d792b8
[ #78 ] layer: Clean up already removed object from tree
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-05-24 17:42:46 +03:00
