TrueCloudLab
/
frostfs-s3-gw
15
2
Fork 16
Code Issues 21 Pull Requests 9 Actions Packages Projects Releases 9 Activity
1,210 Commits
6 Branches
28 Tags
19 MiB
Commit Graph

358 Commits (0850d21ff3097d672fd00796493b8d552bf125cc)

Author SHA1 Message Date
Alexey Vanin f58a0d04ff [#233] Clean tag node in the tree service instead of removal 
With new retry policy of tree service pool, gateway should avoid
deletion of system nodes from tree. Absence of node in the tree
will trigger retry. Other storage in the network may return already
deleted node while tree is not completely synced, and client will
get unexpected result.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-12-07 16:40:56 +03:00
Denis Kirillov ff1ec56d24 [#260] Use namespace as domain when create bucket
/ DCO (pull_request) Successful in 1m26s Details
/ Vulncheck (pull_request) Successful in 2m6s Details
/ Builds (1.20) (pull_request) Successful in 2m20s Details
/ Builds (1.21) (pull_request) Successful in 2m21s Details
/ Lint (pull_request) Successful in 3m15s Details
/ Tests (1.20) (pull_request) Successful in 2m28s Details
/ Tests (1.21) (pull_request) Successful in 2m17s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-27 15:30:12 +03:00
Denis Kirillov 055cc6a22a [#260] Use namespace as domain when resolve bucket 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-23 11:00:11 +03:00
Roman Loginov 861454e499 [#218] Add check content sha256 header 
The X-Amz-Content-Sha256 header check is done only for unencrypted payload.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-11-22 11:33:52 +00:00
Marina Biryukova b28ecef43b [#219] Return ETag in quotes
/ Vulncheck (pull_request) Successful in 9m5s Details
/ Lint (pull_request) Successful in 11m1s Details
/ Tests (1.20) (pull_request) Successful in 9m59s Details
/ Tests (1.21) (pull_request) Successful in 9m53s Details
/ DCO (pull_request) Successful in 11m34s Details
/ Builds (1.20) (pull_request) Successful in 12m24s Details
/ Builds (1.21) (pull_request) Successful in 8m59s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-11-22 11:12:32 +00:00
Marina Biryukova fe796ba538 [#217] Consider Copy-Source-SSE-* headers during copy 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-11-13 13:22:58 +00:00
Denis Kirillov 5ee73fad6a [#248] Correct NextVersionIDMarker in listing versions
/ DCO (pull_request) Successful in 4m40s Details
/ Vulncheck (pull_request) Successful in 5m43s Details
/ Lint (pull_request) Successful in 10m27s Details
/ Tests (1.20) (pull_request) Successful in 5m41s Details
/ Tests (1.21) (pull_request) Successful in 5m47s Details
/ Builds (1.20) (pull_request) Successful in 1m9s Details
/ Builds (1.21) (pull_request) Successful in 4m7s Details 
Despite the spec https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html#API_ListObjectVersions_ResponseElements
says that
"When the number of responses exceeds the value of MaxKeys,
NextVersionIdMarker specifies the first object version not returned
 that satisfies the search criteria. Use this value for the
 version-id-marker request parameter in a subsequent request."
 the actual behavior of AWS S3 is returning NextVersionIdMarker as the last returned object version

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 17:36:24 +03:00
Denis Kirillov b169c5e6c3 [#239] Update test for check goroutines leak 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 13:51:23 +00:00
Denis Kirillov 0938d7ee82 [#226] Fix status code in GET/HEAD delete marker
/ Vulncheck (pull_request) Successful in 2m26s Details
/ Lint (pull_request) Successful in 4m0s Details
/ Tests (1.20) (pull_request) Successful in 2m59s Details
/ Tests (1.21) (pull_request) Successful in 1m52s Details
/ DCO (pull_request) Successful in 4m17s Details
/ Builds (1.20) (pull_request) Successful in 6m33s Details
/ Builds (1.21) (pull_request) Successful in 1m28s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-27 10:58:57 +03:00
Marina Biryukova 25bb581fee [#205] Add md5 checksum in header
/ Vulncheck (pull_request) Successful in 1m50s Details
/ DCO (pull_request) Successful in 3m29s Details
/ Lint (pull_request) Successful in 3m37s Details
/ Tests (1.20) (pull_request) Successful in 2m9s Details
/ Tests (1.21) (pull_request) Successful in 2m5s Details
/ Builds (1.20) (pull_request) Successful in 5m47s Details
/ Builds (1.21) (pull_request) Successful in 1m29s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-25 11:04:19 +03:00
Marina Biryukova 298662df9d [#221] Expand xmlns field ignore
/ Vulncheck (pull_request) Successful in 1m38s Details
/ Lint (pull_request) Successful in 2m49s Details
/ Tests (1.20) (pull_request) Successful in 1m56s Details
/ Tests (1.21) (pull_request) Successful in 1m44s Details
/ DCO (pull_request) Successful in 3m56s Details
/ Builds (1.20) (pull_request) Successful in 6m25s Details
/ Builds (1.21) (pull_request) Successful in 1m36s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-13 16:21:13 +03:00
Denis Kirillov 65412ce1d3 [#197] Configure buffer max size for PUT 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-11 12:32:48 +00:00
Denis Kirillov 7de73f6b73 [#197] Disable homomorphic hash for PUT 
Disable TZ hash for PUT if it's disabled for container itself

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-11 12:32:48 +00:00
Marina Biryukova 51e591877b [#207] Fix list parts with empty list
/ DCO (pull_request) Successful in 1m2s Details
/ Vulncheck (pull_request) Successful in 1m25s Details
/ Builds (1.20) (pull_request) Successful in 1m59s Details
/ Builds (1.21) (pull_request) Successful in 1m52s Details
/ Lint (pull_request) Successful in 8m23s Details
/ Tests (1.20) (pull_request) Successful in 2m1s Details
/ Tests (1.21) (pull_request) Successful in 9m26s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-21 11:27:20 +03:00
Marina Biryukova a4c612614a [#210] Fix multipart object reader
/ Vulncheck (pull_request) Successful in 2m38s Details
/ DCO (pull_request) Successful in 3m40s Details
/ Lint (pull_request) Successful in 5m20s Details
/ Tests (1.20) (pull_request) Successful in 3m19s Details
/ Tests (1.21) (pull_request) Successful in 3m8s Details
/ Builds (1.20) (pull_request) Successful in 6m1s Details
/ Builds (1.21) (pull_request) Successful in 1m33s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-19 16:30:08 +03:00
Marina Biryukova 12cf29aed2 [#207] Fix part-number-marker handling
/ DCO (pull_request) Successful in 1m57s Details
/ Vulncheck (pull_request) Successful in 2m21s Details
/ Builds (1.20) (pull_request) Successful in 1m36s Details
/ Builds (1.21) (pull_request) Successful in 7m55s Details
/ Lint (pull_request) Successful in 13m57s Details
/ Tests (1.20) (pull_request) Successful in 2m34s Details
/ Tests (1.21) (pull_request) Successful in 1m57s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-19 12:43:07 +03:00
Denis Kirillov 6617adc22b [#185] Use correct object size when object is combined or encrypted 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-24 18:03:08 +03:00
Roman Loginov 8efcc957ea [#96] Move log messages to constants
/ DCO (pull_request) Successful in 1m35s Details
/ Builds (1.19) (pull_request) Successful in 2m14s Details
/ Builds (1.20) (pull_request) Successful in 2m9s Details
/ Vulncheck (pull_request) Successful in 5m39s Details
/ Lint (pull_request) Successful in 2m49s Details
/ Tests (1.19) (pull_request) Successful in 7m34s Details
/ Tests (1.20) (pull_request) Successful in 1m44s Details 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-08-23 18:32:31 +03:00
Denis Kirillov 6b728fef87 [#192] Add tests to make sure client_cut flag is passed to sdk 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-23 06:26:55 +00:00
Denis Kirillov 6b1f365e65 [#192] Support client cut 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-23 06:26:55 +00:00
Artem Tataurov 94a42fa273 [#51] Update frostfs-sdk-go 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-08-17 14:26:25 +03:00
Roman Loginov 40d7f844e3 [#137] Refactor context data retrievers 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-08-16 14:05:38 +00:00
Denis Kirillov 18878b66d3 [#175] Use gate owner as object owner
/ DCO (pull_request) Failing after 3s Details
/ Vulncheck (pull_request) Failing after 2s Details
/ Builds (1.19) (pull_request) Failing after 3s Details
/ Builds (1.20) (pull_request) Successful in 7m47s Details
/ Lint (pull_request) Failing after 4s Details
/ Tests (1.19) (pull_request) Successful in 8m8s Details
/ Tests (1.20) (pull_request) Failing after 3s Details 
This is required because node check session token owner
TrueCloudLab/frostfs-node#528
For client cut TrueCloudLab/frostfs-sdk-go#114
such owner will be gate owner

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 12:22:33 +03:00
Denis Kirillov fe897ec588 [#178] wrapReader: Fix goroutine leak
/ Vulncheck (pull_request) Successful in 59s Details
/ Builds (1.19) (pull_request) Successful in 1m28s Details
/ Builds (1.20) (pull_request) Successful in 1m19s Details
/ DCO (pull_request) Successful in 45s Details
/ Lint (pull_request) Successful in 1m45s Details
/ Tests (1.19) (pull_request) Successful in 7m51s Details
/ Tests (1.20) (pull_request) Successful in 1m43s Details 
In case of error in FrostFS.CreateObject wrapped reader
can be blocked because of synchronous pipe. We have to read out all payload in such case.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-01 17:59:34 +03:00
Denis Kirillov 1a09041cd1 [#63] Simplify multiObjectReader and add tests
/ DCO (pull_request) Successful in 1m19s Details
/ Vulncheck (pull_request) Successful in 2m31s Details
/ Builds (1.19) (pull_request) Successful in 2m51s Details
/ Builds (1.20) (pull_request) Successful in 2m34s Details
/ Lint (pull_request) Successful in 5m51s Details
/ Tests (1.19) (pull_request) Successful in 9m36s Details
/ Tests (1.20) (pull_request) Successful in 10m20s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov bcf5a85aab [#63] multipart: Fix copying 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov ad81b599dd [#63] Add fast multipart upload 
Add new flag to object tree meta `isCombined` that means
the object payload is list of parts that forms real payload.
Set this attribute when complete multipart upload not to do unnecessary copying.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov 14ef9ff091 [#158] Separate init object reader from read itself
/ Vulncheck (pull_request) Successful in 1m40s Details
/ Builds (1.19) (pull_request) Successful in 2m54s Details
/ Builds (1.20) (pull_request) Successful in 2m39s Details
/ DCO (pull_request) Successful in 3m48s Details
/ Lint (pull_request) Successful in 3m32s Details
/ Tests (1.19) (pull_request) Successful in 2m38s Details
/ Tests (1.20) (pull_request) Successful in 2m55s Details 
To be able to handle cases and return appropriate http status code
when object missed in storage but gate cache contains its metadata
we need write code after init object reader.
So we separate init reader from actual reading.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:32:05 +03:00
Denis Kirillov fc90981c03 [#149] Update inner imports after moving middlewares 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:09 +03:00
Denis Kirillov d531b13866 [#143] Add more context to some s3 errors
/ DCO (pull_request) Successful in 1m18s Details
/ Vulncheck (pull_request) Successful in 1m50s Details
/ Lint (pull_request) Successful in 3m1s Details
/ Tests (1.19) (pull_request) Successful in 2m47s Details
/ Tests (1.20) (pull_request) Successful in 3m2s Details
/ Builds (1.19) (pull_request) Successful in 2m54s Details
/ Builds (1.20) (pull_request) Successful in 7m58s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-30 12:08:33 +03:00
Denis Kirillov be03c5178f [#143] Fix NoSuchKey error on get/head 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-30 12:04:52 +03:00
Denis Kirillov 23593eee3d [#111] Use request scope logger
/ Lint (pull_request) Failing after 32s Details
/ Tests (1.19) (pull_request) Failing after 32s Details
/ Tests (1.20) (pull_request) Failing after 32s Details
/ Builds (1.19) (pull_request) Failing after 32s Details
/ Builds (1.20) (pull_request) Failing after 32s Details
/ Vulncheck (pull_request) Failing after 57s Details
/ DCO (pull_request) Failing after 1m0s Details 
Store child zap logger with request scope fields into context.
Request scoped fields: request_id, api/method, bucket, object

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-19 13:54:51 +03:00
Denis Kirillov 7a380fa46c [#135] frostfs: Add SEARCH operation 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-16 09:42:41 +03:00
Denis Kirillov 462589fc0c [#103] Return 504 http code on timeout errors
/ Vulncheck (pull_request) Successful in 1m42s Details
/ Lint (pull_request) Successful in 3m31s Details
/ Tests (1.19) (pull_request) Successful in 2m57s Details
/ Tests (1.20) (pull_request) Successful in 3m4s Details
/ Builds (1.19) (pull_request) Successful in 2m35s Details
/ Builds (1.20) (pull_request) Successful in 2m39s Details
/ DCO (pull_request) Successful in 1m7s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-14 09:16:21 +03:00
Denis Kirillov 4a6e3a19ce [#125] Handle negative Content-Length on put 
Add computing actual object size during calculating hash on put.
Use this actual value to save in tree and cache

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-02 14:16:19 +00:00
Alexey Vanin 34bbbcf1ed [#78] Do not show objects missing from object service 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-06-01 16:13:28 +03:00
Alexey Vanin fae03c2b50 [#78] Process 'not found' error when object exists in tree 
When object exists in tree but missing in storage, we can't remove
bucket. While storage node does not sync tree service and object
service, the only way to delete such broken bucket is to ignore
'object not found' error, clear cache and do not include missing
objects in the listing result.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-06-01 15:31:22 +03:00
Alexey Vanin 81e860481d [#122] Fix linter warnings
Builds (1.19) Details
DCO Details
Builds (1.20) Details
Lint Details
Tests (1.19) Details
Tests (1.20) Details 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-05-31 19:43:03 +03:00
Artem Tataurov a0f0d792b8 [#78] layer: Clean up already removed object from tree 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-24 17:42:46 +03:00
Alexey Vanin 3fbad97ae4 [#1] Rename files with mentions of previous project 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-05-19 07:30:04 +00:00
Denis Kirillov b366e75366 [#81] Use impersonate bearer token 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-05-03 17:22:52 +03:00
Artem Tataurov e487ee5b7d [#70] Add arrays of copies numbers for location constraints 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-03 13:48:26 +03:00
Denis Kirillov 70ec5a0a5b [#83] Don't create extra delete marker 
We shouldn't create delete marker if:
1. object doesn't exist at all
2. last version is already a delete marker

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-19 17:56:11 +03:00
Artem Tataurov fb99b26209 [#57] api: Clean up List and Name caches when object is missing in Tree service 2023-04-04 17:15:04 +03:00
Denis Kirillov a0937126cb [#64] Support new system attributes 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-22 11:46:58 +03:00
Alexey Vanin 813aa2f173 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 17:38:08 +03:00
Denis Kirillov 740acadd37 [TrueCloudLab#36] Fix cors object payload 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-20 19:22:48 +03:00
Alexey Vanin 8ac630ee71 [TrueCloudLab#25] Add zone data to BucketInfo 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-02-17 13:47:48 +03:00
Artem Tataurov 8e14ccad9f [#754] Remove duplicating CID/OID logging calls 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
(cherry picked from commit 8c178fc3c7)
 2022-12-28 17:06:32 +03:00
Artem Tataurov af7fbd6f32 [#754] Add CID and OID to logs 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2022-12-28 17:06:32 +03:00