TrueCloudLab
/
frostfs-s3-gw
15
2
Fork 16
Code Issues 21 Pull Requests 9 Actions Packages Projects Releases 9 Activity
1,305 Commits
6 Branches
28 Tags
19 MiB
Commit Graph

698 Commits (0cc6b41b2d3f73ac9422381b62eb29a735120a12)

Author SHA1 Message Date
Marina Biryukova 51e591877b [#207] Fix list parts with empty list
/ DCO (pull_request) Successful in 1m2s Details
/ Vulncheck (pull_request) Successful in 1m25s Details
/ Builds (1.20) (pull_request) Successful in 1m59s Details
/ Builds (1.21) (pull_request) Successful in 1m52s Details
/ Lint (pull_request) Successful in 8m23s Details
/ Tests (1.20) (pull_request) Successful in 2m1s Details
/ Tests (1.21) (pull_request) Successful in 9m26s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-21 11:27:20 +03:00
Marina Biryukova a4c612614a [#210] Fix multipart object reader
/ Vulncheck (pull_request) Successful in 2m38s Details
/ DCO (pull_request) Successful in 3m40s Details
/ Lint (pull_request) Successful in 5m20s Details
/ Tests (1.20) (pull_request) Successful in 3m19s Details
/ Tests (1.21) (pull_request) Successful in 3m8s Details
/ Builds (1.20) (pull_request) Successful in 6m1s Details
/ Builds (1.21) (pull_request) Successful in 1m33s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-19 16:30:08 +03:00
Marina Biryukova 12cf29aed2 [#207] Fix part-number-marker handling
/ DCO (pull_request) Successful in 1m57s Details
/ Vulncheck (pull_request) Successful in 2m21s Details
/ Builds (1.20) (pull_request) Successful in 1m36s Details
/ Builds (1.21) (pull_request) Successful in 7m55s Details
/ Lint (pull_request) Successful in 13m57s Details
/ Tests (1.20) (pull_request) Successful in 2m34s Details
/ Tests (1.21) (pull_request) Successful in 1m57s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-19 12:43:07 +03:00
Marina Biryukova 066b9a0250 [#142] Add trace ID into log when tracing is enabled
/ Vulncheck (pull_request) Successful in 1m29s Details
/ DCO (pull_request) Successful in 2m18s Details
/ Lint (pull_request) Successful in 3m48s Details
/ Tests (1.20) (pull_request) Successful in 2m4s Details
/ Tests (1.21) (pull_request) Successful in 1m42s Details
/ Builds (1.20) (pull_request) Successful in 7m6s Details
/ Builds (1.21) (pull_request) Successful in 1m56s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-07 14:19:37 +03:00
Marina Biryukova 9120e97ac5 [#203] Add go1.21 to CI
/ DCO (pull_request) Successful in 59s Details
/ Vulncheck (pull_request) Successful in 1m35s Details
/ Builds (1.20) (pull_request) Successful in 1m21s Details
/ Builds (1.21) (pull_request) Successful in 7m46s Details
/ Lint (pull_request) Successful in 12m44s Details
/ Tests (1.20) (pull_request) Successful in 2m9s Details
/ Tests (1.21) (pull_request) Successful in 2m0s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-08-31 15:26:07 +03:00
Denis Kirillov b5fce5c8d2 [#168] Skip only invalid policies and copies instead of ignoring all of them
/ DCO (pull_request) Successful in 2m48s Details
/ Vulncheck (pull_request) Successful in 3m30s Details
/ Builds (1.19) (pull_request) Successful in 3m39s Details
/ Builds (1.20) (pull_request) Successful in 3m28s Details
/ Lint (pull_request) Successful in 4m14s Details
/ Tests (1.19) (pull_request) Successful in 3m3s Details
/ Tests (1.20) (pull_request) Successful in 2m45s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-25 12:05:30 +03:00
Denis Kirillov 6617adc22b [#185] Use correct object size when object is combined or encrypted 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-24 18:03:08 +03:00
Denis Kirillov 8898c2ec08 [#185] Add tests for list multipart uploads 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-24 18:01:08 +03:00
Roman Loginov 8efcc957ea [#96] Move log messages to constants
/ DCO (pull_request) Successful in 1m35s Details
/ Builds (1.19) (pull_request) Successful in 2m14s Details
/ Builds (1.20) (pull_request) Successful in 2m9s Details
/ Vulncheck (pull_request) Successful in 5m39s Details
/ Lint (pull_request) Successful in 2m49s Details
/ Tests (1.19) (pull_request) Successful in 7m34s Details
/ Tests (1.20) (pull_request) Successful in 1m44s Details 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-08-23 18:32:31 +03:00
Denis Kirillov 6b728fef87 [#192] Add tests to make sure client_cut flag is passed to sdk 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-23 06:26:55 +00:00
Denis Kirillov 6b1f365e65 [#192] Support client cut 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-23 06:26:55 +00:00
Denis Kirillov fcf1c45ad2 [#188] Fix url escaping
/ Vulncheck (pull_request) Successful in 2m41s Details
/ Builds (1.19) (pull_request) Successful in 3m51s Details
/ Builds (1.20) (pull_request) Successful in 3m22s Details
/ DCO (pull_request) Successful in 5m9s Details
/ Lint (pull_request) Successful in 5m22s Details
/ Tests (1.19) (pull_request) Successful in 5m39s Details
/ Tests (1.20) (pull_request) Successful in 5m18s Details 
Url escaping has already been done in `net/http/request.go`

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-22 11:27:39 +03:00
Artem Tataurov 94a42fa273 [#51] Update frostfs-sdk-go 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-08-17 14:26:25 +03:00
Roman Loginov 40d7f844e3 [#137] Refactor context data retrievers 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-08-16 14:05:38 +00:00
Denis Kirillov 6b109eee92 [#182] Fix parsing signed headers in presigned urls
/ Vulncheck (pull_request) Failing after 3s Details
/ Builds (1.19) (pull_request) Successful in 8m22s Details
/ Builds (1.20) (pull_request) Failing after 3s Details
/ Lint (pull_request) Successful in 9m51s Details
/ Tests (1.19) (pull_request) Failing after 3s Details
/ Tests (1.20) (pull_request) Failing after 2s Details
/ DCO (pull_request) Failing after 2s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 13:04:53 +03:00
Denis Kirillov 18878b66d3 [#175] Use gate owner as object owner
/ DCO (pull_request) Failing after 3s Details
/ Vulncheck (pull_request) Failing after 2s Details
/ Builds (1.19) (pull_request) Failing after 3s Details
/ Builds (1.20) (pull_request) Successful in 7m47s Details
/ Lint (pull_request) Failing after 4s Details
/ Tests (1.19) (pull_request) Successful in 8m8s Details
/ Tests (1.20) (pull_request) Failing after 3s Details 
This is required because node check session token owner
TrueCloudLab/frostfs-node#528
For client cut TrueCloudLab/frostfs-sdk-go#114
such owner will be gate owner

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 12:22:33 +03:00
Denis Kirillov fe897ec588 [#178] wrapReader: Fix goroutine leak
/ Vulncheck (pull_request) Successful in 59s Details
/ Builds (1.19) (pull_request) Successful in 1m28s Details
/ Builds (1.20) (pull_request) Successful in 1m19s Details
/ DCO (pull_request) Successful in 45s Details
/ Lint (pull_request) Successful in 1m45s Details
/ Tests (1.19) (pull_request) Successful in 7m51s Details
/ Tests (1.20) (pull_request) Successful in 1m43s Details 
In case of error in FrostFS.CreateObject wrapped reader
can be blocked because of synchronous pipe. We have to read out all payload in such case.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-01 17:59:34 +03:00
Denis Kirillov 52931663e1 [#176] multipart: Replace part on re-upload
/ Builds (1.19) (pull_request) Successful in 2m57s Details
/ Builds (1.20) (pull_request) Successful in 2m36s Details
/ DCO (pull_request) Successful in 1m15s Details
/ Vulncheck (pull_request) Successful in 5m15s Details
/ Lint (pull_request) Successful in 3m50s Details
/ Tests (1.19) (pull_request) Successful in 8m14s Details
/ Tests (1.20) (pull_request) Successful in 3m17s Details 
We want to have exactly one object and tree node for each part number

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-21 16:20:04 +03:00
Denis Kirillov 1a09041cd1 [#63] Simplify multiObjectReader and add tests
/ DCO (pull_request) Successful in 1m19s Details
/ Vulncheck (pull_request) Successful in 2m31s Details
/ Builds (1.19) (pull_request) Successful in 2m51s Details
/ Builds (1.20) (pull_request) Successful in 2m34s Details
/ Lint (pull_request) Successful in 5m51s Details
/ Tests (1.19) (pull_request) Successful in 9m36s Details
/ Tests (1.20) (pull_request) Successful in 10m20s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov 631b7f67b4 [#63] multipart: Log upload id for every failed request 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov bcf5a85aab [#63] multipart: Fix copying 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov ad81b599dd [#63] Add fast multipart upload 
Add new flag to object tree meta `isCombined` that means
the object payload is list of parts that forms real payload.
Set this attribute when complete multipart upload not to do unnecessary copying.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov 361d10cc78 [#174] Fix query for listing multipart uploads
/ Builds (1.19) (pull_request) Successful in 3m2s Details
/ Builds (1.20) (pull_request) Successful in 2m50s Details
/ DCO (pull_request) Successful in 3m59s Details
/ Vulncheck (pull_request) Successful in 2m39s Details
/ Lint (pull_request) Successful in 3m56s Details
/ Tests (1.19) (pull_request) Successful in 3m5s Details
/ Tests (1.20) (pull_request) Successful in 3m12s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 12:30:17 +03:00
Denis Kirillov 62e6b49254 [#174] Log unmatched requests 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 12:30:17 +03:00
Denis Kirillov 80c4982bd4 [#174] Add router tests 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 12:30:17 +03:00
Denis Kirillov 73ed3f7782 [#174] Fix router filter query matching 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 12:30:17 +03:00
Denis Kirillov 6e3595e35b [#174] Fix object keys with slashes in chi 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 12:30:17 +03:00
Denis Kirillov b59aa06637 [#146] Add kludge.bypass_content_encoding_check_in_chunks flag
/ Builds (1.19) (pull_request) Successful in 3m0s Details
/ Builds (1.20) (pull_request) Successful in 2m50s Details
/ DCO (pull_request) Failing after 1m9s Details
/ Vulncheck (pull_request) Successful in 1m20s Details
/ Lint (pull_request) Successful in 7m50s Details
/ Tests (1.19) (pull_request) Successful in 3m1s Details
/ Tests (1.20) (pull_request) Successful in 3m21s Details 
Flag allows to skip checking `Content-Encoding` for `aws-chunked` value

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-18 14:49:52 +03:00
Denis Kirillov d62aa7b979 [#146] Fix preconditions: trim quotes in etags 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-18 14:49:03 +03:00
Denis Kirillov 751a9be7cc [#146] Move getting chunk payload reader to separate function 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-18 14:49:03 +03:00
Denis Kirillov 14ef9ff091 [#158] Separate init object reader from read itself
/ Vulncheck (pull_request) Successful in 1m40s Details
/ Builds (1.19) (pull_request) Successful in 2m54s Details
/ Builds (1.20) (pull_request) Successful in 2m39s Details
/ DCO (pull_request) Successful in 3m48s Details
/ Lint (pull_request) Successful in 3m32s Details
/ Tests (1.19) (pull_request) Successful in 2m38s Details
/ Tests (1.20) (pull_request) Successful in 2m55s Details 
To be able to handle cases and return appropriate http status code
when object missed in storage but gate cache contains its metadata
we need write code after init object reader.
So we separate init reader from actual reading.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:32:05 +03:00
Denis Kirillov fc90981c03 [#149] Update inner imports after moving middlewares 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:09 +03:00
Denis Kirillov 83cdfbee78 [#149] Move middlewares to separate package 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:09 +03:00
Denis Kirillov 37f2f468fe [#149] Add host bucket router 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:09 +03:00
Denis Kirillov e30a452cdb [#149] Use chi instead of gorilla mux 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:09 +03:00
Denis Kirillov 7f708b3a2d [#111] auth: Get log from real request context 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-06 11:57:04 +00:00
Denis Kirillov d531b13866 [#143] Add more context to some s3 errors
/ DCO (pull_request) Successful in 1m18s Details
/ Vulncheck (pull_request) Successful in 1m50s Details
/ Lint (pull_request) Successful in 3m1s Details
/ Tests (1.19) (pull_request) Successful in 2m47s Details
/ Tests (1.20) (pull_request) Successful in 3m2s Details
/ Builds (1.19) (pull_request) Successful in 2m54s Details
/ Builds (1.20) (pull_request) Successful in 7m58s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-30 12:08:33 +03:00
Denis Kirillov f921bc8af5 [#143] Fix typo 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-30 12:04:52 +03:00
Denis Kirillov be03c5178f [#143] Fix NoSuchKey error on get/head 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-30 12:04:52 +03:00
Denis Kirillov 9df8695463 [#143] Fix transformToS3Error function
/ Builds (1.19) (pull_request) Successful in 3m11s Details
/ Builds (1.20) (pull_request) Successful in 2m57s Details
/ DCO (pull_request) Successful in 4m7s Details
/ Lint (pull_request) Successful in 2m25s Details
/ Tests (1.19) (pull_request) Successful in 3m9s Details
/ Tests (1.20) (pull_request) Successful in 3m18s Details
/ Vulncheck (pull_request) Successful in 1m15s Details 
Unwrap error before checking for s3 error

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-21 17:16:40 +03:00
Artem Tataurov 614d703726 [#106] Add chunk uploading
/ DCO (pull_request) Successful in 1m4s Details
/ Vulncheck (pull_request) Successful in 1m50s Details
/ Builds (1.19) (pull_request) Successful in 9m1s Details
/ Builds (1.20) (pull_request) Successful in 2m20s Details
/ Lint (pull_request) Successful in 10m19s Details
/ Tests (1.19) (pull_request) Successful in 2m45s Details
/ Tests (1.20) (pull_request) Successful in 3m19s Details 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-06-21 10:23:57 +03:00
Denis Kirillov 23593eee3d [#111] Use request scope logger
/ Lint (pull_request) Failing after 32s Details
/ Tests (1.19) (pull_request) Failing after 32s Details
/ Tests (1.20) (pull_request) Failing after 32s Details
/ Builds (1.19) (pull_request) Failing after 32s Details
/ Builds (1.20) (pull_request) Failing after 32s Details
/ Vulncheck (pull_request) Failing after 57s Details
/ DCO (pull_request) Failing after 1m0s Details 
Store child zap logger with request scope fields into context.
Request scoped fields: request_id, api/method, bucket, object

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-19 13:54:51 +03:00
Denis Kirillov 84358f6742 [#135] authmate: Support CRDT GSet for credentials 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-16 09:42:49 +03:00
Denis Kirillov 7a380fa46c [#135] frostfs: Add SEARCH operation 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-16 09:42:41 +03:00
Denis Kirillov 462589fc0c [#103] Return 504 http code on timeout errors
/ Vulncheck (pull_request) Successful in 1m42s Details
/ Lint (pull_request) Successful in 3m31s Details
/ Tests (1.19) (pull_request) Successful in 2m57s Details
/ Tests (1.20) (pull_request) Successful in 3m4s Details
/ Builds (1.19) (pull_request) Successful in 2m35s Details
/ Builds (1.20) (pull_request) Successful in 2m39s Details
/ DCO (pull_request) Successful in 1m7s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-14 09:16:21 +03:00
Denis Kirillov 9dabaf6ecd [#133] Use tree pool from SDK 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-09 09:34:04 +03:00
Pavel Pogodaev 4e1fd9589b [#84] add tracing support
Builds (1.20) Details
Builds (1.19) Details
DCO Details
Tests (1.19) Details
Tests (1.20) Details
Lint Details 
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
 2023-06-06 10:25:27 +03:00
Denis Kirillov 4a6e3a19ce [#125] Handle negative Content-Length on put 
Add computing actual object size during calculating hash on put.
Use this actual value to save in tree and cache

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-02 14:16:19 +00:00
Denis Kirillov b445f7bbf9 [#125] api/auth: DisableURIPathEscaping for presign 
Don't use escaping when presign url.
Escape manually before.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-02 14:16:19 +00:00
Alexey Vanin 868edfdb31 [#78] Add test of bucket removal with object not found error
DCO Details
Builds (1.20) Details
Tests (1.19) Details
Tests (1.20) Details
Builds (1.19) Details
Lint Details 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-06-01 16:13:28 +03:00
Alexey Vanin 34bbbcf1ed [#78] Do not show objects missing from object service 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-06-01 16:13:28 +03:00
Alexey Vanin fae03c2b50 [#78] Process 'not found' error when object exists in tree 
When object exists in tree but missing in storage, we can't remove
bucket. While storage node does not sync tree service and object
service, the only way to delete such broken bucket is to ignore
'object not found' error, clear cache and do not include missing
objects in the listing result.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-06-01 15:31:22 +03:00
Alexey Vanin 81e860481d [#122] Fix linter warnings
Builds (1.19) Details
DCO Details
Builds (1.20) Details
Lint Details
Tests (1.19) Details
Tests (1.20) Details 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-05-31 19:43:03 +03:00
Artem Tataurov a0f0d792b8 [#78] layer: Clean up already removed object from tree 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-24 17:42:46 +03:00
Artem Tataurov 9f186d9aba [#104] app: Reload copies numbers on SIGHUP 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-23 13:19:58 +03:00
Alexey Vanin 3fbad97ae4 [#1] Rename files with mentions of previous project 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-05-19 07:30:04 +00:00
Artem Tataurov e24bc3f2ce [#101] app: Refactor the default copies number setting 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-17 11:36:28 +03:00
Denis Kirillov b366e75366 [#81] Use impersonate bearer token 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-05-03 17:22:52 +03:00
Artem Tataurov e487ee5b7d [#70] Add arrays of copies numbers for location constraints 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-03 13:48:26 +03:00
Denis Kirillov ee8cce662b [#74] service/tree: Add logger 
Log error instead of failing when multiple unversioned nodes are found

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-26 16:39:57 +03:00
Denis Kirillov 9e72fe1662 [#80] Refactor metrics, support dump descriptions 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-20 11:14:52 +03:00
Denis Kirillov 70ec5a0a5b [#83] Don't create extra delete marker 
We shouldn't create delete marker if:
1. object doesn't exist at all
2. last version is already a delete marker

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-19 17:56:11 +03:00
Artem Tataurov a5c89b78bc [#79] Return X-Owner-Id in head-bucket response 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-04-17 10:12:25 +03:00
Artem Tataurov fb99b26209 [#57] api: Clean up List and Name caches when object is missing in Tree service 2023-04-04 17:15:04 +03:00
Denis Kirillov a025f2e9c5 [#59] tree: Make interface for tree service client 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-29 09:34:10 +03:00
Denis Kirillov bd3164c57f [#68] Fix pre-commit issues 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-24 16:22:06 +03:00
Denis Kirillov a0937126cb [#64] Support new system attributes 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-22 11:46:58 +03:00
Alexey Vanin ef556bd8ac [#60] Use session token to set eACL during Complete Multipart Upload 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-22 11:22:11 +03:00
Alexey Vanin 5104683f68 [#60] Refactor start of periodic XML writer 
Reduce code duplication for error handling

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-22 11:02:16 +03:00
Alexey Vanin 8151753eeb [#60] Use periodic white space XML writer in Complete Multipart Upload 
This mechanism is used by Amazon S3 to keep client's
connection alive while object is being constructed from
the upload parts.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-20 13:34:22 +03:00
Alexey Vanin 2282c32822 [#60] Add *NoHeader functions 
Such functions should be used together with periodic white space
XML writer.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-20 12:07:36 +03:00
Alexey Vanin 43685e03d9 [#60] Implement flusher on all http.ResponseWriters 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-20 12:04:34 +03:00
Alexey Vanin cf18158da4 [#60] Implement periodic white space XML writer 
Periodic white space XML writer sends XML header
and white spaces to the io.Writer.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-20 12:04:34 +03:00
Denis Kirillov 0af06c3bd9 [TrueCloudLab#40] Add param to configure xml decoder 
This parameter enables parsing xml body without
xmlns="http://s3.amazonaws.com/doc/2006-03-01/" attribute
for CompleteMultipartUpload requests

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-09 15:44:13 +03:00
Denis Kirillov 64e7356acc [TrueCloudLab#32] Add custom policy unmarshaler 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-09 11:19:01 +00:00
Denis Kirillov 32bf915502 [TrueCloudLab#37] Limit number of objects to delete 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-09 11:16:22 +00:00
Alexey Vanin 813aa2f173 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 17:38:08 +03:00
Denis Kirillov 740acadd37 [TrueCloudLab#36] Fix cors object payload 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-20 19:22:48 +03:00
Alexey Vanin aadefd98b6 [TrueCloudLab#25] Process allow and deny lists of zones in bucket head requests 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-02-17 13:47:48 +03:00
Alexey Vanin 8ac630ee71 [TrueCloudLab#25] Add zone data to BucketInfo 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-02-17 13:47:48 +03:00
Denis Kirillov 9f823bd65a [TrueCloudLab#26] Add billing metrics to separate registry 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-10 10:55:38 +03:00
Denis Kirillov 9dcacc230e [TrueCloudLab#23] Return error on unknown LocationConstraint 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-08 15:04:12 +03:00
Artem Tataurov 5ee4bf80ae [#18] Return container name in head-bucket response 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-02-01 14:25:21 +03:00
Denis Kirillov e278ab9362 [TrueCloudLab#5] Refactor middlewares 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-01-25 11:02:23 +03:00
Denis Kirillov c5570e661d [TrueCloudLab#5] Add traffic metrics per user 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-01-25 11:02:23 +03:00
Denis Kirillov fc5c09c084 [TrueCloudLab#5] Request metrics per user 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-01-25 11:02:23 +03:00
Artem Tataurov 8e14ccad9f [#754] Remove duplicating CID/OID logging calls 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
(cherry picked from commit 8c178fc3c7)
 2022-12-28 17:06:32 +03:00
Artem Tataurov af7fbd6f32 [#754] Add CID and OID to logs 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2022-12-28 17:06:32 +03:00
Denis Kirillov 000d9ed42a [#2] Update issue links 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-21 11:17:45 +03:00
Denis Kirillov 388482e230 [#2] Rename internals 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-21 11:17:45 +03:00
Denis Kirillov 765fd28a49 [#2] Update build 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-20 18:24:30 +03:00
Alexey Vanin 96dff367db [#1] Build S3 Gateway with FrostFS dependencies 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-12-15 12:43:52 +03:00
Denis Kirillov dd4f66712c [#742] Add multiple listeners 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-24 17:48:26 +03:00
Denis Kirillov d2587b21af [#747] Reload policies on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-18 13:14:25 +03:00
Angira Kekteeva f5fe9a9b4b [#741] Fix data race 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-11-16 18:19:31 +03:00
Denis Kirillov 8f1bbb7dc1 [#749] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-15 10:16:41 +03:00
Denis Kirillov 094eb12578 [#726] Use client time on regular requests 
Use `X-Amz-Date` header as `now` when
* compute expiration epoch
* set Timestamp for object and container
* forming locks
* send notifications

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-14 14:53:55 +03:00
Denis Kirillov d47840f137 [#568] Add configuration for region to policy map 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-11 13:12:49 +03:00
Denis Kirillov 2886ac161c [#740] Fix forming policy by ast 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-10 15:33:03 +03:00
Denis Kirillov 16ad0cadcb [#733] Fix container name 
User system name instead of regular.
Set name as CID if name attribute is missed

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-26 17:52:02 +03:00
Denis Kirillov 18a6aca4b4 [#617] api/handler: Simplify tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-25 19:04:37 +03:00
Angira Kekteeva 712740b668 [#728] Remove redundant log message 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-10-25 19:04:11 +03:00
Angira Kekteeva bbd7ba670a [#728] Add bucket and object to method logs 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-10-25 19:04:11 +03:00
Angira Kekteeva da61fd8bcb [#728] Add error message to AbortMultipartUpload 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-10-25 19:04:11 +03:00
Denis Kirillov 0d0ecf5b10 [#650] Drop unused object cache methods 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-25 16:00:21 +03:00
Denis Kirillov 821df3d648 [#669] Optimize getNodeVersion for tags operation 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-20 17:56:09 +03:00
Denis Kirillov 90eb4f0188 [#713] Add check access control for system caches 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 15:26:42 +03:00
Denis Kirillov cb55d36063 [#713] Update tests 
Add bearer token to test context

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 15:26:42 +03:00
Denis Kirillov 3d31c2ab4a [#713] Fix cache access 
User can access only keys that he puts into caches

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 15:26:42 +03:00
Denis Kirillov c6098e53f6 [#719] Add tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 11:13:20 +03:00
Denis Kirillov b3d96e6460 [#719] Improve lock error handling 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 11:13:20 +03:00
Denis Kirillov 2a41929be3 [#702] Reload resolvers and TLS certs on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov be4600988e [#709] Add expiration attribute for legal hold 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-26 16:51:53 +03:00
Denis Kirillov 3d08562843 [#505] Handle access denied from tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-13 18:05:16 +03:00
Denis Kirillov 80beedf13e [#504] Use bktInfo in tree service to check owner 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-13 18:05:16 +03:00
Denis Kirillov 4082cd6b54 [#606] Keep eacl records order on conflict 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-08 12:21:47 +03:00
Denis Kirillov ffd259671a [#704] Fix nil pointer dereference on decReader 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-07 15:51:44 +03:00
Denis Kirillov 9cd4ef1ac4 [#657] Replace FileName with FilePath attribute 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-07 15:50:43 +03:00
Denis Kirillov 6108f8d929 [#624] Update tree service revision 
Remove extra request to tree service to know parentID

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-07 12:17:51 +03:00
Angira Kekteeva 3e4fc52428 [#674] Add AccessKeyID filter 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-09-06 13:47:20 +03:00
Alexey Vanin 272c485706 [#669] Use object cache in PutLockInfo optimization 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-08-31 13:19:00 +03:00
Alexey Vanin 5ffe28a42f [#669] Optimize PutLockInfo 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-08-31 13:19:00 +03:00
Denis Kirillov 96c7b79d1c [#683] Forbid copy to itself in unversioned bucket 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-30 13:31:48 +03:00
Denis Kirillov 44d5878a80 [#656] Read full body on upload error 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-30 09:39:20 +03:00
Alexey Vanin 9590b39333 [#682] Return added node id in `AddVersion` 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-08-29 18:24:00 +03:00
Denis Kirillov 5c7a509cc9 [#684] Fix continuation token for directories 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-29 16:10:01 +03:00
Denis Kirillov e38bdae07a [#676] Fix object acl 
Put object acl always add rules to specific version of object.
Get object acl consider READ rights as FULL_CONTROL
because WRITE cannot be applied to object

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-29 13:20:30 +03:00
Denis Kirillov 163038b37d [#672] Fix handling X-Amz-Copy-Source header 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 17:40:15 +03:00
Denis Kirillov fdc926e70b [#661] Fix updating cache when request fails 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 12:55:49 +03:00
Denis Kirillov 1a580b6fa4 [#666] Support tagging directive for CopyObject 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 12:55:33 +03:00
Denis Kirillov f013282290 [#660] Fix panic on invalid versioning status 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 11:50:44 +03:00
Denis Kirillov 24e1f7e1dd [#545] Drop object search 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 11:49:08 +03:00
Denis Kirillov c3ad6d2faf [#637] Add header to override CopiesNumber 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-19 10:31:26 +03:00
Angira Kekteeva 3b343d1bd5 [#634] Fix tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-16 17:49:15 +03:00
Angira Kekteeva 5307211398 [#634] Add CopiesNumber in NeoFS requests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-16 17:49:15 +03:00
Denis Kirillov 6aab3936d9 [#625] Limit listing generator with maxKeys 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-16 13:47:37 +03:00
Denis Kirillov 3824151699 [#616] Reduce number of requests during listing 
Check if object is directory before request to NeoFS

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-16 13:47:37 +03:00
Denis Kirillov 107d8a9033 [#595] Fix typos and rewording 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov 7ab473a688 [#595] Simplify encryption.Params struct 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov 94a6a55919 [#595] Move decrypter to separate package 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov d824db7f69 [#595] Allow SSE-C only with TLS 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov 87c05eb514 [#595] Add SSE with customer key 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Angira Kekteeva 1987f271f9 [#577] Remove usages of deprecated io/ioutil 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva f66e4a77cd [#577] Add IsDeleteMarker() to NodeVersion 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva 2a933f48be [#577] Remove redundant set of isLatest 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva 3188fd8eb7 [#577] Fix delete from tagging cache 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva 8031baddb6 [#577] Fix cache tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva 245e64900d [#577] Replace ObjInfo with ExtObjInfo in cache 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva dc77ad4863 [#577] Fix tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva 93e0cae743 [#577] Fix version in GetObjectAttributes 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00