TrueCloudLab/frostfs-s3-gw
18
3
Fork 20
Code Issues 29 Pull requests 9 Projects Releases 19 Packages Activity Actions
1331 commits 6 branches 44 tags 23 MiB
Commit graph

64 commits

Author SHA1 Message Date
Denis Kirillov
5d6d5e41d0 [#398] Support retryer
Some checks failed
/ DCO (pull_request) Successful in 1m40s
Details
/ Builds (1.20) (pull_request) Successful in 2m22s
Details
/ Builds (1.21) (pull_request) Successful in 1m56s
Details
/ Vulncheck (pull_request) Failing after 2m40s
Details
/ Lint (pull_request) Successful in 4m31s
Details
/ Tests (1.20) (pull_request) Successful in 2m45s
Details
/ Tests (1.21) (pull_request) Successful in 2m38s
Details 
Add two strategy for PutBucketSettings request retryer:
* exponential backoff (increasing up to `max_backoff` delays with jitter)
* constant backoff (always the same `max_backoff` delay between requests)

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>

(cherry picked from commit bb81afc14a)
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2024-06-19 08:55:45 +03:00
Alex Vanin
b7e15402a1 [#360] Use 'c' prefix for bucket policies instead of 'n' 
With 'c' prefix, acl chains become shorter, thus gateway
receives shorter results and avoids sessions to neo-go.

There is still issue with many IAM rules.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2024-04-10 17:40:25 +03:00
Denis Kirillov
bcfbcdc82f [#345] acl: Update APE and fix using
Some checks failed
/ Lint (pull_request) Successful in 2m53s
Details
/ Tests (1.20) (pull_request) Successful in 2m43s
Details
/ Tests (1.21) (pull_request) Successful in 2m44s
Details
/ DCO (pull_request) Successful in 2m40s
Details
/ Vulncheck (pull_request) Failing after 3m10s
Details
/ Builds (1.20) (pull_request) Successful in 3m18s
Details
/ Builds (1.21) (pull_request) Successful in 3m33s
Details 
* Remove native policy when remove bucket policy
* Allow policies that contain only s3 compatible statements
(now deny rules cannot be converted to native rules)

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-04-02 12:42:43 +00:00
Denis Kirillov
8f89f275bd [#306] Save bucket policy as native chain 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-28 17:50:08 +03:00
Denis Kirillov
c868af8a62 [#306] Add flag to enable old ACL bucket creation 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-28 17:50:08 +03:00
Denis Kirillov
c452d58ce2 [#306] Reduce number of policy contract invocations 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-28 17:50:08 +03:00
Denis Kirillov
d9d12debc3 [#306] Add tests 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-28 17:50:08 +03:00
Denis Kirillov
3d0d2032c6 [#306] acl: Handle put/get acl for APE buckets 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-28 17:50:08 +03:00
Denis Kirillov
88f1acbdfc [#165] Cancel context in outdated list session 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
c7ee628ab0 [#165] Fix versions listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
29ac91dfd5 [#165] Support streaming listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
84af85ed67 [#302] Update APE to support chain id as bytes
All checks were successful
/ DCO (pull_request) Successful in 1m52s
Details
/ Vulncheck (pull_request) Successful in 2m0s
Details
/ Builds (1.20) (pull_request) Successful in 2m49s
Details
/ Builds (1.21) (pull_request) Successful in 1m59s
Details
/ Lint (pull_request) Successful in 2m44s
Details
/ Tests (1.20) (pull_request) Successful in 2m30s
Details
/ Tests (1.21) (pull_request) Successful in 2m24s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 12:53:45 +03:00
Denis Kirillov
8273af8bf8 [#261] Make PutBucketPolicy handler use policy contract 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-18 15:49:54 +03:00
Denis Kirillov
28c6bb4cb8 [#266] Support per namespace placement policies configuration
All checks were successful
/ DCO (pull_request) Successful in 1m34s
Details
/ Builds (1.20) (pull_request) Successful in 3m15s
Details
/ Builds (1.21) (pull_request) Successful in 2m55s
Details
/ Vulncheck (pull_request) Successful in 2m51s
Details
/ Lint (pull_request) Successful in 5m12s
Details
/ Tests (1.20) (pull_request) Successful in 2m57s
Details
/ Tests (1.21) (pull_request) Successful in 2m48s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-28 16:12:42 +03:00
Marina Biryukova
fe796ba538 [#217] Consider Copy-Source-SSE-* headers during copy 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-11-13 13:22:58 +00:00
Marina Biryukova
25bb581fee [#205] Add md5 checksum in header
All checks were successful
/ Vulncheck (pull_request) Successful in 1m50s
Details
/ DCO (pull_request) Successful in 3m29s
Details
/ Lint (pull_request) Successful in 3m37s
Details
/ Tests (1.20) (pull_request) Successful in 2m9s
Details
/ Tests (1.21) (pull_request) Successful in 2m5s
Details
/ Builds (1.20) (pull_request) Successful in 5m47s
Details
/ Builds (1.21) (pull_request) Successful in 1m29s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-25 11:04:19 +03:00
Marina Biryukova
298662df9d [#221] Expand xmlns field ignore
All checks were successful
/ Vulncheck (pull_request) Successful in 1m38s
Details
/ Lint (pull_request) Successful in 2m49s
Details
/ Tests (1.20) (pull_request) Successful in 1m56s
Details
/ Tests (1.21) (pull_request) Successful in 1m44s
Details
/ DCO (pull_request) Successful in 3m56s
Details
/ Builds (1.20) (pull_request) Successful in 6m25s
Details
/ Builds (1.21) (pull_request) Successful in 1m36s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-13 16:21:13 +03:00
Denis Kirillov
7de73f6b73 [#197] Disable homomorphic hash for PUT 
Disable TZ hash for PUT if it's disabled for container itself

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-11 12:32:48 +00:00
Marina Biryukova
b8c93ed391 [#172] Convert handler config to interface 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-04 11:01:27 +00:00
Marina Biryukova
12cf29aed2 [#207] Fix part-number-marker handling
All checks were successful
/ DCO (pull_request) Successful in 1m57s
Details
/ Vulncheck (pull_request) Successful in 2m21s
Details
/ Builds (1.20) (pull_request) Successful in 1m36s
Details
/ Builds (1.21) (pull_request) Successful in 7m55s
Details
/ Lint (pull_request) Successful in 13m57s
Details
/ Tests (1.20) (pull_request) Successful in 2m34s
Details
/ Tests (1.21) (pull_request) Successful in 1m57s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-19 12:43:07 +03:00
Marina Biryukova
9120e97ac5 [#203] Add go1.21 to CI
All checks were successful
/ DCO (pull_request) Successful in 59s
Details
/ Vulncheck (pull_request) Successful in 1m35s
Details
/ Builds (1.20) (pull_request) Successful in 1m21s
Details
/ Builds (1.21) (pull_request) Successful in 7m46s
Details
/ Lint (pull_request) Successful in 12m44s
Details
/ Tests (1.20) (pull_request) Successful in 2m9s
Details
/ Tests (1.21) (pull_request) Successful in 2m0s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-08-31 15:26:07 +03:00
Denis Kirillov
6b728fef87 [#192] Add tests to make sure client_cut flag is passed to sdk 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-23 06:26:55 +00:00
Roman Loginov
40d7f844e3 [#137] Refactor context data retrievers 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-08-16 14:05:38 +00:00
Denis Kirillov
18878b66d3 [#175] Use gate owner as object owner
Some checks failed
/ DCO (pull_request) Failing after 3s
Details
/ Vulncheck (pull_request) Failing after 2s
Details
/ Builds (1.19) (pull_request) Failing after 3s
Details
/ Builds (1.20) (pull_request) Successful in 7m47s
Details
/ Lint (pull_request) Failing after 4s
Details
/ Tests (1.19) (pull_request) Successful in 8m8s
Details
/ Tests (1.20) (pull_request) Failing after 3s
Details 
This is required because node check session token owner
TrueCloudLab/frostfs-node#528
For client cut TrueCloudLab/frostfs-sdk-go#114
such owner will be gate owner

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 12:22:33 +03:00
Denis Kirillov
b59aa06637 [#146] Add kludge.bypass_content_encoding_check_in_chunks flag
Some checks failed
/ Builds (1.19) (pull_request) Successful in 3m0s
Details
/ Builds (1.20) (pull_request) Successful in 2m50s
Details
/ DCO (pull_request) Failing after 1m9s
Details
/ Vulncheck (pull_request) Successful in 1m20s
Details
/ Lint (pull_request) Successful in 7m50s
Details
/ Tests (1.19) (pull_request) Successful in 3m1s
Details
/ Tests (1.20) (pull_request) Successful in 3m21s
Details 
Flag allows to skip checking `Content-Encoding` for `aws-chunked` value

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-18 14:49:52 +03:00
Denis Kirillov
14ef9ff091 [#158] Separate init object reader from read itself
All checks were successful
/ Vulncheck (pull_request) Successful in 1m40s
Details
/ Builds (1.19) (pull_request) Successful in 2m54s
Details
/ Builds (1.20) (pull_request) Successful in 2m39s
Details
/ DCO (pull_request) Successful in 3m48s
Details
/ Lint (pull_request) Successful in 3m32s
Details
/ Tests (1.19) (pull_request) Successful in 2m38s
Details
/ Tests (1.20) (pull_request) Successful in 2m55s
Details 
To be able to handle cases and return appropriate http status code
when object missed in storage but gate cache contains its metadata
we need write code after init object reader.
So we separate init reader from actual reading.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:32:05 +03:00
Denis Kirillov
fc90981c03 [#149] Update inner imports after moving middlewares 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:09 +03:00
Denis Kirillov
be03c5178f [#143] Fix NoSuchKey error on get/head 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-30 12:04:52 +03:00
Denis Kirillov
4a6e3a19ce [#125] Handle negative Content-Length on put 
Add computing actual object size during calculating hash on put.
Use this actual value to save in tree and cache

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-02 14:16:19 +00:00
Artem Tataurov
9f186d9aba [#104] app: Reload copies numbers on SIGHUP 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-23 13:19:58 +03:00
Denis Kirillov
ee8cce662b [#74] service/tree: Add logger 
Log error instead of failing when multiple unversioned nodes are found

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-26 16:39:57 +03:00
Denis Kirillov
70ec5a0a5b [#83] Don't create extra delete marker 
We shouldn't create delete marker if:
1. object doesn't exist at all
2. last version is already a delete marker

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-19 17:56:11 +03:00
Denis Kirillov
a025f2e9c5 [#59] tree: Make interface for tree service client 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-29 09:34:10 +03:00
Denis Kirillov
0af06c3bd9 [TrueCloudLab#40] Add param to configure xml decoder 
This parameter enables parsing xml body without
xmlns="http://s3.amazonaws.com/doc/2006-03-01/" attribute
for CompleteMultipartUpload requests

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-09 15:44:13 +03:00
Alex Vanin
813aa2f173 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 17:38:08 +03:00
Denis Kirillov
388482e230 [#2] Rename internals 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-21 11:17:45 +03:00
Alex Vanin
96dff367db [#1] Build S3 Gateway with FrostFS dependencies 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-12-15 12:43:52 +03:00
Denis Kirillov
dd4f66712c [#742] Add multiple listeners 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-24 17:48:26 +03:00
Denis Kirillov
d2587b21af [#747] Reload policies on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-18 13:14:25 +03:00
Denis Kirillov
821df3d648 [#669] Optimize getNodeVersion for tags operation 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-20 17:56:09 +03:00
Denis Kirillov
cb55d36063 [#713] Update tests 
Add bearer token to test context

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 15:26:42 +03:00
Denis Kirillov
2a41929be3 [#702] Reload resolvers and TLS certs on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov
24e1f7e1dd [#545] Drop object search 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 11:49:08 +03:00
Denis Kirillov
d824db7f69 [#595] Allow SSE-C only with TLS 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov
87c05eb514 [#595] Add SSE with customer key 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Angira Kekteeva
56eb2dc3dc [#569] Rename versioning consts 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
77f466b1d5 [#569] Fix tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
fdf0974679 [#542] Fix object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
85c203e157 [#488] Sync using cid.ID 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
6ad7c988e6 [#532] Fix object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00