TrueCloudLab/frostfs-s3-gw
19
3
Fork 20
Code Issues 29 Pull requests 9 Projects Releases 22 Packages Activity Actions
1488 commits 6 branches 47 tags 24 MiB
Commit graph

82 commits

Author SHA1 Message Date
Roman Loginov
4a4ce00994 [#562] Support TLS termination header for SSE-C 
The TLS termination header added for determining
whether TLS needs to be checked. If the system
requests come through a proxy server and TLS can
terminate at the proxy level, you should use this
header to disable TLS verification at SSE-C.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2024-12-11 16:09:43 +03:00
Marina Biryukova
f215d200e8 [#559] Remove multipart objects using tombstones
All checks were successful
/ DCO (pull_request) Successful in 1m33s
Details
/ Vulncheck (pull_request) Successful in 2m38s
Details
/ Builds (pull_request) Successful in 2m48s
Details
/ Lint (pull_request) Successful in 4m6s
Details
/ Tests (pull_request) Successful in 2m48s
Details
/ Vulncheck (push) Successful in 1m19s
Details
/ Builds (push) Successful in 2m14s
Details
/ Lint (push) Successful in 3m40s
Details
/ Tests (push) Successful in 2m21s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2024-12-04 11:03:01 +03:00
Marina Biryukova
3260308cc0 [#528] Check owner ID before deleting bucket 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2024-11-12 12:47:43 +00:00
Nikita Zinkevich
827ea1a41e [#488] Move layer/frostfs.go to layer/frostfs/frostfs.go 
Signed-off-by: Nikita Zinkevich <n.zinkevich@yadro.com>
 2024-10-02 12:35:04 +03:00
Denis Kirillov
738ce14f50 [#434] Remove container on failed bucket creation 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-09-25 07:15:24 +00:00
Marina Biryukova
f187141ae5 [#486] Fix PUT object with negative Content-Length 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2024-09-16 08:45:46 +00:00
Marina Biryukova
d0e4d55772 [#460] Add network info cache 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2024-09-13 09:56:24 +00:00
Marina Biryukova
d919e6cce2 [#482] Fix containers resolving
All checks were successful
/ DCO (pull_request) Successful in 1m8s
Details
/ Builds (pull_request) Successful in 1m7s
Details
/ Vulncheck (pull_request) Successful in 1m12s
Details
/ Lint (pull_request) Successful in 2m24s
Details
/ Tests (pull_request) Successful in 1m24s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2024-09-05 12:33:14 +03:00
Roman Ognev
664f83b2b7 [#480] Add fuzzing tests
All checks were successful
/ DCO (pull_request) Successful in 1m34s
Details
/ Vulncheck (pull_request) Successful in 1m45s
Details
/ Builds (1.22) (pull_request) Successful in 1m51s
Details
/ Builds (1.23) (pull_request) Successful in 1m56s
Details
/ Lint (pull_request) Successful in 3m3s
Details
/ Tests (1.22) (pull_request) Successful in 1m52s
Details
/ Tests (1.23) (pull_request) Successful in 1m51s
Details 
Signed-off-by: Roman Ognev <r.ognev@yadro.com>
 2024-09-02 15:59:07 +03:00
Roman Loginov
534ae7f0f1 [#446] Add support virtual-hosted-style 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2024-08-23 08:35:05 +00:00
Denis Kirillov
e73f11c251 [#452] tree: Fix logging
All checks were successful
/ DCO (pull_request) Successful in 1m6s
Details
/ Vulncheck (pull_request) Successful in 1m10s
Details
/ Builds (1.21) (pull_request) Successful in 1m32s
Details
/ Builds (1.22) (pull_request) Successful in 1m31s
Details
/ Lint (pull_request) Successful in 1m46s
Details
/ Tests (1.21) (pull_request) Successful in 1m51s
Details
/ Tests (1.22) (pull_request) Successful in 1m45s
Details 
Don't log parsing tags error in listing

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-08-21 10:38:36 +03:00
Denis Kirillov
3dc989d7fe [#451] Support Location in CompleteMultipart response
All checks were successful
/ Vulncheck (pull_request) Successful in 1m15s
Details
/ DCO (pull_request) Successful in 1m13s
Details
/ Builds (1.21) (pull_request) Successful in 1m31s
Details
/ Builds (1.22) (pull_request) Successful in 1m25s
Details
/ Lint (pull_request) Successful in 2m26s
Details
/ Tests (1.21) (pull_request) Successful in 1m40s
Details
/ Tests (1.22) (pull_request) Successful in 1m43s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-08-06 15:45:09 +03:00
Denis Kirillov
77f8bdac58 [#372] Drop kludge.acl_enabled flag 
Now only APE container can be created using s3-gw

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-07-01 16:26:19 +03:00
Denis Kirillov
414f3943e2 [#410] Drop layer.Client interface
All checks were successful
/ DCO (pull_request) Successful in 2m1s
Details
/ Vulncheck (pull_request) Successful in 2m31s
Details
/ Builds (1.20) (pull_request) Successful in 2m39s
Details
/ Builds (1.21) (pull_request) Successful in 2m31s
Details
/ Lint (pull_request) Successful in 3m14s
Details
/ Tests (1.20) (pull_request) Successful in 2m34s
Details
/ Tests (1.21) (pull_request) Successful in 2m10s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-06-25 15:57:55 +03:00
Denis Kirillov
9432782ce6 [#401] Drop notifications
All checks were successful
/ DCO (pull_request) Successful in 2m5s
Details
/ Builds (1.20) (pull_request) Successful in 2m40s
Details
/ Builds (1.21) (pull_request) Successful in 2m33s
Details
/ Vulncheck (pull_request) Successful in 2m22s
Details
/ Lint (pull_request) Successful in 4m24s
Details
/ Tests (1.20) (pull_request) Successful in 2m48s
Details
/ Tests (1.21) (pull_request) Successful in 2m45s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-06-25 15:49:37 +03:00
Denis Kirillov
bb81afc14a [#398] Support retryer 
Add two strategy for PutBucketSettings request retryer:
* exponential backoff (increasing up to `max_backoff` delays with jitter)
* constant backoff (always the same `max_backoff` delay between requests)

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-06-06 13:02:17 +00:00
Marina Biryukova
45f77de8c8 [#371] Add custom Source IP header configuration 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2024-04-22 07:42:45 +00:00
Marina Biryukova
e22ff52165 [#367] Add check of AccessBox attributes 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2024-04-19 06:25:26 +00:00
Alex Vanin
6da1acc554 [#360] Use 'c' prefix for bucket policies instead of 'n' 
With 'c' prefix, acl chains become shorter, thus gateway
receives shorter results and avoids sessions to neo-go.

There is still issue with many IAM rules.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2024-04-10 17:56:47 +03:00
Denis Kirillov
8669bf6b50 [#346] acl: Update APE and fix using
All checks were successful
/ DCO (pull_request) Successful in 2m57s
Details
/ Vulncheck (pull_request) Successful in 3m33s
Details
/ Lint (pull_request) Successful in 4m44s
Details
/ Tests (1.20) (pull_request) Successful in 3m38s
Details
/ Tests (1.21) (pull_request) Successful in 3m29s
Details
/ Builds (1.20) (pull_request) Successful in 1m12s
Details
/ Builds (1.21) (pull_request) Successful in 3m23s
Details 
* Remove native policy when remove bucket policy
* Allow policies that contain only s3 compatible statements
(now deny rules cannot be converted to native rules)

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-04-02 12:43:04 +00:00
Denis Kirillov
fbe7a784e8 [#301] Support GetBucketPolicyStatus 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-03-28 09:13:25 +03:00
Denis Kirillov
8f89f275bd [#306] Save bucket policy as native chain 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-28 17:50:08 +03:00
Denis Kirillov
c868af8a62 [#306] Add flag to enable old ACL bucket creation 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-28 17:50:08 +03:00
Denis Kirillov
c452d58ce2 [#306] Reduce number of policy contract invocations 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-28 17:50:08 +03:00
Denis Kirillov
d9d12debc3 [#306] Add tests 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-28 17:50:08 +03:00
Denis Kirillov
3d0d2032c6 [#306] acl: Handle put/get acl for APE buckets 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-28 17:50:08 +03:00
Denis Kirillov
88f1acbdfc [#165] Cancel context in outdated list session 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
c7ee628ab0 [#165] Fix versions listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
29ac91dfd5 [#165] Support streaming listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
84af85ed67 [#302] Update APE to support chain id as bytes
All checks were successful
/ DCO (pull_request) Successful in 1m52s
Details
/ Vulncheck (pull_request) Successful in 2m0s
Details
/ Builds (1.20) (pull_request) Successful in 2m49s
Details
/ Builds (1.21) (pull_request) Successful in 1m59s
Details
/ Lint (pull_request) Successful in 2m44s
Details
/ Tests (1.20) (pull_request) Successful in 2m30s
Details
/ Tests (1.21) (pull_request) Successful in 2m24s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 12:53:45 +03:00
Denis Kirillov
8273af8bf8 [#261] Make PutBucketPolicy handler use policy contract 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-18 15:49:54 +03:00
Denis Kirillov
28c6bb4cb8 [#266] Support per namespace placement policies configuration
All checks were successful
/ DCO (pull_request) Successful in 1m34s
Details
/ Builds (1.20) (pull_request) Successful in 3m15s
Details
/ Builds (1.21) (pull_request) Successful in 2m55s
Details
/ Vulncheck (pull_request) Successful in 2m51s
Details
/ Lint (pull_request) Successful in 5m12s
Details
/ Tests (1.20) (pull_request) Successful in 2m57s
Details
/ Tests (1.21) (pull_request) Successful in 2m48s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-28 16:12:42 +03:00
Marina Biryukova
fe796ba538 [#217] Consider Copy-Source-SSE-* headers during copy 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-11-13 13:22:58 +00:00
Marina Biryukova
25bb581fee [#205] Add md5 checksum in header
All checks were successful
/ Vulncheck (pull_request) Successful in 1m50s
Details
/ DCO (pull_request) Successful in 3m29s
Details
/ Lint (pull_request) Successful in 3m37s
Details
/ Tests (1.20) (pull_request) Successful in 2m9s
Details
/ Tests (1.21) (pull_request) Successful in 2m5s
Details
/ Builds (1.20) (pull_request) Successful in 5m47s
Details
/ Builds (1.21) (pull_request) Successful in 1m29s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-25 11:04:19 +03:00
Marina Biryukova
298662df9d [#221] Expand xmlns field ignore
All checks were successful
/ Vulncheck (pull_request) Successful in 1m38s
Details
/ Lint (pull_request) Successful in 2m49s
Details
/ Tests (1.20) (pull_request) Successful in 1m56s
Details
/ Tests (1.21) (pull_request) Successful in 1m44s
Details
/ DCO (pull_request) Successful in 3m56s
Details
/ Builds (1.20) (pull_request) Successful in 6m25s
Details
/ Builds (1.21) (pull_request) Successful in 1m36s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-13 16:21:13 +03:00
Denis Kirillov
7de73f6b73 [#197] Disable homomorphic hash for PUT 
Disable TZ hash for PUT if it's disabled for container itself

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-11 12:32:48 +00:00
Marina Biryukova
b8c93ed391 [#172] Convert handler config to interface 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-04 11:01:27 +00:00
Marina Biryukova
12cf29aed2 [#207] Fix part-number-marker handling
All checks were successful
/ DCO (pull_request) Successful in 1m57s
Details
/ Vulncheck (pull_request) Successful in 2m21s
Details
/ Builds (1.20) (pull_request) Successful in 1m36s
Details
/ Builds (1.21) (pull_request) Successful in 7m55s
Details
/ Lint (pull_request) Successful in 13m57s
Details
/ Tests (1.20) (pull_request) Successful in 2m34s
Details
/ Tests (1.21) (pull_request) Successful in 1m57s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-19 12:43:07 +03:00
Marina Biryukova
9120e97ac5 [#203] Add go1.21 to CI
All checks were successful
/ DCO (pull_request) Successful in 59s
Details
/ Vulncheck (pull_request) Successful in 1m35s
Details
/ Builds (1.20) (pull_request) Successful in 1m21s
Details
/ Builds (1.21) (pull_request) Successful in 7m46s
Details
/ Lint (pull_request) Successful in 12m44s
Details
/ Tests (1.20) (pull_request) Successful in 2m9s
Details
/ Tests (1.21) (pull_request) Successful in 2m0s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-08-31 15:26:07 +03:00
Denis Kirillov
6b728fef87 [#192] Add tests to make sure client_cut flag is passed to sdk 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-23 06:26:55 +00:00
Roman Loginov
40d7f844e3 [#137] Refactor context data retrievers 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-08-16 14:05:38 +00:00
Denis Kirillov
18878b66d3 [#175] Use gate owner as object owner
Some checks failed
/ DCO (pull_request) Failing after 3s
Details
/ Vulncheck (pull_request) Failing after 2s
Details
/ Builds (1.19) (pull_request) Failing after 3s
Details
/ Builds (1.20) (pull_request) Successful in 7m47s
Details
/ Lint (pull_request) Failing after 4s
Details
/ Tests (1.19) (pull_request) Successful in 8m8s
Details
/ Tests (1.20) (pull_request) Failing after 3s
Details 
This is required because node check session token owner
TrueCloudLab/frostfs-node#528
For client cut TrueCloudLab/frostfs-sdk-go#114
such owner will be gate owner

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 12:22:33 +03:00
Denis Kirillov
b59aa06637 [#146] Add kludge.bypass_content_encoding_check_in_chunks flag
Some checks failed
/ Builds (1.19) (pull_request) Successful in 3m0s
Details
/ Builds (1.20) (pull_request) Successful in 2m50s
Details
/ DCO (pull_request) Failing after 1m9s
Details
/ Vulncheck (pull_request) Successful in 1m20s
Details
/ Lint (pull_request) Successful in 7m50s
Details
/ Tests (1.19) (pull_request) Successful in 3m1s
Details
/ Tests (1.20) (pull_request) Successful in 3m21s
Details 
Flag allows to skip checking `Content-Encoding` for `aws-chunked` value

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-18 14:49:52 +03:00
Denis Kirillov
14ef9ff091 [#158] Separate init object reader from read itself
All checks were successful
/ Vulncheck (pull_request) Successful in 1m40s
Details
/ Builds (1.19) (pull_request) Successful in 2m54s
Details
/ Builds (1.20) (pull_request) Successful in 2m39s
Details
/ DCO (pull_request) Successful in 3m48s
Details
/ Lint (pull_request) Successful in 3m32s
Details
/ Tests (1.19) (pull_request) Successful in 2m38s
Details
/ Tests (1.20) (pull_request) Successful in 2m55s
Details 
To be able to handle cases and return appropriate http status code
when object missed in storage but gate cache contains its metadata
we need write code after init object reader.
So we separate init reader from actual reading.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:32:05 +03:00
Denis Kirillov
fc90981c03 [#149] Update inner imports after moving middlewares 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:09 +03:00
Denis Kirillov
be03c5178f [#143] Fix NoSuchKey error on get/head 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-30 12:04:52 +03:00
Denis Kirillov
4a6e3a19ce [#125] Handle negative Content-Length on put 
Add computing actual object size during calculating hash on put.
Use this actual value to save in tree and cache

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-02 14:16:19 +00:00
Artem Tataurov
9f186d9aba [#104] app: Reload copies numbers on SIGHUP 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-23 13:19:58 +03:00
Denis Kirillov
ee8cce662b [#74] service/tree: Add logger 
Log error instead of failing when multiple unversioned nodes are found

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-26 16:39:57 +03:00
Denis Kirillov
70ec5a0a5b [#83] Don't create extra delete marker 
We shouldn't create delete marker if:
1. object doesn't exist at all
2. last version is already a delete marker

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-19 17:56:11 +03:00