TrueCloudLab/frostfs-s3-gw
17
3
Fork 19
Code Issues 30 Pull requests 7 Projects Releases 19 Packages Activity Actions
1395 commits 5 branches 44 tags 22 MiB
Commit graph

399 commits

Author SHA1 Message Date
Marina Biryukova
a845e7f798 [#486] Fix PUT object with negative Content-Length 
(cherry picked from commit f187141ae5)

Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2024-09-16 14:30:11 +03:00
Marina Biryukova
c2e313d899 [#486] Fix PUT encrypted object 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2024-09-16 14:30:11 +03:00
Denis Kirillov
689f7ee818 [#437] tree: Support removing old split system nodes
All checks were successful
/ DCO (pull_request) Successful in 1m47s
Details
/ Builds (1.21) (pull_request) Successful in 2m20s
Details
/ Builds (1.22) (pull_request) Successful in 2m22s
Details
/ Vulncheck (pull_request) Successful in 2m6s
Details
/ Lint (pull_request) Successful in 4m43s
Details
/ Tests (1.21) (pull_request) Successful in 2m30s
Details
/ Tests (1.22) (pull_request) Successful in 2m25s
Details 
It's need to fit user expectation on deleting CORs for example.
Previously after removing cors (that was uploaded in split manner)
we can still get some data (from other node)
because deletion worked only for latest node version.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-07-22 10:42:11 +03:00
Denis Kirillov
977a20760b [#430] Delete all split version at once
All checks were successful
/ DCO (pull_request) Successful in 8m40s
Details
/ Vulncheck (pull_request) Successful in 13m10s
Details
/ Builds (1.21) (pull_request) Successful in 14m11s
Details
/ Builds (1.22) (pull_request) Successful in 14m18s
Details
/ Lint (pull_request) Successful in 16m31s
Details
/ Tests (1.21) (pull_request) Successful in 10m48s
Details
/ Tests (1.22) (pull_request) Successful in 14m31s
Details 
Previously after split we can get two `null` versioned object with the same key
and deleting such key removes only one node/object.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-07-19 11:26:51 +03:00
Denis Kirillov
465eaa816a [#372] Drop [e]ACL related code
All checks were successful
/ DCO (pull_request) Successful in 2m15s
Details
/ Vulncheck (pull_request) Successful in 2m55s
Details
/ Builds (1.20) (pull_request) Successful in 3m46s
Details
/ Builds (1.21) (pull_request) Successful in 3m48s
Details
/ Lint (pull_request) Successful in 5m26s
Details
/ Tests (1.20) (pull_request) Successful in 3m34s
Details
/ Tests (1.21) (pull_request) Successful in 3m18s
Details 
Always consider buckets as APE compatible

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-07-01 16:58:44 +03:00
Denis Kirillov
414f3943e2 [#410] Drop layer.Client interface
All checks were successful
/ DCO (pull_request) Successful in 2m1s
Details
/ Vulncheck (pull_request) Successful in 2m31s
Details
/ Builds (1.20) (pull_request) Successful in 2m39s
Details
/ Builds (1.21) (pull_request) Successful in 2m31s
Details
/ Lint (pull_request) Successful in 3m14s
Details
/ Tests (1.20) (pull_request) Successful in 2m34s
Details
/ Tests (1.21) (pull_request) Successful in 2m10s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-06-25 15:57:55 +03:00
Denis Kirillov
9432782ce6 [#401] Drop notifications
All checks were successful
/ DCO (pull_request) Successful in 2m5s
Details
/ Builds (1.20) (pull_request) Successful in 2m40s
Details
/ Builds (1.21) (pull_request) Successful in 2m33s
Details
/ Vulncheck (pull_request) Successful in 2m22s
Details
/ Lint (pull_request) Successful in 4m24s
Details
/ Tests (1.20) (pull_request) Successful in 2m48s
Details
/ Tests (1.21) (pull_request) Successful in 2m45s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-06-25 15:49:37 +03:00
Marina Biryukova
2ab655b909 [#380] Add test for credentials versioning 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2024-05-03 07:24:13 +00:00
Marina Biryukova
e22ff52165 [#367] Add check of AccessBox attributes 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2024-04-19 06:25:26 +00:00
Marina Biryukova
3ff027587c [#357] Add check of request and resource tags 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2024-04-17 07:06:58 +00:00
Denis Kirillov
8307c73fef [#364] Fix removing combined object
Some checks failed
/ Vulncheck (pull_request) Failing after 3m8s
Details
/ DCO (pull_request) Successful in 3m49s
Details
/ Builds (1.20) (pull_request) Successful in 5m35s
Details
/ Builds (1.21) (pull_request) Successful in 4m16s
Details
/ Lint (pull_request) Successful in 6m55s
Details
/ Tests (1.20) (pull_request) Successful in 5m14s
Details
/ Tests (1.21) (pull_request) Successful in 4m29s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-04-12 14:56:38 +03:00
Denis Kirillov
8669bf6b50 [#346] acl: Update APE and fix using
All checks were successful
/ DCO (pull_request) Successful in 2m57s
Details
/ Vulncheck (pull_request) Successful in 3m33s
Details
/ Lint (pull_request) Successful in 4m44s
Details
/ Tests (1.20) (pull_request) Successful in 3m38s
Details
/ Tests (1.21) (pull_request) Successful in 3m29s
Details
/ Builds (1.20) (pull_request) Successful in 1m12s
Details
/ Builds (1.21) (pull_request) Successful in 3m23s
Details 
* Remove native policy when remove bucket policy
* Allow policies that contain only s3 compatible statements
(now deny rules cannot be converted to native rules)

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-04-02 12:43:04 +00:00
Denis Kirillov
80c7b73eb9 [#306] In APE buckets forbid canned acl except private
Some checks failed
/ DCO (pull_request) Successful in 2m50s
Details
/ Vulncheck (pull_request) Failing after 3m15s
Details
/ Builds (1.20) (pull_request) Successful in 3m39s
Details
/ Builds (1.21) (pull_request) Successful in 3m41s
Details
/ Lint (pull_request) Successful in 5m48s
Details
/ Tests (1.20) (pull_request) Successful in 4m0s
Details
/ Tests (1.21) (pull_request) Successful in 3m53s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-03-19 16:57:26 +03:00
Denis Kirillov
f958eef2b3 [#325] Use default empty data.LockInfo in get/head in case of error 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-03-04 09:49:41 +00:00
Denis Kirillov
8050ca2d51 [#306] Use session token for container read operations
All checks were successful
/ DCO (pull_request) Successful in 1m54s
Details
/ Vulncheck (pull_request) Successful in 1m55s
Details
/ Builds (1.20) (pull_request) Successful in 2m49s
Details
/ Builds (1.21) (pull_request) Successful in 1m56s
Details
/ Lint (pull_request) Successful in 3m59s
Details
/ Tests (1.20) (pull_request) Successful in 2m30s
Details
/ Tests (1.21) (pull_request) Successful in 2m19s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-03-01 18:14:33 +03:00
Denis Kirillov
c12e264697 [#306] Simplify cid resolver for metrics 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-03-01 17:46:16 +03:00
Denis Kirillov
e9f38a49e4 [#306] Fix forming key for bucket cache 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-03-01 16:09:40 +03:00
Marina Biryukova
31da31862a [#300] Update error logging in DeleteMultipleObjects 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2024-02-29 14:24:32 +00:00
Denis Kirillov
bac1b3fb2d [#306] Use zero basic acl to mark APE containers 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-28 17:50:08 +03:00
Denis Kirillov
3d0d2032c6 [#306] acl: Handle put/get acl for APE buckets 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-28 17:50:08 +03:00
Denis Kirillov
1f2cf0ed67 [#306] Use APE instead of eACL on bucket creation 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-28 17:50:08 +03:00
Denis Kirillov
a74d498df2 [#165] Return sort after HEAD in listing
Some checks failed
/ DCO (pull_request) Successful in 1m16s
Details
/ Builds (1.20) (pull_request) Successful in 2m9s
Details
/ Builds (1.21) (pull_request) Successful in 1m39s
Details
/ Vulncheck (pull_request) Successful in 1m52s
Details
/ Lint (pull_request) Failing after 1m47s
Details
/ Tests (1.20) (pull_request) Successful in 2m27s
Details
/ Tests (1.21) (pull_request) Successful in 2m13s
Details 
We have to sort object after HEAD because we make request in different goroutines,
so the order is not deterministic.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:15:08 +03:00
Denis Kirillov
69297a4a38 [#165] Delete object from tree in case of storage error 
Extend storage node errors in case of which we continue deleting from tree
with 'object not found' error

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:15:08 +03:00
Denis Kirillov
71d82d1cc8 [#165] Fix lint issues 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:15:08 +03:00
Denis Kirillov
fafe4af529 [#165] Fix real object size in listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:15:03 +03:00
Denis Kirillov
88f1acbdfc [#165] Cancel context in outdated list session 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
4e15452853 [#165] Fix lint errors 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
da642a498a [#165] Listing fix data race 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
2d7973b3f1 [#165] Refactor list versions 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
6d52f46012 [#165] Fix v1 listing bookmark 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
0ae49eaab0 [#165] Generalize allObjectListingParams 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
093de13f54 [#165] Add stream listing tests 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
cf4fc3b602 [#165] Extend error on getting listing containers not in current namespace 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
c7ee628ab0 [#165] Fix versions listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
4ad84b9b94 [#165] listing: Use NodeVersion instead of ObjectInfo 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
3e20f736a6 [#165] Move listing function to one file 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
b52552e8c2 [#165] Add batching in streamin listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
6e8960b2ab [#165] Add list session cache 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
29ac91dfd5 [#165] Support streaming listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Roman Loginov
6f9ee3da76 [#275] Change logic delete multipart upload
All checks were successful
/ Vulncheck (pull_request) Successful in 1m13s
Details
/ DCO (pull_request) Successful in 1m43s
Details
/ Builds (1.20) (pull_request) Successful in 2m49s
Details
/ Builds (1.21) (pull_request) Successful in 1m36s
Details
/ Lint (pull_request) Successful in 3m33s
Details
/ Tests (1.20) (pull_request) Successful in 2m34s
Details
/ Tests (1.21) (pull_request) Successful in 2m29s
Details 
In order not to accidentally take outdated
information about downloaded parts from other
nodes, now when the multipart is abort or complete,
the root node of the multipart upload with the
finish flag remains in the tree.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-12-27 13:06:45 +03:00
Denis Kirillov
43cae9ee04 [#248] Correct object versions response markers
All checks were successful
/ DCO (pull_request) Successful in 1m40s
Details
/ Vulncheck (pull_request) Successful in 1m42s
Details
/ Builds (1.20) (pull_request) Successful in 2m29s
Details
/ Builds (1.21) (pull_request) Successful in 1m37s
Details
/ Lint (pull_request) Successful in 3m55s
Details
/ Tests (1.20) (pull_request) Successful in 2m23s
Details
/ Tests (1.21) (pull_request) Successful in 2m16s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-21 10:53:50 +03:00
Alex Vanin
f58a0d04ff [#233] Clean tag node in the tree service instead of removal 
With new retry policy of tree service pool, gateway should avoid
deletion of system nodes from tree. Absence of node in the tree
will trigger retry. Other storage in the network may return already
deleted node while tree is not completely synced, and client will
get unexpected result.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-12-07 16:40:56 +03:00
Denis Kirillov
ff1ec56d24 [#260] Use namespace as domain when create bucket
All checks were successful
/ DCO (pull_request) Successful in 1m26s
Details
/ Vulncheck (pull_request) Successful in 2m6s
Details
/ Builds (1.20) (pull_request) Successful in 2m20s
Details
/ Builds (1.21) (pull_request) Successful in 2m21s
Details
/ Lint (pull_request) Successful in 3m15s
Details
/ Tests (1.20) (pull_request) Successful in 2m28s
Details
/ Tests (1.21) (pull_request) Successful in 2m17s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-27 15:30:12 +03:00
Denis Kirillov
055cc6a22a [#260] Use namespace as domain when resolve bucket 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-23 11:00:11 +03:00
Roman Loginov
861454e499 [#218] Add check content sha256 header 
The X-Amz-Content-Sha256 header check is done only for unencrypted payload.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-11-22 11:33:52 +00:00
Marina Biryukova
b28ecef43b [#219] Return ETag in quotes
All checks were successful
/ Vulncheck (pull_request) Successful in 9m5s
Details
/ Lint (pull_request) Successful in 11m1s
Details
/ Tests (1.20) (pull_request) Successful in 9m59s
Details
/ Tests (1.21) (pull_request) Successful in 9m53s
Details
/ DCO (pull_request) Successful in 11m34s
Details
/ Builds (1.20) (pull_request) Successful in 12m24s
Details
/ Builds (1.21) (pull_request) Successful in 8m59s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-11-22 11:12:32 +00:00
Marina Biryukova
fe796ba538 [#217] Consider Copy-Source-SSE-* headers during copy 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-11-13 13:22:58 +00:00
Denis Kirillov
5ee73fad6a [#248] Correct NextVersionIDMarker in listing versions
All checks were successful
/ DCO (pull_request) Successful in 4m40s
Details
/ Vulncheck (pull_request) Successful in 5m43s
Details
/ Lint (pull_request) Successful in 10m27s
Details
/ Tests (1.20) (pull_request) Successful in 5m41s
Details
/ Tests (1.21) (pull_request) Successful in 5m47s
Details
/ Builds (1.20) (pull_request) Successful in 1m9s
Details
/ Builds (1.21) (pull_request) Successful in 4m7s
Details 
Despite the spec https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html#API_ListObjectVersions_ResponseElements
says that
"When the number of responses exceeds the value of MaxKeys,
NextVersionIdMarker specifies the first object version not returned
 that satisfies the search criteria. Use this value for the
 version-id-marker request parameter in a subsequent request."
 the actual behavior of AWS S3 is returning NextVersionIdMarker as the last returned object version

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 17:36:24 +03:00
Denis Kirillov
b169c5e6c3 [#239] Update test for check goroutines leak 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 13:51:23 +00:00
Denis Kirillov
0938d7ee82 [#226] Fix status code in GET/HEAD delete marker
All checks were successful
/ Vulncheck (pull_request) Successful in 2m26s
Details
/ Lint (pull_request) Successful in 4m0s
Details
/ Tests (1.20) (pull_request) Successful in 2m59s
Details
/ Tests (1.21) (pull_request) Successful in 1m52s
Details
/ DCO (pull_request) Successful in 4m17s
Details
/ Builds (1.20) (pull_request) Successful in 6m33s
Details
/ Builds (1.21) (pull_request) Successful in 1m28s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-27 10:58:57 +03:00