TrueCloudLab
/
frostfs-s3-gw
15
2
Fork 16
Code Issues 21 Pull Requests 9 Actions Packages Projects Releases 9 Activity
1,249 Commits
6 Branches
28 Tags
19 MiB
Commit Graph

658 Commits (71d82d1cc81a1f8483f4ba1c9cf2410c68d1f2af)

Author SHA1 Message Date
Denis Kirillov 71d82d1cc8 [#165] Fix lint issues 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:15:08 +03:00
Denis Kirillov fafe4af529 [#165] Fix real object size in listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:15:03 +03:00
Denis Kirillov 88f1acbdfc [#165] Cancel context in outdated list session 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov 4e15452853 [#165] Fix lint errors 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov da642a498a [#165] Listing fix data race 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov 2d7973b3f1 [#165] Refactor list versions 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov 6d52f46012 [#165] Fix v1 listing bookmark 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov 0ae49eaab0 [#165] Generalize allObjectListingParams 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov 093de13f54 [#165] Add stream listing tests 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov cf4fc3b602 [#165] Extend error on getting listing containers not in current namespace 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov 739a6ec9df [#165] Support latest only stream listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov c7ee628ab0 [#165] Fix versions listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov 4ad84b9b94 [#165] listing: Use NodeVersion instead of ObjectInfo 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov 3e20f736a6 [#165] Move listing function to one file 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov b52552e8c2 [#165] Add batching in streamin listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov 6e8960b2ab [#165] Add list session cache 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov 29ac91dfd5 [#165] Support streaming listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov 84af85ed67 [#302] Update APE to support chain id as bytes
/ DCO (pull_request) Successful in 1m52s Details
/ Vulncheck (pull_request) Successful in 2m0s Details
/ Builds (1.20) (pull_request) Successful in 2m49s Details
/ Builds (1.21) (pull_request) Successful in 1m59s Details
/ Lint (pull_request) Successful in 2m44s Details
/ Tests (1.20) (pull_request) Successful in 2m30s Details
/ Tests (1.21) (pull_request) Successful in 2m24s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 12:53:45 +03:00
Denis Kirillov e23cc43824 [#299] Drop unused legacy minio code
/ DCO (pull_request) Successful in 2m36s Details
/ Vulncheck (pull_request) Successful in 2m59s Details
/ Lint (pull_request) Successful in 5m23s Details
/ Tests (1.20) (pull_request) Successful in 3m24s Details
/ Tests (1.21) (pull_request) Successful in 3m3s Details
/ Builds (1.20) (pull_request) Successful in 1m16s Details
/ Builds (1.21) (pull_request) Successful in 2m53s Details 
736d8cbac4 (diff-f5a8931b4d5f3b7f583e4cd719bfd2904980518a6f338d463ec76aea814db772)
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-01-29 12:57:18 +03:00
Denis Kirillov eae49908da [#292] authmate: Support custom attributes
/ DCO (pull_request) Successful in 1m36s Details
/ Builds (1.20) (pull_request) Successful in 2m2s Details
/ Builds (1.21) (pull_request) Successful in 1m28s Details
/ Vulncheck (pull_request) Successful in 1m53s Details
/ Lint (pull_request) Successful in 3m19s Details
/ Tests (1.20) (pull_request) Successful in 2m24s Details
/ Tests (1.21) (pull_request) Successful in 2m11s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-01-22 10:02:43 +03:00
Roman Loginov 6f9ee3da76 [#275] Change logic delete multipart upload
/ Vulncheck (pull_request) Successful in 1m13s Details
/ DCO (pull_request) Successful in 1m43s Details
/ Builds (1.20) (pull_request) Successful in 2m49s Details
/ Builds (1.21) (pull_request) Successful in 1m36s Details
/ Lint (pull_request) Successful in 3m33s Details
/ Tests (1.20) (pull_request) Successful in 2m34s Details
/ Tests (1.21) (pull_request) Successful in 2m29s Details 
In order not to accidentally take outdated
information about downloaded parts from other
nodes, now when the multipart is abort or complete,
the root node of the multipart upload with the
finish flag remains in the tree.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-12-27 13:06:45 +03:00
Marina Biryukova 08019f1574 [#280] Add put requests to duration metric
/ DCO (pull_request) Successful in 1m48s Details
/ Builds (1.20) (pull_request) Successful in 2m10s Details
/ Builds (1.21) (pull_request) Successful in 1m24s Details
/ Vulncheck (pull_request) Successful in 2m0s Details
/ Lint (pull_request) Successful in 4m24s Details
/ Tests (1.20) (pull_request) Successful in 2m16s Details
/ Tests (1.21) (pull_request) Successful in 2m6s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-12-22 16:33:05 +03:00
Denis Kirillov 5698d5844e [#283] Support frostfsid groups in policy request checking 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-21 14:21:36 +03:00
Denis Kirillov 43cae9ee04 [#248] Correct object versions response markers
/ DCO (pull_request) Successful in 1m40s Details
/ Vulncheck (pull_request) Successful in 1m42s Details
/ Builds (1.20) (pull_request) Successful in 2m29s Details
/ Builds (1.21) (pull_request) Successful in 1m37s Details
/ Lint (pull_request) Successful in 3m55s Details
/ Tests (1.20) (pull_request) Successful in 2m23s Details
/ Tests (1.21) (pull_request) Successful in 2m16s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-21 10:53:50 +03:00
Denis Kirillov a17ff66975 [#282] policy: Use prefixes to distinguish s3/iam actions/resources
/ DCO (pull_request) Successful in 1m37s Details
/ Vulncheck (pull_request) Successful in 1m50s Details
/ Builds (1.20) (pull_request) Successful in 2m24s Details
/ Builds (1.21) (pull_request) Successful in 2m2s Details
/ Lint (pull_request) Successful in 4m26s Details
/ Tests (1.20) (pull_request) Successful in 2m28s Details
/ Tests (1.21) (pull_request) Successful in 1m58s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-20 10:41:15 +03:00
Denis Kirillov 38c5503a02 [#261] alc: Remove unused
/ Vulncheck (pull_request) Successful in 2m1s Details
/ DCO (pull_request) Successful in 1m49s Details
/ Builds (1.20) (pull_request) Successful in 2m21s Details
/ Builds (1.21) (pull_request) Successful in 2m4s Details
/ Lint (pull_request) Successful in 2m56s Details
/ Tests (1.20) (pull_request) Successful in 2m17s Details
/ Tests (1.21) (pull_request) Successful in 1m46s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-18 15:49:58 +03:00
Denis Kirillov 8273af8bf8 [#261] Make PutBucketPolicy handler use policy contract 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-18 15:49:54 +03:00
Denis Kirillov 6dbb07f0fa [#261] Update policy-engine dependency 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-14 17:52:13 +03:00
Denis Kirillov 9272f4e108 [#259] Support contract based policies
/ DCO (pull_request) Successful in 1m21s Details
/ Vulncheck (pull_request) Successful in 1m41s Details
/ Builds (1.20) (pull_request) Successful in 2m19s Details
/ Builds (1.21) (pull_request) Successful in 2m1s Details
/ Lint (pull_request) Successful in 3m20s Details
/ Tests (1.20) (pull_request) Successful in 2m14s Details
/ Tests (1.21) (pull_request) Successful in 2m10s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-11 10:01:46 +03:00
Denis Kirillov 836874a761 [#262] Set tree request id 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-07 16:40:56 +03:00
Alexey Vanin f58a0d04ff [#233] Clean tag node in the tree service instead of removal 
With new retry policy of tree service pool, gateway should avoid
deletion of system nodes from tree. Absence of node in the tree
will trigger retry. Other storage in the network may return already
deleted node while tree is not completely synced, and client will
get unexpected result.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-12-07 16:40:56 +03:00
Denis Kirillov 43abf58068 [#257] Support flag to deny access if policy rules not found
/ DCO (pull_request) Successful in 1m13s Details
/ Vulncheck (pull_request) Successful in 2m2s Details
/ Builds (1.20) (pull_request) Successful in 2m22s Details
/ Builds (1.21) (pull_request) Successful in 2m16s Details
/ Lint (pull_request) Successful in 3m26s Details
/ Tests (1.20) (pull_request) Successful in 2m21s Details
/ Tests (1.21) (pull_request) Successful in 1m37s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-06 17:47:58 +03:00
Denis Kirillov ca15acf1d3 [#257] router: Use named constants 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-06 17:47:58 +03:00
Denis Kirillov 473239bf36 [#257] Add policy checker 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-06 17:47:51 +03:00
Marina Biryukova 93cf7c462b [#271] Add namespace label to billing metrics
/ DCO (pull_request) Successful in 2m35s Details
/ Vulncheck (pull_request) Successful in 3m3s Details
/ Builds (1.20) (pull_request) Successful in 3m34s Details
/ Builds (1.21) (pull_request) Successful in 2m20s Details
/ Lint (pull_request) Successful in 5m27s Details
/ Tests (1.20) (pull_request) Successful in 3m25s Details
/ Tests (1.21) (pull_request) Successful in 3m12s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-12-04 14:54:40 +03:00
Denis Kirillov 28c6bb4cb8 [#266] Support per namespace placement policies configuration
/ DCO (pull_request) Successful in 1m34s Details
/ Builds (1.20) (pull_request) Successful in 3m15s Details
/ Builds (1.21) (pull_request) Successful in 2m55s Details
/ Vulncheck (pull_request) Successful in 2m51s Details
/ Lint (pull_request) Successful in 5m12s Details
/ Tests (1.20) (pull_request) Successful in 2m57s Details
/ Tests (1.21) (pull_request) Successful in 2m48s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-28 16:12:42 +03:00
Denis Kirillov ff1ec56d24 [#260] Use namespace as domain when create bucket
/ DCO (pull_request) Successful in 1m26s Details
/ Vulncheck (pull_request) Successful in 2m6s Details
/ Builds (1.20) (pull_request) Successful in 2m20s Details
/ Builds (1.21) (pull_request) Successful in 2m21s Details
/ Lint (pull_request) Successful in 3m15s Details
/ Tests (1.20) (pull_request) Successful in 2m28s Details
/ Tests (1.21) (pull_request) Successful in 2m17s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-27 15:30:12 +03:00
Denis Kirillov 055cc6a22a [#260] Use namespace as domain when resolve bucket 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-23 11:00:11 +03:00
Denis Kirillov 6304d7bfda [#260] Support frostfsid validation 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-23 11:00:11 +03:00
Denis Kirillov cf7254f8cd [#260] Refactor api/auth/center.go 
Move the Center interface to middleware package where it's used

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-23 11:00:09 +03:00
Roman Loginov 861454e499 [#218] Add check content sha256 header 
The X-Amz-Content-Sha256 header check is done only for unencrypted payload.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-11-22 11:33:52 +00:00
Marina Biryukova b28ecef43b [#219] Return ETag in quotes
/ Vulncheck (pull_request) Successful in 9m5s Details
/ Lint (pull_request) Successful in 11m1s Details
/ Tests (1.20) (pull_request) Successful in 9m59s Details
/ Tests (1.21) (pull_request) Successful in 9m53s Details
/ DCO (pull_request) Successful in 11m34s Details
/ Builds (1.20) (pull_request) Successful in 12m24s Details
/ Builds (1.21) (pull_request) Successful in 8m59s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-11-22 11:12:32 +00:00
Marina Biryukova fe796ba538 [#217] Consider Copy-Source-SSE-* headers during copy 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-11-13 13:22:58 +00:00
Denis Kirillov 5ee73fad6a [#248] Correct NextVersionIDMarker in listing versions
/ DCO (pull_request) Successful in 4m40s Details
/ Vulncheck (pull_request) Successful in 5m43s Details
/ Lint (pull_request) Successful in 10m27s Details
/ Tests (1.20) (pull_request) Successful in 5m41s Details
/ Tests (1.21) (pull_request) Successful in 5m47s Details
/ Builds (1.20) (pull_request) Successful in 1m9s Details
/ Builds (1.21) (pull_request) Successful in 4m7s Details 
Despite the spec https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html#API_ListObjectVersions_ResponseElements
says that
"When the number of responses exceeds the value of MaxKeys,
NextVersionIdMarker specifies the first object version not returned
 that satisfies the search criteria. Use this value for the
 version-id-marker request parameter in a subsequent request."
 the actual behavior of AWS S3 is returning NextVersionIdMarker as the last returned object version

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 17:36:24 +03:00
Marina Biryukova 890a8ed237 [#227] Add versionID header after complete multipart 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-31 14:07:08 +00:00
Denis Kirillov 0bed25816c [#224] Add conditional escaping for object name 
Chi gives inconsistent results in terms of whether
the strings returned are URL coded or not
See:
* https://github.com/go-chi/chi/issues/641
* https://github.com/go-chi/chi/issues/642

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 13:58:51 +00:00
Denis Kirillov b169c5e6c3 [#239] Update test for check goroutines leak 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 13:51:23 +00:00
Marina Biryukova cf13aae342 [#225] Add default storage class to responses 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-31 13:37:07 +00:00
Denis Kirillov 0938d7ee82 [#226] Fix status code in GET/HEAD delete marker
/ Vulncheck (pull_request) Successful in 2m26s Details
/ Lint (pull_request) Successful in 4m0s Details
/ Tests (1.20) (pull_request) Successful in 2m59s Details
/ Tests (1.21) (pull_request) Successful in 1m52s Details
/ DCO (pull_request) Successful in 4m17s Details
/ Builds (1.20) (pull_request) Successful in 6m33s Details
/ Builds (1.21) (pull_request) Successful in 1m28s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-27 10:58:57 +03:00
Denis Kirillov 4f5f5fb5c8 [#222] Fix marshaling errors in `DeleteObjects` method 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-25 14:54:02 +00:00