TrueCloudLab/frostfs-s3-gw
17
3
Fork 19
Code Issues 29 Pull requests 7 Projects Releases 18 Packages Activity Actions
1317 commits 5 branches 43 tags 21 MiB
Commit graph

704 commits

Author SHA1 Message Date
Denis Kirillov
da642a498a [#165] Listing fix data race 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
2d7973b3f1 [#165] Refactor list versions 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
6d52f46012 [#165] Fix v1 listing bookmark 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
0ae49eaab0 [#165] Generalize allObjectListingParams 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
093de13f54 [#165] Add stream listing tests 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
cf4fc3b602 [#165] Extend error on getting listing containers not in current namespace 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
739a6ec9df [#165] Support latest only stream listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
c7ee628ab0 [#165] Fix versions listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
4ad84b9b94 [#165] listing: Use NodeVersion instead of ObjectInfo 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
3e20f736a6 [#165] Move listing function to one file 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
b52552e8c2 [#165] Add batching in streamin listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
6e8960b2ab [#165] Add list session cache 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
29ac91dfd5 [#165] Support streaming listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
84af85ed67 [#302] Update APE to support chain id as bytes
All checks were successful
/ DCO (pull_request) Successful in 1m52s
Details
/ Vulncheck (pull_request) Successful in 2m0s
Details
/ Builds (1.20) (pull_request) Successful in 2m49s
Details
/ Builds (1.21) (pull_request) Successful in 1m59s
Details
/ Lint (pull_request) Successful in 2m44s
Details
/ Tests (1.20) (pull_request) Successful in 2m30s
Details
/ Tests (1.21) (pull_request) Successful in 2m24s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 12:53:45 +03:00
Denis Kirillov
e23cc43824 [#299] Drop unused legacy minio code
All checks were successful
/ DCO (pull_request) Successful in 2m36s
Details
/ Vulncheck (pull_request) Successful in 2m59s
Details
/ Lint (pull_request) Successful in 5m23s
Details
/ Tests (1.20) (pull_request) Successful in 3m24s
Details
/ Tests (1.21) (pull_request) Successful in 3m3s
Details
/ Builds (1.20) (pull_request) Successful in 1m16s
Details
/ Builds (1.21) (pull_request) Successful in 2m53s
Details 
736d8cbac4 (diff-f5a8931b4d5f3b7f583e4cd719bfd2904980518a6f338d463ec76aea814db772)
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-01-29 12:57:18 +03:00
Denis Kirillov
eae49908da [#292] authmate: Support custom attributes
All checks were successful
/ DCO (pull_request) Successful in 1m36s
Details
/ Builds (1.20) (pull_request) Successful in 2m2s
Details
/ Builds (1.21) (pull_request) Successful in 1m28s
Details
/ Vulncheck (pull_request) Successful in 1m53s
Details
/ Lint (pull_request) Successful in 3m19s
Details
/ Tests (1.20) (pull_request) Successful in 2m24s
Details
/ Tests (1.21) (pull_request) Successful in 2m11s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-01-22 10:02:43 +03:00
Roman Loginov
6f9ee3da76 [#275] Change logic delete multipart upload
All checks were successful
/ Vulncheck (pull_request) Successful in 1m13s
Details
/ DCO (pull_request) Successful in 1m43s
Details
/ Builds (1.20) (pull_request) Successful in 2m49s
Details
/ Builds (1.21) (pull_request) Successful in 1m36s
Details
/ Lint (pull_request) Successful in 3m33s
Details
/ Tests (1.20) (pull_request) Successful in 2m34s
Details
/ Tests (1.21) (pull_request) Successful in 2m29s
Details 
In order not to accidentally take outdated
information about downloaded parts from other
nodes, now when the multipart is abort or complete,
the root node of the multipart upload with the
finish flag remains in the tree.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-12-27 13:06:45 +03:00
Marina Biryukova
08019f1574 [#280] Add put requests to duration metric
All checks were successful
/ DCO (pull_request) Successful in 1m48s
Details
/ Builds (1.20) (pull_request) Successful in 2m10s
Details
/ Builds (1.21) (pull_request) Successful in 1m24s
Details
/ Vulncheck (pull_request) Successful in 2m0s
Details
/ Lint (pull_request) Successful in 4m24s
Details
/ Tests (1.20) (pull_request) Successful in 2m16s
Details
/ Tests (1.21) (pull_request) Successful in 2m6s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-12-22 16:33:05 +03:00
Denis Kirillov
5698d5844e [#283] Support frostfsid groups in policy request checking 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-21 14:21:36 +03:00
Denis Kirillov
43cae9ee04 [#248] Correct object versions response markers
All checks were successful
/ DCO (pull_request) Successful in 1m40s
Details
/ Vulncheck (pull_request) Successful in 1m42s
Details
/ Builds (1.20) (pull_request) Successful in 2m29s
Details
/ Builds (1.21) (pull_request) Successful in 1m37s
Details
/ Lint (pull_request) Successful in 3m55s
Details
/ Tests (1.20) (pull_request) Successful in 2m23s
Details
/ Tests (1.21) (pull_request) Successful in 2m16s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-21 10:53:50 +03:00
Denis Kirillov
a17ff66975 [#282] policy: Use prefixes to distinguish s3/iam actions/resources
All checks were successful
/ DCO (pull_request) Successful in 1m37s
Details
/ Vulncheck (pull_request) Successful in 1m50s
Details
/ Builds (1.20) (pull_request) Successful in 2m24s
Details
/ Builds (1.21) (pull_request) Successful in 2m2s
Details
/ Lint (pull_request) Successful in 4m26s
Details
/ Tests (1.20) (pull_request) Successful in 2m28s
Details
/ Tests (1.21) (pull_request) Successful in 1m58s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-20 10:41:15 +03:00
Denis Kirillov
38c5503a02 [#261] alc: Remove unused
All checks were successful
/ Vulncheck (pull_request) Successful in 2m1s
Details
/ DCO (pull_request) Successful in 1m49s
Details
/ Builds (1.20) (pull_request) Successful in 2m21s
Details
/ Builds (1.21) (pull_request) Successful in 2m4s
Details
/ Lint (pull_request) Successful in 2m56s
Details
/ Tests (1.20) (pull_request) Successful in 2m17s
Details
/ Tests (1.21) (pull_request) Successful in 1m46s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-18 15:49:58 +03:00
Denis Kirillov
8273af8bf8 [#261] Make PutBucketPolicy handler use policy contract 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-18 15:49:54 +03:00
Denis Kirillov
6dbb07f0fa [#261] Update policy-engine dependency 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-14 17:52:13 +03:00
Denis Kirillov
9272f4e108 [#259] Support contract based policies
All checks were successful
/ DCO (pull_request) Successful in 1m21s
Details
/ Vulncheck (pull_request) Successful in 1m41s
Details
/ Builds (1.20) (pull_request) Successful in 2m19s
Details
/ Builds (1.21) (pull_request) Successful in 2m1s
Details
/ Lint (pull_request) Successful in 3m20s
Details
/ Tests (1.20) (pull_request) Successful in 2m14s
Details
/ Tests (1.21) (pull_request) Successful in 2m10s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-11 10:01:46 +03:00
Denis Kirillov
836874a761 [#262] Set tree request id 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-07 16:40:56 +03:00
Alex Vanin
f58a0d04ff [#233] Clean tag node in the tree service instead of removal 
With new retry policy of tree service pool, gateway should avoid
deletion of system nodes from tree. Absence of node in the tree
will trigger retry. Other storage in the network may return already
deleted node while tree is not completely synced, and client will
get unexpected result.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-12-07 16:40:56 +03:00
Denis Kirillov
43abf58068 [#257] Support flag to deny access if policy rules not found
All checks were successful
/ DCO (pull_request) Successful in 1m13s
Details
/ Vulncheck (pull_request) Successful in 2m2s
Details
/ Builds (1.20) (pull_request) Successful in 2m22s
Details
/ Builds (1.21) (pull_request) Successful in 2m16s
Details
/ Lint (pull_request) Successful in 3m26s
Details
/ Tests (1.20) (pull_request) Successful in 2m21s
Details
/ Tests (1.21) (pull_request) Successful in 1m37s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-06 17:47:58 +03:00
Denis Kirillov
ca15acf1d3 [#257] router: Use named constants 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-06 17:47:58 +03:00
Denis Kirillov
473239bf36 [#257] Add policy checker 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-12-06 17:47:51 +03:00
Marina Biryukova
93cf7c462b [#271] Add namespace label to billing metrics
All checks were successful
/ DCO (pull_request) Successful in 2m35s
Details
/ Vulncheck (pull_request) Successful in 3m3s
Details
/ Builds (1.20) (pull_request) Successful in 3m34s
Details
/ Builds (1.21) (pull_request) Successful in 2m20s
Details
/ Lint (pull_request) Successful in 5m27s
Details
/ Tests (1.20) (pull_request) Successful in 3m25s
Details
/ Tests (1.21) (pull_request) Successful in 3m12s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-12-04 14:54:40 +03:00
Denis Kirillov
28c6bb4cb8 [#266] Support per namespace placement policies configuration
All checks were successful
/ DCO (pull_request) Successful in 1m34s
Details
/ Builds (1.20) (pull_request) Successful in 3m15s
Details
/ Builds (1.21) (pull_request) Successful in 2m55s
Details
/ Vulncheck (pull_request) Successful in 2m51s
Details
/ Lint (pull_request) Successful in 5m12s
Details
/ Tests (1.20) (pull_request) Successful in 2m57s
Details
/ Tests (1.21) (pull_request) Successful in 2m48s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-28 16:12:42 +03:00
Denis Kirillov
ff1ec56d24 [#260] Use namespace as domain when create bucket
All checks were successful
/ DCO (pull_request) Successful in 1m26s
Details
/ Vulncheck (pull_request) Successful in 2m6s
Details
/ Builds (1.20) (pull_request) Successful in 2m20s
Details
/ Builds (1.21) (pull_request) Successful in 2m21s
Details
/ Lint (pull_request) Successful in 3m15s
Details
/ Tests (1.20) (pull_request) Successful in 2m28s
Details
/ Tests (1.21) (pull_request) Successful in 2m17s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-27 15:30:12 +03:00
Denis Kirillov
055cc6a22a [#260] Use namespace as domain when resolve bucket 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-23 11:00:11 +03:00
Denis Kirillov
6304d7bfda [#260] Support frostfsid validation 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-23 11:00:11 +03:00
Denis Kirillov
cf7254f8cd [#260] Refactor api/auth/center.go 
Move the Center interface to middleware package where it's used

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-23 11:00:09 +03:00
Roman Loginov
861454e499 [#218] Add check content sha256 header 
The X-Amz-Content-Sha256 header check is done only for unencrypted payload.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-11-22 11:33:52 +00:00
Marina Biryukova
b28ecef43b [#219] Return ETag in quotes
All checks were successful
/ Vulncheck (pull_request) Successful in 9m5s
Details
/ Lint (pull_request) Successful in 11m1s
Details
/ Tests (1.20) (pull_request) Successful in 9m59s
Details
/ Tests (1.21) (pull_request) Successful in 9m53s
Details
/ DCO (pull_request) Successful in 11m34s
Details
/ Builds (1.20) (pull_request) Successful in 12m24s
Details
/ Builds (1.21) (pull_request) Successful in 8m59s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-11-22 11:12:32 +00:00
Marina Biryukova
fe796ba538 [#217] Consider Copy-Source-SSE-* headers during copy 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-11-13 13:22:58 +00:00
Denis Kirillov
5ee73fad6a [#248] Correct NextVersionIDMarker in listing versions
All checks were successful
/ DCO (pull_request) Successful in 4m40s
Details
/ Vulncheck (pull_request) Successful in 5m43s
Details
/ Lint (pull_request) Successful in 10m27s
Details
/ Tests (1.20) (pull_request) Successful in 5m41s
Details
/ Tests (1.21) (pull_request) Successful in 5m47s
Details
/ Builds (1.20) (pull_request) Successful in 1m9s
Details
/ Builds (1.21) (pull_request) Successful in 4m7s
Details 
Despite the spec https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html#API_ListObjectVersions_ResponseElements
says that
"When the number of responses exceeds the value of MaxKeys,
NextVersionIdMarker specifies the first object version not returned
 that satisfies the search criteria. Use this value for the
 version-id-marker request parameter in a subsequent request."
 the actual behavior of AWS S3 is returning NextVersionIdMarker as the last returned object version

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 17:36:24 +03:00
Marina Biryukova
890a8ed237 [#227] Add versionID header after complete multipart 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-31 14:07:08 +00:00
Denis Kirillov
0bed25816c [#224] Add conditional escaping for object name 
Chi gives inconsistent results in terms of whether
the strings returned are URL coded or not
See:
* https://github.com/go-chi/chi/issues/641
* https://github.com/go-chi/chi/issues/642

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 13:58:51 +00:00
Denis Kirillov
b169c5e6c3 [#239] Update test for check goroutines leak 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 13:51:23 +00:00
Marina Biryukova
cf13aae342 [#225] Add default storage class to responses 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-31 13:37:07 +00:00
Denis Kirillov
0938d7ee82 [#226] Fix status code in GET/HEAD delete marker
All checks were successful
/ Vulncheck (pull_request) Successful in 2m26s
Details
/ Lint (pull_request) Successful in 4m0s
Details
/ Tests (1.20) (pull_request) Successful in 2m59s
Details
/ Tests (1.21) (pull_request) Successful in 1m52s
Details
/ DCO (pull_request) Successful in 4m17s
Details
/ Builds (1.20) (pull_request) Successful in 6m33s
Details
/ Builds (1.21) (pull_request) Successful in 1m28s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-27 10:58:57 +03:00
Denis Kirillov
4f5f5fb5c8 [#222] Fix marshaling errors in DeleteObjects method 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-25 14:54:02 +00:00
Marina Biryukova
25bb581fee [#205] Add md5 checksum in header
All checks were successful
/ Vulncheck (pull_request) Successful in 1m50s
Details
/ DCO (pull_request) Successful in 3m29s
Details
/ Lint (pull_request) Successful in 3m37s
Details
/ Tests (1.20) (pull_request) Successful in 2m9s
Details
/ Tests (1.21) (pull_request) Successful in 2m5s
Details
/ Builds (1.20) (pull_request) Successful in 5m47s
Details
/ Builds (1.21) (pull_request) Successful in 1m29s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-25 11:04:19 +03:00
Denis Kirillov
8d6aa0d40a [#243] Fix list object versions marker param
All checks were successful
/ Vulncheck (pull_request) Successful in 2m8s
Details
/ DCO (pull_request) Successful in 2m5s
Details
/ Builds (1.20) (pull_request) Successful in 2m39s
Details
/ Builds (1.21) (pull_request) Successful in 2m15s
Details
/ Lint (pull_request) Successful in 4m23s
Details
/ Tests (1.20) (pull_request) Successful in 2m53s
Details
/ Tests (1.21) (pull_request) Successful in 2m18s
Details 
According to https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html
we have to use `key-marker`

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-18 10:35:47 +03:00
Roman Loginov
7e91f62c28 [#223] Add store content language 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-10-17 14:42:02 +00:00
Roman Loginov
01323ca8e0 [#216] Add check tag key uniqueness 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-10-17 14:40:29 +00:00
Marina Biryukova
298662df9d [#221] Expand xmlns field ignore
All checks were successful
/ Vulncheck (pull_request) Successful in 1m38s
Details
/ Lint (pull_request) Successful in 2m49s
Details
/ Tests (1.20) (pull_request) Successful in 1m56s
Details
/ Tests (1.21) (pull_request) Successful in 1m44s
Details
/ DCO (pull_request) Successful in 3m56s
Details
/ Builds (1.20) (pull_request) Successful in 6m25s
Details
/ Builds (1.21) (pull_request) Successful in 1m36s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-13 16:21:13 +03:00
Denis Kirillov
65412ce1d3 [#197] Configure buffer max size for PUT 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-11 12:32:48 +00:00
Denis Kirillov
7de73f6b73 [#197] Disable homomorphic hash for PUT 
Disable TZ hash for PUT if it's disabled for container itself

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-11 12:32:48 +00:00
Roman Loginov
e3f2d59565 [#154] Rename access key to secret key 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-10-06 09:20:39 +00:00
Roman Loginov
c4af1dc4ad [#171] Update message error auth header malformed 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-10-04 11:13:12 +00:00
Marina Biryukova
b8c93ed391 [#172] Convert handler config to interface 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-04 11:01:27 +00:00
Marina Biryukova
51e591877b [#207] Fix list parts with empty list
All checks were successful
/ DCO (pull_request) Successful in 1m2s
Details
/ Vulncheck (pull_request) Successful in 1m25s
Details
/ Builds (1.20) (pull_request) Successful in 1m59s
Details
/ Builds (1.21) (pull_request) Successful in 1m52s
Details
/ Lint (pull_request) Successful in 8m23s
Details
/ Tests (1.20) (pull_request) Successful in 2m1s
Details
/ Tests (1.21) (pull_request) Successful in 9m26s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-21 11:27:20 +03:00
Marina Biryukova
a4c612614a [#210] Fix multipart object reader
All checks were successful
/ Vulncheck (pull_request) Successful in 2m38s
Details
/ DCO (pull_request) Successful in 3m40s
Details
/ Lint (pull_request) Successful in 5m20s
Details
/ Tests (1.20) (pull_request) Successful in 3m19s
Details
/ Tests (1.21) (pull_request) Successful in 3m8s
Details
/ Builds (1.20) (pull_request) Successful in 6m1s
Details
/ Builds (1.21) (pull_request) Successful in 1m33s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-19 16:30:08 +03:00
Marina Biryukova
12cf29aed2 [#207] Fix part-number-marker handling
All checks were successful
/ DCO (pull_request) Successful in 1m57s
Details
/ Vulncheck (pull_request) Successful in 2m21s
Details
/ Builds (1.20) (pull_request) Successful in 1m36s
Details
/ Builds (1.21) (pull_request) Successful in 7m55s
Details
/ Lint (pull_request) Successful in 13m57s
Details
/ Tests (1.20) (pull_request) Successful in 2m34s
Details
/ Tests (1.21) (pull_request) Successful in 1m57s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-19 12:43:07 +03:00
Marina Biryukova
066b9a0250 [#142] Add trace ID into log when tracing is enabled
All checks were successful
/ Vulncheck (pull_request) Successful in 1m29s
Details
/ DCO (pull_request) Successful in 2m18s
Details
/ Lint (pull_request) Successful in 3m48s
Details
/ Tests (1.20) (pull_request) Successful in 2m4s
Details
/ Tests (1.21) (pull_request) Successful in 1m42s
Details
/ Builds (1.20) (pull_request) Successful in 7m6s
Details
/ Builds (1.21) (pull_request) Successful in 1m56s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-07 14:19:37 +03:00
Marina Biryukova
9120e97ac5 [#203] Add go1.21 to CI
All checks were successful
/ DCO (pull_request) Successful in 59s
Details
/ Vulncheck (pull_request) Successful in 1m35s
Details
/ Builds (1.20) (pull_request) Successful in 1m21s
Details
/ Builds (1.21) (pull_request) Successful in 7m46s
Details
/ Lint (pull_request) Successful in 12m44s
Details
/ Tests (1.20) (pull_request) Successful in 2m9s
Details
/ Tests (1.21) (pull_request) Successful in 2m0s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-08-31 15:26:07 +03:00
Denis Kirillov
b5fce5c8d2 [#168] Skip only invalid policies and copies instead of ignoring all of them
All checks were successful
/ DCO (pull_request) Successful in 2m48s
Details
/ Vulncheck (pull_request) Successful in 3m30s
Details
/ Builds (1.19) (pull_request) Successful in 3m39s
Details
/ Builds (1.20) (pull_request) Successful in 3m28s
Details
/ Lint (pull_request) Successful in 4m14s
Details
/ Tests (1.19) (pull_request) Successful in 3m3s
Details
/ Tests (1.20) (pull_request) Successful in 2m45s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-25 12:05:30 +03:00
Denis Kirillov
6617adc22b [#185] Use correct object size when object is combined or encrypted 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-24 18:03:08 +03:00
Denis Kirillov
8898c2ec08 [#185] Add tests for list multipart uploads 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-24 18:01:08 +03:00
Roman Loginov
8efcc957ea [#96] Move log messages to constants
All checks were successful
/ DCO (pull_request) Successful in 1m35s
Details
/ Builds (1.19) (pull_request) Successful in 2m14s
Details
/ Builds (1.20) (pull_request) Successful in 2m9s
Details
/ Vulncheck (pull_request) Successful in 5m39s
Details
/ Lint (pull_request) Successful in 2m49s
Details
/ Tests (1.19) (pull_request) Successful in 7m34s
Details
/ Tests (1.20) (pull_request) Successful in 1m44s
Details 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-08-23 18:32:31 +03:00
Denis Kirillov
6b728fef87 [#192] Add tests to make sure client_cut flag is passed to sdk 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-23 06:26:55 +00:00
Denis Kirillov
6b1f365e65 [#192] Support client cut 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-23 06:26:55 +00:00
Denis Kirillov
fcf1c45ad2 [#188] Fix url escaping
All checks were successful
/ Vulncheck (pull_request) Successful in 2m41s
Details
/ Builds (1.19) (pull_request) Successful in 3m51s
Details
/ Builds (1.20) (pull_request) Successful in 3m22s
Details
/ DCO (pull_request) Successful in 5m9s
Details
/ Lint (pull_request) Successful in 5m22s
Details
/ Tests (1.19) (pull_request) Successful in 5m39s
Details
/ Tests (1.20) (pull_request) Successful in 5m18s
Details 
Url escaping has already been done in `net/http/request.go`

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-22 11:27:39 +03:00
Artem Tataurov
94a42fa273 [#51] Update frostfs-sdk-go 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-08-17 14:26:25 +03:00
Roman Loginov
40d7f844e3 [#137] Refactor context data retrievers 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-08-16 14:05:38 +00:00
Denis Kirillov
6b109eee92 [#182] Fix parsing signed headers in presigned urls
Some checks failed
/ Vulncheck (pull_request) Failing after 3s
Details
/ Builds (1.19) (pull_request) Successful in 8m22s
Details
/ Builds (1.20) (pull_request) Failing after 3s
Details
/ Lint (pull_request) Successful in 9m51s
Details
/ Tests (1.19) (pull_request) Failing after 3s
Details
/ Tests (1.20) (pull_request) Failing after 2s
Details
/ DCO (pull_request) Failing after 2s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 13:04:53 +03:00
Denis Kirillov
18878b66d3 [#175] Use gate owner as object owner
Some checks failed
/ DCO (pull_request) Failing after 3s
Details
/ Vulncheck (pull_request) Failing after 2s
Details
/ Builds (1.19) (pull_request) Failing after 3s
Details
/ Builds (1.20) (pull_request) Successful in 7m47s
Details
/ Lint (pull_request) Failing after 4s
Details
/ Tests (1.19) (pull_request) Successful in 8m8s
Details
/ Tests (1.20) (pull_request) Failing after 3s
Details 
This is required because node check session token owner
TrueCloudLab/frostfs-node#528
For client cut TrueCloudLab/frostfs-sdk-go#114
such owner will be gate owner

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 12:22:33 +03:00
Denis Kirillov
fe897ec588 [#178] wrapReader: Fix goroutine leak
All checks were successful
/ Vulncheck (pull_request) Successful in 59s
Details
/ Builds (1.19) (pull_request) Successful in 1m28s
Details
/ Builds (1.20) (pull_request) Successful in 1m19s
Details
/ DCO (pull_request) Successful in 45s
Details
/ Lint (pull_request) Successful in 1m45s
Details
/ Tests (1.19) (pull_request) Successful in 7m51s
Details
/ Tests (1.20) (pull_request) Successful in 1m43s
Details 
In case of error in FrostFS.CreateObject wrapped reader
can be blocked because of synchronous pipe. We have to read out all payload in such case.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-01 17:59:34 +03:00
Denis Kirillov
52931663e1 [#176] multipart: Replace part on re-upload
All checks were successful
/ Builds (1.19) (pull_request) Successful in 2m57s
Details
/ Builds (1.20) (pull_request) Successful in 2m36s
Details
/ DCO (pull_request) Successful in 1m15s
Details
/ Vulncheck (pull_request) Successful in 5m15s
Details
/ Lint (pull_request) Successful in 3m50s
Details
/ Tests (1.19) (pull_request) Successful in 8m14s
Details
/ Tests (1.20) (pull_request) Successful in 3m17s
Details 
We want to have exactly one object and tree node for each part number

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-21 16:20:04 +03:00
Denis Kirillov
1a09041cd1 [#63] Simplify multiObjectReader and add tests
All checks were successful
/ DCO (pull_request) Successful in 1m19s
Details
/ Vulncheck (pull_request) Successful in 2m31s
Details
/ Builds (1.19) (pull_request) Successful in 2m51s
Details
/ Builds (1.20) (pull_request) Successful in 2m34s
Details
/ Lint (pull_request) Successful in 5m51s
Details
/ Tests (1.19) (pull_request) Successful in 9m36s
Details
/ Tests (1.20) (pull_request) Successful in 10m20s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov
631b7f67b4 [#63] multipart: Log upload id for every failed request 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov
bcf5a85aab [#63] multipart: Fix copying 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov
ad81b599dd [#63] Add fast multipart upload 
Add new flag to object tree meta `isCombined` that means
the object payload is list of parts that forms real payload.
Set this attribute when complete multipart upload not to do unnecessary copying.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov
361d10cc78 [#174] Fix query for listing multipart uploads
All checks were successful
/ Builds (1.19) (pull_request) Successful in 3m2s
Details
/ Builds (1.20) (pull_request) Successful in 2m50s
Details
/ DCO (pull_request) Successful in 3m59s
Details
/ Vulncheck (pull_request) Successful in 2m39s
Details
/ Lint (pull_request) Successful in 3m56s
Details
/ Tests (1.19) (pull_request) Successful in 3m5s
Details
/ Tests (1.20) (pull_request) Successful in 3m12s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 12:30:17 +03:00
Denis Kirillov
62e6b49254 [#174] Log unmatched requests 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 12:30:17 +03:00
Denis Kirillov
80c4982bd4 [#174] Add router tests 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 12:30:17 +03:00
Denis Kirillov
73ed3f7782 [#174] Fix router filter query matching 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 12:30:17 +03:00
Denis Kirillov
6e3595e35b [#174] Fix object keys with slashes in chi 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 12:30:17 +03:00
Denis Kirillov
b59aa06637 [#146] Add kludge.bypass_content_encoding_check_in_chunks flag
Some checks failed
/ Builds (1.19) (pull_request) Successful in 3m0s
Details
/ Builds (1.20) (pull_request) Successful in 2m50s
Details
/ DCO (pull_request) Failing after 1m9s
Details
/ Vulncheck (pull_request) Successful in 1m20s
Details
/ Lint (pull_request) Successful in 7m50s
Details
/ Tests (1.19) (pull_request) Successful in 3m1s
Details
/ Tests (1.20) (pull_request) Successful in 3m21s
Details 
Flag allows to skip checking `Content-Encoding` for `aws-chunked` value

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-18 14:49:52 +03:00
Denis Kirillov
d62aa7b979 [#146] Fix preconditions: trim quotes in etags 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-18 14:49:03 +03:00
Denis Kirillov
751a9be7cc [#146] Move getting chunk payload reader to separate function 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-18 14:49:03 +03:00
Denis Kirillov
14ef9ff091 [#158] Separate init object reader from read itself
All checks were successful
/ Vulncheck (pull_request) Successful in 1m40s
Details
/ Builds (1.19) (pull_request) Successful in 2m54s
Details
/ Builds (1.20) (pull_request) Successful in 2m39s
Details
/ DCO (pull_request) Successful in 3m48s
Details
/ Lint (pull_request) Successful in 3m32s
Details
/ Tests (1.19) (pull_request) Successful in 2m38s
Details
/ Tests (1.20) (pull_request) Successful in 2m55s
Details 
To be able to handle cases and return appropriate http status code
when object missed in storage but gate cache contains its metadata
we need write code after init object reader.
So we separate init reader from actual reading.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:32:05 +03:00
Denis Kirillov
fc90981c03 [#149] Update inner imports after moving middlewares 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:09 +03:00
Denis Kirillov
83cdfbee78 [#149] Move middlewares to separate package 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:09 +03:00
Denis Kirillov
37f2f468fe [#149] Add host bucket router 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:09 +03:00
Denis Kirillov
e30a452cdb [#149] Use chi instead of gorilla mux 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:25:09 +03:00
Denis Kirillov
7f708b3a2d [#111] auth: Get log from real request context 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-06 11:57:04 +00:00
Denis Kirillov
d531b13866 [#143] Add more context to some s3 errors
All checks were successful
/ DCO (pull_request) Successful in 1m18s
Details
/ Vulncheck (pull_request) Successful in 1m50s
Details
/ Lint (pull_request) Successful in 3m1s
Details
/ Tests (1.19) (pull_request) Successful in 2m47s
Details
/ Tests (1.20) (pull_request) Successful in 3m2s
Details
/ Builds (1.19) (pull_request) Successful in 2m54s
Details
/ Builds (1.20) (pull_request) Successful in 7m58s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-30 12:08:33 +03:00
Denis Kirillov
f921bc8af5 [#143] Fix typo 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-30 12:04:52 +03:00
Denis Kirillov
be03c5178f [#143] Fix NoSuchKey error on get/head 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-30 12:04:52 +03:00
Denis Kirillov
9df8695463 [#143] Fix transformToS3Error function
All checks were successful
/ Builds (1.19) (pull_request) Successful in 3m11s
Details
/ Builds (1.20) (pull_request) Successful in 2m57s
Details
/ DCO (pull_request) Successful in 4m7s
Details
/ Lint (pull_request) Successful in 2m25s
Details
/ Tests (1.19) (pull_request) Successful in 3m9s
Details
/ Tests (1.20) (pull_request) Successful in 3m18s
Details
/ Vulncheck (pull_request) Successful in 1m15s
Details 
Unwrap error before checking for s3 error

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-21 17:16:40 +03:00
Artem Tataurov
614d703726 [#106] Add chunk uploading
All checks were successful
/ DCO (pull_request) Successful in 1m4s
Details
/ Vulncheck (pull_request) Successful in 1m50s
Details
/ Builds (1.19) (pull_request) Successful in 9m1s
Details
/ Builds (1.20) (pull_request) Successful in 2m20s
Details
/ Lint (pull_request) Successful in 10m19s
Details
/ Tests (1.19) (pull_request) Successful in 2m45s
Details
/ Tests (1.20) (pull_request) Successful in 3m19s
Details 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-06-21 10:23:57 +03:00
Denis Kirillov
23593eee3d [#111] Use request scope logger
Some checks failed
/ Lint (pull_request) Failing after 32s
Details
/ Tests (1.19) (pull_request) Failing after 32s
Details
/ Tests (1.20) (pull_request) Failing after 32s
Details
/ Builds (1.19) (pull_request) Failing after 32s
Details
/ Builds (1.20) (pull_request) Failing after 32s
Details
/ Vulncheck (pull_request) Failing after 57s
Details
/ DCO (pull_request) Failing after 1m0s
Details 
Store child zap logger with request scope fields into context.
Request scoped fields: request_id, api/method, bucket, object

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-19 13:54:51 +03:00
Denis Kirillov
84358f6742 [#135] authmate: Support CRDT GSet for credentials 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-16 09:42:49 +03:00
Denis Kirillov
7a380fa46c [#135] frostfs: Add SEARCH operation 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-16 09:42:41 +03:00