Commit graph

1322 commits

Author SHA1 Message Date
6f9ee3da76 [#275] Change logic delete multipart upload
All checks were successful
/ Vulncheck (pull_request) Successful in 1m13s
/ DCO (pull_request) Successful in 1m43s
/ Builds (1.20) (pull_request) Successful in 2m49s
/ Builds (1.21) (pull_request) Successful in 1m36s
/ Lint (pull_request) Successful in 3m33s
/ Tests (1.20) (pull_request) Successful in 2m34s
/ Tests (1.21) (pull_request) Successful in 2m29s
In order not to accidentally take outdated
information about downloaded parts from other
nodes, now when the multipart is abort or complete,
the root node of the multipart upload with the
finish flag remains in the tree.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-12-27 13:06:45 +03:00
08019f1574 [#280] Add put requests to duration metric
All checks were successful
/ DCO (pull_request) Successful in 1m48s
/ Builds (1.20) (pull_request) Successful in 2m10s
/ Builds (1.21) (pull_request) Successful in 1m24s
/ Vulncheck (pull_request) Successful in 2m0s
/ Lint (pull_request) Successful in 4m24s
/ Tests (1.20) (pull_request) Successful in 2m16s
/ Tests (1.21) (pull_request) Successful in 2m6s
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-12-22 16:33:05 +03:00
899213b3f3 [#287] Support proxy for frostfsid and policy contracts
All checks were successful
/ Builds (1.20) (pull_request) Successful in 1m52s
/ Builds (1.21) (pull_request) Successful in 1m12s
/ DCO (pull_request) Successful in 1m46s
/ Vulncheck (pull_request) Successful in 1m43s
/ Lint (pull_request) Successful in 4m3s
/ Tests (1.20) (pull_request) Successful in 2m24s
/ Tests (1.21) (pull_request) Successful in 2m23s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-22 15:45:10 +03:00
3b6d2bc522 [#287] authmate: Support frostfsid proxy and namespace
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-22 15:43:14 +03:00
6509639540 [#283] control: Changed type of chainID to bytes
All checks were successful
/ DCO (pull_request) Successful in 1m38s
/ Builds (1.20) (pull_request) Successful in 2m11s
/ Builds (1.21) (pull_request) Successful in 2m7s
/ Vulncheck (pull_request) Successful in 1m54s
/ Lint (pull_request) Successful in 4m2s
/ Tests (1.20) (pull_request) Successful in 2m23s
/ Tests (1.21) (pull_request) Successful in 2m13s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-21 18:07:17 +03:00
5698d5844e [#283] Support frostfsid groups in policy request checking
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-21 14:21:36 +03:00
43cae9ee04 [#248] Correct object versions response markers
All checks were successful
/ DCO (pull_request) Successful in 1m40s
/ Vulncheck (pull_request) Successful in 1m42s
/ Builds (1.20) (pull_request) Successful in 2m29s
/ Builds (1.21) (pull_request) Successful in 1m37s
/ Lint (pull_request) Successful in 3m55s
/ Tests (1.20) (pull_request) Successful in 2m23s
/ Tests (1.21) (pull_request) Successful in 2m16s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-21 10:53:50 +03:00
a17ff66975 [#282] policy: Use prefixes to distinguish s3/iam actions/resources
All checks were successful
/ DCO (pull_request) Successful in 1m37s
/ Vulncheck (pull_request) Successful in 1m50s
/ Builds (1.20) (pull_request) Successful in 2m24s
/ Builds (1.21) (pull_request) Successful in 2m2s
/ Lint (pull_request) Successful in 4m26s
/ Tests (1.20) (pull_request) Successful in 2m28s
/ Tests (1.21) (pull_request) Successful in 1m58s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-20 10:41:15 +03:00
38c5503a02 [#261] alc: Remove unused
All checks were successful
/ Vulncheck (pull_request) Successful in 2m1s
/ DCO (pull_request) Successful in 1m49s
/ Builds (1.20) (pull_request) Successful in 2m21s
/ Builds (1.21) (pull_request) Successful in 2m4s
/ Lint (pull_request) Successful in 2m56s
/ Tests (1.20) (pull_request) Successful in 2m17s
/ Tests (1.21) (pull_request) Successful in 1m46s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-18 15:49:58 +03:00
8273af8bf8 [#261] Make PutBucketPolicy handler use policy contract
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-18 15:49:54 +03:00
6dbb07f0fa [#261] Update policy-engine dependency
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-14 17:52:13 +03:00
340e6b807b [#266] Update params that being reloaded in one place
All checks were successful
/ DCO (pull_request) Successful in 1m33s
/ Builds (1.20) (pull_request) Successful in 1m57s
/ Builds (1.21) (pull_request) Successful in 1m39s
/ Vulncheck (pull_request) Successful in 1m55s
/ Lint (pull_request) Successful in 4m0s
/ Tests (1.20) (pull_request) Successful in 2m30s
/ Tests (1.21) (pull_request) Successful in 2m22s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-14 16:21:09 +03:00
0850d21ff3 [#266] Move trimming namespaces name into fetch function
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-14 16:12:10 +03:00
9272f4e108 [#259] Support contract based policies
All checks were successful
/ DCO (pull_request) Successful in 1m21s
/ Vulncheck (pull_request) Successful in 1m41s
/ Builds (1.20) (pull_request) Successful in 2m19s
/ Builds (1.21) (pull_request) Successful in 2m1s
/ Lint (pull_request) Successful in 3m20s
/ Tests (1.20) (pull_request) Successful in 2m14s
/ Tests (1.21) (pull_request) Successful in 2m10s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-11 10:01:46 +03:00
be6a37ada5 [#262] Support configuring max tree request attempts
All checks were successful
/ DCO (pull_request) Successful in 1m15s
/ Vulncheck (pull_request) Successful in 1m39s
/ Builds (1.20) (pull_request) Successful in 1m19s
/ Builds (1.21) (pull_request) Successful in 2m27s
/ Lint (pull_request) Successful in 4m24s
/ Tests (1.20) (pull_request) Successful in 2m17s
/ Tests (1.21) (pull_request) Successful in 2m36s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-12-07 16:41:21 +03:00
836874a761 [#262] Set tree request id
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-07 16:40:56 +03:00
7a285d1464 [#267] Update api-go to fix stable marshal of empty structs
Newer version of api-go does not ignore non-nil empty
structures in protobuf messages, so compatibility with
previous version is preserved.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-12-07 16:40:56 +03:00
f58a0d04ff [#233] Clean tag node in the tree service instead of removal
With new retry policy of tree service pool, gateway should avoid
deletion of system nodes from tree. Absence of node in the tree
will trigger retry. Other storage in the network may return already
deleted node while tree is not completely synced, and client will
get unexpected result.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-12-07 16:40:56 +03:00
94c2674f44 [#233] Update tree service pool in SDK
Tree service pool now produces retries if
tree service returns 'not found' errors
or empty result on 'GetNodeByPath'

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-12-07 16:40:53 +03:00
6b1b43a364 Release v0.28.0
All checks were successful
/ Vulncheck (pull_request) Successful in 4m23s
/ DCO (pull_request) Successful in 4m25s
/ Builds (1.20) (pull_request) Successful in 6m12s
/ Builds (1.21) (pull_request) Successful in 5m58s
/ Lint (pull_request) Successful in 6m49s
/ Tests (1.20) (pull_request) Successful in 5m45s
/ Tests (1.21) (pull_request) Successful in 5m26s
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-12-07 15:47:31 +03:00
43abf58068 [#257] Support flag to deny access if policy rules not found
All checks were successful
/ DCO (pull_request) Successful in 1m13s
/ Vulncheck (pull_request) Successful in 2m2s
/ Builds (1.20) (pull_request) Successful in 2m22s
/ Builds (1.21) (pull_request) Successful in 2m16s
/ Lint (pull_request) Successful in 3m26s
/ Tests (1.20) (pull_request) Successful in 2m21s
/ Tests (1.21) (pull_request) Successful in 1m37s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-06 17:47:58 +03:00
ca15acf1d3 [#257] router: Use named constants
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-06 17:47:58 +03:00
473239bf36 [#257] Add policy checker
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-06 17:47:51 +03:00
93cf7c462b [#271] Add namespace label to billing metrics
All checks were successful
/ DCO (pull_request) Successful in 2m35s
/ Vulncheck (pull_request) Successful in 3m3s
/ Builds (1.20) (pull_request) Successful in 3m34s
/ Builds (1.21) (pull_request) Successful in 2m20s
/ Lint (pull_request) Successful in 5m27s
/ Tests (1.20) (pull_request) Successful in 3m25s
/ Tests (1.21) (pull_request) Successful in 3m12s
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-12-04 14:54:40 +03:00
6c5f9b2764 [#266] Fix namespace config initialization
Don't use nil Namespaces map in case when file isn't provided or invalid

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-04 09:40:47 +00:00
42862fd69e [#258] Support policy management in control svc
All checks were successful
/ Vulncheck (pull_request) Successful in 3m20s
/ DCO (pull_request) Successful in 3m56s
/ Builds (1.20) (pull_request) Successful in 4m13s
/ Builds (1.21) (pull_request) Successful in 4m9s
/ Lint (pull_request) Successful in 3m9s
/ Tests (1.20) (pull_request) Successful in 4m14s
/ Tests (1.21) (pull_request) Successful in 3m59s
Add PutPolicies, RemovePolicies, GetPolicy, ListPolicies methods

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-01 15:47:12 +03:00
c7a65bd075 [#258] Add control service
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-12-01 14:17:06 +03:00
28c6bb4cb8 [#266] Support per namespace placement policies configuration
All checks were successful
/ DCO (pull_request) Successful in 1m34s
/ Builds (1.20) (pull_request) Successful in 3m15s
/ Builds (1.21) (pull_request) Successful in 2m55s
/ Vulncheck (pull_request) Successful in 2m51s
/ Lint (pull_request) Successful in 5m12s
/ Tests (1.20) (pull_request) Successful in 2m57s
/ Tests (1.21) (pull_request) Successful in 2m48s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-11-28 16:12:42 +03:00
0db6cd6727 [#268] Add dirty version check
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-11-28 10:42:49 +00:00
ff1ec56d24 [#260] Use namespace as domain when create bucket
All checks were successful
/ DCO (pull_request) Successful in 1m26s
/ Vulncheck (pull_request) Successful in 2m6s
/ Builds (1.20) (pull_request) Successful in 2m20s
/ Builds (1.21) (pull_request) Successful in 2m21s
/ Lint (pull_request) Successful in 3m15s
/ Tests (1.20) (pull_request) Successful in 2m28s
/ Tests (1.21) (pull_request) Successful in 2m17s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-11-27 15:30:12 +03:00
9ebfca654b [#260] Update CHANGELOG.md
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-11-23 11:00:19 +03:00
055cc6a22a [#260] Use namespace as domain when resolve bucket
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-11-23 11:00:11 +03:00
a61ff3b8cb [#260] authmate: Support key registration in frostfsid contract
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-11-23 11:00:11 +03:00
6304d7bfda [#260] Support frostfsid validation
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-11-23 11:00:11 +03:00
cf7254f8cd [#260] Refactor api/auth/center.go
Move the Center interface to middleware package where it's used

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-11-23 11:00:09 +03:00
861454e499 [#218] Add check content sha256 header
The X-Amz-Content-Sha256 header check is done only for unencrypted payload.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-11-22 11:33:52 +00:00
b28ecef43b [#219] Return ETag in quotes
All checks were successful
/ Vulncheck (pull_request) Successful in 9m5s
/ Lint (pull_request) Successful in 11m1s
/ Tests (1.20) (pull_request) Successful in 9m59s
/ Tests (1.21) (pull_request) Successful in 9m53s
/ DCO (pull_request) Successful in 11m34s
/ Builds (1.20) (pull_request) Successful in 12m24s
/ Builds (1.21) (pull_request) Successful in 8m59s
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-11-22 11:12:32 +00:00
406075aebb [#236] Add support zapjournald logger configuration
All checks were successful
/ DCO (pull_request) Successful in 1m34s
/ Vulncheck (pull_request) Successful in 2m6s
/ Builds (1.20) (pull_request) Successful in 2m15s
/ Builds (1.21) (pull_request) Successful in 2m2s
/ Lint (pull_request) Successful in 2m59s
/ Tests (1.20) (pull_request) Successful in 2m22s
/ Tests (1.21) (pull_request) Successful in 1m43s
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-11-13 16:31:11 +03:00
fe796ba538 [#217] Consider Copy-Source-SSE-* headers during copy
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-11-13 13:22:58 +00:00
5ee73fad6a [#248] Correct NextVersionIDMarker in listing versions
All checks were successful
/ DCO (pull_request) Successful in 4m40s
/ Vulncheck (pull_request) Successful in 5m43s
/ Lint (pull_request) Successful in 10m27s
/ Tests (1.20) (pull_request) Successful in 5m41s
/ Tests (1.21) (pull_request) Successful in 5m47s
/ Builds (1.20) (pull_request) Successful in 1m9s
/ Builds (1.21) (pull_request) Successful in 4m7s
Despite the spec https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html#API_ListObjectVersions_ResponseElements
says that
"When the number of responses exceeds the value of MaxKeys,
NextVersionIdMarker specifies the first object version not returned
 that satisfies the search criteria. Use this value for the
 version-id-marker request parameter in a subsequent request."
 the actual behavior of AWS S3 is returning NextVersionIdMarker as the last returned object version

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-31 17:36:24 +03:00
890a8ed237 [#227] Add versionID header after complete multipart
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-31 14:07:08 +00:00
0bed25816c [#224] Add conditional escaping for object name
Chi gives inconsistent results in terms of whether
the strings returned are URL coded or not
See:
* https://github.com/go-chi/chi/issues/641
* https://github.com/go-chi/chi/issues/642

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-31 13:58:51 +00:00
b169c5e6c3 [#239] Update test for check goroutines leak
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-31 13:51:23 +00:00
122af0b5a7 [#220] Support configuring web server timeout params
Set IdleTimeout and ReadHeaderTimeout to `30s`.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-31 13:48:08 +00:00
cf13aae342 [#225] Add default storage class to responses
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-31 13:37:07 +00:00
0938d7ee82 [#226] Fix status code in GET/HEAD delete marker
All checks were successful
/ Vulncheck (pull_request) Successful in 2m26s
/ Lint (pull_request) Successful in 4m0s
/ Tests (1.20) (pull_request) Successful in 2m59s
/ Tests (1.21) (pull_request) Successful in 1m52s
/ DCO (pull_request) Successful in 4m17s
/ Builds (1.20) (pull_request) Successful in 6m33s
/ Builds (1.21) (pull_request) Successful in 1m28s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-27 10:58:57 +03:00
4f5f5fb5c8 [#222] Fix marshaling errors in DeleteObjects method
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-25 14:54:02 +00:00
25bb581fee [#205] Add md5 checksum in header
All checks were successful
/ Vulncheck (pull_request) Successful in 1m50s
/ DCO (pull_request) Successful in 3m29s
/ Lint (pull_request) Successful in 3m37s
/ Tests (1.20) (pull_request) Successful in 2m9s
/ Tests (1.21) (pull_request) Successful in 2m5s
/ Builds (1.20) (pull_request) Successful in 5m47s
/ Builds (1.21) (pull_request) Successful in 1m29s
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-25 11:04:19 +03:00
8d6aa0d40a [#243] Fix list object versions marker param
All checks were successful
/ Vulncheck (pull_request) Successful in 2m8s
/ DCO (pull_request) Successful in 2m5s
/ Builds (1.20) (pull_request) Successful in 2m39s
/ Builds (1.21) (pull_request) Successful in 2m15s
/ Lint (pull_request) Successful in 4m23s
/ Tests (1.20) (pull_request) Successful in 2m53s
/ Tests (1.21) (pull_request) Successful in 2m18s
According to https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html
we have to use `key-marker`

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-18 10:35:47 +03:00
7e91f62c28 [#223] Add store content language
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-10-17 14:42:02 +00:00