TrueCloudLab
/
frostfs-s3-gw
16
2
Fork 14
Code Issues 19 Pull Requests 6 Actions Packages Releases 7 Activity
1,195 Commits
4 Branches
25 Tags
18 MiB
Commit Graph

106 Commits (28c6bb4cb85f1de7c3408da13086293ea1dc3ae5)

Author SHA1 Message Date
Denis Kirillov 28c6bb4cb8 [#266] Support per namespace placement policies configuration
/ DCO (pull_request) Successful in 1m34s Details
/ Builds (1.20) (pull_request) Successful in 3m15s Details
/ Builds (1.21) (pull_request) Successful in 2m55s Details
/ Vulncheck (pull_request) Successful in 2m51s Details
/ Lint (pull_request) Successful in 5m12s Details
/ Tests (1.20) (pull_request) Successful in 2m57s Details
/ Tests (1.21) (pull_request) Successful in 2m48s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-28 16:12:42 +03:00
Denis Kirillov 9ebfca654b [#260] Update CHANGELOG.md 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-11-23 11:00:19 +03:00
Roman Loginov 861454e499 [#218] Add check content sha256 header 
The X-Amz-Content-Sha256 header check is done only for unencrypted payload.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-11-22 11:33:52 +00:00
Marina Biryukova b28ecef43b [#219] Return ETag in quotes
/ Vulncheck (pull_request) Successful in 9m5s Details
/ Lint (pull_request) Successful in 11m1s Details
/ Tests (1.20) (pull_request) Successful in 9m59s Details
/ Tests (1.21) (pull_request) Successful in 9m53s Details
/ DCO (pull_request) Successful in 11m34s Details
/ Builds (1.20) (pull_request) Successful in 12m24s Details
/ Builds (1.21) (pull_request) Successful in 8m59s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-11-22 11:12:32 +00:00
Roman Loginov 406075aebb [#236] Add support zapjournald logger configuration
/ DCO (pull_request) Successful in 1m34s Details
/ Vulncheck (pull_request) Successful in 2m6s Details
/ Builds (1.20) (pull_request) Successful in 2m15s Details
/ Builds (1.21) (pull_request) Successful in 2m2s Details
/ Lint (pull_request) Successful in 2m59s Details
/ Tests (1.20) (pull_request) Successful in 2m22s Details
/ Tests (1.21) (pull_request) Successful in 1m43s Details 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-11-13 16:31:11 +03:00
Marina Biryukova fe796ba538 [#217] Consider Copy-Source-SSE-* headers during copy 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-11-13 13:22:58 +00:00
Denis Kirillov 5ee73fad6a [#248] Correct NextVersionIDMarker in listing versions
/ DCO (pull_request) Successful in 4m40s Details
/ Vulncheck (pull_request) Successful in 5m43s Details
/ Lint (pull_request) Successful in 10m27s Details
/ Tests (1.20) (pull_request) Successful in 5m41s Details
/ Tests (1.21) (pull_request) Successful in 5m47s Details
/ Builds (1.20) (pull_request) Successful in 1m9s Details
/ Builds (1.21) (pull_request) Successful in 4m7s Details 
Despite the spec https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html#API_ListObjectVersions_ResponseElements
says that
"When the number of responses exceeds the value of MaxKeys,
NextVersionIdMarker specifies the first object version not returned
 that satisfies the search criteria. Use this value for the
 version-id-marker request parameter in a subsequent request."
 the actual behavior of AWS S3 is returning NextVersionIdMarker as the last returned object version

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 17:36:24 +03:00
Marina Biryukova 890a8ed237 [#227] Add versionID header after complete multipart 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-31 14:07:08 +00:00
Denis Kirillov 0bed25816c [#224] Add conditional escaping for object name 
Chi gives inconsistent results in terms of whether
the strings returned are URL coded or not
See:
* https://github.com/go-chi/chi/issues/641
* https://github.com/go-chi/chi/issues/642

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 13:58:51 +00:00
Denis Kirillov 122af0b5a7 [#220] Support configuring web server timeout params 
Set IdleTimeout and ReadHeaderTimeout to `30s`.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-31 13:48:08 +00:00
Denis Kirillov 0938d7ee82 [#226] Fix status code in GET/HEAD delete marker
/ Vulncheck (pull_request) Successful in 2m26s Details
/ Lint (pull_request) Successful in 4m0s Details
/ Tests (1.20) (pull_request) Successful in 2m59s Details
/ Tests (1.21) (pull_request) Successful in 1m52s Details
/ DCO (pull_request) Successful in 4m17s Details
/ Builds (1.20) (pull_request) Successful in 6m33s Details
/ Builds (1.21) (pull_request) Successful in 1m28s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-27 10:58:57 +03:00
Denis Kirillov 4f5f5fb5c8 [#222] Fix marshaling errors in `DeleteObjects` method 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-25 14:54:02 +00:00
Denis Kirillov 8d6aa0d40a [#243] Fix list object versions marker param
/ Vulncheck (pull_request) Successful in 2m8s Details
/ DCO (pull_request) Successful in 2m5s Details
/ Builds (1.20) (pull_request) Successful in 2m39s Details
/ Builds (1.21) (pull_request) Successful in 2m15s Details
/ Lint (pull_request) Successful in 4m23s Details
/ Tests (1.20) (pull_request) Successful in 2m53s Details
/ Tests (1.21) (pull_request) Successful in 2m18s Details 
According to https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html
we have to use `key-marker`

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-18 10:35:47 +03:00
Marina Biryukova 298662df9d [#221] Expand xmlns field ignore
/ Vulncheck (pull_request) Successful in 1m38s Details
/ Lint (pull_request) Successful in 2m49s Details
/ Tests (1.20) (pull_request) Successful in 1m56s Details
/ Tests (1.21) (pull_request) Successful in 1m44s Details
/ DCO (pull_request) Successful in 3m56s Details
/ Builds (1.20) (pull_request) Successful in 6m25s Details
/ Builds (1.21) (pull_request) Successful in 1m36s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-13 16:21:13 +03:00
Denis Kirillov 10a03faeb4 [#197] Update CHANGELOG.md 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-11 12:32:48 +00:00
Marina Biryukova 066b9a0250 [#142] Add trace ID into log when tracing is enabled
/ Vulncheck (pull_request) Successful in 1m29s Details
/ DCO (pull_request) Successful in 2m18s Details
/ Lint (pull_request) Successful in 3m48s Details
/ Tests (1.20) (pull_request) Successful in 2m4s Details
/ Tests (1.21) (pull_request) Successful in 1m42s Details
/ Builds (1.20) (pull_request) Successful in 7m6s Details
/ Builds (1.21) (pull_request) Successful in 1m56s Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-07 14:19:37 +03:00
Artem Tataurov 54e1c333a1 [#152] authmate: Add basic error types and exit codes
/ DCO (pull_request) Successful in 1m9s Details
/ Vulncheck (pull_request) Successful in 5m17s Details
/ Builds (1.20) (pull_request) Successful in 2m1s Details
/ Builds (1.21) (pull_request) Successful in 7m18s Details
/ Lint (pull_request) Successful in 15m19s Details
/ Tests (1.20) (pull_request) Successful in 1m14s Details
/ Tests (1.21) (pull_request) Successful in 11m45s Details 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-09-06 23:56:56 +03:00
Denis Kirillov b5fce5c8d2 [#168] Skip only invalid policies and copies instead of ignoring all of them
/ DCO (pull_request) Successful in 2m48s Details
/ Vulncheck (pull_request) Successful in 3m30s Details
/ Builds (1.19) (pull_request) Successful in 3m39s Details
/ Builds (1.20) (pull_request) Successful in 3m28s Details
/ Lint (pull_request) Successful in 4m14s Details
/ Tests (1.19) (pull_request) Successful in 3m3s Details
/ Tests (1.20) (pull_request) Successful in 2m45s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-25 12:05:30 +03:00
Denis Kirillov 41a128b1aa [#185] Update CHANGELOG.md
/ Vulncheck (pull_request) Successful in 1m47s Details
/ Lint (pull_request) Successful in 2m46s Details
/ Tests (1.19) (pull_request) Successful in 2m9s Details
/ Tests (1.20) (pull_request) Successful in 2m6s Details
/ DCO (pull_request) Successful in 4m17s Details
/ Builds (1.19) (pull_request) Successful in 6m36s Details
/ Builds (1.20) (pull_request) Successful in 1m43s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-24 18:03:11 +03:00
Denis Kirillov 6b1f365e65 [#192] Support client cut 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-23 06:26:55 +00:00
Denis Kirillov fcf1c45ad2 [#188] Fix url escaping
/ Vulncheck (pull_request) Successful in 2m41s Details
/ Builds (1.19) (pull_request) Successful in 3m51s Details
/ Builds (1.20) (pull_request) Successful in 3m22s Details
/ DCO (pull_request) Successful in 5m9s Details
/ Lint (pull_request) Successful in 5m22s Details
/ Tests (1.19) (pull_request) Successful in 5m39s Details
/ Tests (1.20) (pull_request) Successful in 5m18s Details 
Url escaping has already been done in `net/http/request.go`

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-22 11:27:39 +03:00
Artem Tataurov c750c87a61 [#51] metrics: Add a metric of currently used nodes
/ DCO (pull_request) Successful in 1m18s Details
/ Builds (1.19) (pull_request) Successful in 2m23s Details
/ Builds (1.20) (pull_request) Successful in 2m18s Details
/ Vulncheck (pull_request) Successful in 5m26s Details
/ Lint (pull_request) Failing after 2m4s Details
/ Tests (1.19) (pull_request) Successful in 7m42s Details
/ Tests (1.20) (pull_request) Successful in 2m11s Details 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-08-17 14:26:25 +03:00
Denis Kirillov 6b109eee92 [#182] Fix parsing signed headers in presigned urls
/ Vulncheck (pull_request) Failing after 3s Details
/ Builds (1.19) (pull_request) Successful in 8m22s Details
/ Builds (1.20) (pull_request) Failing after 3s Details
/ Lint (pull_request) Successful in 9m51s Details
/ Tests (1.19) (pull_request) Failing after 3s Details
/ Tests (1.20) (pull_request) Failing after 2s Details
/ DCO (pull_request) Failing after 2s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 13:04:53 +03:00
Denis Kirillov 18878b66d3 [#175] Use gate owner as object owner
/ DCO (pull_request) Failing after 3s Details
/ Vulncheck (pull_request) Failing after 2s Details
/ Builds (1.19) (pull_request) Failing after 3s Details
/ Builds (1.20) (pull_request) Successful in 7m47s Details
/ Lint (pull_request) Failing after 4s Details
/ Tests (1.19) (pull_request) Successful in 8m8s Details
/ Tests (1.20) (pull_request) Failing after 3s Details 
This is required because node check session token owner
TrueCloudLab/frostfs-node#528
For client cut TrueCloudLab/frostfs-sdk-go#114
such owner will be gate owner

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 12:22:33 +03:00
Denis Kirillov fe897ec588 [#178] wrapReader: Fix goroutine leak
/ Vulncheck (pull_request) Successful in 59s Details
/ Builds (1.19) (pull_request) Successful in 1m28s Details
/ Builds (1.20) (pull_request) Successful in 1m19s Details
/ DCO (pull_request) Successful in 45s Details
/ Lint (pull_request) Successful in 1m45s Details
/ Tests (1.19) (pull_request) Successful in 7m51s Details
/ Tests (1.20) (pull_request) Successful in 1m43s Details 
In case of error in FrostFS.CreateObject wrapped reader
can be blocked because of synchronous pipe. We have to read out all payload in such case.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-01 17:59:34 +03:00
Denis Kirillov 52931663e1 [#176] multipart: Replace part on re-upload
/ Builds (1.19) (pull_request) Successful in 2m57s Details
/ Builds (1.20) (pull_request) Successful in 2m36s Details
/ DCO (pull_request) Successful in 1m15s Details
/ Vulncheck (pull_request) Successful in 5m15s Details
/ Lint (pull_request) Successful in 3m50s Details
/ Tests (1.19) (pull_request) Successful in 8m14s Details
/ Tests (1.20) (pull_request) Successful in 3m17s Details 
We want to have exactly one object and tree node for each part number

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-21 16:20:04 +03:00
Denis Kirillov 8ca2998297 [#63] Update CHANGELOG.md 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov b59aa06637 [#146] Add kludge.bypass_content_encoding_check_in_chunks flag
/ Builds (1.19) (pull_request) Successful in 3m0s Details
/ Builds (1.20) (pull_request) Successful in 2m50s Details
/ DCO (pull_request) Failing after 1m9s Details
/ Vulncheck (pull_request) Successful in 1m20s Details
/ Lint (pull_request) Successful in 7m50s Details
/ Tests (1.19) (pull_request) Successful in 3m1s Details
/ Tests (1.20) (pull_request) Successful in 3m21s Details 
Flag allows to skip checking `Content-Encoding` for `aws-chunked` value

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-18 14:49:52 +03:00
Alexey Vanin e58ea40463 Release v0.27.0
/ DCO (pull_request) Failing after 1m14s Details
/ Vulncheck (pull_request) Successful in 1m34s Details
/ Builds (1.19) (pull_request) Successful in 2m59s Details
/ Builds (1.20) (pull_request) Successful in 8m35s Details
/ Lint (pull_request) Successful in 3m22s Details
/ Tests (1.19) (pull_request) Successful in 2m56s Details
/ Tests (1.20) (pull_request) Successful in 2m47s Details 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-07-14 10:53:28 +03:00
Denis Kirillov 14ef9ff091 [#158] Separate init object reader from read itself
/ Vulncheck (pull_request) Successful in 1m40s Details
/ Builds (1.19) (pull_request) Successful in 2m54s Details
/ Builds (1.20) (pull_request) Successful in 2m39s Details
/ DCO (pull_request) Successful in 3m48s Details
/ Lint (pull_request) Successful in 3m32s Details
/ Tests (1.19) (pull_request) Successful in 2m38s Details
/ Tests (1.20) (pull_request) Successful in 2m55s Details 
To be able to handle cases and return appropriate http status code
when object missed in storage but gate cache contains its metadata
we need write code after init object reader.
So we separate init reader from actual reading.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:32:05 +03:00
Denis Kirillov 2cbe3b9a27 [#131] Update docs
/ Builds (1.19) (pull_request) Successful in 3m3s Details
/ Builds (1.20) (pull_request) Successful in 2m44s Details
/ DCO (pull_request) Successful in 3m59s Details
/ Vulncheck (pull_request) Successful in 1m34s Details
/ Lint (pull_request) Successful in 3m41s Details
/ Tests (1.19) (pull_request) Successful in 3m9s Details
/ Tests (1.20) (pull_request) Successful in 2m36s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-27 14:18:16 +03:00
Denis Kirillov 9df8695463 [#143] Fix transformToS3Error function
/ Builds (1.19) (pull_request) Successful in 3m11s Details
/ Builds (1.20) (pull_request) Successful in 2m57s Details
/ DCO (pull_request) Successful in 4m7s Details
/ Lint (pull_request) Successful in 2m25s Details
/ Tests (1.19) (pull_request) Successful in 3m9s Details
/ Tests (1.20) (pull_request) Successful in 3m18s Details
/ Vulncheck (pull_request) Successful in 1m15s Details 
Unwrap error before checking for s3 error

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-21 17:16:40 +03:00
Artem Tataurov 614d703726 [#106] Add chunk uploading
/ DCO (pull_request) Successful in 1m4s Details
/ Vulncheck (pull_request) Successful in 1m50s Details
/ Builds (1.19) (pull_request) Successful in 9m1s Details
/ Builds (1.20) (pull_request) Successful in 2m20s Details
/ Lint (pull_request) Successful in 10m19s Details
/ Tests (1.19) (pull_request) Successful in 2m45s Details
/ Tests (1.20) (pull_request) Successful in 3m19s Details 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-06-21 10:23:57 +03:00
Denis Kirillov 23593eee3d [#111] Use request scope logger
/ Lint (pull_request) Failing after 32s Details
/ Tests (1.19) (pull_request) Failing after 32s Details
/ Tests (1.20) (pull_request) Failing after 32s Details
/ Builds (1.19) (pull_request) Failing after 32s Details
/ Builds (1.20) (pull_request) Failing after 32s Details
/ Vulncheck (pull_request) Failing after 57s Details
/ DCO (pull_request) Failing after 1m0s Details 
Store child zap logger with request scope fields into context.
Request scoped fields: request_id, api/method, bucket, object

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-19 13:54:51 +03:00
Denis Kirillov dfc4476afd [#135] authmate: Update docs
/ Vulncheck (pull_request) Successful in 1m30s Details
/ Builds (1.19) (pull_request) Successful in 2m37s Details
/ Builds (1.20) (pull_request) Successful in 2m31s Details
/ DCO (pull_request) Successful in 3m6s Details
/ Lint (pull_request) Successful in 2m46s Details
/ Tests (1.19) (pull_request) Successful in 5m26s Details
/ Tests (1.20) (pull_request) Successful in 2m36s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-16 09:42:49 +03:00
Denis Kirillov 462589fc0c [#103] Return 504 http code on timeout errors
/ Vulncheck (pull_request) Successful in 1m42s Details
/ Lint (pull_request) Successful in 3m31s Details
/ Tests (1.19) (pull_request) Successful in 2m57s Details
/ Tests (1.20) (pull_request) Successful in 3m4s Details
/ Builds (1.19) (pull_request) Successful in 2m35s Details
/ Builds (1.20) (pull_request) Successful in 2m39s Details
/ DCO (pull_request) Successful in 1m7s Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-14 09:16:21 +03:00
Denis Kirillov 8fcaf76f41 [#132] authmate: Add bearer token to `obtain-secret` result 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-13 15:14:19 +00:00
Denis Kirillov 19c89b38e6 [#133] Update docs
Builds (1.20) Details
Builds (1.19) Details
DCO Details
Tests (1.19) Details
Tests (1.20) Details
Vulncheck Details
Lint Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-09 09:34:49 +03:00
Alexey Vanin e4b1d07185 [#2] Update CHANGELOG 
Replace changelog history before the fork
with the link to the fork source.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-06-07 15:35:57 +00:00
Denis Kirillov bd898ad59e [#125] Update CHANGELOG.md 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-02 14:16:19 +00:00
Artem Tataurov a0f0d792b8 [#78] layer: Clean up already removed object from tree 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-24 17:42:46 +03:00
Denis Kirillov 43e336e155 [#118] go.mod: Update min go version to 1.19 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-05-23 17:32:05 +03:00
Artem Tataurov 9f186d9aba [#104] app: Reload copies numbers on SIGHUP 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-23 13:19:58 +03:00
Artem Tataurov fa5889a0f5 [#94] Update prometheus to v1.15.0 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-18 17:36:32 +03:00
Denis Kirillov b366e75366 [#81] Use impersonate bearer token 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-05-03 17:22:52 +03:00
Artem Tataurov e487ee5b7d [#70] Add arrays of copies numbers for location constraints 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-03 13:48:26 +03:00
Denis Kirillov f200dd310e [#74] Update docs 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-26 16:39:57 +03:00
Artem Tataurov 3da2d40fa8 syncTree: Update file filter to 'frostfs' 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-04-21 17:08:37 +03:00
Denis Kirillov ee6118c3d7 [#80] Update CHANGELOG.md 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-20 11:14:52 +03:00
Denis Kirillov 70ec5a0a5b [#83] Don't create extra delete marker 
We shouldn't create delete marker if:
1. object doesn't exist at all
2. last version is already a delete marker

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-19 17:56:11 +03:00