TrueCloudLab/frostfs-s3-gw
17
3
Fork 19
Code Issues 28 Pull requests 7 Projects Releases 18 Packages Activity Actions
1172 commits 5 branches 43 tags 21 MiB
Commit graph

94 commits

Author SHA1 Message Date
Alex Vanin
962c120125 Release v0.28.0
All checks were successful
/ DCO (pull_request) Successful in 1m50s
Details
/ Vulncheck (pull_request) Successful in 2m40s
Details
/ Builds (1.20) (pull_request) Successful in 2m41s
Details
/ Builds (1.21) (pull_request) Successful in 2m8s
Details
/ Lint (pull_request) Successful in 5m8s
Details
/ Tests (1.20) (pull_request) Successful in 2m18s
Details
/ Tests (1.21) (pull_request) Successful in 2m14s
Details 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-12-07 13:48:06 +03:00
Denis Kirillov
5f9555afad [#237] Fix list object versions marker param
All checks were successful
/ DCO (pull_request) Successful in 1m33s
Details
/ Builds (1.20) (pull_request) Successful in 1m59s
Details
/ Builds (1.21) (pull_request) Successful in 1m33s
Details
/ Lint (pull_request) Successful in 4m20s
Details
/ Tests (1.20) (pull_request) Successful in 21m13s
Details
/ Tests (1.21) (pull_request) Successful in 1m43s
Details
/ Vulncheck (pull_request) Successful in 1m4s
Details 
According to https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html
we have to use `key-marker`

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-12 11:36:39 +03:00
Roman Loginov
670576248a [#231] Fix get latest version node
All checks were successful
/ Vulncheck (pull_request) Successful in 1m33s
Details
/ DCO (pull_request) Successful in 1m38s
Details
/ Builds (1.20) (pull_request) Successful in 2m57s
Details
/ Builds (1.21) (pull_request) Successful in 2m48s
Details
/ Lint (pull_request) Successful in 4m53s
Details
/ Tests (1.20) (pull_request) Successful in 3m19s
Details
/ Tests (1.21) (pull_request) Successful in 3m16s
Details 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-10-06 11:48:22 +03:00
Marina Biryukova
066b9a0250 [#142] Add trace ID into log when tracing is enabled
All checks were successful
/ Vulncheck (pull_request) Successful in 1m29s
Details
/ DCO (pull_request) Successful in 2m18s
Details
/ Lint (pull_request) Successful in 3m48s
Details
/ Tests (1.20) (pull_request) Successful in 2m4s
Details
/ Tests (1.21) (pull_request) Successful in 1m42s
Details
/ Builds (1.20) (pull_request) Successful in 7m6s
Details
/ Builds (1.21) (pull_request) Successful in 1m56s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-09-07 14:19:37 +03:00
Artem Tataurov
54e1c333a1 [#152] authmate: Add basic error types and exit codes
All checks were successful
/ DCO (pull_request) Successful in 1m9s
Details
/ Vulncheck (pull_request) Successful in 5m17s
Details
/ Builds (1.20) (pull_request) Successful in 2m1s
Details
/ Builds (1.21) (pull_request) Successful in 7m18s
Details
/ Lint (pull_request) Successful in 15m19s
Details
/ Tests (1.20) (pull_request) Successful in 1m14s
Details
/ Tests (1.21) (pull_request) Successful in 11m45s
Details 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-09-06 23:56:56 +03:00
Denis Kirillov
b5fce5c8d2 [#168] Skip only invalid policies and copies instead of ignoring all of them
All checks were successful
/ DCO (pull_request) Successful in 2m48s
Details
/ Vulncheck (pull_request) Successful in 3m30s
Details
/ Builds (1.19) (pull_request) Successful in 3m39s
Details
/ Builds (1.20) (pull_request) Successful in 3m28s
Details
/ Lint (pull_request) Successful in 4m14s
Details
/ Tests (1.19) (pull_request) Successful in 3m3s
Details
/ Tests (1.20) (pull_request) Successful in 2m45s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-25 12:05:30 +03:00
Denis Kirillov
41a128b1aa [#185] Update CHANGELOG.md
All checks were successful
/ Vulncheck (pull_request) Successful in 1m47s
Details
/ Lint (pull_request) Successful in 2m46s
Details
/ Tests (1.19) (pull_request) Successful in 2m9s
Details
/ Tests (1.20) (pull_request) Successful in 2m6s
Details
/ DCO (pull_request) Successful in 4m17s
Details
/ Builds (1.19) (pull_request) Successful in 6m36s
Details
/ Builds (1.20) (pull_request) Successful in 1m43s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-24 18:03:11 +03:00
Denis Kirillov
6b1f365e65 [#192] Support client cut 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-23 06:26:55 +00:00
Denis Kirillov
fcf1c45ad2 [#188] Fix url escaping
All checks were successful
/ Vulncheck (pull_request) Successful in 2m41s
Details
/ Builds (1.19) (pull_request) Successful in 3m51s
Details
/ Builds (1.20) (pull_request) Successful in 3m22s
Details
/ DCO (pull_request) Successful in 5m9s
Details
/ Lint (pull_request) Successful in 5m22s
Details
/ Tests (1.19) (pull_request) Successful in 5m39s
Details
/ Tests (1.20) (pull_request) Successful in 5m18s
Details 
Url escaping has already been done in `net/http/request.go`

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-22 11:27:39 +03:00
Artem Tataurov
c750c87a61 [#51] metrics: Add a metric of currently used nodes
Some checks failed
/ DCO (pull_request) Successful in 1m18s
Details
/ Builds (1.19) (pull_request) Successful in 2m23s
Details
/ Builds (1.20) (pull_request) Successful in 2m18s
Details
/ Vulncheck (pull_request) Successful in 5m26s
Details
/ Lint (pull_request) Failing after 2m4s
Details
/ Tests (1.19) (pull_request) Successful in 7m42s
Details
/ Tests (1.20) (pull_request) Successful in 2m11s
Details 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-08-17 14:26:25 +03:00
Denis Kirillov
6b109eee92 [#182] Fix parsing signed headers in presigned urls
Some checks failed
/ Vulncheck (pull_request) Failing after 3s
Details
/ Builds (1.19) (pull_request) Successful in 8m22s
Details
/ Builds (1.20) (pull_request) Failing after 3s
Details
/ Lint (pull_request) Successful in 9m51s
Details
/ Tests (1.19) (pull_request) Failing after 3s
Details
/ Tests (1.20) (pull_request) Failing after 2s
Details
/ DCO (pull_request) Failing after 2s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 13:04:53 +03:00
Denis Kirillov
18878b66d3 [#175] Use gate owner as object owner
Some checks failed
/ DCO (pull_request) Failing after 3s
Details
/ Vulncheck (pull_request) Failing after 2s
Details
/ Builds (1.19) (pull_request) Failing after 3s
Details
/ Builds (1.20) (pull_request) Successful in 7m47s
Details
/ Lint (pull_request) Failing after 4s
Details
/ Tests (1.19) (pull_request) Successful in 8m8s
Details
/ Tests (1.20) (pull_request) Failing after 3s
Details 
This is required because node check session token owner
TrueCloudLab/frostfs-node#528
For client cut TrueCloudLab/frostfs-sdk-go#114
such owner will be gate owner

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 12:22:33 +03:00
Denis Kirillov
fe897ec588 [#178] wrapReader: Fix goroutine leak
All checks were successful
/ Vulncheck (pull_request) Successful in 59s
Details
/ Builds (1.19) (pull_request) Successful in 1m28s
Details
/ Builds (1.20) (pull_request) Successful in 1m19s
Details
/ DCO (pull_request) Successful in 45s
Details
/ Lint (pull_request) Successful in 1m45s
Details
/ Tests (1.19) (pull_request) Successful in 7m51s
Details
/ Tests (1.20) (pull_request) Successful in 1m43s
Details 
In case of error in FrostFS.CreateObject wrapped reader
can be blocked because of synchronous pipe. We have to read out all payload in such case.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-01 17:59:34 +03:00
Denis Kirillov
52931663e1 [#176] multipart: Replace part on re-upload
All checks were successful
/ Builds (1.19) (pull_request) Successful in 2m57s
Details
/ Builds (1.20) (pull_request) Successful in 2m36s
Details
/ DCO (pull_request) Successful in 1m15s
Details
/ Vulncheck (pull_request) Successful in 5m15s
Details
/ Lint (pull_request) Successful in 3m50s
Details
/ Tests (1.19) (pull_request) Successful in 8m14s
Details
/ Tests (1.20) (pull_request) Successful in 3m17s
Details 
We want to have exactly one object and tree node for each part number

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-21 16:20:04 +03:00
Denis Kirillov
8ca2998297 [#63] Update CHANGELOG.md 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov
b59aa06637 [#146] Add kludge.bypass_content_encoding_check_in_chunks flag
Some checks failed
/ Builds (1.19) (pull_request) Successful in 3m0s
Details
/ Builds (1.20) (pull_request) Successful in 2m50s
Details
/ DCO (pull_request) Failing after 1m9s
Details
/ Vulncheck (pull_request) Successful in 1m20s
Details
/ Lint (pull_request) Successful in 7m50s
Details
/ Tests (1.19) (pull_request) Successful in 3m1s
Details
/ Tests (1.20) (pull_request) Successful in 3m21s
Details 
Flag allows to skip checking `Content-Encoding` for `aws-chunked` value

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-18 14:49:52 +03:00
Alex Vanin
e58ea40463 Release v0.27.0
Some checks failed
/ DCO (pull_request) Failing after 1m14s
Details
/ Vulncheck (pull_request) Successful in 1m34s
Details
/ Builds (1.19) (pull_request) Successful in 2m59s
Details
/ Builds (1.20) (pull_request) Successful in 8m35s
Details
/ Lint (pull_request) Successful in 3m22s
Details
/ Tests (1.19) (pull_request) Successful in 2m56s
Details
/ Tests (1.20) (pull_request) Successful in 2m47s
Details 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-07-14 10:53:28 +03:00
Denis Kirillov
14ef9ff091 [#158] Separate init object reader from read itself
All checks were successful
/ Vulncheck (pull_request) Successful in 1m40s
Details
/ Builds (1.19) (pull_request) Successful in 2m54s
Details
/ Builds (1.20) (pull_request) Successful in 2m39s
Details
/ DCO (pull_request) Successful in 3m48s
Details
/ Lint (pull_request) Successful in 3m32s
Details
/ Tests (1.19) (pull_request) Successful in 2m38s
Details
/ Tests (1.20) (pull_request) Successful in 2m55s
Details 
To be able to handle cases and return appropriate http status code
when object missed in storage but gate cache contains its metadata
we need write code after init object reader.
So we separate init reader from actual reading.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-11 17:32:05 +03:00
Denis Kirillov
2cbe3b9a27 [#131] Update docs
All checks were successful
/ Builds (1.19) (pull_request) Successful in 3m3s
Details
/ Builds (1.20) (pull_request) Successful in 2m44s
Details
/ DCO (pull_request) Successful in 3m59s
Details
/ Vulncheck (pull_request) Successful in 1m34s
Details
/ Lint (pull_request) Successful in 3m41s
Details
/ Tests (1.19) (pull_request) Successful in 3m9s
Details
/ Tests (1.20) (pull_request) Successful in 2m36s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-27 14:18:16 +03:00
Denis Kirillov
9df8695463 [#143] Fix transformToS3Error function
All checks were successful
/ Builds (1.19) (pull_request) Successful in 3m11s
Details
/ Builds (1.20) (pull_request) Successful in 2m57s
Details
/ DCO (pull_request) Successful in 4m7s
Details
/ Lint (pull_request) Successful in 2m25s
Details
/ Tests (1.19) (pull_request) Successful in 3m9s
Details
/ Tests (1.20) (pull_request) Successful in 3m18s
Details
/ Vulncheck (pull_request) Successful in 1m15s
Details 
Unwrap error before checking for s3 error

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-21 17:16:40 +03:00
Artem Tataurov
614d703726 [#106] Add chunk uploading
All checks were successful
/ DCO (pull_request) Successful in 1m4s
Details
/ Vulncheck (pull_request) Successful in 1m50s
Details
/ Builds (1.19) (pull_request) Successful in 9m1s
Details
/ Builds (1.20) (pull_request) Successful in 2m20s
Details
/ Lint (pull_request) Successful in 10m19s
Details
/ Tests (1.19) (pull_request) Successful in 2m45s
Details
/ Tests (1.20) (pull_request) Successful in 3m19s
Details 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-06-21 10:23:57 +03:00
Denis Kirillov
23593eee3d [#111] Use request scope logger
Some checks failed
/ Lint (pull_request) Failing after 32s
Details
/ Tests (1.19) (pull_request) Failing after 32s
Details
/ Tests (1.20) (pull_request) Failing after 32s
Details
/ Builds (1.19) (pull_request) Failing after 32s
Details
/ Builds (1.20) (pull_request) Failing after 32s
Details
/ Vulncheck (pull_request) Failing after 57s
Details
/ DCO (pull_request) Failing after 1m0s
Details 
Store child zap logger with request scope fields into context.
Request scoped fields: request_id, api/method, bucket, object

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-19 13:54:51 +03:00
Denis Kirillov
dfc4476afd [#135] authmate: Update docs
All checks were successful
/ Vulncheck (pull_request) Successful in 1m30s
Details
/ Builds (1.19) (pull_request) Successful in 2m37s
Details
/ Builds (1.20) (pull_request) Successful in 2m31s
Details
/ DCO (pull_request) Successful in 3m6s
Details
/ Lint (pull_request) Successful in 2m46s
Details
/ Tests (1.19) (pull_request) Successful in 5m26s
Details
/ Tests (1.20) (pull_request) Successful in 2m36s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-16 09:42:49 +03:00
Denis Kirillov
462589fc0c [#103] Return 504 http code on timeout errors
All checks were successful
/ Vulncheck (pull_request) Successful in 1m42s
Details
/ Lint (pull_request) Successful in 3m31s
Details
/ Tests (1.19) (pull_request) Successful in 2m57s
Details
/ Tests (1.20) (pull_request) Successful in 3m4s
Details
/ Builds (1.19) (pull_request) Successful in 2m35s
Details
/ Builds (1.20) (pull_request) Successful in 2m39s
Details
/ DCO (pull_request) Successful in 1m7s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-14 09:16:21 +03:00
Denis Kirillov
8fcaf76f41 [#132] authmate: Add bearer token to obtain-secret result 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-13 15:14:19 +00:00
Denis Kirillov
19c89b38e6 [#133] Update docs
All checks were successful
Builds (1.20)
Details
Builds (1.19)
Details
DCO
Details
Tests (1.19)
Details
Tests (1.20)
Details
Vulncheck
Details
Lint
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-09 09:34:49 +03:00
Alex Vanin
e4b1d07185 [#2] Update CHANGELOG 
Replace changelog history before the fork
with the link to the fork source.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-06-07 15:35:57 +00:00
Denis Kirillov
bd898ad59e [#125] Update CHANGELOG.md 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-02 14:16:19 +00:00
Artem Tataurov
a0f0d792b8 [#78] layer: Clean up already removed object from tree 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-24 17:42:46 +03:00
Denis Kirillov
43e336e155 [#118] go.mod: Update min go version to 1.19 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-05-23 17:32:05 +03:00
Artem Tataurov
9f186d9aba [#104] app: Reload copies numbers on SIGHUP 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-23 13:19:58 +03:00
Artem Tataurov
fa5889a0f5 [#94] Update prometheus to v1.15.0 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-18 17:36:32 +03:00
Denis Kirillov
b366e75366 [#81] Use impersonate bearer token 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-05-03 17:22:52 +03:00
Artem Tataurov
e487ee5b7d [#70] Add arrays of copies numbers for location constraints 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-03 13:48:26 +03:00
Denis Kirillov
f200dd310e [#74] Update docs 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-26 16:39:57 +03:00
Artem Tataurov
3da2d40fa8 syncTree: Update file filter to 'frostfs' 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-04-21 17:08:37 +03:00
Denis Kirillov
ee6118c3d7 [#80] Update CHANGELOG.md 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-20 11:14:52 +03:00
Denis Kirillov
70ec5a0a5b [#83] Don't create extra delete marker 
We shouldn't create delete marker if:
1. object doesn't exist at all
2. last version is already a delete marker

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-19 17:56:11 +03:00
Denis Kirillov
4a8c382491 [#91] Update values for health metric 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-04-18 10:25:11 +03:00
Artem Tataurov
a5c89b78bc [#79] Return X-Owner-Id in head-bucket response 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-04-17 10:12:25 +03:00
Artem Tataurov
fb99b26209 [#57] api: Clean up List and Name caches when object is missing in Tree service 2023-04-04 17:15:04 +03:00
Denis Kirillov
bd3164c57f [#68] Fix pre-commit issues 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-24 16:22:06 +03:00
Denis Kirillov
a0937126cb [#64] Support new system attributes 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-22 11:46:58 +03:00
Alex Vanin
655889a1a2 [#60] Update docs 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-22 11:22:11 +03:00
Denis Kirillov
5c62010331 [#35] Update SDK to not count error on client aborting 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-10 11:08:21 +03:00
Denis Kirillov
0af06c3bd9 [TrueCloudLab#40] Add param to configure xml decoder 
This parameter enables parsing xml body without
xmlns="http://s3.amazonaws.com/doc/2006-03-01/" attribute
for CompleteMultipartUpload requests

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-09 15:44:13 +03:00
Denis Kirillov
596381c382 [TrueCloudLab#32] Update CHANGELOG.md 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-09 11:19:01 +00:00
Denis Kirillov
32bf915502 [TrueCloudLab#37] Limit number of objects to delete 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-03-09 11:16:22 +00:00
Denis Kirillov
2dcb3c283d [TrueCloudLab#36] Update CHANGELOG.md 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-02-20 19:22:48 +03:00
Alex Vanin
3ab77c8990 [TrueCloudLab#25] Update changelog 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-02-17 13:47:48 +03:00