Commit graph

607 commits

Author SHA1 Message Date
8d6aa0d40a [#243] Fix list object versions marker param
All checks were successful
/ Vulncheck (pull_request) Successful in 2m8s
/ DCO (pull_request) Successful in 2m5s
/ Builds (1.20) (pull_request) Successful in 2m39s
/ Builds (1.21) (pull_request) Successful in 2m15s
/ Lint (pull_request) Successful in 4m23s
/ Tests (1.20) (pull_request) Successful in 2m53s
/ Tests (1.21) (pull_request) Successful in 2m18s
According to https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html
we have to use `key-marker`

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-18 10:35:47 +03:00
7e91f62c28 [#223] Add store content language
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-10-17 14:42:02 +00:00
01323ca8e0 [#216] Add check tag key uniqueness
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-10-17 14:40:29 +00:00
298662df9d [#221] Expand xmlns field ignore
All checks were successful
/ Vulncheck (pull_request) Successful in 1m38s
/ Lint (pull_request) Successful in 2m49s
/ Tests (1.20) (pull_request) Successful in 1m56s
/ Tests (1.21) (pull_request) Successful in 1m44s
/ DCO (pull_request) Successful in 3m56s
/ Builds (1.20) (pull_request) Successful in 6m25s
/ Builds (1.21) (pull_request) Successful in 1m36s
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-13 16:21:13 +03:00
65412ce1d3 [#197] Configure buffer max size for PUT
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-11 12:32:48 +00:00
7de73f6b73 [#197] Disable homomorphic hash for PUT
Disable TZ hash for PUT if it's disabled for container itself

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-11 12:32:48 +00:00
e3f2d59565 [#154] Rename access key to secret key
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-10-06 09:20:39 +00:00
c4af1dc4ad [#171] Update message error auth header malformed
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-10-04 11:13:12 +00:00
b8c93ed391 [#172] Convert handler config to interface
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-04 11:01:27 +00:00
51e591877b [#207] Fix list parts with empty list
All checks were successful
/ DCO (pull_request) Successful in 1m2s
/ Vulncheck (pull_request) Successful in 1m25s
/ Builds (1.20) (pull_request) Successful in 1m59s
/ Builds (1.21) (pull_request) Successful in 1m52s
/ Lint (pull_request) Successful in 8m23s
/ Tests (1.20) (pull_request) Successful in 2m1s
/ Tests (1.21) (pull_request) Successful in 9m26s
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-09-21 11:27:20 +03:00
a4c612614a [#210] Fix multipart object reader
All checks were successful
/ Vulncheck (pull_request) Successful in 2m38s
/ DCO (pull_request) Successful in 3m40s
/ Lint (pull_request) Successful in 5m20s
/ Tests (1.20) (pull_request) Successful in 3m19s
/ Tests (1.21) (pull_request) Successful in 3m8s
/ Builds (1.20) (pull_request) Successful in 6m1s
/ Builds (1.21) (pull_request) Successful in 1m33s
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-09-19 16:30:08 +03:00
12cf29aed2 [#207] Fix part-number-marker handling
All checks were successful
/ DCO (pull_request) Successful in 1m57s
/ Vulncheck (pull_request) Successful in 2m21s
/ Builds (1.20) (pull_request) Successful in 1m36s
/ Builds (1.21) (pull_request) Successful in 7m55s
/ Lint (pull_request) Successful in 13m57s
/ Tests (1.20) (pull_request) Successful in 2m34s
/ Tests (1.21) (pull_request) Successful in 1m57s
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-09-19 12:43:07 +03:00
066b9a0250 [#142] Add trace ID into log when tracing is enabled
All checks were successful
/ Vulncheck (pull_request) Successful in 1m29s
/ DCO (pull_request) Successful in 2m18s
/ Lint (pull_request) Successful in 3m48s
/ Tests (1.20) (pull_request) Successful in 2m4s
/ Tests (1.21) (pull_request) Successful in 1m42s
/ Builds (1.20) (pull_request) Successful in 7m6s
/ Builds (1.21) (pull_request) Successful in 1m56s
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-09-07 14:19:37 +03:00
9120e97ac5 [#203] Add go1.21 to CI
All checks were successful
/ DCO (pull_request) Successful in 59s
/ Vulncheck (pull_request) Successful in 1m35s
/ Builds (1.20) (pull_request) Successful in 1m21s
/ Builds (1.21) (pull_request) Successful in 7m46s
/ Lint (pull_request) Successful in 12m44s
/ Tests (1.20) (pull_request) Successful in 2m9s
/ Tests (1.21) (pull_request) Successful in 2m0s
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-08-31 15:26:07 +03:00
b5fce5c8d2 [#168] Skip only invalid policies and copies instead of ignoring all of them
All checks were successful
/ DCO (pull_request) Successful in 2m48s
/ Vulncheck (pull_request) Successful in 3m30s
/ Builds (1.19) (pull_request) Successful in 3m39s
/ Builds (1.20) (pull_request) Successful in 3m28s
/ Lint (pull_request) Successful in 4m14s
/ Tests (1.19) (pull_request) Successful in 3m3s
/ Tests (1.20) (pull_request) Successful in 2m45s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-25 12:05:30 +03:00
6617adc22b [#185] Use correct object size when object is combined or encrypted
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-24 18:03:08 +03:00
8898c2ec08 [#185] Add tests for list multipart uploads
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-24 18:01:08 +03:00
8efcc957ea [#96] Move log messages to constants
All checks were successful
/ DCO (pull_request) Successful in 1m35s
/ Builds (1.19) (pull_request) Successful in 2m14s
/ Builds (1.20) (pull_request) Successful in 2m9s
/ Vulncheck (pull_request) Successful in 5m39s
/ Lint (pull_request) Successful in 2m49s
/ Tests (1.19) (pull_request) Successful in 7m34s
/ Tests (1.20) (pull_request) Successful in 1m44s
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-08-23 18:32:31 +03:00
6b728fef87 [#192] Add tests to make sure client_cut flag is passed to sdk
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-23 06:26:55 +00:00
6b1f365e65 [#192] Support client cut
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-23 06:26:55 +00:00
fcf1c45ad2 [#188] Fix url escaping
All checks were successful
/ Vulncheck (pull_request) Successful in 2m41s
/ Builds (1.19) (pull_request) Successful in 3m51s
/ Builds (1.20) (pull_request) Successful in 3m22s
/ DCO (pull_request) Successful in 5m9s
/ Lint (pull_request) Successful in 5m22s
/ Tests (1.19) (pull_request) Successful in 5m39s
/ Tests (1.20) (pull_request) Successful in 5m18s
Url escaping has already been done in `net/http/request.go`

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-22 11:27:39 +03:00
94a42fa273 [#51] Update frostfs-sdk-go
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-08-17 14:26:25 +03:00
40d7f844e3 [#137] Refactor context data retrievers
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-08-16 14:05:38 +00:00
6b109eee92 [#182] Fix parsing signed headers in presigned urls
Some checks failed
/ Vulncheck (pull_request) Failing after 3s
/ Builds (1.19) (pull_request) Successful in 8m22s
/ Builds (1.20) (pull_request) Failing after 3s
/ Lint (pull_request) Successful in 9m51s
/ Tests (1.19) (pull_request) Failing after 3s
/ Tests (1.20) (pull_request) Failing after 2s
/ DCO (pull_request) Failing after 2s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-08 13:04:53 +03:00
18878b66d3 [#175] Use gate owner as object owner
Some checks failed
/ DCO (pull_request) Failing after 3s
/ Vulncheck (pull_request) Failing after 2s
/ Builds (1.19) (pull_request) Failing after 3s
/ Builds (1.20) (pull_request) Successful in 7m47s
/ Lint (pull_request) Failing after 4s
/ Tests (1.19) (pull_request) Successful in 8m8s
/ Tests (1.20) (pull_request) Failing after 3s
This is required because node check session token owner
TrueCloudLab/frostfs-node#528
For client cut TrueCloudLab/frostfs-sdk-go#114
such owner will be gate owner

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-08 12:22:33 +03:00
fe897ec588 [#178] wrapReader: Fix goroutine leak
All checks were successful
/ Vulncheck (pull_request) Successful in 59s
/ Builds (1.19) (pull_request) Successful in 1m28s
/ Builds (1.20) (pull_request) Successful in 1m19s
/ DCO (pull_request) Successful in 45s
/ Lint (pull_request) Successful in 1m45s
/ Tests (1.19) (pull_request) Successful in 7m51s
/ Tests (1.20) (pull_request) Successful in 1m43s
In case of error in FrostFS.CreateObject wrapped reader
can be blocked because of synchronous pipe. We have to read out all payload in such case.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-01 17:59:34 +03:00
52931663e1 [#176] multipart: Replace part on re-upload
All checks were successful
/ Builds (1.19) (pull_request) Successful in 2m57s
/ Builds (1.20) (pull_request) Successful in 2m36s
/ DCO (pull_request) Successful in 1m15s
/ Vulncheck (pull_request) Successful in 5m15s
/ Lint (pull_request) Successful in 3m50s
/ Tests (1.19) (pull_request) Successful in 8m14s
/ Tests (1.20) (pull_request) Successful in 3m17s
We want to have exactly one object and tree node for each part number

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-21 16:20:04 +03:00
1a09041cd1 [#63] Simplify multiObjectReader and add tests
All checks were successful
/ DCO (pull_request) Successful in 1m19s
/ Vulncheck (pull_request) Successful in 2m31s
/ Builds (1.19) (pull_request) Successful in 2m51s
/ Builds (1.20) (pull_request) Successful in 2m34s
/ Lint (pull_request) Successful in 5m51s
/ Tests (1.19) (pull_request) Successful in 9m36s
/ Tests (1.20) (pull_request) Successful in 10m20s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 14:00:15 +03:00
631b7f67b4 [#63] multipart: Log upload id for every failed request
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 14:00:15 +03:00
bcf5a85aab [#63] multipart: Fix copying
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 14:00:15 +03:00
ad81b599dd [#63] Add fast multipart upload
Add new flag to object tree meta `isCombined` that means
the object payload is list of parts that forms real payload.
Set this attribute when complete multipart upload not to do unnecessary copying.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 14:00:15 +03:00
361d10cc78 [#174] Fix query for listing multipart uploads
All checks were successful
/ Builds (1.19) (pull_request) Successful in 3m2s
/ Builds (1.20) (pull_request) Successful in 2m50s
/ DCO (pull_request) Successful in 3m59s
/ Vulncheck (pull_request) Successful in 2m39s
/ Lint (pull_request) Successful in 3m56s
/ Tests (1.19) (pull_request) Successful in 3m5s
/ Tests (1.20) (pull_request) Successful in 3m12s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 12:30:17 +03:00
62e6b49254 [#174] Log unmatched requests
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 12:30:17 +03:00
80c4982bd4 [#174] Add router tests
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 12:30:17 +03:00
73ed3f7782 [#174] Fix router filter query matching
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 12:30:17 +03:00
6e3595e35b [#174] Fix object keys with slashes in chi
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 12:30:17 +03:00
b59aa06637 [#146] Add kludge.bypass_content_encoding_check_in_chunks flag
Some checks failed
/ Builds (1.19) (pull_request) Successful in 3m0s
/ Builds (1.20) (pull_request) Successful in 2m50s
/ DCO (pull_request) Failing after 1m9s
/ Vulncheck (pull_request) Successful in 1m20s
/ Lint (pull_request) Successful in 7m50s
/ Tests (1.19) (pull_request) Successful in 3m1s
/ Tests (1.20) (pull_request) Successful in 3m21s
Flag allows to skip checking `Content-Encoding` for `aws-chunked` value

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-18 14:49:52 +03:00
d62aa7b979 [#146] Fix preconditions: trim quotes in etags
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-18 14:49:03 +03:00
751a9be7cc [#146] Move getting chunk payload reader to separate function
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-18 14:49:03 +03:00
14ef9ff091 [#158] Separate init object reader from read itself
All checks were successful
/ Vulncheck (pull_request) Successful in 1m40s
/ Builds (1.19) (pull_request) Successful in 2m54s
/ Builds (1.20) (pull_request) Successful in 2m39s
/ DCO (pull_request) Successful in 3m48s
/ Lint (pull_request) Successful in 3m32s
/ Tests (1.19) (pull_request) Successful in 2m38s
/ Tests (1.20) (pull_request) Successful in 2m55s
To be able to handle cases and return appropriate http status code
when object missed in storage but gate cache contains its metadata
we need write code after init object reader.
So we separate init reader from actual reading.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:32:05 +03:00
fc90981c03 [#149] Update inner imports after moving middlewares
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:25:09 +03:00
83cdfbee78 [#149] Move middlewares to separate package
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:25:09 +03:00
37f2f468fe [#149] Add host bucket router
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:25:09 +03:00
e30a452cdb [#149] Use chi instead of gorilla mux
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:25:09 +03:00
7f708b3a2d [#111] auth: Get log from real request context
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-06 11:57:04 +00:00
d531b13866 [#143] Add more context to some s3 errors
All checks were successful
/ DCO (pull_request) Successful in 1m18s
/ Vulncheck (pull_request) Successful in 1m50s
/ Lint (pull_request) Successful in 3m1s
/ Tests (1.19) (pull_request) Successful in 2m47s
/ Tests (1.20) (pull_request) Successful in 3m2s
/ Builds (1.19) (pull_request) Successful in 2m54s
/ Builds (1.20) (pull_request) Successful in 7m58s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-30 12:08:33 +03:00
f921bc8af5 [#143] Fix typo
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-30 12:04:52 +03:00
be03c5178f [#143] Fix NoSuchKey error on get/head
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-30 12:04:52 +03:00
9df8695463 [#143] Fix transformToS3Error function
All checks were successful
/ Builds (1.19) (pull_request) Successful in 3m11s
/ Builds (1.20) (pull_request) Successful in 2m57s
/ DCO (pull_request) Successful in 4m7s
/ Lint (pull_request) Successful in 2m25s
/ Tests (1.19) (pull_request) Successful in 3m9s
/ Tests (1.20) (pull_request) Successful in 3m18s
/ Vulncheck (pull_request) Successful in 1m15s
Unwrap error before checking for s3 error

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-21 17:16:40 +03:00
614d703726 [#106] Add chunk uploading
All checks were successful
/ DCO (pull_request) Successful in 1m4s
/ Vulncheck (pull_request) Successful in 1m50s
/ Builds (1.19) (pull_request) Successful in 9m1s
/ Builds (1.20) (pull_request) Successful in 2m20s
/ Lint (pull_request) Successful in 10m19s
/ Tests (1.19) (pull_request) Successful in 2m45s
/ Tests (1.20) (pull_request) Successful in 3m19s
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-06-21 10:23:57 +03:00