bcf5a85aab
[ #63 ] multipart: Fix copying
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 14:00:15 +03:00
ad81b599dd
[ #63 ] Add fast multipart upload
...
Add new flag to object tree meta `isCombined` that means
the object payload is list of parts that forms real payload.
Set this attribute when complete multipart upload not to do unnecessary copying.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 14:00:15 +03:00
361d10cc78
[ #174 ] Fix query for listing multipart uploads
...
/ Builds (1.19) (pull_request) Successful in 3m2s
/ Builds (1.20) (pull_request) Successful in 2m50s
/ DCO (pull_request) Successful in 3m59s
/ Vulncheck (pull_request) Successful in 2m39s
/ Lint (pull_request) Successful in 3m56s
/ Tests (1.19) (pull_request) Successful in 3m5s
/ Tests (1.20) (pull_request) Successful in 3m12s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 12:30:17 +03:00
62e6b49254
[ #174 ] Log unmatched requests
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 12:30:17 +03:00
80c4982bd4
[ #174 ] Add router tests
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 12:30:17 +03:00
73ed3f7782
[ #174 ] Fix router filter query matching
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 12:30:17 +03:00
6e3595e35b
[ #174 ] Fix object keys with slashes in chi
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 12:30:17 +03:00
b59aa06637
[ #146 ] Add kludge.bypass_content_encoding_check_in_chunks flag
...
/ Builds (1.19) (pull_request) Successful in 3m0s
/ Builds (1.20) (pull_request) Successful in 2m50s
/ DCO (pull_request) Failing after 1m9s
/ Vulncheck (pull_request) Successful in 1m20s
/ Lint (pull_request) Successful in 7m50s
/ Tests (1.19) (pull_request) Successful in 3m1s
/ Tests (1.20) (pull_request) Successful in 3m21s
Flag allows to skip checking `Content-Encoding` for `aws-chunked` value
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-18 14:49:52 +03:00
d62aa7b979
[ #146 ] Fix preconditions: trim quotes in etags
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-18 14:49:03 +03:00
751a9be7cc
[ #146 ] Move getting chunk payload reader to separate function
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-18 14:49:03 +03:00
14ef9ff091
[ #158 ] Separate init object reader from read itself
...
/ Vulncheck (pull_request) Successful in 1m40s
/ Builds (1.19) (pull_request) Successful in 2m54s
/ Builds (1.20) (pull_request) Successful in 2m39s
/ DCO (pull_request) Successful in 3m48s
/ Lint (pull_request) Successful in 3m32s
/ Tests (1.19) (pull_request) Successful in 2m38s
/ Tests (1.20) (pull_request) Successful in 2m55s
To be able to handle cases and return appropriate http status code
when object missed in storage but gate cache contains its metadata
we need write code after init object reader.
So we separate init reader from actual reading.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:32:05 +03:00
fc90981c03
[ #149 ] Update inner imports after moving middlewares
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:25:09 +03:00
83cdfbee78
[ #149 ] Move middlewares to separate package
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:25:09 +03:00
37f2f468fe
[ #149 ] Add host bucket router
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:25:09 +03:00
e30a452cdb
[ #149 ] Use chi instead of gorilla mux
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-11 17:25:09 +03:00
7f708b3a2d
[ #111 ] auth: Get log from real request context
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-06 11:57:04 +00:00
d531b13866
[ #143 ] Add more context to some s3 errors
...
/ DCO (pull_request) Successful in 1m18s
/ Vulncheck (pull_request) Successful in 1m50s
/ Lint (pull_request) Successful in 3m1s
/ Tests (1.19) (pull_request) Successful in 2m47s
/ Tests (1.20) (pull_request) Successful in 3m2s
/ Builds (1.19) (pull_request) Successful in 2m54s
/ Builds (1.20) (pull_request) Successful in 7m58s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-30 12:08:33 +03:00
f921bc8af5
[ #143 ] Fix typo
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-30 12:04:52 +03:00
be03c5178f
[ #143 ] Fix NoSuchKey error on get/head
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-30 12:04:52 +03:00
9df8695463
[ #143 ] Fix transformToS3Error function
...
/ Builds (1.19) (pull_request) Successful in 3m11s
/ Builds (1.20) (pull_request) Successful in 2m57s
/ DCO (pull_request) Successful in 4m7s
/ Lint (pull_request) Successful in 2m25s
/ Tests (1.19) (pull_request) Successful in 3m9s
/ Tests (1.20) (pull_request) Successful in 3m18s
/ Vulncheck (pull_request) Successful in 1m15s
Unwrap error before checking for s3 error
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-21 17:16:40 +03:00
614d703726
[ #106 ] Add chunk uploading
...
/ DCO (pull_request) Successful in 1m4s
/ Vulncheck (pull_request) Successful in 1m50s
/ Builds (1.19) (pull_request) Successful in 9m1s
/ Builds (1.20) (pull_request) Successful in 2m20s
/ Lint (pull_request) Successful in 10m19s
/ Tests (1.19) (pull_request) Successful in 2m45s
/ Tests (1.20) (pull_request) Successful in 3m19s
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-06-21 10:23:57 +03:00
23593eee3d
[ #111 ] Use request scope logger
...
/ Lint (pull_request) Failing after 32s
/ Tests (1.19) (pull_request) Failing after 32s
/ Tests (1.20) (pull_request) Failing after 32s
/ Builds (1.19) (pull_request) Failing after 32s
/ Builds (1.20) (pull_request) Failing after 32s
/ Vulncheck (pull_request) Failing after 57s
/ DCO (pull_request) Failing after 1m0s
Store child zap logger with request scope fields into context.
Request scoped fields: request_id, api/method, bucket, object
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-19 13:54:51 +03:00
84358f6742
[ #135 ] authmate: Support CRDT GSet for credentials
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-16 09:42:49 +03:00
7a380fa46c
[ #135 ] frostfs: Add SEARCH operation
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-16 09:42:41 +03:00
462589fc0c
[ #103 ] Return 504 http code on timeout errors
...
/ Vulncheck (pull_request) Successful in 1m42s
/ Lint (pull_request) Successful in 3m31s
/ Tests (1.19) (pull_request) Successful in 2m57s
/ Tests (1.20) (pull_request) Successful in 3m4s
/ Builds (1.19) (pull_request) Successful in 2m35s
/ Builds (1.20) (pull_request) Successful in 2m39s
/ DCO (pull_request) Successful in 1m7s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-14 09:16:21 +03:00
9dabaf6ecd
[ #133 ] Use tree pool from SDK
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-09 09:34:04 +03:00
4e1fd9589b
[ #84 ] add tracing support
...
Signed-off-by: Pavel Pogodaev <p.pogodaev@yadro.com>
2023-06-06 10:25:27 +03:00
4a6e3a19ce
[ #125 ] Handle negative Content-Length on put
...
Add computing actual object size during calculating hash on put.
Use this actual value to save in tree and cache
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-02 14:16:19 +00:00
b445f7bbf9
[ #125 ] api/auth: DisableURIPathEscaping for presign
...
Don't use escaping when presign url.
Escape manually before.
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-02 14:16:19 +00:00
868edfdb31
[ #78 ] Add test of bucket removal with object not found error
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-06-01 16:13:28 +03:00
34bbbcf1ed
[ #78 ] Do not show objects missing from object service
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-06-01 16:13:28 +03:00
fae03c2b50
[ #78 ] Process 'not found' error when object exists in tree
...
When object exists in tree but missing in storage, we can't remove
bucket. While storage node does not sync tree service and object
service, the only way to delete such broken bucket is to ignore
'object not found' error, clear cache and do not include missing
objects in the listing result.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-06-01 15:31:22 +03:00
81e860481d
[ #122 ] Fix linter warnings
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-05-31 19:43:03 +03:00
a0f0d792b8
[ #78 ] layer: Clean up already removed object from tree
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-05-24 17:42:46 +03:00
9f186d9aba
[ #104 ] app: Reload copies numbers on SIGHUP
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-05-23 13:19:58 +03:00
3fbad97ae4
[ #1 ] Rename files with mentions of previous project
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-05-19 07:30:04 +00:00
e24bc3f2ce
[ #101 ] app: Refactor the default copies number setting
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-05-17 11:36:28 +03:00
Denis Kirillov
b366e75366
[ #81 ] Use impersonate bearer token
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-05-03 17:22:52 +03:00
e487ee5b7d
[ #70 ] Add arrays of copies numbers for location constraints
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-05-03 13:48:26 +03:00
ee8cce662b
[ #74 ] service/tree: Add logger
...
Log error instead of failing when multiple unversioned nodes are found
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-26 16:39:57 +03:00
9e72fe1662
[ #80 ] Refactor metrics, support dump descriptions
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-20 11:14:52 +03:00
70ec5a0a5b
[ #83 ] Don't create extra delete marker
...
We shouldn't create delete marker if:
1. object doesn't exist at all
2. last version is already a delete marker
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-04-19 17:56:11 +03:00
a5c89b78bc
[ #79 ] Return X-Owner-Id in head-bucket response
...
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-04-17 10:12:25 +03:00
fb99b26209
[ #57 ] api: Clean up List and Name caches when object is missing in Tree service
2023-04-04 17:15:04 +03:00
a025f2e9c5
[ #59 ] tree: Make interface for tree service client
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-29 09:34:10 +03:00
bd3164c57f
[ #68 ] Fix pre-commit issues
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-24 16:22:06 +03:00
a0937126cb
[ #64 ] Support new system attributes
...
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-03-22 11:46:58 +03:00
ef556bd8ac
[ #60 ] Use session token to set eACL during Complete Multipart Upload
...
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-22 11:22:11 +03:00
5104683f68
[ #60 ] Refactor start of periodic XML writer
...
Reduce code duplication for error handling
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-22 11:02:16 +03:00
8151753eeb
[ #60 ] Use periodic white space XML writer in Complete Multipart Upload
...
This mechanism is used by Amazon S3 to keep client's
connection alive while object is being constructed from
the upload parts.
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-20 13:34:22 +03:00
