TrueCloudLab/frostfs-s3-gw
15
2
Fork 16
Code Issues 20 Pull requests 4 Projects Releases 13 Packages Activity Actions
1274 commits 6 branches 32 tags 17 MiB
Commit graph

160 commits

Author SHA1 Message Date
Denis Kirillov
4ad84b9b94 [#165] listing: Use NodeVersion instead of ObjectInfo 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
3e20f736a6 [#165] Move listing function to one file 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
b52552e8c2 [#165] Add batching in streamin listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
6e8960b2ab [#165] Add list session cache 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Denis Kirillov
29ac91dfd5 [#165] Support streaming listing 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2024-02-02 16:09:48 +03:00
Roman Loginov
861454e499 [#218] Add check content sha256 header 
The X-Amz-Content-Sha256 header check is done only for unencrypted payload.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-11-22 11:33:52 +00:00
Denis Kirillov
0938d7ee82 [#226] Fix status code in GET/HEAD delete marker
All checks were successful
/ Vulncheck (pull_request) Successful in 2m26s
Details
/ Lint (pull_request) Successful in 4m0s
Details
/ Tests (1.20) (pull_request) Successful in 2m59s
Details
/ Tests (1.21) (pull_request) Successful in 1m52s
Details
/ DCO (pull_request) Successful in 4m17s
Details
/ Builds (1.20) (pull_request) Successful in 6m33s
Details
/ Builds (1.21) (pull_request) Successful in 1m28s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-27 10:58:57 +03:00
Marina Biryukova
25bb581fee [#205] Add md5 checksum in header
All checks were successful
/ Vulncheck (pull_request) Successful in 1m50s
Details
/ DCO (pull_request) Successful in 3m29s
Details
/ Lint (pull_request) Successful in 3m37s
Details
/ Tests (1.20) (pull_request) Successful in 2m9s
Details
/ Tests (1.21) (pull_request) Successful in 2m5s
Details
/ Builds (1.20) (pull_request) Successful in 5m47s
Details
/ Builds (1.21) (pull_request) Successful in 1m29s
Details 
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
 2023-10-25 11:04:19 +03:00
Denis Kirillov
65412ce1d3 [#197] Configure buffer max size for PUT 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-11 12:32:48 +00:00
Denis Kirillov
7de73f6b73 [#197] Disable homomorphic hash for PUT 
Disable TZ hash for PUT if it's disabled for container itself

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-10-11 12:32:48 +00:00
Roman Loginov
8efcc957ea [#96] Move log messages to constants
All checks were successful
/ DCO (pull_request) Successful in 1m35s
Details
/ Builds (1.19) (pull_request) Successful in 2m14s
Details
/ Builds (1.20) (pull_request) Successful in 2m9s
Details
/ Vulncheck (pull_request) Successful in 5m39s
Details
/ Lint (pull_request) Successful in 2m49s
Details
/ Tests (1.19) (pull_request) Successful in 7m34s
Details
/ Tests (1.20) (pull_request) Successful in 1m44s
Details 
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
 2023-08-23 18:32:31 +03:00
Denis Kirillov
6b1f365e65 [#192] Support client cut 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-23 06:26:55 +00:00
Denis Kirillov
18878b66d3 [#175] Use gate owner as object owner
Some checks failed
/ DCO (pull_request) Failing after 3s
Details
/ Vulncheck (pull_request) Failing after 2s
Details
/ Builds (1.19) (pull_request) Failing after 3s
Details
/ Builds (1.20) (pull_request) Successful in 7m47s
Details
/ Lint (pull_request) Failing after 4s
Details
/ Tests (1.19) (pull_request) Successful in 8m8s
Details
/ Tests (1.20) (pull_request) Failing after 3s
Details 
This is required because node check session token owner
TrueCloudLab/frostfs-node#528
For client cut TrueCloudLab/frostfs-sdk-go#114
such owner will be gate owner

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-08 12:22:33 +03:00
Denis Kirillov
fe897ec588 [#178] wrapReader: Fix goroutine leak
All checks were successful
/ Vulncheck (pull_request) Successful in 59s
Details
/ Builds (1.19) (pull_request) Successful in 1m28s
Details
/ Builds (1.20) (pull_request) Successful in 1m19s
Details
/ DCO (pull_request) Successful in 45s
Details
/ Lint (pull_request) Successful in 1m45s
Details
/ Tests (1.19) (pull_request) Successful in 7m51s
Details
/ Tests (1.20) (pull_request) Successful in 1m43s
Details 
In case of error in FrostFS.CreateObject wrapped reader
can be blocked because of synchronous pipe. We have to read out all payload in such case.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-08-01 17:59:34 +03:00
Denis Kirillov
1a09041cd1 [#63] Simplify multiObjectReader and add tests
All checks were successful
/ DCO (pull_request) Successful in 1m19s
Details
/ Vulncheck (pull_request) Successful in 2m31s
Details
/ Builds (1.19) (pull_request) Successful in 2m51s
Details
/ Builds (1.20) (pull_request) Successful in 2m34s
Details
/ Lint (pull_request) Successful in 5m51s
Details
/ Tests (1.19) (pull_request) Successful in 9m36s
Details
/ Tests (1.20) (pull_request) Successful in 10m20s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov
ad81b599dd [#63] Add fast multipart upload 
Add new flag to object tree meta `isCombined` that means
the object payload is list of parts that forms real payload.
Set this attribute when complete multipart upload not to do unnecessary copying.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-07-20 14:00:15 +03:00
Denis Kirillov
d531b13866 [#143] Add more context to some s3 errors
All checks were successful
/ DCO (pull_request) Successful in 1m18s
Details
/ Vulncheck (pull_request) Successful in 1m50s
Details
/ Lint (pull_request) Successful in 3m1s
Details
/ Tests (1.19) (pull_request) Successful in 2m47s
Details
/ Tests (1.20) (pull_request) Successful in 3m2s
Details
/ Builds (1.19) (pull_request) Successful in 2m54s
Details
/ Builds (1.20) (pull_request) Successful in 7m58s
Details 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-30 12:08:33 +03:00
Denis Kirillov
be03c5178f [#143] Fix NoSuchKey error on get/head 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-30 12:04:52 +03:00
Denis Kirillov
23593eee3d [#111] Use request scope logger
Some checks failed
/ Lint (pull_request) Failing after 32s
Details
/ Tests (1.19) (pull_request) Failing after 32s
Details
/ Tests (1.20) (pull_request) Failing after 32s
Details
/ Builds (1.19) (pull_request) Failing after 32s
Details
/ Builds (1.20) (pull_request) Failing after 32s
Details
/ Vulncheck (pull_request) Failing after 57s
Details
/ DCO (pull_request) Failing after 1m0s
Details 
Store child zap logger with request scope fields into context.
Request scoped fields: request_id, api/method, bucket, object

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-19 13:54:51 +03:00
Denis Kirillov
4a6e3a19ce [#125] Handle negative Content-Length on put 
Add computing actual object size during calculating hash on put.
Use this actual value to save in tree and cache

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2023-06-02 14:16:19 +00:00
Alex Vanin
34bbbcf1ed [#78] Do not show objects missing from object service 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-06-01 16:13:28 +03:00
Artem Tataurov
e487ee5b7d [#70] Add arrays of copies numbers for location constraints 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2023-05-03 13:48:26 +03:00
Alex Vanin
813aa2f173 Rename package name 
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2023-03-07 17:38:08 +03:00
Artem Tataurov
af7fbd6f32 [#754] Add CID and OID to logs 
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
 2022-12-28 17:06:32 +03:00
Denis Kirillov
388482e230 [#2] Rename internals 
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
 2022-12-21 11:17:45 +03:00
Alex Vanin
96dff367db [#1] Build S3 Gateway with FrostFS dependencies 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-12-15 12:43:52 +03:00
Angira Kekteeva
f5fe9a9b4b [#741] Fix data race 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-11-16 18:19:31 +03:00
Denis Kirillov
094eb12578 [#726] Use client time on regular requests 
Use `X-Amz-Date` header as `now` when
* compute expiration epoch
* set Timestamp for object and container
* forming locks
* send notifications

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-14 14:53:55 +03:00
Denis Kirillov
821df3d648 [#669] Optimize getNodeVersion for tags operation 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-20 17:56:09 +03:00
Denis Kirillov
3d31c2ab4a [#713] Fix cache access 
User can access only keys that he puts into caches

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 15:26:42 +03:00
Denis Kirillov
3d08562843 [#505] Handle access denied from tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-13 18:05:16 +03:00
Denis Kirillov
80beedf13e [#504] Use bktInfo in tree service to check owner 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-13 18:05:16 +03:00
Denis Kirillov
9cd4ef1ac4 [#657] Replace FileName with FilePath attribute 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-07 15:50:43 +03:00
Alex Vanin
5ffe28a42f [#669] Optimize PutLockInfo 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-08-31 13:19:00 +03:00
Alex Vanin
9590b39333 [#682] Return added node id in AddVersion 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-08-29 18:24:00 +03:00
Denis Kirillov
5c7a509cc9 [#684] Fix continuation token for directories 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-29 16:10:01 +03:00
Angira Kekteeva
5307211398 [#634] Add CopiesNumber in NeoFS requests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-16 17:49:15 +03:00
Denis Kirillov
6aab3936d9 [#625] Limit listing generator with maxKeys 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-16 13:47:37 +03:00
Denis Kirillov
3824151699 [#616] Reduce number of requests during listing 
Check if object is directory before request to NeoFS

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-16 13:47:37 +03:00
Denis Kirillov
94a6a55919 [#595] Move decrypter to separate package 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov
87c05eb514 [#595] Add SSE with customer key 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Angira Kekteeva
f66e4a77cd [#577] Add IsDeleteMarker() to NodeVersion 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
2a933f48be [#577] Remove redundant set of isLatest 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
245e64900d [#577] Replace ObjInfo with ExtObjInfo in cache 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
37c46e926e [#577] Add Version() to ExtendedObjectInfo 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Denis Kirillov
5dd658302d [#612] Fix comment 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-26 17:36:21 +03:00
Denis Kirillov
b2e8b1cfb3 [#612] Make listing more robust 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-26 17:36:21 +03:00
Angira Kekteeva
4483c6f57a [#619] Filter system metadata 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-26 15:37:13 +03:00
Alex Vanin
e0136feb73 [#611] Split name processing from object info creation procedure 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-25 14:23:30 +03:00
Alex Vanin
97f81d3270 [#611] Process delimiters on the object info cache hit 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-25 14:23:30 +03:00