Commit graph

161 commits

Author SHA1 Message Date
bcfbcdc82f [#345] acl: Update APE and fix using
Some checks failed
/ Lint (pull_request) Successful in 2m53s
/ Tests (1.20) (pull_request) Successful in 2m43s
/ Tests (1.21) (pull_request) Successful in 2m44s
/ DCO (pull_request) Successful in 2m40s
/ Vulncheck (pull_request) Failing after 3m10s
/ Builds (1.20) (pull_request) Successful in 3m18s
/ Builds (1.21) (pull_request) Successful in 3m33s
* Remove native policy when remove bucket policy
* Allow policies that contain only s3 compatible statements
(now deny rules cannot be converted to native rules)

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-04-02 12:42:43 +00:00
4ad84b9b94 [#165] listing: Use NodeVersion instead of ObjectInfo
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
3e20f736a6 [#165] Move listing function to one file
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
b52552e8c2 [#165] Add batching in streamin listing
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
6e8960b2ab [#165] Add list session cache
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
29ac91dfd5 [#165] Support streaming listing
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2024-02-02 16:09:48 +03:00
861454e499 [#218] Add check content sha256 header
The X-Amz-Content-Sha256 header check is done only for unencrypted payload.

Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-11-22 11:33:52 +00:00
0938d7ee82 [#226] Fix status code in GET/HEAD delete marker
All checks were successful
/ Vulncheck (pull_request) Successful in 2m26s
/ Lint (pull_request) Successful in 4m0s
/ Tests (1.20) (pull_request) Successful in 2m59s
/ Tests (1.21) (pull_request) Successful in 1m52s
/ DCO (pull_request) Successful in 4m17s
/ Builds (1.20) (pull_request) Successful in 6m33s
/ Builds (1.21) (pull_request) Successful in 1m28s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-27 10:58:57 +03:00
25bb581fee [#205] Add md5 checksum in header
All checks were successful
/ Vulncheck (pull_request) Successful in 1m50s
/ DCO (pull_request) Successful in 3m29s
/ Lint (pull_request) Successful in 3m37s
/ Tests (1.20) (pull_request) Successful in 2m9s
/ Tests (1.21) (pull_request) Successful in 2m5s
/ Builds (1.20) (pull_request) Successful in 5m47s
/ Builds (1.21) (pull_request) Successful in 1m29s
Signed-off-by: Marina Biryukova <m.biryukova@yadro.com>
2023-10-25 11:04:19 +03:00
65412ce1d3 [#197] Configure buffer max size for PUT
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-11 12:32:48 +00:00
7de73f6b73 [#197] Disable homomorphic hash for PUT
Disable TZ hash for PUT if it's disabled for container itself

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-10-11 12:32:48 +00:00
8efcc957ea [#96] Move log messages to constants
All checks were successful
/ DCO (pull_request) Successful in 1m35s
/ Builds (1.19) (pull_request) Successful in 2m14s
/ Builds (1.20) (pull_request) Successful in 2m9s
/ Vulncheck (pull_request) Successful in 5m39s
/ Lint (pull_request) Successful in 2m49s
/ Tests (1.19) (pull_request) Successful in 7m34s
/ Tests (1.20) (pull_request) Successful in 1m44s
Signed-off-by: Roman Loginov <r.loginov@yadro.com>
2023-08-23 18:32:31 +03:00
6b1f365e65 [#192] Support client cut
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-23 06:26:55 +00:00
18878b66d3 [#175] Use gate owner as object owner
Some checks failed
/ DCO (pull_request) Failing after 3s
/ Vulncheck (pull_request) Failing after 2s
/ Builds (1.19) (pull_request) Failing after 3s
/ Builds (1.20) (pull_request) Successful in 7m47s
/ Lint (pull_request) Failing after 4s
/ Tests (1.19) (pull_request) Successful in 8m8s
/ Tests (1.20) (pull_request) Failing after 3s
This is required because node check session token owner
TrueCloudLab/frostfs-node#528
For client cut TrueCloudLab/frostfs-sdk-go#114
such owner will be gate owner

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-08 12:22:33 +03:00
fe897ec588 [#178] wrapReader: Fix goroutine leak
All checks were successful
/ Vulncheck (pull_request) Successful in 59s
/ Builds (1.19) (pull_request) Successful in 1m28s
/ Builds (1.20) (pull_request) Successful in 1m19s
/ DCO (pull_request) Successful in 45s
/ Lint (pull_request) Successful in 1m45s
/ Tests (1.19) (pull_request) Successful in 7m51s
/ Tests (1.20) (pull_request) Successful in 1m43s
In case of error in FrostFS.CreateObject wrapped reader
can be blocked because of synchronous pipe. We have to read out all payload in such case.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-08-01 17:59:34 +03:00
1a09041cd1 [#63] Simplify multiObjectReader and add tests
All checks were successful
/ DCO (pull_request) Successful in 1m19s
/ Vulncheck (pull_request) Successful in 2m31s
/ Builds (1.19) (pull_request) Successful in 2m51s
/ Builds (1.20) (pull_request) Successful in 2m34s
/ Lint (pull_request) Successful in 5m51s
/ Tests (1.19) (pull_request) Successful in 9m36s
/ Tests (1.20) (pull_request) Successful in 10m20s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 14:00:15 +03:00
ad81b599dd [#63] Add fast multipart upload
Add new flag to object tree meta `isCombined` that means
the object payload is list of parts that forms real payload.
Set this attribute when complete multipart upload not to do unnecessary copying.

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-07-20 14:00:15 +03:00
d531b13866 [#143] Add more context to some s3 errors
All checks were successful
/ DCO (pull_request) Successful in 1m18s
/ Vulncheck (pull_request) Successful in 1m50s
/ Lint (pull_request) Successful in 3m1s
/ Tests (1.19) (pull_request) Successful in 2m47s
/ Tests (1.20) (pull_request) Successful in 3m2s
/ Builds (1.19) (pull_request) Successful in 2m54s
/ Builds (1.20) (pull_request) Successful in 7m58s
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-30 12:08:33 +03:00
be03c5178f [#143] Fix NoSuchKey error on get/head
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-30 12:04:52 +03:00
23593eee3d [#111] Use request scope logger
Some checks failed
/ Lint (pull_request) Failing after 32s
/ Tests (1.19) (pull_request) Failing after 32s
/ Tests (1.20) (pull_request) Failing after 32s
/ Builds (1.19) (pull_request) Failing after 32s
/ Builds (1.20) (pull_request) Failing after 32s
/ Vulncheck (pull_request) Failing after 57s
/ DCO (pull_request) Failing after 1m0s
Store child zap logger with request scope fields into context.
Request scoped fields: request_id, api/method, bucket, object

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-19 13:54:51 +03:00
4a6e3a19ce [#125] Handle negative Content-Length on put
Add computing actual object size during calculating hash on put.
Use this actual value to save in tree and cache

Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2023-06-02 14:16:19 +00:00
34bbbcf1ed [#78] Do not show objects missing from object service
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-06-01 16:13:28 +03:00
e487ee5b7d [#70] Add arrays of copies numbers for location constraints
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2023-05-03 13:48:26 +03:00
813aa2f173 Rename package name
Due to source code relocation from GitHub.

Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2023-03-07 17:38:08 +03:00
af7fbd6f32 [#754] Add CID and OID to logs
Signed-off-by: Artem Tataurov <a.tataurov@yadro.com>
2022-12-28 17:06:32 +03:00
388482e230 [#2] Rename internals
Signed-off-by: Denis Kirillov <d.kirillov@yadro.com>
2022-12-21 11:17:45 +03:00
96dff367db [#1] Build S3 Gateway with FrostFS dependencies
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-12-15 12:43:52 +03:00
Angira Kekteeva
f5fe9a9b4b [#741] Fix data race
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-11-16 18:19:31 +03:00
Denis Kirillov
094eb12578 [#726] Use client time on regular requests
Use `X-Amz-Date` header as `now` when
* compute expiration epoch
* set Timestamp for object and container
* forming locks
* send notifications

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-11-14 14:53:55 +03:00
Denis Kirillov
821df3d648 [#669] Optimize getNodeVersion for tags operation
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-10-20 17:56:09 +03:00
Denis Kirillov
3d31c2ab4a [#713] Fix cache access
User can access only keys that he puts into caches

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-10-11 15:26:42 +03:00
Denis Kirillov
3d08562843 [#505] Handle access denied from tree service
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-13 18:05:16 +03:00
Denis Kirillov
80beedf13e [#504] Use bktInfo in tree service to check owner
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-13 18:05:16 +03:00
Denis Kirillov
9cd4ef1ac4 [#657] Replace FileName with FilePath attribute
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-09-07 15:50:43 +03:00
5ffe28a42f [#669] Optimize PutLockInfo
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-08-31 13:19:00 +03:00
9590b39333 [#682] Return added node id in AddVersion
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
2022-08-29 18:24:00 +03:00
Denis Kirillov
5c7a509cc9 [#684] Fix continuation token for directories
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-29 16:10:01 +03:00
Angira Kekteeva
5307211398 [#634] Add CopiesNumber in NeoFS requests
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-16 17:49:15 +03:00
Denis Kirillov
6aab3936d9 [#625] Limit listing generator with maxKeys
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-16 13:47:37 +03:00
Denis Kirillov
3824151699 [#616] Reduce number of requests during listing
Check if object is directory before request to NeoFS

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-16 13:47:37 +03:00
Denis Kirillov
94a6a55919 [#595] Move decrypter to separate package
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-13 10:26:00 +03:00
Denis Kirillov
87c05eb514 [#595] Add SSE with customer key
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-08-13 10:26:00 +03:00
Angira Kekteeva
f66e4a77cd [#577] Add IsDeleteMarker() to NodeVersion
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-09 17:20:40 +04:00
Angira Kekteeva
2a933f48be [#577] Remove redundant set of isLatest
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-09 17:20:40 +04:00
Angira Kekteeva
245e64900d [#577] Replace ObjInfo with ExtObjInfo in cache
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-09 17:20:40 +04:00
Angira Kekteeva
37c46e926e [#577] Add Version() to ExtendedObjectInfo
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-08-09 17:20:40 +04:00
Denis Kirillov
5dd658302d [#612] Fix comment
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-26 17:36:21 +03:00
Denis Kirillov
b2e8b1cfb3 [#612] Make listing more robust
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
2022-07-26 17:36:21 +03:00
Angira Kekteeva
4483c6f57a [#619] Filter system metadata
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
2022-07-26 15:37:13 +03:00
Alex Vanin
e0136feb73 [#611] Split name processing from object info creation procedure
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
2022-07-25 14:23:30 +03:00