EliChin/frostfs-s3-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-s3-gw
Code Issues Pull requests Projects Releases Packages Wiki Activity
500 commits 6 branches 19 tags 19 MiB
Commit graph

221 commits

Author SHA1 Message Date
Kirillov Denis
b555a1b1d8
Merge pull request #181 from masterSplinter01/feature/179-add-cache-to-lov1 
Add cache to ListObjectsV1
 2021-08-09 14:20:20 +03:00
Angira Kekteeva
3cdcbf6e21 [#179] Add unittests for cache 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-09 13:52:11 +03:00
Angira Kekteeva
8a69c7cca0 [#179] api: Add cache for ListObjectsV1/V2 
Refactored cache for ListObjects:
made cache common for all versions,
simplified: remove dependendence on token/startafter
add mitable cachelifetime.

Refactored listobjects

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-09 13:52:11 +03:00
Denis Kirillov
a89add8ff3 [#198] Fix object metadata 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-06 15:06:40 +03:00
Denis Kirillov
e5b1dae750 [#194] Fix range header handling 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-06 13:14:50 +03:00
Denis Kirillov
45110b4c5d [#184] Fix typo 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-05 12:29:31 +03:00
Denis Kirillov
f4c29cd300 [#184] Unify error handling 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-05 12:18:52 +03:00
Angira Kekteeva
eaec271283 [#180] api: Add unittests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-03 12:40:28 +03:00
Angira Kekteeva
88aaad361a [#180] api: Add continuation-token check 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-03 12:40:02 +03:00
Angira Kekteeva
7291b81040 [#180] api: Format object_list.go 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-29 17:31:48 +03:00
Angira Kekteeva
afff616675 [#180] api: Remove extra parses of url requests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-29 17:18:51 +03:00
Angira Kekteeva
0ceea95e11 [#112] Add cache to ListObjects and layer 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-28 17:05:11 +03:00
Angira Kekteeva
c24fe5cc21 [#112] Move getBoxData from handler to layer 
And made it exported

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-28 16:27:06 +03:00
Angira Kekteeva
97a7d16f68 [#112] api: Refactor allObjectsParams struct 
Replaced Bucket (means name) by BucketInfo struct

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-26 23:22:05 +03:00
Denis Kirillov
2479650739 [#118] Correct NotFound status 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-26 14:03:26 +03:00
Alex Vanin
f4cd1e4c38
Merge pull request #170 from KirillovDenis/feature/89-placement_policy 
[#89] Add placement policy
 2021-07-26 13:43:38 +03:00
Denis Kirillov
8aac61652f [#89] Move BoxData to CreateBucketParams 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-26 13:35:52 +03:00
Alex Vanin
3f7941fdf3
Merge pull request #174 from nspcc-dev/feature/159-optional_owner 
[#159] Add fetch-owner param
 2021-07-26 13:32:04 +03:00
Denis Kirillov
c7cf5afd2f [#149] Add benchmarks 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-22 09:55:35 +03:00
Denis Kirillov
60bc0037fd [#149] Remove unused 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-22 09:55:35 +03:00
Denis Kirillov
28974474f2 [#149] Refactoring 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-22 09:55:35 +03:00
Denis Kirillov
63652d213c [#149] Removed grpc errors 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-22 09:54:12 +03:00
Denis Kirillov
b695e6a3b4 [#89] Refactor error logging 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-22 09:45:24 +03:00
Denis Kirillov
65a61a9f7d [#159] Add fetch-owner param 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-20 15:40:38 +03:00
Denis Kirillov
c05e2d6757 [#160] Fix head dir object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-20 12:36:41 +03:00
Angira Kekteeva
b1cda2a714 [#154] api: refactor EncodeV1 and EncodeV2 
Move common parts of Encode to separate functions.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-19 16:38:51 +03:00
Angira Kekteeva
ee84062154 [#154] api: refactor ListObjectsV1 and V2 
Separate ListObject for different versions.
Remove useless grouping of keys on V2.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-19 10:12:47 +03:00
Denis Kirillov
1fbd192bd7 [#89] Add placement policy 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-16 16:01:35 +03:00
Angira Kekteeva
b7ebe4127f [#164] Change delimiter in acesskeyID 
Replaced delimiter '_' by '0'

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-16 14:17:01 +03:00
Alex Vanin
22faaadc32 Merge pull request #161 from nspcc-dev/feature/158-s3_tests_conditional_headers 
[#158] Handled s3 errors on conditional headers
 2021-07-16 09:39:17 +03:00
Denis Kirillov
352d5345fc [#158] Handled s3 errors on conditional headers 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-16 09:30:08 +03:00
Alex Vanin
3a7876e292 Merge pull request #157 from KirillovDenis/feature/155-s3_ceph_listObject_compatibility 
[#155] Improved s3 listObjects compatibility
 2021-07-15 18:48:26 +03:00
Denis Kirillov
f2eeed0b85 [#155] Fix error handling 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-14 19:06:58 +03:00
Denis Kirillov
1ecf32c302 [#155] Fixed invalid max-keys handling 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-14 12:21:09 +03:00
Denis Kirillov
9f57756351 [#155] Added s3 url encoder 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-14 11:34:49 +03:00
Denis Kirillov
288f6edce8 [#155] Fixed default maxKey value 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-13 17:57:48 +03:00
Denis Kirillov
5fb4c4fad6 [#155] Fixed empty delimiter response 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-13 17:46:32 +03:00
Denis Kirillov
b004996d51 [#155] Fix KeyCount 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-13 17:45:38 +03:00
Denis Kirillov
e1c2cee365 [#152] Removed redundant GetBucketInfo call 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-13 14:22:12 +03:00
Alex Vanin
7b1058a9bd Merge pull request #143 from KirillovDenis/bugfix/125-check_bucket_unique 
[#125] Fixed bucket creation
 2021-07-09 17:44:28 +03:00
Alex Vanin
1f656aac23 Merge pull request #148 from masterSplinter01/bugfix/131-add-metadata 
Fix metadata
 2021-07-09 17:20:59 +03:00
Denis Kirillov
acc2cdd96c [#125] Added exact eacl checking 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-09 17:06:35 +03:00
Angira Kekteeva
d5dfc0006a [#131] Remove metadata from Object in ListObjects 
Because ListObjects v1 and v2 shouldn't show metadata. Also aws cli
doesn't show metadata even we return it.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-09 13:44:24 +03:00
Angira Kekteeva
da9a6e4b33 [#131] Fix metadata in HeadObject and GetObject 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-09 13:44:24 +03:00
Angira Kekteeva
e8bf46a2fa [#131] Add meta to PutObject 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-09 13:44:24 +03:00
Denis Kirillov
e78543adf3 [#125] Updated error handling 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-09 11:57:44 +03:00
Angira Kekteeva
d09a4e4e6c [#145] Fix util tests 
Fixed contentType attribute difference in expected and actual results

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-09 00:36:19 +03:00
Denis Kirillov
e11b1b76ba [#125] Added eacl to newly created container 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-08 13:10:46 +03:00
Denis Kirillov
6ba5167f5d [#125] Fixed acl rule 
Container must be public (basic acl) to enable bearer token.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-08 12:22:02 +03:00
Denis Kirillov
0aae8c595a [#125] Fixed bucket creation 
Bucket should has unique name.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-08 12:22:02 +03:00
Alex Vanin
f873be8019 Merge pull request #136 from masterSplinter01/bugfix/135-max-keys-check 
[#135] Add max-keys check to ListObjects
 2021-07-06 15:03:43 +03:00
Alex Vanin
2956f34fb7 Merge pull request #139 from KirillovDenis/bugfix/133-wrong_time_format 
[#133] Fix response time format
 2021-07-06 15:03:15 +03:00
Denis Kirillov
bdc7f9acba [#133] Fix response time format 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-06 14:07:09 +03:00
Angira Kekteeva
fd41c325f7 [#135] Add max-keys check to ListObjects 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-06 12:23:46 +03:00
Angira Kekteeva
aac1ac2868 [#119] Add null versioning and handler 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-06 12:11:03 +03:00
Angira Kekteeva
cc0e2e7636 Merge pull request #132 from KirillovDenis/bugfix/129-fix_listing 
[#129] Fix root objects listing
 2021-07-05 12:20:26 +03:00
Denis Kirillov
e145effd17 [#129] Fix root objects listing 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-04 23:25:53 +03:00
Denis Kirillov
95476b1c9c [#95] Supported copy match headers 
Supported x-amz-copy-source-if-match
and x-amz-copy-source-if-none-match.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-02 19:21:53 +03:00
Denis Kirillov
568d7ac810 [#95] Supported get match headers 
Supported If-Match and If-None-Match.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-02 19:05:43 +03:00
Roman Khimov
daed0978a6 Merge pull request #126 from KirillovDenis/feature/94-time_based_conditional 
[#94] CopyObject/GetObject support conditional headers
 2021-07-02 17:19:16 +03:00
Denis Kirillov
42384730a0 [#121] Fixed directory listing 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-01 14:41:20 +03:00
Denis Kirillov
116ffbb438 [#94] Fix time format 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-01 14:25:16 +03:00
Denis Kirillov
c4631e5806 [#94] Refactoring 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-01 14:24:00 +03:00
Denis Kirillov
ab8dd4201c [#94] CopyObject support conditional headers 
Supported X-Amz-Copy-Source-If-Modified-Since and
X-Amz-Copy-Source-If-Unmodified-Since.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-01 10:45:55 +03:00
Denis Kirillov
da4eca5da5 [#94] GetObject support conditional headers 
Supported If-Modified-Since and If-Unmodified-Since.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-01 08:39:19 +03:00
Denis Kirillov
2af1b16b59 [#93] Removed unnecessary 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-30 17:25:46 +03:00
Denis Kirillov
a6ec27b40d [#93] Object ETag support 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-30 12:55:31 +03:00
Denis Kirillov
b565045649 [#93] Fixed order headers set 
Writing headers had no effect early.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-30 12:55:31 +03:00
Denis Kirillov
65be38831c [#98] Supported listObjects delimiter 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-30 10:50:47 +03:00
Denis Kirillov
23dfa0c46e [#97] List object paging 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-28 17:05:14 +03:00
Roman Khimov
ce65a47d1b Merge pull request #109 from KirillovDenis/feature/104-drop_neofs_crypto 
[#104] Support NEP-6 for authmate
 2021-06-26 16:40:44 +03:00
Denis Kirillov
c4fef3d948 [#96] Support ranges for GetObject 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-25 12:18:52 +03:00
Denis Kirillov
52c63d4c44 [#104] Support NEP-6 for authmate 
Drop neofs-crypto.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-25 12:16:24 +03:00
Angira Kekteeva
26d6fd95c7 [#47] authmate, layer: use sdk-go Container impl 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-06-23 23:36:39 +03:00
Angira Kekteeva
3aa9aceda5 [#47] handler, layer: Add DeleteBucket 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-06-23 23:25:00 +03:00
Angira Kekteeva
c12abf6243 [#47] handler, layer: Add CreateBucket 
Renamed PutBucket to CreateBucket

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-06-23 23:21:15 +03:00
Angira Kekteeva
95f75ec880 [#47] Add session token to context 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-06-21 13:56:26 +03:00
Denis Kirillov
d1594b586e [#83] Use multiple bearer tokens 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-18 11:19:07 +03:00
Denis Kirillov
b86d20265d [#75] Using secp256r1 instead of curve25519 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-17 16:36:23 +03:00
Angira Kekteeva
0814bf3d01 Merge pull request #87 from roman-khimov/lint 
Linter fixes
 2021-06-16 23:01:29 +03:00
Roman Khimov
996b2d8667 auth: add missing comment 
Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-06-15 18:50:10 +03:00
Denis Kirillov
6a135a6bb4 [#82] Using bearer token 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-15 17:16:26 +03:00
Denis Kirillov
fba29a9a66 [#78] Using client.Object from sdk 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-15 17:06:19 +03:00
Roman Khimov
ee078eabcc Merge pull request #80 from masterSplinter01/feature/48-protobuf-format-authbox 
Protobuf format of accessbox
 2021-06-15 16:57:22 +03:00
Angira Kekteeva
fe2d507121 [#48] creds,authmate:Replace old accessbox by new 
Removed encoder, decoder wraps.
Made changes in api, authmate and creds via new accessbox.
Updated bearer_token_tests via new accessbox.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-06-15 14:54:11 +03:00
Denis Kirillov
47fc10a869 [#65] Using containerList falls back 
Looking container up using the owner list if its name is not cid

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-13 14:32:13 +03:00
Denis Kirillov
4df647baac [#65] Added NoAuthorizationHeader error 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-11 19:29:55 +03:00
Denis Kirillov
8185b71462 [#65] Allow no sign requests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-11 14:52:03 +03:00
Angira Kekteeva
b3b190adc0 [#68] *: Replace deprecated 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-06-04 16:01:42 +03:00
Angira Kekteeva
1a818bac33 [#64] authmate, auth: Fix access key id 
Replaced forbidden symbol '/' in access key id by '_'
SecretKeyAddress and SecretKeyID are different things now
Renamed param of authmate from secretAddressFlag to accessKeyIDFlag,
that is more correct, imo.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-06-03 22:24:02 +03:00
Roman Khimov
d15687f37c *: drop creds/neofs 
Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-28 23:48:39 +03:00
Roman Khimov
35ffc1fbc1 *: use sdk-go instead of http-gw 
Now that the packages have moved there we can do it.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-28 14:05:46 +03:00
Roman Khimov
1c277bd97b api: fix golint suggestion for proper context usage 
api/user-auth.go:27:5  golint  should not use basic type string as key in context.WithValue

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-26 21:25:37 +03:00
Roman Khimov
184c26551a *: fix comments for golint 
Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-26 21:23:48 +03:00
Roman Khimov
d19ce03072 *: drop old sdk dependecies, bump neofs-api-go version 
I'm not sure it works, but it's enough code-wise for now. We're reusing some
http-gw components here that are to be moved into sdk-go in future.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-26 21:01:46 +03:00
Roman Khimov
dbe65ae602 creds: move credential management into s3 gate 
Mostly taken from old SDK (abe47687cd11266f946cad57f07572cc10c67226), but
error handling adapted to eliminate pkg/errors and internal packages.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-25 23:00:19 +03:00
Roman Khimov
ce7c8932d4 authmate: drop creds/s3 dependency 
Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-25 19:52:51 +03:00
Roman Khimov
92c8cce933 misc: move to internal/version, use for all binaries 
No libmisc, please.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-20 16:26:41 +03:00
Roman Khimov
ef8684c11d s3-gw: drop build time setting 
It's not properly set anyway and it's a bad practice for build
reproducibility.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-20 16:26:20 +03:00
Angira Kekteeva
9d496d70a7 [#53] *: Remove external pkg/errors dependency 
Replaced functions from pkg/errors by functions from errors, fixed not fully correct comment

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-05-20 15:50:14 +03:00