EliChin/frostfs-s3-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-s3-gw
Code Issues Pull requests Projects Releases Packages Wiki Activity
946 commits 6 branches 19 tags 19 MiB
Commit graph

301 commits

Author SHA1 Message Date
Angira Kekteeva
72013e08ec [#569] Refactor versioning in tree service 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9f019c7470 [#542] Add test to check delete marker 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
b380d4ab53 [#542] Clear list cache on object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
f1a0288e3a [#542] Handle unversioned mode 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
fdf0974679 [#542] Fix object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
88c392d024 [#490] Optimize GetObjectTaggingAndLock 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
c88a9842db [#488] Sync using oid.ID 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
85c203e157 [#488] Sync using cid.ID 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
42a761c1f2 [#547] Cache ObjectInfo instead of Object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
5984f2c2bc [#525] Add comment to clarify anonymous func 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
ad95d1745c [#525] Parallelize listing 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
6ad7c988e6 [#532] Fix object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
12a2060dd0 [#524] Optimize listing 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
3f5afabd4a [#515] Fix nil reference when a part is missing 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
93a7c2eaeb [#514] Missed version id in tags 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
59570ac075 [#513] Don't remove delete mark object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Alex Vanin
4f3d206422 [#478] Don't fetch epoch at object upload 
Creation epoch was used for versioning. With
tree service, versioning is done on the NeoFS
side in the tree.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9dfc7e043f [#476] Save full parts info to multipart object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Alex Vanin
7fd27e04fe [#477] Provide aggregation of tags and locks in layer.Client 
Adopts GetObjectTaggingAndLock method from
TreeClient in layer.Client with caching.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Alex Vanin
f8496973b8 [#477] Aggregate fetching tags and lock in layer.TreeService 
New method aggregates GetObjectTagging and GetLock
methods in single RPC to the tree service.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
a02900a4f7 [#474] Use appropriate null version during listing 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
c8e8ba9f6a [#476] Fix parts info for GetObjectAttributes 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9b1ccd39be [#475] Add extra attributes to partInfo 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
5b9a5cadef [#452] Fix lint issues 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
1a456eaa8b [#452] Remove unused 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9fb3fb1274 [#452] Fix versions tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
dd534e8738 [#451] Handle lock objects using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
70957d75fd [#417] Complete multipart upload using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
6b2ddcadd0 [#417] Abort multipart upload using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
58f2bf44b1 [#417] List parts using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
bc0bdc7767 [#417] Upload part using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
e1b9a4432a [#417] List multipart uploads using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
13e01164d7 [#417] Create multipart upload using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
b6a43c6f4c [#449] Replace error checks and literals 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
385a2102f0 [#449] Add tree service for bucket tagging 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
99feb1d936 [#449] Add tree service for object tagging 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9c74cca9af [#420] Using tree service to list object versions 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
25477cdaf8 [#416] Use tree service to list objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
7e8b57605a [#413] Fix delete object without version 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
37b1baed41 [#413] Refactor 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
977f176713 [#413] Use tree service to delete object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
49bd77d9cf [#413] Use tree service to head objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
36f3c43af5 [#413] Use tree service to put object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
ab5c44ac14 [#429] Refactor TreeService for notif,cors,settings 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
8eff857e41 [#429] Add tree service for cors 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
7520952792 [#429] Add tree service for notifications 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
b03ae827fb [#429] Refactor tree service 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9ed081db3e [#407] Add service tree client prototype 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Alex Vanin
aad4862c59 [#564] Don't parse lock attribute if it is omitted in container 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-15 01:33:48 +04:00
Leonard Lyubich
5bfc549746 [#564] neofs: Merge if with same condition in CreateContainer 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-07-04 23:42:50 +04:00
Leonard Lyubich
4a8a248f34 [#564] Upgrade NeoFS SDK Go with changed container API 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-07-04 23:42:50 +04:00
Denis Kirillov
9f740b9683 [#289] Add detect mimetype by extension 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-29 19:13:16 +03:00
Denis Kirillov
7ca519cb32 [#539] Add context to errors 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-27 02:23:19 +04:00
Leonard Lyubich
e1f1e6d960 [#541] Upgrade NeoFS SDK Go with changed basic ACL API 
SDK now provides dedicated type for basic ACL with convenient interface.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 16:43:50 +03:00
Leonard Lyubich
a8fc313ff5 [#544] layer: Remove no longer needed deleteContainer method 
Call `DeleteContainer` on `NeoFS` component directly.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Leonard Lyubich
f596c8be06 [#544] layer: Add session token parameter to DeleteBucketParams 
Add `DeleteBucketParams.SessionToken` field in order to unify the
parameters with `CreateBucketParams` and `PutBucketACLParams`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Leonard Lyubich
028a152e04 [#544] Upgrade NeoFS SDK Go with another approach of container sessions 
After recent changes in NeoFS SDK Go library session tokens aren't
embedded into `container.Container` and `eacl.Table` structures.
Instead, the operations of storing given values in NeoFS are
parameterized by elements of the corresponding type.

Add dedicated session parameters to operations of bucket and eACL
setting.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Leonard Lyubich
f0749fd23e [#537] Upgrade NeoFS SDK Go with changed netmap package 
`PlacementPolicy` type now provides methods to work with QL-encoded
policies. System network parameters can be read using dedicated method
without iterating. Applications can work with `PlacementPolicy`
variables directly so there is no need to use pointers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 08:28:59 +03:00
Leonard Lyubich
880ffe7108 [#492] layer: Don't parameterize basic ACL of created containers 
`CreateBucket` handler always creates containers with extended public
ACL, so there is no need to configure it in `NeoFS.CreateContainer`.

Make internal `NeoFS` implementation to create containers with
`eacl-public-read-write` basic ACL if corresponding parameter is unset.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-16 10:46:50 +03:00
Denis Kirillov
300d4359d8 [#465] Handle cache cast failure 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-07 17:20:26 +03:00
Denis Kirillov
f00ca1b6c4 [#438] Drop layer/neofs package 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-07 13:59:45 +03:00
Alex Vanin
4ed939773b [#487] Use updated SDK method to get bearer issuer 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-06-03 11:20:47 +03:00
Angira Kekteeva
dbfac29171 [#487] Fix tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-03 10:57:56 +03:00
Angira Kekteeva
e904ed51c7 [#487] Optimize bucketInfo in initObjectPayloadReader 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-03 10:57:56 +03:00
Angira Kekteeva
4767eeed8c [#487] Remove attach of bearer token 
When bucket owner is not an issuer of the bearer token

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-03 10:57:56 +03:00
Leonard Lyubich
4f43aad495 [#485] Upgrade SDK with latest bearer package API 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-03 09:38:49 +03:00
Alex Vanin
12d9eb62cb [#489] Sanitize log records that may contain user input 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-06-02 19:42:31 +04:00
Denis Kirillov
933ef2bc71 [#441] Remove unused 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-31 14:48:58 +03:00
Denis Kirillov
2bca4755f9 [#441] Optimize put objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-31 14:48:58 +03:00
Denis Kirillov
faa3c65290 [#462] Fix None versioning status 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-30 14:44:21 +03:00
Alex Vanin
0e37242b65 [#463] Move DefaultLocationConstraint constant into api 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-30 14:41:32 +03:00
Alex Vanin
80d4d071d8 [#463] Restrict overriding default location constraint in authmate 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-30 14:41:32 +03:00
Alex Vanin
89ff89a32b [#463] Define default location constraint 
Fixes `test_bucket_get_location` from s3-tests.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-30 14:41:32 +03:00
Leonard Lyubich
087d500c5f [#458] *: Refactor working with NeoFS identities 
Pull latest changes from NeoFS SDK Go library. Decrease redundant and
unsafe usage of ID pointers. Use `EncodeToString` method in order to
calculate protocol strings.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-27 17:22:37 +04:00
Alex Vanin
385437aace [#441] Don't produce extra object.Head request at regular object upload 
Hash can be calculated locally in S3 gateway.
Creation epoch used for versioning and will be
fetched during get and list requests. To avoid
conflicts, put method do not update cache anymore.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-20 10:56:36 +03:00
Alex Vanin
2575462daa [#439] Less cache misses for empty bucket setting 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-20 10:56:16 +03:00
Angira Kekteeva
7d69f9f74b [#432] Add get-object-attributes 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-05-13 11:42:43 +03:00
Alex Vanin
b778c2e072 [#431] Fix ETag value 
ETag is a string with SHA256 of NeoFS object payload.
Hash is taken from object header by neofs-sdk-go getter.
Checksum type in neofs-sdk-go has changes `String()`
output from `<hash>` to `SHA256:<hash>` in latest update.

S3 gateway should not be relied on unstable string format
implementations and use raw value.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-12 21:19:59 +04:00
Alex Vanin
1c33f06bfe [#428] Update SDK 
Includes:
- container removal fix
- new session token structure: authmate does not
  parse session context anymore, instead it is
  application defined flexible structure with
  container ID encoded in human-readable format

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-06 14:47:06 +04:00
Denis Kirillov
94caa2247e [#391] Refactor notifications 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-06 10:59:31 +03:00
Denis Kirillov
e3c16a32dd [#409] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-26 12:51:52 +04:00
Denis Kirillov
6e91074b50 [#367] Check errors using status 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-22 15:30:42 +04:00
Elizaveta Chichindaeva
bf38007692 [#405] English Check 
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
 2022-04-22 14:01:40 +03:00
Angira Kekteeva
f274747e83 [#400] Make multipart-upload parts system objects 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-19 12:51:04 +03:00
Denis Kirillov
3c5c2f20d8 [#399] Use bearer token for system objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-13 16:44:32 +04:00
Denis Kirillov
7710de39ec [#365] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-08 17:12:52 +03:00
Leonard Lyubich
01c721ee53 [#380] cmd/s3-gw: Refactor logger 
Do not import `logger` package from NeoFS SDK Go. Discard unusable
`name` and `version` configuration values from `app` section. Discard
all unusable onfiguration values from `logger` section except `level`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-04-06 17:31:01 +03:00
Angira Kekteeva
ed47bc1596 Fixed deletes for failed tests 
test_multi_object_delete
test_multi_objectv2_delete

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-06 09:55:00 +03:00
Angira Kekteeva
e319f2422e [#357] Remove checks of bucket settings is not nil 
Via #389
Also fixed test

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
a718b92652 [#357] Add ObjectDelete notifications 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
021f5d4dd0 [#357] Refactor delete objects 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
67c1ba2c61 [#357] Add tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
371eb2feda [#357] Add events and sending of events 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
40e7dbf768 [#357] Add test events and check of bucket notif conf 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
2b6843f8fa [#389] Fix checks of bucket settings 
In Put/CopyObject and PutBucketVersioning

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:22:10 +04:00
Denis Kirillov
c399cfbdda [#382] Refactor neofs related types 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-31 11:47:24 +03:00
Angira Kekteeva
83bb6fc020 [#384] Add check of Notificator interface value 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-03-30 10:29:28 +03:00
Denis Kirillov
f0914b8a43 [#377] Reuse BucketInfo in layer 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-29 02:43:28 +04:00
Denis Kirillov
9d19acadcd [#195] Use exp epoch attribute instead of tick 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
e0b8bc6cd6 [#195] Fix object deletion 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00