dkirillov/frostfs-s3-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-s3-gw
Code Issues Pull requests Projects Releases Packages Wiki Activity
891 commits 15 branches 31 tags 22 MiB
Commit graph

468 commits

Author SHA1 Message Date
Denis Kirillov
c88a9842db [#488] Sync using oid.ID 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
85c203e157 [#488] Sync using cid.ID 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
dca7b5bc36 [#547] Fix tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
42a761c1f2 [#547] Cache ObjectInfo instead of Object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
127b91a97f [#547] Cache empty lists without error 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
5984f2c2bc [#525] Add comment to clarify anonymous func 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
ad95d1745c [#525] Parallelize listing 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
6ad7c988e6 [#532] Fix object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
12a2060dd0 [#524] Optimize listing 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
3f5afabd4a [#515] Fix nil reference when a part is missing 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
93a7c2eaeb [#514] Missed version id in tags 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
59570ac075 [#513] Don't remove delete mark object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Alex Vanin
4f3d206422 [#478] Don't fetch epoch at object upload 
Creation epoch was used for versioning. With
tree service, versioning is done on the NeoFS
side in the tree.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9dfc7e043f [#476] Save full parts info to multipart object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Alex Vanin
8731dcfd3c [#477] Use tags and lock aggregation in handler 
Reduce amount of tree service RPCs
by using new method in layer package.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Alex Vanin
7fd27e04fe [#477] Provide aggregation of tags and locks in layer.Client 
Adopts GetObjectTaggingAndLock method from
TreeClient in layer.Client with caching.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Alex Vanin
f8496973b8 [#477] Aggregate fetching tags and lock in layer.TreeService 
New method aggregates GetObjectTagging and GetLock
methods in single RPC to the tree service.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
a02900a4f7 [#474] Use appropriate null version during listing 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
c8e8ba9f6a [#476] Fix parts info for GetObjectAttributes 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9b1ccd39be [#475] Add extra attributes to partInfo 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
5b9a5cadef [#452] Fix lint issues 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
1a456eaa8b [#452] Remove unused 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9fb3fb1274 [#452] Fix versions tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
5529fb914e [#452] Fix lock tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
dd534e8738 [#451] Handle lock objects using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
70957d75fd [#417] Complete multipart upload using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
6b2ddcadd0 [#417] Abort multipart upload using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
58f2bf44b1 [#417] List parts using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
bc0bdc7767 [#417] Upload part using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
e1b9a4432a [#417] List multipart uploads using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
13e01164d7 [#417] Create multipart upload using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
b6a43c6f4c [#449] Replace error checks and literals 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
385a2102f0 [#449] Add tree service for bucket tagging 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
99feb1d936 [#449] Add tree service for object tagging 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9c74cca9af [#420] Using tree service to list object versions 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
25477cdaf8 [#416] Use tree service to list objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
7e8b57605a [#413] Fix delete object without version 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
37b1baed41 [#413] Refactor 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
977f176713 [#413] Use tree service to delete object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
49bd77d9cf [#413] Use tree service to head objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
36f3c43af5 [#413] Use tree service to put object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
ab5c44ac14 [#429] Refactor TreeService for notif,cors,settings 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
8eff857e41 [#429] Add tree service for cors 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
7520952792 [#429] Add tree service for notifications 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
b03ae827fb [#429] Refactor tree service 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9ed081db3e [#407] Add service tree client prototype 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Alex Vanin
d7f77ce874 [#574] Produce deny records for private objects in put-object-acl 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-21 17:55:15 +03:00
Alex Vanin
66fe3fee7b [#574] Produce deny records for private objects 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-21 17:55:15 +03:00
Denis Kirillov
7ba7e7dc4d [#590] Make service records valid 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-20 19:10:23 +03:00
Denis Kirillov
1e26cf1541 [#590] Use service records to save resource info 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-20 19:10:23 +03:00
Angira Kekteeva
b144e50f7f [#584] Refactor formRecords func 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-19 11:54:27 +03:00
Angira Kekteeva
3f4a55f39e [#584] Fix order in astToTable 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-19 11:54:27 +03:00
Angira Kekteeva
260fb95677 [#584] Fix order in tableToAst 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-19 11:54:27 +03:00
Angira Kekteeva
74300a75a9 [#584] Add tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-19 11:54:27 +03:00
Alex Vanin
aad4862c59 [#564] Don't parse lock attribute if it is omitted in container 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-15 01:33:48 +04:00
Denis Kirillov
1575da65a4 [#573] Fix object acl filters 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-14 13:33:11 +03:00
Alex Vanin
a57b8d34d3 [#553] Add more comments about eacl.RoleUnknown 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-07 12:05:25 +03:00
Alex Vanin
06d043e1eb [#553] Optimize target formation with multiple keys 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-07 12:05:25 +03:00
Alex Vanin
d6065c64c4 [#553] Check group grantee based on stored list of users 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-07 12:05:25 +03:00
Alex Vanin
c7de7d2928 [#553] Do not use user role with public keys in eacl target 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-07 12:05:25 +03:00
Alex Vanin
36029ca864 [#580] Fix user removal in astOperation 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-06 17:40:41 +03:00
Leonard Lyubich
5bfc549746 [#564] neofs: Merge if with same condition in CreateContainer 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-07-04 23:42:50 +04:00
Leonard Lyubich
4a8a248f34 [#564] Upgrade NeoFS SDK Go with changed container API 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-07-04 23:42:50 +04:00
Denis Kirillov
6e1a1f3839 [#522] Suppress CodeQL error 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-04 11:03:55 +03:00
Denis Kirillov
9f740b9683 [#289] Add detect mimetype by extension 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-29 19:13:16 +03:00
Denis Kirillov
f72bc538b9 [#551] Refactor notifications logs 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-27 17:31:17 +03:00
Denis Kirillov
35f55c5af5 [#539] Fix tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-27 02:23:19 +04:00
Denis Kirillov
7ca519cb32 [#539] Add context to errors 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-27 02:23:19 +04:00
Leonard Lyubich
e1f1e6d960 [#541] Upgrade NeoFS SDK Go with changed basic ACL API 
SDK now provides dedicated type for basic ACL with convenient interface.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 16:43:50 +03:00
Leonard Lyubich
a8fc313ff5 [#544] layer: Remove no longer needed deleteContainer method 
Call `DeleteContainer` on `NeoFS` component directly.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Leonard Lyubich
f596c8be06 [#544] layer: Add session token parameter to DeleteBucketParams 
Add `DeleteBucketParams.SessionToken` field in order to unify the
parameters with `CreateBucketParams` and `PutBucketACLParams`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Leonard Lyubich
028a152e04 [#544] Upgrade NeoFS SDK Go with another approach of container sessions 
After recent changes in NeoFS SDK Go library session tokens aren't
embedded into `container.Container` and `eacl.Table` structures.
Instead, the operations of storing given values in NeoFS are
parameterized by elements of the corresponding type.

Add dedicated session parameters to operations of bucket and eACL
setting.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Denis Kirillov
818176e7e1 [#528] Adopt aws v4signer 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-20 09:44:25 +03:00
Leonard Lyubich
f0749fd23e [#537] Upgrade NeoFS SDK Go with changed netmap package 
`PlacementPolicy` type now provides methods to work with QL-encoded
policies. System network parameters can be read using dedicated method
without iterating. Applications can work with `PlacementPolicy`
variables directly so there is no need to use pointers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 08:28:59 +03:00
Angira Kekteeva
cfe7591cf7 [#523] Add putObjectACL notification 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-17 01:43:46 +04:00
Angira Kekteeva
dd0d21b690 [#523] Fix typo 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-17 01:43:46 +04:00
Angira Kekteeva
bd5fd041b7 [#523] Add object tagging notifications 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-17 01:43:46 +04:00
Denis Kirillov
d521af2065 [#529] Add presign URLs support 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-16 17:17:45 +03:00
Leonard Lyubich
880ffe7108 [#492] layer: Don't parameterize basic ACL of created containers 
`CreateBucket` handler always creates containers with extended public
ACL, so there is no need to configure it in `NeoFS.CreateContainer`.

Make internal `NeoFS` implementation to create containers with
`eacl-public-read-write` basic ACL if corresponding parameter is unset.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-16 10:46:50 +03:00
Angira Kekteeva
a8bff13801 [#495] Add acl to copy-object 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-07 17:36:12 +03:00
Denis Kirillov
de7281ac58 [#465] Unify log messages 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-07 17:20:26 +03:00
Denis Kirillov
300d4359d8 [#465] Handle cache cast failure 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-07 17:20:26 +03:00
Denis Kirillov
f00ca1b6c4 [#438] Drop layer/neofs package 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-07 13:59:45 +03:00
Alex Vanin
4ed939773b [#487] Use updated SDK method to get bearer issuer 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-06-03 11:20:47 +03:00
Angira Kekteeva
dbfac29171 [#487] Fix tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-03 10:57:56 +03:00
Angira Kekteeva
e904ed51c7 [#487] Optimize bucketInfo in initObjectPayloadReader 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-03 10:57:56 +03:00
Angira Kekteeva
4767eeed8c [#487] Remove attach of bearer token 
When bucket owner is not an issuer of the bearer token

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-03 10:57:56 +03:00
Leonard Lyubich
4f43aad495 [#485] Upgrade SDK with latest bearer package API 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-03 09:38:49 +03:00
Denis Kirillov
ea252421f5 [#484] Add tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-03 09:35:06 +03:00
Denis Kirillov
f282e877e2 [#484] Handle conditional headers 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-03 09:35:06 +03:00
Alex Vanin
ee0f3fb196 [#489] Avoid sensitive data logging 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-06-02 19:42:31 +04:00
Alex Vanin
12d9eb62cb [#489] Sanitize log records that may contain user input 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-06-02 19:42:31 +04:00
Denis Kirillov
933ef2bc71 [#441] Remove unused 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-31 14:48:58 +03:00
Denis Kirillov
2bca4755f9 [#441] Optimize put objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-31 14:48:58 +03:00
Denis Kirillov
faa3c65290 [#462] Fix None versioning status 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-30 14:44:21 +03:00
Alex Vanin
0e37242b65 [#463] Move DefaultLocationConstraint constant into api 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-30 14:41:32 +03:00
Alex Vanin
80d4d071d8 [#463] Restrict overriding default location constraint in authmate 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-30 14:41:32 +03:00
Alex Vanin
89ff89a32b [#463] Define default location constraint 
Fixes `test_bucket_get_location` from s3-tests.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-30 14:41:32 +03:00
Leonard Lyubich
087d500c5f [#458] *: Refactor working with NeoFS identities 
Pull latest changes from NeoFS SDK Go library. Decrease redundant and
unsafe usage of ID pointers. Use `EncodeToString` method in order to
calculate protocol strings.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-27 17:22:37 +04:00
Alex Vanin
80c6122f55 [#450] Do not pass nil error to logAndSendError 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-25 00:36:42 +04:00