r.loginov/frostfs-s3-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-s3-gw
Code Pull requests Activity
1144 commits 7 branches 15 tags 21 MiB
Commit graph

345 commits

Author SHA1 Message Date
Alex Vanin
96dff367db [#1] Build S3 Gateway with FrostFS dependencies 
Signed-off-by: Alex Vanin <a.vanin@yadro.com>
 2022-12-15 12:43:52 +03:00
Denis Kirillov
dd4f66712c [#742] Add multiple listeners 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-24 17:48:26 +03:00
Denis Kirillov
d2587b21af [#747] Reload policies on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-18 13:14:25 +03:00
Denis Kirillov
094eb12578 [#726] Use client time on regular requests 
Use `X-Amz-Date` header as `now` when
* compute expiration epoch
* set Timestamp for object and container
* forming locks
* send notifications

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-14 14:53:55 +03:00
Denis Kirillov
d47840f137 [#568] Add configuration for region to policy map 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-11 13:12:49 +03:00
Denis Kirillov
2886ac161c [#740] Fix forming policy by ast 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-11-10 15:33:03 +03:00
Denis Kirillov
18a6aca4b4 [#617] api/handler: Simplify tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-25 19:04:37 +03:00
Angira Kekteeva
712740b668 [#728] Remove redundant log message 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-10-25 19:04:11 +03:00
Denis Kirillov
821df3d648 [#669] Optimize getNodeVersion for tags operation 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-20 17:56:09 +03:00
Denis Kirillov
90eb4f0188 [#713] Add check access control for system caches 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 15:26:42 +03:00
Denis Kirillov
cb55d36063 [#713] Update tests 
Add bearer token to test context

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 15:26:42 +03:00
Denis Kirillov
c6098e53f6 [#719] Add tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 11:13:20 +03:00
Denis Kirillov
b3d96e6460 [#719] Improve lock error handling 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-10-11 11:13:20 +03:00
Denis Kirillov
2a41929be3 [#702] Reload resolvers and TLS certs on SIGHUP 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-28 15:07:31 +03:00
Denis Kirillov
3d08562843 [#505] Handle access denied from tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-13 18:05:16 +03:00
Denis Kirillov
80beedf13e [#504] Use bktInfo in tree service to check owner 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-13 18:05:16 +03:00
Denis Kirillov
4082cd6b54 [#606] Keep eacl records order on conflict 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-08 12:21:47 +03:00
Denis Kirillov
9cd4ef1ac4 [#657] Replace FileName with FilePath attribute 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-09-07 15:50:43 +03:00
Denis Kirillov
96c7b79d1c [#683] Forbid copy to itself in unversioned bucket 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-30 13:31:48 +03:00
Denis Kirillov
44d5878a80 [#656] Read full body on upload error 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-30 09:39:20 +03:00
Denis Kirillov
5c7a509cc9 [#684] Fix continuation token for directories 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-29 16:10:01 +03:00
Denis Kirillov
e38bdae07a [#676] Fix object acl 
Put object acl always add rules to specific version of object.
Get object acl consider READ rights as FULL_CONTROL
because WRITE cannot be applied to object

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-29 13:20:30 +03:00
Denis Kirillov
163038b37d [#672] Fix handling X-Amz-Copy-Source header 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 17:40:15 +03:00
Denis Kirillov
fdc926e70b [#661] Fix updating cache when request fails 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 12:55:49 +03:00
Denis Kirillov
1a580b6fa4 [#666] Support tagging directive for CopyObject 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 12:55:33 +03:00
Denis Kirillov
f013282290 [#660] Fix panic on invalid versioning status 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 11:50:44 +03:00
Denis Kirillov
24e1f7e1dd [#545] Drop object search 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-25 11:49:08 +03:00
Denis Kirillov
c3ad6d2faf [#637] Add header to override CopiesNumber 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-19 10:31:26 +03:00
Angira Kekteeva
5307211398 [#634] Add CopiesNumber in NeoFS requests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-16 17:49:15 +03:00
Denis Kirillov
3824151699 [#616] Reduce number of requests during listing 
Check if object is directory before request to NeoFS

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-16 13:47:37 +03:00
Denis Kirillov
107d8a9033 [#595] Fix typos and rewording 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov
7ab473a688 [#595] Simplify encryption.Params struct 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov
94a6a55919 [#595] Move decrypter to separate package 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov
d824db7f69 [#595] Allow SSE-C only with TLS 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Denis Kirillov
87c05eb514 [#595] Add SSE with customer key 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-08-13 10:26:00 +03:00
Angira Kekteeva
f66e4a77cd [#577] Add IsDeleteMarker() to NodeVersion 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
dc77ad4863 [#577] Fix tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
93e0cae743 [#577] Fix version in GetObjectAttributes 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
659e7bfc97 [#577] Fix version in GetObject and HeadObject 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
dfd734b9ec [#577] Separate GetObjectInfo and GetExtendedObjectInfo 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
37c46e926e [#577] Add Version() to ExtendedObjectInfo 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
3ac3f1cc9d [#577] Rename objectInfo.Version() to VersionID() 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
eaf76afe50 [#577] Remove ObjectVersionInfo 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-09 17:20:40 +04:00
Angira Kekteeva
1ee1b8bb79 [#643] Fix missing const api.QueryVersionID 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-08-05 15:32:46 +03:00
Alex Vanin
5219965105 [#498] Remove log string sanitizer 
`zap` lib does all the job anyway.
Suppress CWE-117 from CodeQL.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-29 09:32:37 +03:00
Denis Kirillov
1fd943ee88 [#610] Fix deleted object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-29 09:31:54 +03:00
Denis Kirillov
b2e8b1cfb3 [#612] Make listing more robust 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-26 17:36:21 +03:00
Alex Vanin
757600608c [#539] Fix imports 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
85eacdb970 [#602] Fix removal non-empty bucket 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
0057f6b7db [#546] Add size and etag in nodeVersionInfo 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
56eb2dc3dc [#569] Rename versioning consts 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
77f466b1d5 [#569] Fix tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
72013e08ec [#569] Refactor versioning in tree service 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9f019c7470 [#542] Add test to check delete marker 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
b380d4ab53 [#542] Clear list cache on object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
f1a0288e3a [#542] Handle unversioned mode 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
fdf0974679 [#542] Fix object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
88c392d024 [#490] Optimize GetObjectTaggingAndLock 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
c88a9842db [#488] Sync using oid.ID 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
85c203e157 [#488] Sync using cid.ID 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
dca7b5bc36 [#547] Fix tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
6ad7c988e6 [#532] Fix object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
93a7c2eaeb [#514] Missed version id in tags 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9dfc7e043f [#476] Save full parts info to multipart object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Alex Vanin
8731dcfd3c [#477] Use tags and lock aggregation in handler 
Reduce amount of tree service RPCs
by using new method in layer package.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
a02900a4f7 [#474] Use appropriate null version during listing 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
c8e8ba9f6a [#476] Fix parts info for GetObjectAttributes 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
1a456eaa8b [#452] Remove unused 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9fb3fb1274 [#452] Fix versions tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
5529fb914e [#452] Fix lock tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
dd534e8738 [#451] Handle lock objects using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
70957d75fd [#417] Complete multipart upload using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
6b2ddcadd0 [#417] Abort multipart upload using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
bc0bdc7767 [#417] Upload part using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
13e01164d7 [#417] Create multipart upload using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
385a2102f0 [#449] Add tree service for bucket tagging 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
99feb1d936 [#449] Add tree service for object tagging 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
25477cdaf8 [#416] Use tree service to list objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Alex Vanin
d7f77ce874 [#574] Produce deny records for private objects in put-object-acl 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-21 17:55:15 +03:00
Alex Vanin
66fe3fee7b [#574] Produce deny records for private objects 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-21 17:55:15 +03:00
Denis Kirillov
7ba7e7dc4d [#590] Make service records valid 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-20 19:10:23 +03:00
Denis Kirillov
1e26cf1541 [#590] Use service records to save resource info 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-20 19:10:23 +03:00
Angira Kekteeva
b144e50f7f [#584] Refactor formRecords func 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-19 11:54:27 +03:00
Angira Kekteeva
3f4a55f39e [#584] Fix order in astToTable 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-19 11:54:27 +03:00
Angira Kekteeva
260fb95677 [#584] Fix order in tableToAst 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-19 11:54:27 +03:00
Angira Kekteeva
74300a75a9 [#584] Add tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-19 11:54:27 +03:00
Denis Kirillov
1575da65a4 [#573] Fix object acl filters 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-14 13:33:11 +03:00
Alex Vanin
a57b8d34d3 [#553] Add more comments about eacl.RoleUnknown 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-07 12:05:25 +03:00
Alex Vanin
06d043e1eb [#553] Optimize target formation with multiple keys 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-07 12:05:25 +03:00
Alex Vanin
d6065c64c4 [#553] Check group grantee based on stored list of users 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-07 12:05:25 +03:00
Alex Vanin
c7de7d2928 [#553] Do not use user role with public keys in eacl target 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-07 12:05:25 +03:00
Alex Vanin
36029ca864 [#580] Fix user removal in astOperation 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-06 17:40:41 +03:00
Leonard Lyubich
4a8a248f34 [#564] Upgrade NeoFS SDK Go with changed container API 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-07-04 23:42:50 +04:00
Denis Kirillov
6e1a1f3839 [#522] Suppress CodeQL error 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-04 11:03:55 +03:00
Denis Kirillov
9f740b9683 [#289] Add detect mimetype by extension 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-29 19:13:16 +03:00
Denis Kirillov
f72bc538b9 [#551] Refactor notifications logs 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-27 17:31:17 +03:00
Denis Kirillov
7ca519cb32 [#539] Add context to errors 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-27 02:23:19 +04:00
Leonard Lyubich
f596c8be06 [#544] layer: Add session token parameter to DeleteBucketParams 
Add `DeleteBucketParams.SessionToken` field in order to unify the
parameters with `CreateBucketParams` and `PutBucketACLParams`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Leonard Lyubich
028a152e04 [#544] Upgrade NeoFS SDK Go with another approach of container sessions 
After recent changes in NeoFS SDK Go library session tokens aren't
embedded into `container.Container` and `eacl.Table` structures.
Instead, the operations of storing given values in NeoFS are
parameterized by elements of the corresponding type.

Add dedicated session parameters to operations of bucket and eACL
setting.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Leonard Lyubich
f0749fd23e [#537] Upgrade NeoFS SDK Go with changed netmap package 
`PlacementPolicy` type now provides methods to work with QL-encoded
policies. System network parameters can be read using dedicated method
without iterating. Applications can work with `PlacementPolicy`
variables directly so there is no need to use pointers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 08:28:59 +03:00