Anoke/frostfs-s3-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-s3-gw
Code Pull requests Activity
1054 commits 4 branches 17 tags 13 MiB
Commit graph

542 commits

Author SHA1 Message Date
Denis Kirillov
6b2ddcadd0 [#417] Abort multipart upload using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
58f2bf44b1 [#417] List parts using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
bc0bdc7767 [#417] Upload part using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
e1b9a4432a [#417] List multipart uploads using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
13e01164d7 [#417] Create multipart upload using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
b6a43c6f4c [#449] Replace error checks and literals 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
385a2102f0 [#449] Add tree service for bucket tagging 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
99feb1d936 [#449] Add tree service for object tagging 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9c74cca9af [#420] Using tree service to list object versions 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
25477cdaf8 [#416] Use tree service to list objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
7e8b57605a [#413] Fix delete object without version 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
37b1baed41 [#413] Refactor 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
977f176713 [#413] Use tree service to delete object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
49bd77d9cf [#413] Use tree service to head objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
36f3c43af5 [#413] Use tree service to put object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
ab5c44ac14 [#429] Refactor TreeService for notif,cors,settings 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
8eff857e41 [#429] Add tree service for cors 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
7520952792 [#429] Add tree service for notifications 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
b03ae827fb [#429] Refactor tree service 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9ed081db3e [#407] Add service tree client prototype 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Alex Vanin
d7f77ce874 [#574] Produce deny records for private objects in put-object-acl 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-21 17:55:15 +03:00
Alex Vanin
66fe3fee7b [#574] Produce deny records for private objects 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-21 17:55:15 +03:00
Denis Kirillov
7ba7e7dc4d [#590] Make service records valid 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-20 19:10:23 +03:00
Denis Kirillov
1e26cf1541 [#590] Use service records to save resource info 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-20 19:10:23 +03:00
Angira Kekteeva
b144e50f7f [#584] Refactor formRecords func 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-19 11:54:27 +03:00
Angira Kekteeva
3f4a55f39e [#584] Fix order in astToTable 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-19 11:54:27 +03:00
Angira Kekteeva
260fb95677 [#584] Fix order in tableToAst 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-19 11:54:27 +03:00
Angira Kekteeva
74300a75a9 [#584] Add tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-19 11:54:27 +03:00
Alex Vanin
aad4862c59 [#564] Don't parse lock attribute if it is omitted in container 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-15 01:33:48 +04:00
Denis Kirillov
1575da65a4 [#573] Fix object acl filters 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-14 13:33:11 +03:00
Alex Vanin
a57b8d34d3 [#553] Add more comments about eacl.RoleUnknown 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-07 12:05:25 +03:00
Alex Vanin
06d043e1eb [#553] Optimize target formation with multiple keys 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-07 12:05:25 +03:00
Alex Vanin
d6065c64c4 [#553] Check group grantee based on stored list of users 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-07 12:05:25 +03:00
Alex Vanin
c7de7d2928 [#553] Do not use user role with public keys in eacl target 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-07 12:05:25 +03:00
Alex Vanin
36029ca864 [#580] Fix user removal in astOperation 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-06 17:40:41 +03:00
Leonard Lyubich
5bfc549746 [#564] neofs: Merge if with same condition in CreateContainer 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-07-04 23:42:50 +04:00
Leonard Lyubich
4a8a248f34 [#564] Upgrade NeoFS SDK Go with changed container API 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-07-04 23:42:50 +04:00
Denis Kirillov
6e1a1f3839 [#522] Suppress CodeQL error 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-04 11:03:55 +03:00
Denis Kirillov
9f740b9683 [#289] Add detect mimetype by extension 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-29 19:13:16 +03:00
Denis Kirillov
f72bc538b9 [#551] Refactor notifications logs 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-27 17:31:17 +03:00
Denis Kirillov
35f55c5af5 [#539] Fix tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-27 02:23:19 +04:00
Denis Kirillov
7ca519cb32 [#539] Add context to errors 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-27 02:23:19 +04:00
Leonard Lyubich
e1f1e6d960 [#541] Upgrade NeoFS SDK Go with changed basic ACL API 
SDK now provides dedicated type for basic ACL with convenient interface.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 16:43:50 +03:00
Leonard Lyubich
a8fc313ff5 [#544] layer: Remove no longer needed deleteContainer method 
Call `DeleteContainer` on `NeoFS` component directly.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Leonard Lyubich
f596c8be06 [#544] layer: Add session token parameter to DeleteBucketParams 
Add `DeleteBucketParams.SessionToken` field in order to unify the
parameters with `CreateBucketParams` and `PutBucketACLParams`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Leonard Lyubich
028a152e04 [#544] Upgrade NeoFS SDK Go with another approach of container sessions 
After recent changes in NeoFS SDK Go library session tokens aren't
embedded into `container.Container` and `eacl.Table` structures.
Instead, the operations of storing given values in NeoFS are
parameterized by elements of the corresponding type.

Add dedicated session parameters to operations of bucket and eACL
setting.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Denis Kirillov
818176e7e1 [#528] Adopt aws v4signer 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-20 09:44:25 +03:00
Leonard Lyubich
f0749fd23e [#537] Upgrade NeoFS SDK Go with changed netmap package 
`PlacementPolicy` type now provides methods to work with QL-encoded
policies. System network parameters can be read using dedicated method
without iterating. Applications can work with `PlacementPolicy`
variables directly so there is no need to use pointers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 08:28:59 +03:00
Angira Kekteeva
cfe7591cf7 [#523] Add putObjectACL notification 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-17 01:43:46 +04:00
Angira Kekteeva
dd0d21b690 [#523] Fix typo 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-17 01:43:46 +04:00
Angira Kekteeva
bd5fd041b7 [#523] Add object tagging notifications 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-17 01:43:46 +04:00
Denis Kirillov
d521af2065 [#529] Add presign URLs support 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-16 17:17:45 +03:00
Leonard Lyubich
880ffe7108 [#492] layer: Don't parameterize basic ACL of created containers 
`CreateBucket` handler always creates containers with extended public
ACL, so there is no need to configure it in `NeoFS.CreateContainer`.

Make internal `NeoFS` implementation to create containers with
`eacl-public-read-write` basic ACL if corresponding parameter is unset.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-16 10:46:50 +03:00
Angira Kekteeva
a8bff13801 [#495] Add acl to copy-object 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-07 17:36:12 +03:00
Denis Kirillov
de7281ac58 [#465] Unify log messages 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-07 17:20:26 +03:00
Denis Kirillov
300d4359d8 [#465] Handle cache cast failure 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-07 17:20:26 +03:00
Denis Kirillov
f00ca1b6c4 [#438] Drop layer/neofs package 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-07 13:59:45 +03:00
Alex Vanin
4ed939773b [#487] Use updated SDK method to get bearer issuer 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-06-03 11:20:47 +03:00
Angira Kekteeva
dbfac29171 [#487] Fix tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-03 10:57:56 +03:00
Angira Kekteeva
e904ed51c7 [#487] Optimize bucketInfo in initObjectPayloadReader 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-03 10:57:56 +03:00
Angira Kekteeva
4767eeed8c [#487] Remove attach of bearer token 
When bucket owner is not an issuer of the bearer token

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-03 10:57:56 +03:00
Leonard Lyubich
4f43aad495 [#485] Upgrade SDK with latest bearer package API 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-03 09:38:49 +03:00
Denis Kirillov
ea252421f5 [#484] Add tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-03 09:35:06 +03:00
Denis Kirillov
f282e877e2 [#484] Handle conditional headers 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-03 09:35:06 +03:00
Alex Vanin
ee0f3fb196 [#489] Avoid sensitive data logging 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-06-02 19:42:31 +04:00
Alex Vanin
12d9eb62cb [#489] Sanitize log records that may contain user input 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-06-02 19:42:31 +04:00
Denis Kirillov
933ef2bc71 [#441] Remove unused 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-31 14:48:58 +03:00
Denis Kirillov
2bca4755f9 [#441] Optimize put objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-31 14:48:58 +03:00
Denis Kirillov
faa3c65290 [#462] Fix None versioning status 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-30 14:44:21 +03:00
Alex Vanin
0e37242b65 [#463] Move DefaultLocationConstraint constant into api 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-30 14:41:32 +03:00
Alex Vanin
80d4d071d8 [#463] Restrict overriding default location constraint in authmate 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-30 14:41:32 +03:00
Alex Vanin
89ff89a32b [#463] Define default location constraint 
Fixes `test_bucket_get_location` from s3-tests.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-30 14:41:32 +03:00
Leonard Lyubich
087d500c5f [#458] *: Refactor working with NeoFS identities 
Pull latest changes from NeoFS SDK Go library. Decrease redundant and
unsafe usage of ID pointers. Use `EncodeToString` method in order to
calculate protocol strings.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-27 17:22:37 +04:00
Alex Vanin
80c6122f55 [#450] Do not pass nil error to logAndSendError 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-25 00:36:42 +04:00
Alex Vanin
385437aace [#441] Don't produce extra object.Head request at regular object upload 
Hash can be calculated locally in S3 gateway.
Creation epoch used for versioning and will be
fetched during get and list requests. To avoid
conflicts, put method do not update cache anymore.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-20 10:56:36 +03:00
Alex Vanin
2575462daa [#439] Less cache misses for empty bucket setting 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-20 10:56:16 +03:00
Angira Kekteeva
7d69f9f74b [#432] Add get-object-attributes 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-05-13 11:42:43 +03:00
Alex Vanin
b778c2e072 [#431] Fix ETag value 
ETag is a string with SHA256 of NeoFS object payload.
Hash is taken from object header by neofs-sdk-go getter.
Checksum type in neofs-sdk-go has changes `String()`
output from `<hash>` to `SHA256:<hash>` in latest update.

S3 gateway should not be relied on unstable string format
implementations and use raw value.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-12 21:19:59 +04:00
Alex Vanin
1c33f06bfe [#428] Update SDK 
Includes:
- container removal fix
- new session token structure: authmate does not
  parse session context anymore, instead it is
  application defined flexible structure with
  container ID encoded in human-readable format

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-06 14:47:06 +04:00
Denis Kirillov
94caa2247e [#391] Refactor notifications 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-06 10:59:31 +03:00
Denis Kirillov
e3c16a32dd [#409] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-26 12:51:52 +04:00
Denis Kirillov
6e91074b50 [#367] Check errors using status 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-22 15:30:42 +04:00
Elizaveta Chichindaeva
bf38007692 [#405] English Check 
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
 2022-04-22 14:01:40 +03:00
Angira Kekteeva
a0a04a73bd [#347] Add setEACL session token checks 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-20 12:42:33 +03:00
Angira Kekteeva
f274747e83 [#400] Make multipart-upload parts system objects 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-19 12:51:04 +03:00
Denis Kirillov
3c5c2f20d8 [#399] Use bearer token for system objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-13 16:44:32 +04:00
Denis Kirillov
7710de39ec [#365] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-08 17:12:52 +03:00
Leonard Lyubich
01c721ee53 [#380] cmd/s3-gw: Refactor logger 
Do not import `logger` package from NeoFS SDK Go. Discard unusable
`name` and `version` configuration values from `app` section. Discard
all unusable onfiguration values from `logger` section except `level`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-04-06 17:31:01 +03:00
Angira Kekteeva
f3df5ff633 [#395] Fix grantee in ACL 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-06 15:30:17 +03:00
Angira Kekteeva
ed47bc1596 Fixed deletes for failed tests 
test_multi_object_delete
test_multi_objectv2_delete

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-06 09:55:00 +03:00
Angira Kekteeva
e319f2422e [#357] Remove checks of bucket settings is not nil 
Via #389
Also fixed test

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
02bcbe9754 [#357] Replace prepareEvent params by struct 
By struct SendNotificationParams
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
a718b92652 [#357] Add ObjectDelete notifications 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
021f5d4dd0 [#357] Refactor delete objects 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
b7aac223df [#357] Add ObjectCreated notifications 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
67c1ba2c61 [#357] Add tests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
371eb2feda [#357] Add events and sending of events 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
40e7dbf768 [#357] Add test events and check of bucket notif conf 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
2b6843f8fa [#389] Fix checks of bucket settings 
In Put/CopyObject and PutBucketVersioning

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:22:10 +04:00
Denis Kirillov
c399cfbdda [#382] Refactor neofs related types 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-31 11:47:24 +03:00
Angira Kekteeva
83bb6fc020 [#384] Add check of Notificator interface value 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-03-30 10:29:28 +03:00
Denis Kirillov
f0914b8a43 [#377] Reuse BucketInfo in layer 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-29 02:43:28 +04:00
Denis Kirillov
46e4b28489 [#195] Add logger to notification controller 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
9d19acadcd [#195] Use exp epoch attribute instead of tick 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
cfcc8933e4 [#195] Add response lock headers 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
e0b8bc6cd6 [#195] Fix object deletion 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
4a67e4b311 [#195] Set tick attribute to lock objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
4c3c4b6bee [#195] Add Lock tick listening 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
32e83db064 [#195] Set Lock type to system lock objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
13080d6d96 [#195] Refactor TestNeoFS 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
5c530123ab [#195] Fix lint errors 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
6cf01bed14 [#195] Add tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
7d6271be8a [#195] Implement PUT, GET locks to certain object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
8553158b81 [#195] Add handling lock headers for PUT and COPY 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
fe9eb9cedc [#195] Add PUT and GET default lock configuration 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
b96c3c5a33 [#195] Refactor 
Using object settings to save bucket versioning

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
3046d80b37 [#195] Check object locking when disable versioning 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
e98c663bd6 [#195] Support enabling object locking for bucket 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
94406b08f3 [#376] Update NeoFS SDK to v1.0.0-rc.3 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-16 15:44:19 +03:00
Denis Kirillov
02f4524d67 [#368] Unify cache value deletion 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-16 10:59:42 +03:00
Denis Kirillov
e7b742df4d [#366] Add wait for container removed 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-04 10:41:15 +03:00
Leonard Lyubich
20b8e3d249 [#346] Upgrade NeoFS SDK Go to 2nd release candidate v1.0.0 
Avoid using the deprecated elements.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
8fb3835250 [#346] api: Do not use io.Pipe in CompleteMultipartUpload 
Replace `layer.objectWritePayload` method with `initObjectPayloadReader`
which returns `io.Reader` of the object payload. Copy payload data to
the parameterized `io.Writer` in `layer.GetObject`. Remove `io.Pipe`
from `CompleteMultipartUpload` implementation and build analogue of
`io.MultiReader` for the part list.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
eac4c4d849 [#346] api: Remove unused and no longer needed layer.Get method 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
1517b5c638 [#346] neofs: Fix belated updateCRDT2PSetHeaders call 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
cd64f41ce8 [#346] *: Refactor communication with NeoFS at the protocol level 
Make `tokens`, `authmate` and `layer` packages to depend from locally
defined `NeoFS` interface of the virtual connection to NeoFS network.
Create internal `neofs` package and implement these interfaces through
`pool.Pool` there. Implement mediators between `NeoFS` interfaces and
`neofs.NeoFS` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
34a221c5c9 [#346] Upgrade NeoFS SDK Go library 
Core changes:
  - `object.ID` moved to new package `oid`;
  - `object.Address` moved to new package `address`;
  - `pool.Object` interface changes.

Additionally:
  - Set container owner in `Agent.IssueSecret`.
  - Remove no longer needed fields from `GetObjectParams`
  - `Length` and `Offset` are never assigned. These values
  are set in `Range` field.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Angira Kekteeva
4454821285 [#340] Add notification configuration handlers 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-18 16:59:42 +03:00
Angira Kekteeva
4cbce87eac [#340] Make nats tls and ca params optional 
nats.Connect returned error when tls and ca files were not set, what
made these params required, but establishing of unsecured connection
for debug is acceptable.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-18 16:59:42 +03:00
Angira Kekteeva
e0c6544567 [#340] Move notification controller to layer 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-18 16:59:42 +03:00
Denis Kirillov
f04815c32f [#354] Check container owner if bucket exists 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-02-15 17:08:46 +03:00
Angira Kekteeva
3277293bb3 [#341] Add notifications controller 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-04 14:41:22 +03:00
Denis Kirillov
c383f332d5 [#329] Use suitable tokens for container requests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-31 11:05:35 +03:00
Denis Kirillov
9be4bbbed4 [#312] Support Expires and Cache-Control headers 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-31 00:04:41 +03:00
Denis Kirillov
8fc19b11db [#333] Update sdk 
Update sdk to fix "invalid session token owner" error

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-28 12:42:40 +03:00
Angira Kekteeva
1502857dba [#313] Remove extra QueryUnescapes 
Removed extra QueryUnescapes in search and objectPut

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-01-27 10:25:11 +03:00
Denis Kirillov
4f5b316211 [#319] Fix tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-26 16:22:58 +03:00
Denis Kirillov
8985681493 [#319] Delete null versions in different modes 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-26 16:22:58 +03:00
Denis Kirillov
f5d365af1d [#319] Update CRDT headers 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-26 16:22:58 +03:00
Denis Kirillov
58df410111 [#319] Head and delete null versions 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-26 16:22:58 +03:00
Denis Kirillov
a46726a545 [#331] Use UTC Last-Modified time 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-26 16:04:14 +03:00
Denis Kirillov
962d136e73 [#308] Correct access denied status code 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-24 18:47:03 +03:00
Denis Kirillov
2b4ed8487b [#310] Support response headers overriding 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-21 16:02:06 +03:00
Angira Kekteeva
bffc09167a [#316] Remove minio mentions in commits 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-01-21 10:44:54 +03:00
Angira Kekteeva
e5a256df0c [#316] Return badrequest err if couldnt parse body 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-01-21 10:44:54 +03:00
Angira Kekteeva
3307953a4c [#316] Remove MinIO related errors 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-01-21 10:44:54 +03:00
Denis Kirillov
c202893563 [#307] Filter system metadata 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-21 10:28:51 +03:00
Denis Kirillov
d532eb710f [#306] Add test to check empty policy 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-18 13:23:32 +03:00
Denis Kirillov
76042f6728 [#306] Handle empty post policy 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-18 13:23:32 +03:00
Denis Kirillov
e8e07147a8 [#318] Add x-amz-version-id header 
Support version id in CompleteMultipartUpload method

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-17 18:18:17 +03:00
Denis Kirillov
8872b6f196 [#285] Add resolving order 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-17 18:14:19 +03:00
Denis Kirillov
097f745d3e [#305] Fix UnknownAPIRequest handling 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-17 17:29:06 +03:00
Denis Kirillov
b9f77b3d96 [#301] Implement GetBucketLocation 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-13 20:24:21 +03:00
Denis Kirillov
9b14340816 [#299] Fix creating named containers 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-13 13:12:20 +03:00
Denis Kirillov
12c15504ef [#297] Update neofs-sdk-go and neo-go 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-12-30 13:00:23 +03:00
Angira Kekteeva
8127d68128 [#186] Fix errors 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-11-30 11:32:59 +03:00
Angira Kekteeva
8c80b16360 [#186] Add unittests for MultipartUpload 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-11-30 11:32:59 +03:00
Angira Kekteeva
d6f0ab8ea4 [#186] Fix versioning issues in UploadPart 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-11-30 11:32:59 +03:00
Angira Kekteeva
873622d4d5 [#186] Add MultipartUpload support 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-11-30 11:32:59 +03:00
Angira Kekteeva
284a560ea6 [#186] Refactor objectSearch, fix objectPut 
Fixed panic in objectPut() if input reader is nil
Made objectSearch searching by any attribute

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-11-30 11:32:59 +03:00
Denis Kirillov
befe084900 [#219] Add container name resolving 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-24 13:24:39 +03:00
Denis Kirillov
d36dfe8c61 [#271] Update neo-sdk-go to the latest version 
Refactoring invoking pool methods for anonymous requests.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-16 14:42:08 +03:00
Denis Kirillov
ae87effb28 [#271] Add random key for no sign requests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-16 14:42:08 +03:00
Angira Kekteeva
91bed76010 [#274] Refactor system cache and cors 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-15 17:44:40 +03:00
Angira Kekteeva
5b4b9df031 [#217] Add errors for cors 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-12 09:03:09 +03:00
Angira Kekteeva
f4afe41086 [#217] Add headers in response to simple requests 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-12 09:03:09 +03:00
Angira Kekteeva
7d0bc1e992 [#217] Add CORS support 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-12 09:03:09 +03:00
Angira Kekteeva
ccf5db95a5 [#217] Refactor system objects 
Moved into a separate file
getSystemObject renamed to headSystemObject, implemented getSystemObject
for system objects with payload
Refactor putSystemObjects
Moved systemCacheKey from data system_object

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-12 09:03:09 +03:00
Angira Kekteeva
0460225869 [#272] Fix linter warning 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-01 12:25:54 +03:00
Denis Kirillov
3231ecab03 [#263] Add LWW to system objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-23 20:37:55 +03:00
Denis Kirillov
d616d9e2d9 [#262] Fix status codes 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-23 14:22:24 +03:00
Alex Vanin
16296490ee [#264] Adopt minio code distributed with AGPLv3 license in gateway components 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2021-09-20 10:38:28 +03:00
Alex Vanin
15023b9371 [#264] Add minio code distributed with AGPLv3 license 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2021-09-20 10:38:28 +03:00
Alex Vanin
5d11d8f556 [#264] Remove minio code distributed with Apache 2.0 license 
NeoFS S3 Gate based on Minio RELEASE.2020-07-02T00-15-09Z
code which distributed with Apache 2.0 license. This commit
removes remaining minio code with that license.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2021-09-20 10:38:28 +03:00
Denis Kirillov
19b917e3b5 [#253] Caches refactoring 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 10:46:18 +03:00
Denis Kirillov
951eb6fda8 [#253] Add access box cache 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 10:46:18 +03:00
Denis Kirillov
3c2e25f977 [#248] Support delete marker 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 09:44:39 +03:00
Denis Kirillov
458f9cf17b [#242] Fix versions sorting 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-07 17:06:22 +03:00
Denis Kirillov
42ed6a16ea [#190] Add POST object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-07 11:10:48 +03:00
Angira Kekteeva
b2bf81cec7 [#231] cache: Add unittests for ListObjectsCache 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Angira Kekteeva
d4a6a3ea2c [#236] cache: Update comments for ListObjectsCache 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Angira Kekteeva
be08596c22 [#231] cache: Add invalidation of ListObjectsCache 
In put operations ListObjectsCache remove entries which can contain
put object

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Angira Kekteeva
1ece42b23f [#236] cache: Refactor ListObjectsCache 
Replaced map in ListObjectsCache by gcache.
Now ListObjectsCache keeps only objectIDs and
requests ObjectInfo from cache or NeoFS.
Refactored ListObjectsCache keys: removed delimiter and method fields.
Now ListObjectsCache keeps cache with all objects versions.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Angira Kekteeva
1bc2e51cbc [#236] api: Refactor caches: ObjectsList, Objects 
Move ObjectsList from layer to cache package
Rename object_cache.go to objects.go

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Angira Kekteeva
239742f413 [#236] api: Create info.go for basic structs 
Moved BucketInfo and ObjectInfo from layer and handler to api

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Denis Kirillov
a9be642eaf [#213] Add object acl versioning 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-31 12:38:23 +03:00
Denis Kirillov
5502fb97c3 [#196] Add tagging caching 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-31 10:46:42 +03:00
Denis Kirillov
987185b9e1 [#196] Add bucket tagging 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-31 10:46:42 +03:00
Denis Kirillov
16da1aba64 [#196] Add DeleteObjectTagging 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-31 10:46:42 +03:00
Denis Kirillov
4a2575fbf3 [#196] Add GetObjectTagging 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-31 10:46:42 +03:00
Denis Kirillov
8b5ebe2ec2 [#196] Add PutObjectTagging 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-31 10:46:42 +03:00
Angira Kekteeva
44a2f1b471 [#234] *: Update go.mod 
Remove neofs-node dep
Update versions neofs reps

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-30 17:01:41 +03:00
Angira Kekteeva
dcac1aa3e3 [#230] layer: Change objects_list_cache's key 
Remove access_key from objectsListCache key

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-26 12:54:58 +03:00
Kirillov Denis
04b7958ab3
Merge pull request #207 from KirillovDenis/feature/122-poc_versioning 
[#122] Add enabling versioning
 2021-08-25 15:01:33 +03:00
Denis Kirillov
4bb885d526 [#122] Refactor 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-25 14:49:21 +03:00
Angira Kekteeva
2299db4e81 [#218] handler,s3-gw: Make policy configurable 
Now default policy of placing containers can be set via config/env
variable.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-25 12:18:23 +03:00
Angira Kekteeva
6d4fe34f3d [#218] handler: Remove unused struct 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-25 12:18:23 +03:00
Denis Kirillov
11558124cd [#122] Add versioning cache 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-25 09:15:00 +03:00
Denis Kirillov
f6c51cc9ee [#122] Update listObjectVerions 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-25 09:15:00 +03:00
Denis Kirillov
9c058a70fd [#122] Add tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-25 09:15:00 +03:00