Anoke/frostfs-s3-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-s3-gw
Code Pull requests Activity
1276 commits 4 branches 17 tags 13 MiB
Commit graph

223 commits

Author SHA1 Message Date
Denis Kirillov
1fd943ee88 [#610] Fix deleted object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-29 09:31:54 +03:00
Angira Kekteeva
4380406602 [#619] Remove unused PutSystemObjectParams 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-26 15:37:13 +03:00
Denis Kirillov
85eacdb970 [#602] Fix removal non-empty bucket 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
0057f6b7db [#546] Add size and etag in nodeVersionInfo 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
72013e08ec [#569] Refactor versioning in tree service 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9f019c7470 [#542] Add test to check delete marker 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
b380d4ab53 [#542] Clear list cache on object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
f1a0288e3a [#542] Handle unversioned mode 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
fdf0974679 [#542] Fix object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
88c392d024 [#490] Optimize GetObjectTaggingAndLock 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
c88a9842db [#488] Sync using oid.ID 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
85c203e157 [#488] Sync using cid.ID 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
6ad7c988e6 [#532] Fix object removal 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
12a2060dd0 [#524] Optimize listing 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
93a7c2eaeb [#514] Missed version id in tags 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
59570ac075 [#513] Don't remove delete mark object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Alex Vanin
7fd27e04fe [#477] Provide aggregation of tags and locks in layer.Client 
Adopts GetObjectTaggingAndLock method from
TreeClient in layer.Client with caching.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
a02900a4f7 [#474] Use appropriate null version during listing 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9b1ccd39be [#475] Add extra attributes to partInfo 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
1a456eaa8b [#452] Remove unused 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9fb3fb1274 [#452] Fix versions tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
dd534e8738 [#451] Handle lock objects using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
70957d75fd [#417] Complete multipart upload using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
bc0bdc7767 [#417] Upload part using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
13e01164d7 [#417] Create multipart upload using tree service 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
385a2102f0 [#449] Add tree service for bucket tagging 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Angira Kekteeva
99feb1d936 [#449] Add tree service for object tagging 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9c74cca9af [#420] Using tree service to list object versions 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
7e8b57605a [#413] Fix delete object without version 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
37b1baed41 [#413] Refactor 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
977f176713 [#413] Use tree service to delete object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
9ed081db3e [#407] Add service tree client prototype 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-07-22 15:19:16 +03:00
Denis Kirillov
7ca519cb32 [#539] Add context to errors 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-27 02:23:19 +04:00
Leonard Lyubich
a8fc313ff5 [#544] layer: Remove no longer needed deleteContainer method 
Call `DeleteContainer` on `NeoFS` component directly.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Leonard Lyubich
f596c8be06 [#544] layer: Add session token parameter to DeleteBucketParams 
Add `DeleteBucketParams.SessionToken` field in order to unify the
parameters with `CreateBucketParams` and `PutBucketACLParams`.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Leonard Lyubich
028a152e04 [#544] Upgrade NeoFS SDK Go with another approach of container sessions 
After recent changes in NeoFS SDK Go library session tokens aren't
embedded into `container.Container` and `eacl.Table` structures.
Instead, the operations of storing given values in NeoFS are
parameterized by elements of the corresponding type.

Add dedicated session parameters to operations of bucket and eACL
setting.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-24 13:00:26 +03:00
Leonard Lyubich
f0749fd23e [#537] Upgrade NeoFS SDK Go with changed netmap package 
`PlacementPolicy` type now provides methods to work with QL-encoded
policies. System network parameters can be read using dedicated method
without iterating. Applications can work with `PlacementPolicy`
variables directly so there is no need to use pointers.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-17 08:28:59 +03:00
Leonard Lyubich
880ffe7108 [#492] layer: Don't parameterize basic ACL of created containers 
`CreateBucket` handler always creates containers with extended public
ACL, so there is no need to configure it in `NeoFS.CreateContainer`.

Make internal `NeoFS` implementation to create containers with
`eacl-public-read-write` basic ACL if corresponding parameter is unset.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-16 10:46:50 +03:00
Denis Kirillov
300d4359d8 [#465] Handle cache cast failure 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-07 17:20:26 +03:00
Denis Kirillov
f00ca1b6c4 [#438] Drop layer/neofs package 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-06-07 13:59:45 +03:00
Alex Vanin
4ed939773b [#487] Use updated SDK method to get bearer issuer 
Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-06-03 11:20:47 +03:00
Angira Kekteeva
e904ed51c7 [#487] Optimize bucketInfo in initObjectPayloadReader 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-03 10:57:56 +03:00
Angira Kekteeva
4767eeed8c [#487] Remove attach of bearer token 
When bucket owner is not an issuer of the bearer token

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-06-03 10:57:56 +03:00
Leonard Lyubich
4f43aad495 [#485] Upgrade SDK with latest bearer package API 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-06-03 09:38:49 +03:00
Denis Kirillov
2bca4755f9 [#441] Optimize put objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-31 14:48:58 +03:00
Leonard Lyubich
087d500c5f [#458] *: Refactor working with NeoFS identities 
Pull latest changes from NeoFS SDK Go library. Decrease redundant and
unsafe usage of ID pointers. Use `EncodeToString` method in order to
calculate protocol strings.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-05-27 17:22:37 +04:00
Alex Vanin
1c33f06bfe [#428] Update SDK 
Includes:
- container removal fix
- new session token structure: authmate does not
  parse session context anymore, instead it is
  application defined flexible structure with
  container ID encoded in human-readable format

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-06 14:47:06 +04:00
Denis Kirillov
94caa2247e [#391] Refactor notifications 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-05-06 10:59:31 +03:00
Denis Kirillov
e3c16a32dd [#409] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-26 12:51:52 +04:00
Elizaveta Chichindaeva
bf38007692 [#405] English Check 
Signed-off-by: Elizaveta Chichindaeva <elizaveta@nspcc.ru>
 2022-04-22 14:01:40 +03:00
Denis Kirillov
7710de39ec [#365] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-08 17:12:52 +03:00
Angira Kekteeva
ed47bc1596 Fixed deletes for failed tests 
test_multi_object_delete
test_multi_objectv2_delete

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-06 09:55:00 +03:00
Angira Kekteeva
e319f2422e [#357] Remove checks of bucket settings is not nil 
Via #389
Also fixed test

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
a718b92652 [#357] Add ObjectDelete notifications 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
021f5d4dd0 [#357] Refactor delete objects 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
371eb2feda [#357] Add events and sending of events 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
40e7dbf768 [#357] Add test events and check of bucket notif conf 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-04-01 14:57:19 +03:00
Angira Kekteeva
83bb6fc020 [#384] Add check of Notificator interface value 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-03-30 10:29:28 +03:00
Denis Kirillov
f0914b8a43 [#377] Reuse BucketInfo in layer 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-29 02:43:28 +04:00
Denis Kirillov
9d19acadcd [#195] Use exp epoch attribute instead of tick 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
e0b8bc6cd6 [#195] Fix object deletion 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
4a67e4b311 [#195] Set tick attribute to lock objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
4c3c4b6bee [#195] Add Lock tick listening 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
13080d6d96 [#195] Refactor TestNeoFS 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
7d6271be8a [#195] Implement PUT, GET locks to certain object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
8553158b81 [#195] Add handling lock headers for PUT and COPY 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
b96c3c5a33 [#195] Refactor 
Using object settings to save bucket versioning

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
e98c663bd6 [#195] Support enabling object locking for bucket 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
02f4524d67 [#368] Unify cache value deletion 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-16 10:59:42 +03:00
Leonard Lyubich
8fb3835250 [#346] api: Do not use io.Pipe in CompleteMultipartUpload 
Replace `layer.objectWritePayload` method with `initObjectPayloadReader`
which returns `io.Reader` of the object payload. Copy payload data to
the parameterized `io.Writer` in `layer.GetObject`. Remove `io.Pipe`
from `CompleteMultipartUpload` implementation and build analogue of
`io.MultiReader` for the part list.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
eac4c4d849 [#346] api: Remove unused and no longer needed layer.Get method 
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
cd64f41ce8 [#346] *: Refactor communication with NeoFS at the protocol level 
Make `tokens`, `authmate` and `layer` packages to depend from locally
defined `NeoFS` interface of the virtual connection to NeoFS network.
Create internal `neofs` package and implement these interfaces through
`pool.Pool` there. Implement mediators between `NeoFS` interfaces and
`neofs.NeoFS` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
34a221c5c9 [#346] Upgrade NeoFS SDK Go library 
Core changes:
  - `object.ID` moved to new package `oid`;
  - `object.Address` moved to new package `address`;
  - `pool.Object` interface changes.

Additionally:
  - Set container owner in `Agent.IssueSecret`.
  - Remove no longer needed fields from `GetObjectParams`
  - `Length` and `Offset` are never assigned. These values
  are set in `Range` field.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Angira Kekteeva
4454821285 [#340] Add notification configuration handlers 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-18 16:59:42 +03:00
Angira Kekteeva
e0c6544567 [#340] Move notification controller to layer 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-02-18 16:59:42 +03:00
Denis Kirillov
f04815c32f [#354] Check container owner if bucket exists 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-02-15 17:08:46 +03:00
Denis Kirillov
c383f332d5 [#329] Use suitable tokens for container requests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-31 11:05:35 +03:00
Denis Kirillov
8fc19b11db [#333] Update sdk 
Update sdk to fix "invalid session token owner" error

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-28 12:42:40 +03:00
Denis Kirillov
8985681493 [#319] Delete null versions in different modes 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-26 16:22:58 +03:00
Denis Kirillov
58df410111 [#319] Head and delete null versions 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-26 16:22:58 +03:00
Denis Kirillov
962d136e73 [#308] Correct access denied status code 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-24 18:47:03 +03:00
Denis Kirillov
2b4ed8487b [#310] Support response headers overriding 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-21 16:02:06 +03:00
Denis Kirillov
c202893563 [#307] Filter system metadata 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-21 10:28:51 +03:00
Denis Kirillov
8872b6f196 [#285] Add resolving order 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-17 18:14:19 +03:00
Denis Kirillov
b9f77b3d96 [#301] Implement GetBucketLocation 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-13 20:24:21 +03:00
Denis Kirillov
12c15504ef [#297] Update neofs-sdk-go and neo-go 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-12-30 13:00:23 +03:00
Angira Kekteeva
d6f0ab8ea4 [#186] Fix versioning issues in UploadPart 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-11-30 11:32:59 +03:00
Angira Kekteeva
873622d4d5 [#186] Add MultipartUpload support 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-11-30 11:32:59 +03:00
Angira Kekteeva
284a560ea6 [#186] Refactor objectSearch, fix objectPut 
Fixed panic in objectPut() if input reader is nil
Made objectSearch searching by any attribute

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-11-30 11:32:59 +03:00
Denis Kirillov
befe084900 [#219] Add container name resolving 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-24 13:24:39 +03:00
Denis Kirillov
d36dfe8c61 [#271] Update neo-sdk-go to the latest version 
Refactoring invoking pool methods for anonymous requests.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-16 14:42:08 +03:00
Denis Kirillov
ae87effb28 [#271] Add random key for no sign requests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-16 14:42:08 +03:00
Angira Kekteeva
91bed76010 [#274] Refactor system cache and cors 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-15 17:44:40 +03:00
Angira Kekteeva
7d0bc1e992 [#217] Add CORS support 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-12 09:03:09 +03:00
Angira Kekteeva
ccf5db95a5 [#217] Refactor system objects 
Moved into a separate file
getSystemObject renamed to headSystemObject, implemented getSystemObject
for system objects with payload
Refactor putSystemObjects
Moved systemCacheKey from data system_object

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-12 09:03:09 +03:00
Denis Kirillov
3231ecab03 [#263] Add LWW to system objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-23 20:37:55 +03:00
Denis Kirillov
19b917e3b5 [#253] Caches refactoring 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 10:46:18 +03:00
Denis Kirillov
3c2e25f977 [#248] Support delete marker 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 09:44:39 +03:00
Angira Kekteeva
be08596c22 [#231] cache: Add invalidation of ListObjectsCache 
In put operations ListObjectsCache remove entries which can contain
put object

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Angira Kekteeva
1ece42b23f [#236] cache: Refactor ListObjectsCache 
Replaced map in ListObjectsCache by gcache.
Now ListObjectsCache keeps only objectIDs and
requests ObjectInfo from cache or NeoFS.
Refactored ListObjectsCache keys: removed delimiter and method fields.
Now ListObjectsCache keeps cache with all objects versions.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00