aarifullin/frostfs-s3-gw
1
0
Fork 0
forked from TrueCloudLab/frostfs-s3-gw
Code Pull requests Activity
913 commits 3 branches 21 tags 12 MiB
Commit graph

127 commits

Author SHA1 Message Date
Alex Vanin
b778c2e072 [#431] Fix ETag value 
ETag is a string with SHA256 of NeoFS object payload.
Hash is taken from object header by neofs-sdk-go getter.
Checksum type in neofs-sdk-go has changes `String()`
output from `<hash>` to `SHA256:<hash>` in latest update.

S3 gateway should not be relied on unstable string format
implementations and use raw value.

Signed-off-by: Alex Vanin <alexey@nspcc.ru>
 2022-05-12 21:19:59 +04:00
Denis Kirillov
e3c16a32dd [#409] Update SDK 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-26 12:51:52 +04:00
Denis Kirillov
6e91074b50 [#367] Check errors using status 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-22 15:30:42 +04:00
Denis Kirillov
3c5c2f20d8 [#399] Use bearer token for system objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-04-13 16:44:32 +04:00
Denis Kirillov
f0914b8a43 [#377] Reuse BucketInfo in layer 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-29 02:43:28 +04:00
Denis Kirillov
32e83db064 [#195] Set Lock type to system lock objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
13080d6d96 [#195] Refactor TestNeoFS 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
6cf01bed14 [#195] Add tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
7d6271be8a [#195] Implement PUT, GET locks to certain object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
8553158b81 [#195] Add handling lock headers for PUT and COPY 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-23 14:32:13 +04:00
Denis Kirillov
02f4524d67 [#368] Unify cache value deletion 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-03-16 10:59:42 +03:00
Leonard Lyubich
8fb3835250 [#346] api: Do not use io.Pipe in CompleteMultipartUpload 
Replace `layer.objectWritePayload` method with `initObjectPayloadReader`
which returns `io.Reader` of the object payload. Copy payload data to
the parameterized `io.Writer` in `layer.GetObject`. Remove `io.Pipe`
from `CompleteMultipartUpload` implementation and build analogue of
`io.MultiReader` for the part list.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
cd64f41ce8 [#346] *: Refactor communication with NeoFS at the protocol level 
Make `tokens`, `authmate` and `layer` packages to depend from locally
defined `NeoFS` interface of the virtual connection to NeoFS network.
Create internal `neofs` package and implement these interfaces through
`pool.Pool` there. Implement mediators between `NeoFS` interfaces and
`neofs.NeoFS` implementation.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Leonard Lyubich
34a221c5c9 [#346] Upgrade NeoFS SDK Go library 
Core changes:
  - `object.ID` moved to new package `oid`;
  - `object.Address` moved to new package `address`;
  - `pool.Object` interface changes.

Additionally:
  - Set container owner in `Agent.IssueSecret`.
  - Remove no longer needed fields from `GetObjectParams`
  - `Length` and `Offset` are never assigned. These values
  are set in `Range` field.
Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
 2022-03-04 00:14:30 +03:00
Angira Kekteeva
1502857dba [#313] Remove extra QueryUnescapes 
Removed extra QueryUnescapes in search and objectPut

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2022-01-27 10:25:11 +03:00
Denis Kirillov
f5d365af1d [#319] Update CRDT headers 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-26 16:22:58 +03:00
Denis Kirillov
58df410111 [#319] Head and delete null versions 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-26 16:22:58 +03:00
Denis Kirillov
962d136e73 [#308] Correct access denied status code 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-24 18:47:03 +03:00
Denis Kirillov
c202893563 [#307] Filter system metadata 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2022-01-21 10:28:51 +03:00
Angira Kekteeva
d6f0ab8ea4 [#186] Fix versioning issues in UploadPart 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-11-30 11:32:59 +03:00
Angira Kekteeva
284a560ea6 [#186] Refactor objectSearch, fix objectPut 
Fixed panic in objectPut() if input reader is nil
Made objectSearch searching by any attribute

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-11-30 11:32:59 +03:00
Denis Kirillov
d36dfe8c61 [#271] Update neo-sdk-go to the latest version 
Refactoring invoking pool methods for anonymous requests.

Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-16 14:42:08 +03:00
Denis Kirillov
ae87effb28 [#271] Add random key for no sign requests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-11-16 14:42:08 +03:00
Angira Kekteeva
7d0bc1e992 [#217] Add CORS support 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-12 09:03:09 +03:00
Angira Kekteeva
ccf5db95a5 [#217] Refactor system objects 
Moved into a separate file
getSystemObject renamed to headSystemObject, implemented getSystemObject
for system objects with payload
Refactor putSystemObjects
Moved systemCacheKey from data system_object

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-10-12 09:03:09 +03:00
Denis Kirillov
3231ecab03 [#263] Add LWW to system objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-23 20:37:55 +03:00
Denis Kirillov
19b917e3b5 [#253] Caches refactoring 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-14 10:46:18 +03:00
Denis Kirillov
458f9cf17b [#242] Fix versions sorting 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-09-07 17:06:22 +03:00
Angira Kekteeva
be08596c22 [#231] cache: Add invalidation of ListObjectsCache 
In put operations ListObjectsCache remove entries which can contain
put object

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Angira Kekteeva
1ece42b23f [#236] cache: Refactor ListObjectsCache 
Replaced map in ListObjectsCache by gcache.
Now ListObjectsCache keeps only objectIDs and
requests ObjectInfo from cache or NeoFS.
Refactored ListObjectsCache keys: removed delimiter and method fields.
Now ListObjectsCache keeps cache with all objects versions.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Angira Kekteeva
1bc2e51cbc [#236] api: Refactor caches: ObjectsList, Objects 
Move ObjectsList from layer to cache package
Rename object_cache.go to objects.go

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Angira Kekteeva
239742f413 [#236] api: Create info.go for basic structs 
Moved BucketInfo and ObjectInfo from layer and handler to api

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-09-07 11:08:48 +03:00
Denis Kirillov
8b5ebe2ec2 [#196] Add PutObjectTagging 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-31 10:46:42 +03:00
Angira Kekteeva
dcac1aa3e3 [#230] layer: Change objects_list_cache's key 
Remove access_key from objectsListCache key

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-26 12:54:58 +03:00
Denis Kirillov
4bb885d526 [#122] Refactor 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-25 14:49:21 +03:00
Denis Kirillov
11558124cd [#122] Add versioning cache 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-25 09:15:00 +03:00
Denis Kirillov
f6c51cc9ee [#122] Update listObjectVerions 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-25 09:15:00 +03:00
Denis Kirillov
9c058a70fd [#122] Add tests 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-25 09:15:00 +03:00
Denis Kirillov
43185de52a [#122] Add list object versions 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-25 09:15:00 +03:00
Denis Kirillov
d5aef7566f [#122] Add delete versioned object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-25 09:15:00 +03:00
Denis Kirillov
3130784ee6 [#122] Add getting specific object version 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-25 09:13:12 +03:00
Denis Kirillov
f463522f34 [#122] Add versioning put object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-25 09:10:46 +03:00
Denis Kirillov
feb45d0633 [#122] Add replacing objects 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-25 09:10:46 +03:00
Angira Kekteeva
295be71f00 [#206] api: Add objects cache 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-18 10:16:27 +03:00
Angira Kekteeva
ae19eb4ad4 [#206] Refactor ListObjectsCache 
Replaced conflicting names

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-18 10:16:27 +03:00
Denis Kirillov
d332096598 [#199] Refactor 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-08-11 13:17:37 +03:00
Angira Kekteeva
8a69c7cca0 [#179] api: Add cache for ListObjectsV1/V2 
Refactored cache for ListObjects:
made cache common for all versions,
simplified: remove dependendence on token/startafter
add mitable cachelifetime.

Refactored listobjects

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-08-09 13:52:11 +03:00
Angira Kekteeva
0ceea95e11 [#112] Add cache to ListObjects and layer 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-28 17:05:11 +03:00
Angira Kekteeva
97a7d16f68 [#112] api: Refactor allObjectsParams struct 
Replaced Bucket (means name) by BucketInfo struct

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-26 23:22:05 +03:00
Alex Vanin
3f7941fdf3
Merge pull request #174 from nspcc-dev/feature/159-optional_owner 
[#159] Add fetch-owner param
 2021-07-26 13:32:04 +03:00
Denis Kirillov
63652d213c [#149] Removed grpc errors 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-22 09:54:12 +03:00
Denis Kirillov
65a61a9f7d [#159] Add fetch-owner param 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-20 15:40:38 +03:00
Denis Kirillov
c05e2d6757 [#160] Fix head dir object 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-07-20 12:36:41 +03:00
Angira Kekteeva
ee84062154 [#154] api: refactor ListObjectsV1 and V2 
Separate ListObject for different versions.
Remove useless grouping of keys on V2.

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-07-19 10:12:47 +03:00
Denis Kirillov
a6ec27b40d [#93] Object ETag support 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-30 12:55:31 +03:00
Denis Kirillov
c4fef3d948 [#96] Support ranges for GetObject 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-25 12:18:52 +03:00
Denis Kirillov
6a135a6bb4 [#82] Using bearer token 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-15 17:16:26 +03:00
Denis Kirillov
fba29a9a66 [#78] Using client.Object from sdk 
Signed-off-by: Denis Kirillov <denis@nspcc.ru>
 2021-06-15 17:06:19 +03:00
Angira Kekteeva
b3b190adc0 [#68] *: Replace deprecated 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-06-04 16:01:42 +03:00
Roman Khimov
d15687f37c *: drop creds/neofs 
Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-28 23:48:39 +03:00
Roman Khimov
d19ce03072 *: drop old sdk dependecies, bump neofs-api-go version 
I'm not sure it works, but it's enough code-wise for now. We're reusing some
http-gw components here that are to be moved into sdk-go in future.

Signed-off-by: Roman Khimov <roman@nspcc.ru>
 2021-05-26 21:01:46 +03:00
Angira Kekteeva
9d496d70a7 [#53] *: Remove external pkg/errors dependency 
Replaced functions from pkg/errors by functions from errors, fixed not fully correct comment

Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-05-20 15:50:14 +03:00
Angira Kekteeva
a399590204 Replace s3-gate by s3-gw 
Signed-off-by: Angira Kekteeva <kira@nspcc.ru>
 2021-05-18 14:12:24 +03:00
Evgeniy Kulikov
2a93a216f8 Migrate to SDK 0.3.0 and fixes 
- fix displaying list objects
- simplify `ListObjects`
- simplify `GetObjectHandler`
- simplify `HeadObjectHandler`
- add method for `GetBucketVersioningHandler`
- add method for `ListMultipartUploadsHandler`
- improvements for `HeadObjectHandler`, to display folders meta
- update dependencies
  - github.com/aws/aws-sdk-go v1.36.26
  - github.com/google/uuid v1.1.4
  - github.com/gorilla/mux v1.8.0
  - github.com/nspcc-dev/cdn-sdk v0.3.0
  - github.com/nspcc-dev/neofs-api-go v1.22.0
  - github.com/prometheus/client_golang v1.9.0
  - github.com/stretchr/testify v1.7.0
  - google.golang.org/grpc v1.35.0

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2021-01-14 20:39:48 +03:00
Evgeniy Kulikov
8168181658 Remove bufio writer and double creation of OffsetWriter 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-12-10 18:15:18 +03:00
Evgeniy Kulikov
f4263d50a7 Refactoring api/layer 
- don't require token.BearerToken
- simplify method layer.objectPut
- add method to check that object exists
- add method to fetch Owner (from context or sdk client)

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-11-27 15:36:15 +03:00
Evgeniy Kulikov
0284bf136b Migrate to Credentials client 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-11-24 12:17:36 +03:00
Evgeniy Kulikov
0ee3a5f9ba Migrate api/layer to SDK 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-11-24 10:01:38 +03:00
Evgeniy Kulikov
256850b8fe [#25] Refactoring and make fixes 
closes #25
closes #33

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-10-24 16:09:22 +03:00
Evgeniy Kulikov
4d605d1113 [#25] Refactoring and make fixes 
closes #25
closes #32

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-10-23 03:21:14 +03:00
Evgeniy Kulikov
fbd4a83602 [#25] Refactoring over api/layer 
closes #25

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-10-22 03:19:16 +03:00
Evgeniy Kulikov
7deeb68d47 [#25] Migrate layer to NeoFS API v2 
Naive migration, without any tries to start application

closes #25

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-10-19 04:04:37 +03:00
Evgeniy Kulikov
5eae2aac78 Change NotFound error for objectFindID 
method should return gRPC status error

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-08-22 05:39:12 +03:00
Evgeniy Kulikov
21591ebe65 Add method to find all ID's for the nice names 
Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-08-20 22:33:59 +03:00
Evgeniy Kulikov
edd44ea981 Check that aws filename wasn't set 
we should check that filename header wasn't set, otherwise we should not change custom filename

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-08-20 02:32:42 +03:00
Evgeniy Kulikov
64429d736d Refactoring API layer 
- logging RequestID
- should return error, when headers already received

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-08-11 14:32:04 +03:00
Evgeniy Kulikov
0161d2fbd3 [#13] Rename go module name according to NSPCC standards 
- refactoring s3 gate structure
- cleanup unused code
- rename go module to `github.com/nspcc-dev/neofs-s3-gate`

closes #13

Signed-off-by: Evgeniy Kulikov <kim@nspcc.ru>
 2020-08-06 15:02:13 +03:00
Renamed from neofs/layer/object.go (Browse further)